Re: [gentoo-user] [OT] - command line read *.csv create new file

2009-02-25 Thread Etaoin Shrdlu
On Tuesday 24 February 2009, 23:51, Mark Knecht wrote: Looks like I'm running into one more problem and then I'm ready to give it a try for real. Unfortunately one vendor platform is putting quotes around the names in the header row so your _N increment looks like High_4 instead of High_4 or

Re: [gentoo-user] [OT] - command line read *.csv create new file

2009-02-24 Thread Etaoin Shrdlu
On Tuesday 24 February 2009, 03:26, Mark Knecht wrote: If I drop columns - and I do need to - then something like how cut works would be good, but it needs to repeat across all the rows being used. For instance, if I'm dropping columns 6 12 from a 20 column wide data set, then I'm dropping 6

Re: [gentoo-user] [OT] - command line read *.csv create new file

2009-02-24 Thread Mark Knecht
On Tue, Feb 24, 2009 at 2:56 AM, Etaoin Shrdlu shr...@unlimitedmail.org wrote: SNIP So, in my understanding this is what we want to accomplish so far: given an input of the form D1,T1,a1,b1,c1,d1,...,R1 D2,T2,a2,b2,c2,d2,...,R2 D3,T3,a3,b3,c3,d3,...,R3 D4,T4,a4,b4,c4,d4,...,R4

Re: [gentoo-user] [OT] - command line read *.csv create new file

2009-02-24 Thread Etaoin Shrdlu
On Tuesday 24 February 2009, 15:41, Mark Knecht wrote: On Tue, Feb 24, 2009 at 2:56 AM, Etaoin Shrdlu shr...@unlimitedmail.org wrote: SNIP So, in my understanding this is what we want to accomplish so far: given an input of the form D1,T1,a1,b1,c1,d1,...,R1 D2,T2,a2,b2,c2,d2,...,R2

Re: [gentoo-user] [OT] - command line read *.csv create new file

2009-02-24 Thread Mark Knecht
On Tue, Feb 24, 2009 at 9:48 AM, Etaoin Shrdlu shr...@unlimitedmail.org wrote: On Tuesday 24 February 2009, 15:41, Mark Knecht wrote: On Tue, Feb 24, 2009 at 2:56 AM, Etaoin Shrdlu shr...@unlimitedmail.org wrote: SNIP So, in my understanding this is what we want to accomplish so far:

Re: [gentoo-user] [OT] - command line read *.csv create new file

2009-02-23 Thread Etaoin Shrdlu
On Monday 23 February 2009, 00:31, Mark Knecht wrote: Yeah, that's probably almost usable as it is . I tried it with n=3 and n=10. Worked both times just fine. The initial issue might be (as with Willie's sed code) that the first line wasn't quite right and required some hand editing. I'd

Re: [gentoo-user] [OT] - command line read *.csv create new file

2009-02-23 Thread Mark Knecht
On Mon, Feb 23, 2009 at 1:57 AM, Etaoin Shrdlu shr...@unlimitedmail.org wrote: On Monday 23 February 2009, 00:31, Mark Knecht wrote: Yeah, that's probably almost usable as it is . I tried it with n=3 and n=10. Worked both times just fine. The initial issue might be (as with Willie's sed code)

Re: [gentoo-user] [OT] - command line read *.csv create new file

2009-02-23 Thread Etaoin Shrdlu
On Monday 23 February 2009, 17:05, Mark Knecht wrote: I'm attaching a small (100 line) data file out of TradeStation. Zipped it's about 2K. It should expand to about 10K. When I run the command to get 10 lines put together it works correctly and gives me a file with 91 lines and about 100K in

Re: [gentoo-user] [OT] - command line read *.csv create new file

2009-02-23 Thread Mark Knecht
On Mon, Feb 23, 2009 at 2:18 PM, Etaoin Shrdlu shr...@unlimitedmail.org wrote: On Monday 23 February 2009, 17:05, Mark Knecht wrote: I'm attaching a small (100 line) data file out of TradeStation. Zipped it's about 2K. It should expand to about 10K. When I run the command to get 10 lines put

Re: [gentoo-user] [OT] - command line read *.csv create new file

2009-02-22 Thread Etaoin Shrdlu
On Sunday 22 February 2009, 20:06, Mark Knecht wrote: Hi, Very off topic other than I'd do this on my Gentoo box prior to using R on my Gentoo box. Please ignore if not of interest. I've got a really big data file in essentially a *.csv format. (comma delimited) I need to scan this

Re: [gentoo-user] [OT] - command line read *.csv create new file

2009-02-22 Thread Willie Wong
On Sun, Feb 22, 2009 at 11:06:31AM -0800, Penguin Lover Mark Knecht squawked: I've got a really big data file in essentially a *.csv format. (comma delimited) I need to scan this file and create a new output file. I'm wondering if there is a reasonably easy command line way of doing this

Re: [gentoo-user] [OT] - command line read *.csv create new file

2009-02-22 Thread Mark Knecht
On Sun, Feb 22, 2009 at 12:15 PM, Etaoin Shrdlu shr...@unlimitedmail.org wrote: On Sunday 22 February 2009, 20:06, Mark Knecht wrote: Hi, Very off topic other than I'd do this on my Gentoo box prior to using R on my Gentoo box. Please ignore if not of interest. I've got a really big

Re: [gentoo-user] [OT] - command line read *.csv create new file

2009-02-22 Thread Etaoin Shrdlu
On Sunday 22 February 2009, 23:28, Mark Knecht wrote: concatenation of attributes of lines 1..n result of line n concatenation of attributes of lines 2..n+1 result of line n+1 concatenation of attributes of lines 3..n+2 result of line n+1 concatenation of attributes of lines 4..n+3 result

Re: [gentoo-user] [OT] - command line read *.csv create new file

2009-02-22 Thread Mark Knecht
On Sun, Feb 22, 2009 at 12:59 PM, Willie Wong ww...@princeton.edu wrote: On Sun, Feb 22, 2009 at 11:06:31AM -0800, Penguin Lover Mark Knecht squawked: I've got a really big data file in essentially a *.csv format. (comma delimited) I need to scan this file and create a new output file. I'm

Re: [gentoo-user] [OT] - command line read *.csv create new file

2009-02-22 Thread Mark Knecht
On Sun, Feb 22, 2009 at 2:57 PM, Etaoin Shrdlu shr...@unlimitedmail.org wrote: On Sunday 22 February 2009, 23:28, Mark Knecht wrote: concatenation of attributes of lines 1..n result of line n concatenation of attributes of lines 2..n+1 result of line n+1 concatenation of attributes of

Re: [gentoo-user] [OT] - command line read *.csv create new file

2009-02-22 Thread Willie Wong
On Sun, Feb 22, 2009 at 03:15:09PM -0800, Penguin Lover Mark Knecht squawked: 1) My actual input data starts with two fields which date time. For lines 2 3 I need exclude the 2nd 3rd date time from the output corresponding to line 1, so these 3 lines: Date1,Time1,A,B,C,D,0

Re: [gentoo-user] [OT] - command line read *.csv create new file

2009-02-22 Thread Mark Knecht
On Sun, Feb 22, 2009 at 4:57 PM, Willie Wong ww...@princeton.edu wrote: On Sun, Feb 22, 2009 at 03:15:09PM -0800, Penguin Lover Mark Knecht squawked: 1) My actual input data starts with two fields which date time. For lines 2 3 I need exclude the 2nd 3rd date time from the output

Re: [gentoo-user] [OT] - command line read *.csv create new file

2009-02-22 Thread Paul Hartman
On Sun, Feb 22, 2009 at 5:31 PM, Mark Knecht markkne...@gmail.com wrote: The down side is the output file is 10x larger than the input file - roughly - and my current input files are 40-60MB so the output files will be 600MB. Not huge but if they grew too much more I might get beyond what a