Ole - thanks for the education - Malcolm
> -----Original Message----- > From: [email protected] [mailto:[email protected]] On > Behalf Of Ole Tange > Sent: Tuesday, March 29, 2011 5:03 PM > To: Cook, Malcolm > Cc: Jay Hacker; [email protected] > Subject: Re: Processing files from a tar archive in parallel > > On Tue, Mar 29, 2011 at 11:41 PM, Cook, Malcolm > <[email protected]> wrote: > > ooops, more like: > > > > tar -t big-file.tar.gz | parallel tar -O -x -f > big-file.tar.gz > > '|' someCommandThatReadsFromStdIn > > You probably mean: > > tar -tf big-file.tar.gz | parallel tar -O -x -f > big-file.tar.gz {} '|' someCommandThatReadsFromStdIn > > I am afraid this is highly inefficient. If big-file.tar.gz > contains 1000 files, then to extract the last file the second > tar will have to read through the first 999 files (as you > cannot seek into tar.gz files). In total you will be reading > big-file.tar.gz 1000*999/2 times. > > > /Ole >
