All those small files - hrmmmm, those wouldn't be Lois Lerner's missing Emails, would they?
On Mon, Jun 16, 2014 at 4:17 PM, andrew mcelroy <[email protected]> wrote: > it would have been **much** faster if I had use the -m flag and set a > higher memory ( in ram) amount. > > > > On Mon, Jun 16, 2014 at 4:11 PM, andrew mcelroy <[email protected]> > wrote: > >> >> >> >> On Mon, Jun 16, 2014 at 4:06 PM, Jack Coats <[email protected]> wrote: >> >>> Sow how long did it take for how many fiels? >>> >> >> $ tar -c XXXX | pbzip2 -vc > XXXX.tar.bz2 >> Parallel BZIP2 v1.1.8 - by: Jeff Gilchrist [http://compression.ca] >> [Jun. 10, 2012] (uses libbzip2 by Julian Seward) >> Major contributions: Yavor Nikolov <[email protected]> >> >> # CPUs: 4 >> BWT Block Size: 900 KB >> File Block Size: 900 KB >> Maximum Memory: 100 MB >> ------------------------------------------- >> File #: 1 of 1 >> Input Name: <stdin> >> Output Name: <stdout> >> >> Compressing data... >> Output Size: 205714360 bytes >> ------------------------------------------- >> >> Wall Clock: 5115.167256 seconds >> >> Don't let the files 1 of 1 fool you, that was because it was receiving >> input from 1 descriptor (tar). >> >> This was on a 4 core i7 2.7 GHZ iMac. >> >> Respectfully, >> Andrew McElroy >> >> >> >> >> >>> >>> >>> On Mon, Jun 16, 2014 at 2:27 PM, andrew mcelroy <[email protected]> >>> wrote: >>> >>>> I found myself having to compress a million files less than 12k in size >>>> each. >>>> tar was likely to take the better part of a full working day (possibly >>>> longer, I killed the process and didn't use pv (another great asleep at the >>>> prompt topic). >>>> >>>> Enter: pbzip2 >>>> http://compression.ca/pbzip2/ >>>> >>>> I'll let the website do it's own talking. It's a parallel bzip2 >>>> implementation which can auto detect the number of cores it has to work >>>> with. Fantastic stuff. >>>> >>>> Respectfully, >>>> Andrew McElroy >>>> >>>> -- >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "NLUG" group. >>>> To post to this group, send email to [email protected] >>>> To unsubscribe from this group, send email to >>>> [email protected] >>>> For more options, visit this group at >>>> http://groups.google.com/group/nlug-talk?hl=en >>>> >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "NLUG" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> ><> ... Jack >>> >>> "Whatever you do, work at it with all your heart"... Colossians 3:23 >>> "If you are not part of the solution, you are part of the precipitate" - >>> Henry J. Tillman >>> "Anyone who has never made a mistake, has never tried anything new." - >>> Albert Einstein >>> "You don't manage people; you manage things. You lead people." - Admiral >>> Grace Hopper, USN >>> "a nanosecond is the time it takes electrons to propigate 11.8 inches" - >>> " - http://youtu.be/JEpsKnWZrJ8 >>> "Life is complex: it has a real part and an imaginary part." - Martin >>> Terma >>> >>> -- >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "NLUG" group. >>> To post to this group, send email to [email protected] >>> To unsubscribe from this group, send email to >>> [email protected] >>> For more options, visit this group at >>> http://groups.google.com/group/nlug-talk?hl=en >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "NLUG" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> > -- > -- > You received this message because you are subscribed to the Google Groups > "NLUG" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nlug-talk?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "NLUG" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- -- You received this message because you are subscribed to the Google Groups "NLUG" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nlug-talk?hl=en --- You received this message because you are subscribed to the Google Groups "NLUG" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
