Is bzip2 not advisable? I think it can split too and is supported out of the box.
On Thu, Mar 29, 2012 at 8:08 PM, 帝归 <[email protected]> wrote: > When I use LzoPigStorage, it will load all files under a directory. But I > want compress every file under a directory and keep the file name > unchanged, just with a .lzo extension name. How can I do this? Maybe I must > write a mapreduce job? > > 2012/3/30 Jonathan Coveney <[email protected]> > > > check out: > > > > > https://github.com/kevinweil/elephant-bird/tree/master/src/java/com/twitter/elephantbird/pig/store > > > > 2012/3/29 Mohit Anchlia <[email protected]> > > > > > Thanks! When I store output how can I tell pig to compress it in LZO > > > format? > > > > > > On Thu, Mar 29, 2012 at 4:02 PM, Dmitriy Ryaboy <[email protected]> > > > wrote: > > > > > > > You might find the elephant-bird project helpful for reading and > > > > creating LZO files, in raw hadoop or using Pig. > > > > (disclaimer: I'm a committer on elephant-bird) > > > > > > > > D > > > > > > > > On Wed, Mar 28, 2012 at 9:49 AM, Prashant Kommireddi > > > > <[email protected]> wrote: > > > > > Pig support LZO for splittable compression. > > > > > > > > > > Thanks, > > > > > Prashant > > > > > > > > > > On Mar 28, 2012, at 9:45 AM, Mohit Anchlia <[email protected] > > > > > > wrote: > > > > > > > > > >> We currently have 100s of GB of uncompressed data which we would > > like > > > to > > > > >> zip using some compression that is block compression so that we > can > > > use > > > > >> multiple input splits. Does pig support any such compression? > > > > > > > > > > > > > -- > ‘(hello world) >
