Hi, On Wed, Feb 29, 2012 at 16:52, Edward Capriolo <edlinuxg...@gmail.com>wrote: ...
> But being able to generate split info for them and processing them > would be good as well. I remember that was a hot thing to do with lzo > back in the day. The pain of once overing the gz files to generate the > split info is detracting but it is nice to know it is there if you > want it. > Note that the solution I created (HADOOP-7076) does not require any preprocessing. It can split ANY gzipped file as-is. The downside is that this effectively costs some additional performance because the task has to decompress the first part of the file that is to be discarded. The other two ways of splitting gzipped files either require - creating come kind of "compression index" before actually using the file (HADOOP-6153) - creating a file in a format that is gerenated in such a way that it is really a set of concatenated gzipped files. (HADOOP-7909) -- Best regards / Met vriendelijke groeten, Niels Basjes
