Hello Pig mailing list, I have around 10 TB of apache log files (1 TB as .gz compressed files) and analyze these files with pig. Obviously apache log files can be compressed pretty good with gzip, so it would be great if Pig would accept the log files in compressed form.
Is this possible with the CombinedLogLoader from contrib/piggybank or is there any other way to do this? It is pretty easy with the normal TextLoader. It automatically detects if the file is a .gz file. If there is no way, would the RegExLoader be the correct class to extend? Regards Dirk
