Re: Getting an error when trying to read a GZIPPED file

Are you doing a .cache after the sc.textFile? If so, you can set the StorageLevel to MEMORY_AND_DISK to avoid that. Thanks Best Regards On Thu, Sep 3, 2015 at 10:11 AM, Spark Enthusiast wrote: > Folks, > > I have an input file which is gzipped. I use

Getting an error when trying to read a GZIPPED file

Folks, I have an input file which is gzipped. I use sc.textFile("foo.gz") when I see the following problem. Can someone help me how to fix this? 15/09/03 10:05:32 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id15/09/03 10:05:32 INFO CodecPool: Got brand-new