Hello,
I am using Pig 0.17.0 and I am trying to enable Snappy compression for
temporary files.
I installed Snappy on all the Hadoop nodes:
sudo yum install snappy snappy-devel
ln -sf /usr/lib64/libsnappy.so /opt/hadoop/lib/native/
Yum installed the following packages:
Installed Packages
Name : snappy
Arch : x86_64
Version : 1.1.0
Release : 3.el7
Size : 117 k
Repo : installed
From repo : anaconda
Summary : Fast compression and decompression library
URL : http://code.google.com/p/snappy/
License : BSD
Description : Snappy is a compression/decompression library. It does not aim
for maximum
: compression, or compatibility with any other compression library;
instead, it
: aims for very high speeds and reasonable compression. For
instance, compared to
: the fastest mode of zlib, Snappy is an order of magnitude faster
for most
: inputs, but the resulting compressed files are anywhere from 20%
to 100%
: bigger.
Name : snappy-devel
Arch : x86_64
Version : 1.1.0
Release : 3.el7
Size : 29 k
Repo : installed
From repo : base
Summary : Development files for snappy
URL : http://code.google.com/p/snappy/
License : BSD
Description : The snappy-devel package contains libraries and header files for
: developing applications that use snappy.
In the pig.properties file, I made the following changes:
pig.tmpfilecompression=true
pig.tmpfilecompression.storage=seqfile
pig.tmpfilecompression.codec=snappy
Now, when I run my script, I get this error:
java.lang.ClassCastException: class org.xerial.snappy.SnappyCodec
In my Pig’s lib directory I can see this jar:
snappy-java-1.1.1.3.jar
Looks like incompatible version, but I can’t find any information regarding
which version of Snappy is compatible with this version of Pig.
I will appreciate any input on this.
Best regards,
Alex soto
