Yes, it should be part of your Hadoop distribution. Since the zstd will call native implementations to speed up the decompression these libraries need to be there at compile-time to wire everything correctly.
You can check if they are: $ hadoop checknative -a 14/12/06 01:30:45 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version 14/12/06 01:30:45 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library Native library checking: hadoop: true /home/ozawa/hadoop/lib/native/libhadoop.so.1.0.0 zlib: true /lib/x86_64-linux-gnu/libz.so.1 snappy: true /usr/lib/libsnappy.so.1 zstd: true /usr/lib/libzstd.so.1 lz4: true revision:99 bzip2: false Hope this helps. Cheers, Fokko Op wo 31 jul. 2019 om 19:03 schreef Manik Singla <[email protected]>: > I downloaded and added libHadoop.so to make it work. > I will compile it separately for mac. Lemme know if we can make it work > using just Jars or readymade soln > > > Regards > Manik Singla > +91-9996008893 > +91-9665639677 > > "Life doesn't consist in holding good cards but playing those you hold > well." > > > On Wed, Jul 31, 2019 at 3:34 PM Manik Singla <[email protected]> wrote: > > > Hadoop common fails with native zStandard library not available: this > > version of libhadoop was built without zstd support. > > > > I don't want to compile libhadoop for myself > > > > On Wed, 31 Jul, 2019, 3:19 PM Driesprong, Fokko, <[email protected]> > > wrote: > > > >> Hi Manik, > >> > >> Can you explain how you en- and decode the messages? Please keep in mind > >> that with Parquet only the actual data is compressed. For example, the > >> metadata in the footer isn't compressed. > >> > >> Cheers, Fokko > >> > >> Op wo 31 jul. 2019 om 11:33 schreef Manik Singla <[email protected]>: > >> > >> > Hi Guys > >> > > >> > We are trying to use zstd but there are compatibility issues with > >> standard > >> > binaries. > >> > I am using latest of parquet jars. > >> > > >> > Am i missing something? > >> > > >> > Regards > >> > Manik Singla > >> > +91-9996008893 > >> > +91-9665639677 > >> > > >> > "Life doesn't consist in holding good cards but playing those you hold > >> > well." > >> > > >> > > >
