Hi, Thanks... It ended up being more involved than that due to all the shared library dependencies, but I figured it out (at least with an older version of Hadoop). I ended up writing a short post about it
https://creechy.wordpress.com/2022/03/22/building-hadoop-spark-jupyter-on-macos/ --joe On Thu, Mar 24, 2022 at 3:14 PM Andrew Purtell <andrew.purt...@gmail.com> wrote: > If you build with -Dbundle.snappy -Dbundle.zstd on the Maven command line > this would produce a tarball containing copies of the native shared > libraries in lib/native/ and this would be like your symlink workaround but > perhaps less hacky and something the build supports already. Does this work > for you? > > On Mar 19, 2022, at 8:09 AM, Joe Mocker <jmoc...@magnite.com.invalid> > wrote: > > Hi, > > Curious if anyone has tips for building Hadoop on macOS Monterey, for > Apple Silicon? My goal is to be able to use native (compression) libraries. > After some gymnastics, I have been able to compile Hadoop 2.9.1 but > problems arise locating and loading dynamic libraries. > > For example running hadoop checknative results in the following > > 22/03/19 07:57:00 WARN bzip2.Bzip2Factory: Failed to load/initialize > native-bzip2 library system-native, will use pure-Java version > 22/03/19 07:57:00 INFO zlib.ZlibFactory: Successfully loaded & initialized > native-zlib library > 22/03/19 07:57:00 ERROR snappy.SnappyCompressor: failed to load > SnappyCompressor > java.lang.UnsatisfiedLinkError: Cannot load libsnappy.1.dylib > (dlopen(libsnappy.1.dylib, 0x0009): tried: > '/Volumes/work/zulu8.60.0.21-ca-jdk8.0.322-macosx_aarch64/zulu-8.jdk/Contents/Home/bin/./libsnappy.1.dylib' > (no such file), 'libsnappy.1.dylib' (no such file), > '/usr/lib/libsnappy.1.dylib' (no such file), > '/Volumes/work/hadoop-2.9.1/libsnappy.1.dylib' (no such file))! > at org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs(Native > Method) > at > org.apache.hadoop.io.compress.snappy.SnappyCompressor.<clinit>(SnappyCompressor.java:57) > at > org.apache.hadoop.io.compress.SnappyCodec.isNativeCodeLoaded(SnappyCodec.java:82) > at > org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:92) > 22/03/19 07:57:00 WARN zstd.ZStandardCompressor: Error loading zstandard > native libraries: java.lang.InternalError: Cannot load libzstd.1.dylib > (dlopen(libzstd.1.dylib, 0x0009): tried: > '/Volumes/work/zulu8.60.0.21-ca-jdk8.0.322-macosx_aarch64/zulu-8.jdk/Contents/Home/bin/./libzstd.1.dylib' > (no such file), 'libzstd.1.dylib' (no such file), > '/usr/lib/libzstd.1.dylib' (no such file), > '/Volumes/work/hadoop-2.9.1/libzstd.1.dylib' (no such file))! > WARNING: /work/zulu8.60.0.21-ca-jdk8.0.322-macosx_aarch64//bin/java is > loading libcrypto in an unsafe way > Abort trap: 6 > > No matter what combination I try of setting LD_LIBRARY_PATH, > DYLD_LIBRARY_PATH and/or DYLD_FALLBACK_LIBRARY_PATH it will not find the > necessary libraries. I think this has to do with restrictions due to > Apple’s System Integrity Protection (SIP). > > The only way I have figured out how to work around this so far is to > symlink all the dynamic libraries in one location then run hadoop from that > working directory, for example > > lrwxrwxr-x 1 mock staff 59 Mar 18 17:55 libcrypto.dylib@ -> > /opt/homebrew/Cellar/openssl@1.1/1.1.1m/lib/libcrypto.dylib > lrwxrwxr-x 1 mock staff 45 Mar 18 18:09 libhadoop.dylib@ -> > /work/hadoop-2.9.1/lib/native/libhadoop.dylib > lrwxrwxr-x 1 mock staff 53 Mar 18 17:55 libsnappy.1.dylib@ -> > /opt/homebrew/Cellar/snappy/1.1.9/lib/libsnappy.dylib > lrwxrwxr-x 1 mock staff 51 Mar 18 18:05 libzstd.1.dylib@ -> > /opt/homebrew/Cellar/zstd/1.5.2/lib/libzstd.1.dylib > > % $HADOOP_HOME/bin/hadoop checknative > 22/03/19 08:05:55 WARN bzip2.Bzip2Factory: Failed to load/initialize > native-bzip2 library system-native, will use pure-Java version > 22/03/19 08:05:55 INFO zlib.ZlibFactory: Successfully loaded & initialized > native-zlib library > Native library checking: > hadoop: true /Volumes/work/hadoop-2.9.1/lib/native/libhadoop.dylib > zlib: true /usr/lib/libz.1.dylib > snappy: true /opt/homebrew/Cellar/snappy/1.1.9/lib/libsnappy.1.1.9.dylib > zstd : true /opt/homebrew/Cellar/zstd/1.5.2/lib/libzstd.1.5.2.dylib > lz4: true revision:10301 > bzip2: false > openssl: false EVP_CIPHER_CTX_cleanup > > What am really looking to do is use Spark (and Jupyter), with native > libraries, which adds even more wrinkles to it. > > Any suggestions would be appreciated. > > —joe > >
