The stripping of DYLD* and LD* variables is a "feature" that's part of Apple's SIP. So another option to stop this is to disable SIP - https://developer.apple.com/documentation/security/disabling_and_enabling_system_integrity_protection
Apple doesn't recommend this, but I've been running different macbooks with SIP disabled for years, and haven't noticed any bad side effects. As an aside, Apple has a history of crippling various functionalities behind SIP. For example, in the latest versions of Monterey, you can't run certain `dscl` commands unless you either disable SIP or provide "Full Disk Access" to bash. Thanks, Hariharan On Fri, 25 Mar 2022, 21:46 Andrew Purtell, <andrew.purt...@gmail.com> wrote: > Thank you for sharing that blog post. > > > The TL;DR is that as soon as macOS executes one if its trusted > executables, like /bin/sh or /usr/bin/env, it cripples anything you might > have done like setting DYLD_LIBRARY_PATH to dynamic library folders, and > results in failure to load them. > > On the one hand I can see the security requirements that led to this > decision, but this is so contrary to the UNIX philosophy, IMHO, it's no > wonder it violates the principle of least surprise here and I bet in many > many other situations. This reminds me why 'step 1' of setting up for dev > on my new M1 macbook was to install Parallels and a Linux aarch64 VM. That > environment is quite sane and the VM overheads are manageable. > > > On Fri, Mar 25, 2022 at 9:03 AM Joe Mocker <jmoc...@magnite.com> wrote: > >> Hi, Thanks... >> >> It ended up being more involved than that due to all the shared library >> dependencies, but I figured it out (at least with an older version of >> Hadoop). I ended up writing a short post about it >> >> >> https://creechy.wordpress.com/2022/03/22/building-hadoop-spark-jupyter-on-macos/ >> >> --joe >> >> On Thu, Mar 24, 2022 at 3:14 PM Andrew Purtell <andrew.purt...@gmail.com> >> wrote: >> >>> If you build with -Dbundle.snappy -Dbundle.zstd on the Maven command >>> line this would produce a tarball containing copies of the native shared >>> libraries in lib/native/ and this would be like your symlink workaround but >>> perhaps less hacky and something the build supports already. Does this work >>> for you? >>> >>> On Mar 19, 2022, at 8:09 AM, Joe Mocker <jmoc...@magnite.com.invalid> >>> wrote: >>> >>> Hi, >>> >>> Curious if anyone has tips for building Hadoop on macOS Monterey, for >>> Apple Silicon? My goal is to be able to use native (compression) libraries. >>> After some gymnastics, I have been able to compile Hadoop 2.9.1 but >>> problems arise locating and loading dynamic libraries. >>> >>> For example running hadoop checknative results in the following >>> >>> 22/03/19 07:57:00 WARN bzip2.Bzip2Factory: Failed to load/initialize >>> native-bzip2 library system-native, will use pure-Java version >>> 22/03/19 07:57:00 INFO zlib.ZlibFactory: Successfully loaded & >>> initialized native-zlib library >>> 22/03/19 07:57:00 ERROR snappy.SnappyCompressor: failed to load >>> SnappyCompressor >>> java.lang.UnsatisfiedLinkError: Cannot load libsnappy.1.dylib >>> (dlopen(libsnappy.1.dylib, 0x0009): tried: >>> '/Volumes/work/zulu8.60.0.21-ca-jdk8.0.322-macosx_aarch64/zulu-8.jdk/Contents/Home/bin/./libsnappy.1.dylib' >>> (no such file), 'libsnappy.1.dylib' (no such file), >>> '/usr/lib/libsnappy.1.dylib' (no such file), >>> '/Volumes/work/hadoop-2.9.1/libsnappy.1.dylib' (no such file))! >>> at org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs(Native >>> Method) >>> at >>> org.apache.hadoop.io.compress.snappy.SnappyCompressor.<clinit>(SnappyCompressor.java:57) >>> at >>> org.apache.hadoop.io.compress.SnappyCodec.isNativeCodeLoaded(SnappyCodec.java:82) >>> at >>> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:92) >>> 22/03/19 07:57:00 WARN zstd.ZStandardCompressor: Error loading zstandard >>> native libraries: java.lang.InternalError: Cannot load libzstd.1.dylib >>> (dlopen(libzstd.1.dylib, 0x0009): tried: >>> '/Volumes/work/zulu8.60.0.21-ca-jdk8.0.322-macosx_aarch64/zulu-8.jdk/Contents/Home/bin/./libzstd.1.dylib' >>> (no such file), 'libzstd.1.dylib' (no such file), >>> '/usr/lib/libzstd.1.dylib' (no such file), >>> '/Volumes/work/hadoop-2.9.1/libzstd.1.dylib' (no such file))! >>> WARNING: /work/zulu8.60.0.21-ca-jdk8.0.322-macosx_aarch64//bin/java is >>> loading libcrypto in an unsafe way >>> Abort trap: 6 >>> >>> No matter what combination I try of setting LD_LIBRARY_PATH, >>> DYLD_LIBRARY_PATH and/or DYLD_FALLBACK_LIBRARY_PATH it will not find the >>> necessary libraries. I think this has to do with restrictions due to >>> Apple’s System Integrity Protection (SIP). >>> >>> The only way I have figured out how to work around this so far is to >>> symlink all the dynamic libraries in one location then run hadoop from that >>> working directory, for example >>> >>> lrwxrwxr-x 1 mock staff 59 Mar 18 17:55 libcrypto.dylib@ -> >>> /opt/homebrew/Cellar/openssl@1.1/1.1.1m/lib/libcrypto.dylib >>> lrwxrwxr-x 1 mock staff 45 Mar 18 18:09 libhadoop.dylib@ -> >>> /work/hadoop-2.9.1/lib/native/libhadoop.dylib >>> lrwxrwxr-x 1 mock staff 53 Mar 18 17:55 libsnappy.1.dylib@ -> >>> /opt/homebrew/Cellar/snappy/1.1.9/lib/libsnappy.dylib >>> lrwxrwxr-x 1 mock staff 51 Mar 18 18:05 libzstd.1.dylib@ -> >>> /opt/homebrew/Cellar/zstd/1.5.2/lib/libzstd.1.dylib >>> >>> % $HADOOP_HOME/bin/hadoop checknative >>> 22/03/19 08:05:55 WARN bzip2.Bzip2Factory: Failed to load/initialize >>> native-bzip2 library system-native, will use pure-Java version >>> 22/03/19 08:05:55 INFO zlib.ZlibFactory: Successfully loaded & >>> initialized native-zlib library >>> Native library checking: >>> hadoop: true /Volumes/work/hadoop-2.9.1/lib/native/libhadoop.dylib >>> zlib: true /usr/lib/libz.1.dylib >>> snappy: true /opt/homebrew/Cellar/snappy/1.1.9/lib/libsnappy.1.1.9.dylib >>> zstd : true /opt/homebrew/Cellar/zstd/1.5.2/lib/libzstd.1.5.2.dylib >>> lz4: true revision:10301 >>> bzip2: false >>> openssl: false EVP_CIPHER_CTX_cleanup >>> >>> What am really looking to do is use Spark (and Jupyter), with native >>> libraries, which adds even more wrinkles to it. >>> >>> Any suggestions would be appreciated. >>> >>> —joe >>> >>> > > -- > Best regards, > Andrew > > Unrest, ignorance distilled, nihilistic imbeciles - > It's what we’ve earned > Welcome, apocalypse, what’s taken you so long? > Bring us the fitting end that we’ve been counting on > - A23, Welcome, Apocalypse >
