casperhart commented on issue #45369: URL: https://github.com/apache/arrow/issues/45369#issuecomment-2692890879
I have a similar issue with `libhdfs.dylib` on mac, even when it's installed and in the right place. I think the root of the problem is twofold: 1. `libhdfs.dylib` not being compiled for the specific architecture means it can't be loaded 2. The error message is misleading and says that the file can't be found, when it is actually in the right place and environment variables are set correctly. To test this, I created a docker container setup. Using a native ARM docker container, I can reproduce the same issue (but with `libhdfs.so` instead of `libhdfs.dylib`). If I run an emulated amd64 version of the same container, this issue goes away. (but different issues with emulation mean that it's not a viable workaround). Another symptom of this is when running `hdfs dfs -ls <some path>` from the mac or aarch64 container, I get this warning: ``` WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable ``` Would it be possible to include a platform-specific version of libhdfs with the wheel, and fall back to that if the hadoop native one can't be loaded? I've come across this same issue on a couple of platforms which has taken a long time to narrow down. It would also be nice if the source of the problem was clearer from the error message: file not found vs. found but unable to load. It's worth noting that I can't seem to find an aarch64 version of hadoop native libraries anywhere, even the release page for hadoop gives "file not found" for the aarch64 release https://hadoop.apache.org/release/3.4.1.html, so I don't know how to fully fix this on my system yet. I've also encountered this error on Linux systems, and I suspect that one is a mismatch between 32 and 64 bit libraries. It doesn't stop HDFS or Spark from working, but it does stop pyarrow. I had this issue > 1 year ago and temporarily gave up on pyarrow, so I'm glad I've finally figured out the problem! :). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
