jorisvandenbossche commented on issue #15019: URL: https://github.com/apache/arrow/issues/15019#issuecomment-1360992352
The Arrow HDFS implementation is based on the JNI libhdfs, so it is expected that it doesn't work with libhdfs3. And since libhdfs doesn't seem to be packages by conda-forge, I don't think there is a way to run the hdfs tests in the conda-forge build purely based on conda packages (our own tests install the JNI library manually on top of the conda env, see eg ci/docker/conda-python-hdfs.dockerfile) We have had integration with libhdfs3 as well in the past (and you could switch between both drivers), but this was removed almost 3 years ago (https://github.com/apache/arrow/pull/6432), because the libhdfs3 project is unmaintained. Also the dask filesystem wrapper using libhdfs3 is archived (https://github.com/dask/hdfs3). Unless the libhdfs3 project would be revived, I don't think we should currently consider adding support for it again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
