[
https://issues.apache.org/jira/browse/ARROW-7841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036450#comment-17036450
]
Krisztian Szucs commented on ARROW-7841:
----------------------------------------
This regression was probably introduced with
https://github.com/apache/arrow/commit/e12d2851fe60a3628d900f432950dc321db65ff9#diff-29a7b8eebea6dfdb3246dd2b853ba8dcR145
Could you please try to set either ARROW_LIBHDFS_DIR=$HADOOP_HOME/lib/native to
see whether that resolves the issue or not?
> pyarrow release 0.16.0 breaks `libhdfs.so` loading mechanism
> ------------------------------------------------------------
>
> Key: ARROW-7841
> URL: https://issues.apache.org/jira/browse/ARROW-7841
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.16.0
> Reporter: Jack Fan
> Priority: Major
> Fix For: 0.15.1
>
>
> I have my env variable setup correctly according to the pyarrow README
> {code:java}
> $ ls $HADOOP_HOME/lib/native
> libhadoop.a libhadooppipes.a libhadoop.so libhadoop.so.1.0.0
> libhadooputils.a libhdfs.a libhdfs.so libhdfs.so.0.0.0 {code}
> Use the following script to reproduce
> {code:java}
> import pyarrow
> pyarrow.hdfs.connect('hdfs://localhost'){code}
> With pyarrow version 0.15.1 it is fine.
> However, version 0.16.0 will give error
> {code:java}
> Traceback (most recent call last):
> File "<string>", line 2, in <module>
> File
> "/home/jackwindows/anaconda2/lib/python2.7/site-packages/pyarrow/hdfs.py",
> line 215, in connect
> extra_conf=extra_conf)
> File
> "/home/jackwindows/anaconda2/lib/python2.7/site-packages/pyarrow/hdfs.py",
> line 40, in __init__
> self._connect(host, port, user, kerb_ticket, driver, extra_conf)
> File "pyarrow/io-hdfs.pxi", line 89, in
> pyarrow.lib.HadoopFileSystem._connect
> File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
> IOError: Unable to load libhdfs: /opt/hadoop/latest/libhdfs.so: cannot open
> shared object file: No such file or directory {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
