Micah Kornfield created ARROW-4802:
--------------------------------------
Summary: [Python] Hadoop classpath discovery broken in 0.12 when
HADOOP_HOME is a symlin
Key: ARROW-4802
URL: https://issues.apache.org/jira/browse/ARROW-4802
Project: Apache Arrow
Issue Type: Bug
Components: Python
Reporter: Micah Kornfield
From [https://github.com/apache/arrow/issues/3748]:
CLASSPATH discovery was recently changed in
[{{d911850}}|https://github.com/apache/arrow/commit/d91185000945cec96abad41a230d05d3cdefff93]
to resolve ARROW-2113 and ARROW-3768.
Specifically, the logic used to find all jars under HADOOP_HOME uses the find
command directly
[arrow/python/pyarrow/hdfs.py|https://github.com/apache/arrow/blob/d91185000945cec96abad41a230d05d3cdefff93/python/pyarrow/hdfs.py#L144]
Line 144 in
[d911850|https://github.com/apache/arrow/commit/d91185000945cec96abad41a230d05d3cdefff93]
| |find_args = ('find', os.environ['HADOOP_HOME'], '-name', '*.jar')|
This will not work when HADOOP_HOME is a symlink, in which case '-L' needs to
be passed to the find command.
CLASSPATH can still be set explicitly, but this is a change in behavior as
HADOOP_HOME symlinks worked without issue before.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
