Hi Arrow devs,
I'm not sure if this is the correct place to ask, if not, please point
me in the right direction.
I wanted to use the HDFS client with PySpark (for now). My Spark
cluster is on Amazon EMR, so the nodes use Amazon Linux (2017.03). On
my dev machine (Fedora 25) with Python 3.5.1, a simple `pip3 install
pyarrow` works just fine, and I can use the HDFS client. But when I do
the same on Amazon Linux, it complains about incompatible Python
version, then pip tries to download an old tarball (0.3), and finally
cmake fails with missing cmake files (PythonLibsNewConfig.cmake).
Here's a snippet from the failure with verbose on:
$ sudo python3.5 -m pip -vvv install pyarrow
[..]
Skipping
https://pypi.python.org/../pyarrow-0.4.1-cp35-cp35m-manylinux1_x86_64.whl#md5=..
(from https://pypi.python.org/simple/pyarrow/) because it is not compatible
with this Python
Skipping
https://pypi.python.org/../pyarrow-0.4.0-cp27-cp27mu-manylinux1_x86_64.whl#md5=..
(from https://pypi.python.org/simple/pyarrow/) because it is not compatible
with this Python
Skipping
https://pypi.python.org/../pyarrow-0.2.0-cp34-cp34m-manylinux1_x86_64.whl#md5=..
(from https://pypi.python.org/simple/pyarrow/) because it is not compatible
with this Python
Using version 0.3.0 (newest of versions: 0.3.0, 0.2.0)
[..]
When I explicitly specify which wheel file to use, it fails like this:
$ sudo python3.5 -m pip -vvv install
https://pypi.python.org/../pyarrow-0.4.1-cp35-cp35m-manylinux1_x86_64.whl#md5=..
pyarrow-0.4.1-cp35-cp35m-manylinux1_x86_64.whl is not a supported wheel on
this platform.
Exception information:
Traceback (most recent call last):
File "/usr/lib/python3.5/dist-packages/pip/basecommand.py", line 246, in
main
status = self.run(options, args)
File "/usr/lib/python3.5/dist-packages/pip/commands/install.py", line 308,
in run
name, None, isolated=options.isolated_mode,
File "/usr/lib/python3.5/dist-packages/pip/req/req_install.py", line 207,
in from_line
wheel.filename
pip.exceptions.UnsupportedWheel:
pyarrow-0.4.1-cp35-cp35m-manylinux1_x86_64.whl is not a supported wheel on this
platform.
Any ideas what is going wrong? Thanks for any help.
Cheers,
--
Suvayu
Open source is the future. It sets us free.
