Hello, there are two possibilities that directly come to my mind:
1. Your pip is too old, you need at least 8.1.2 2. You're on a 32bit system Given that https://aws.amazon.com/amazon-linux-ami/2017.03-packages/ states python-pip-6.1.1, mostly likely option 1 is your problem and can be solved by upgrading to the latest pip version. Uwe On Sun, Jul 2, 2017, at 02:01 PM, Suvayu Ali wrote: > Hi Arrow devs, > > I'm not sure if this is the correct place to ask, if not, please point > me in the right direction. > > I wanted to use the HDFS client with PySpark (for now). My Spark > cluster is on Amazon EMR, so the nodes use Amazon Linux (2017.03). On > my dev machine (Fedora 25) with Python 3.5.1, a simple `pip3 install > pyarrow` works just fine, and I can use the HDFS client. But when I do > the same on Amazon Linux, it complains about incompatible Python > version, then pip tries to download an old tarball (0.3), and finally > cmake fails with missing cmake files (PythonLibsNewConfig.cmake). > > Here's a snippet from the failure with verbose on: > > $ sudo python3.5 -m pip -vvv install pyarrow > [..] > Skipping > > https://pypi.python.org/../pyarrow-0.4.1-cp35-cp35m-manylinux1_x86_64.whl#md5=.. > > (from https://pypi.python.org/simple/pyarrow/) because it is not > compatible with this Python > Skipping > > https://pypi.python.org/../pyarrow-0.4.0-cp27-cp27mu-manylinux1_x86_64.whl#md5=.. > (from https://pypi.python.org/simple/pyarrow/) because it is not > compatible with this Python > Skipping > > https://pypi.python.org/../pyarrow-0.2.0-cp34-cp34m-manylinux1_x86_64.whl#md5=.. > (from https://pypi.python.org/simple/pyarrow/) because it is not > compatible with this Python > Using version 0.3.0 (newest of versions: 0.3.0, 0.2.0) > [..] > > When I explicitly specify which wheel file to use, it fails like this: > > $ sudo python3.5 -m pip -vvv install > > https://pypi.python.org/../pyarrow-0.4.1-cp35-cp35m-manylinux1_x86_64.whl#md5=.. > pyarrow-0.4.1-cp35-cp35m-manylinux1_x86_64.whl is not a supported wheel > on this platform. > Exception information: > Traceback (most recent call last): > File "/usr/lib/python3.5/dist-packages/pip/basecommand.py", line 246, > in main > status = self.run(options, args) > File "/usr/lib/python3.5/dist-packages/pip/commands/install.py", line > 308, in run > name, None, isolated=options.isolated_mode, > File "/usr/lib/python3.5/dist-packages/pip/req/req_install.py", line > 207, in from_line > wheel.filename > pip.exceptions.UnsupportedWheel: > pyarrow-0.4.1-cp35-cp35m-manylinux1_x86_64.whl is not a supported wheel > on this platform. > > Any ideas what is going wrong? Thanks for any help. > > Cheers, > > -- > Suvayu > > Open source is the future. It sets us free.
