Julius created ARROW-14973:
------------------------------
Summary: How to build wheels depending on PyArrow that are
compatible with multiple PyArrow versions
Key: ARROW-14973
URL: https://issues.apache.org/jira/browse/ARROW-14973
Project: Apache Arrow
Issue Type: New Feature
Reporter: Julius
Currently, for MongoDB's package [PyMongoArrow,
|https://github.com/mongodb-labs/mongo-arrow/tree/main/bindings/python]we build
our wheels following the instructions
[here|https://arrow.apache.org/docs/python/extending.html]. (here is the
[script|https://github.com/mongodb-labs/mongo-arrow/blob/main/bindings/python/release.sh]
we use) However, once these wheels are built they are no longer compatible
with versions of PyArrow other than the one that it was built with. Before we
added Python 3.10 support we built all of our wheels with PyArrow 3.0. Those
same wheels now break with the following message after upgrading PyArrow to 6.0:
{code:java}
➜ ~ pip install --upgrade pyarrow
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: pyarrow in
./Library/Python/3.8/lib/python/site-packages (3.0.0)
Collecting pyarrow
Downloading pyarrow-6.0.1-cp38-cp38-macosx_10_13_x86_64.whl (19.1 MB)
|████████████████████████████████| 19.1 MB 5.1 MB/s
Requirement already satisfied: numpy>=1.16.6 in
./Library/Python/3.8/lib/python/site-packages (from pyarrow) (1.21.4)
Installing collected packages: pyarrow
Attempting uninstall: pyarrow
Found existing installation: pyarrow 3.0.0
Uninstalling pyarrow-3.0.0:
Successfully uninstalled pyarrow-3.0.0
ERROR: pip's dependency resolver does not currently take into account all the
packages that are installed. This behaviour is the source of the following
dependency conflicts.
pymongoarrow 0.1.1 requires pyarrow<3.1,>=3, but you have pyarrow 6.0.1 which
is incompatible.
Successfully installed pyarrow-6.0.1
➜ ~ cd Work/mongo-arrow/bindings/python/test
➜ test git:(PYTHON-3008) ✗ python3 -m unittest discover EEEEEEEEEE
====================================================================== ERROR:
test_arrow (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_arrow Traceback (most recent
call last): File
"/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/unittest/loader.py",
line 436, in _find_test_path module = self._get_module_from_name(name)
File
"/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/unittest/loader.py",
line 377, in _get_module_from_name __import__(name) File
"/Users/julius/Work/mongo-arrow/bindings/python/test/test_arrow.py", line 20,
in <module> from pymongoarrow.api import aggregate_arrow_all,
find_arrow_all, Schema File
"/Users/julius/Library/Python/3.8/lib/python/site-packages/pymongoarrow/__init__.py",
line 18, in <module> from pymongoarrow.lib import libbson_version
ImportError:
dlopen(/Users/julius/Library/Python/3.8/lib/python/site-packages/pymongoarrow/lib.cpython-38-darwin.so,
2): Library not loaded: @rpath/libarrow.300.dylib Referenced from:
/Users/julius/Library/Python/3.8/lib/python/site-packages/pymongoarrow/lib.cpython-38-darwin.so
Reason: image not found
{code}
The same error occurs when using our newer wheels built with PyArrow 6.0 when
you downgrade to PyArrow 3.0:
{code:java}
➜ python git:(main) ✗ pip install pyarrow==3.0
Defaulting to user installation because normal site-packages is not writeable
Collecting pyarrow==3.0
Using cached pyarrow-3.0.0-cp38-cp38-macosx_10_13_x86_64.whl (14.1 MB)
Requirement already satisfied: numpy>=1.16.6 in
/Users/julius/Library/Python/3.8/lib/python/site-packages (from pyarrow==3.0)
(1.21.4)
Installing collected packages: pyarrow
Attempting uninstall: pyarrow
Found existing installation: pyarrow 6.0.1
Uninstalling pyarrow-6.0.1:
Successfully uninstalled pyarrow-6.0.1
ERROR: pip's dependency resolver does not currently take into account all the
packages that are installed. This behaviour is the source of the following
dependency conflicts.
pymongoarrow 0.2.0.dev0 requires pyarrow<6.1,>=6, but you have pyarrow 3.0.0
which is incompatible.
Successfully installed pyarrow-3.0.0
➜ python git:(main) ✗ cd tests
cd: no such file or directory: tests
➜ python git:(main) ✗ cd test
➜ test git:(main) ✗ python3 -m unittest discover
EEEEEEEEEE
======================================================================
ERROR: test_arrow (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_arrow
Traceback (most recent call last):
File
"/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/unittest/loader.py",
line 436, in _find_test_path
module = self._get_module_from_name(name)
File
"/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/unittest/loader.py",
line 377, in _get_module_from_name
__import__(name)
File "/Users/julius/Work/mongo-arrow/bindings/python/test/test_arrow.py",
line 20, in <module>
from pymongoarrow.api import aggregate_arrow_all, find_arrow_all, Schema
File
"/Users/julius/Library/Python/3.8/lib/python/site-packages/pymongoarrow/__init__.py",
line 18, in <module>
from pymongoarrow.lib import libbson_version
ImportError:
dlopen(/Users/julius/Library/Python/3.8/lib/python/site-packages/pymongoarrow/lib.cpython-38-darwin.so,
2): Library not loaded: @rpath/libarrow.600.dylib
Referenced from:
/Users/julius/Library/Python/3.8/lib/python/site-packages/pymongoarrow/lib.cpython-38-darwin.so
Reason: image not found
{code}
Is there a way to build wheels that support multiple PyArrow versions?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
