[
https://issues.apache.org/jira/browse/ARROW-14973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weston Pace updated ARROW-14973:
--------------------------------
Summary: [python][CI]How to build wheels depending on PyArrow that are
compatible with multiple PyArrow versions (was: How to build wheels depending
on PyArrow that are compatible with multiple PyArrow versions)
> [python][CI]How to build wheels depending on PyArrow that are compatible with
> multiple PyArrow versions
> -------------------------------------------------------------------------------------------------------
>
> Key: ARROW-14973
> URL: https://issues.apache.org/jira/browse/ARROW-14973
> Project: Apache Arrow
> Issue Type: New Feature
> Reporter: Julius
> Priority: Major
>
> Currently, for MongoDB's package [PyMongoArrow,
> |https://github.com/mongodb-labs/mongo-arrow/tree/main/bindings/python]we
> build our wheels following the instructions
> [here|https://arrow.apache.org/docs/python/extending.html]. (here is the
> [script|https://github.com/mongodb-labs/mongo-arrow/blob/main/bindings/python/release.sh]
> we use) However, once these wheels are built they are no longer compatible
> with versions of PyArrow other than the one that it was built with. Before we
> added Python 3.10 support we built all of our wheels with PyArrow 3.0. Those
> same wheels now break with the following message after upgrading PyArrow to
> 6.0:
> {code:java}
> ➜ ~ pip install --upgrade pyarrow
> Defaulting to user installation because normal site-packages is not writeable
> Requirement already satisfied: pyarrow in
> ./Library/Python/3.8/lib/python/site-packages (3.0.0)
> Collecting pyarrow
> Downloading pyarrow-6.0.1-cp38-cp38-macosx_10_13_x86_64.whl (19.1 MB)
> |████████████████████████████████| 19.1 MB 5.1 MB/s
> Requirement already satisfied: numpy>=1.16.6 in
> ./Library/Python/3.8/lib/python/site-packages (from pyarrow) (1.21.4)
> Installing collected packages: pyarrow
> Attempting uninstall: pyarrow
> Found existing installation: pyarrow 3.0.0
> Uninstalling pyarrow-3.0.0:
> Successfully uninstalled pyarrow-3.0.0
> ERROR: pip's dependency resolver does not currently take into account all the
> packages that are installed. This behaviour is the source of the following
> dependency conflicts.
> pymongoarrow 0.1.1 requires pyarrow<3.1,>=3, but you have pyarrow 6.0.1 which
> is incompatible.
> Successfully installed pyarrow-6.0.1
> ➜ ~ cd Work/mongo-arrow/bindings/python/test
> ➜ test git:(PYTHON-3008) ✗ python3 -m unittest discover EEEEEEEEEE
> ====================================================================== ERROR:
> test_arrow (unittest.loader._FailedTest)
> ----------------------------------------------------------------------
> ImportError: Failed to import test module: test_arrow Traceback (most recent
> call last): File
> "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/unittest/loader.py",
> line 436, in _find_test_path module = self._get_module_from_name(name)
> File
> "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/unittest/loader.py",
> line 377, in _get_module_from_name __import__(name) File
> "/Users/julius/Work/mongo-arrow/bindings/python/test/test_arrow.py", line 20,
> in <module> from pymongoarrow.api import aggregate_arrow_all,
> find_arrow_all, Schema File
> "/Users/julius/Library/Python/3.8/lib/python/site-packages/pymongoarrow/__init__.py",
> line 18, in <module> from pymongoarrow.lib import libbson_version
> ImportError:
> dlopen(/Users/julius/Library/Python/3.8/lib/python/site-packages/pymongoarrow/lib.cpython-38-darwin.so,
> 2): Library not loaded: @rpath/libarrow.300.dylib Referenced from:
> /Users/julius/Library/Python/3.8/lib/python/site-packages/pymongoarrow/lib.cpython-38-darwin.so
> Reason: image not found
> {code}
> The same error occurs when using our newer wheels built with PyArrow 6.0 when
> you downgrade to PyArrow 3.0:
> {code:java}
> ➜ python git:(main) ✗ pip install pyarrow==3.0
> Defaulting to user installation because normal site-packages is not writeable
> Collecting pyarrow==3.0
> Using cached pyarrow-3.0.0-cp38-cp38-macosx_10_13_x86_64.whl (14.1 MB)
> Requirement already satisfied: numpy>=1.16.6 in
> /Users/julius/Library/Python/3.8/lib/python/site-packages (from pyarrow==3.0)
> (1.21.4)
> Installing collected packages: pyarrow
> Attempting uninstall: pyarrow
> Found existing installation: pyarrow 6.0.1
> Uninstalling pyarrow-6.0.1:
> Successfully uninstalled pyarrow-6.0.1
> ERROR: pip's dependency resolver does not currently take into account all the
> packages that are installed. This behaviour is the source of the following
> dependency conflicts.
> pymongoarrow 0.2.0.dev0 requires pyarrow<6.1,>=6, but you have pyarrow 3.0.0
> which is incompatible.
> Successfully installed pyarrow-3.0.0
> ➜ python git:(main) ✗ cd tests
> cd: no such file or directory: tests
> ➜ python git:(main) ✗ cd test
> ➜ test git:(main) ✗ python3 -m unittest discover
> EEEEEEEEEE
> ======================================================================
> ERROR: test_arrow (unittest.loader._FailedTest)
> ----------------------------------------------------------------------
> ImportError: Failed to import test module: test_arrow
> Traceback (most recent call last):
> File
> "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/unittest/loader.py",
> line 436, in _find_test_path
> module = self._get_module_from_name(name)
> File
> "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/unittest/loader.py",
> line 377, in _get_module_from_name
> __import__(name)
> File "/Users/julius/Work/mongo-arrow/bindings/python/test/test_arrow.py",
> line 20, in <module>
> from pymongoarrow.api import aggregate_arrow_all, find_arrow_all, Schema
> File
> "/Users/julius/Library/Python/3.8/lib/python/site-packages/pymongoarrow/__init__.py",
> line 18, in <module>
> from pymongoarrow.lib import libbson_version
> ImportError:
> dlopen(/Users/julius/Library/Python/3.8/lib/python/site-packages/pymongoarrow/lib.cpython-38-darwin.so,
> 2): Library not loaded: @rpath/libarrow.600.dylib
> Referenced from:
> /Users/julius/Library/Python/3.8/lib/python/site-packages/pymongoarrow/lib.cpython-38-darwin.so
> Reason: image not found
> {code}
> Is there a way to build wheels that support multiple PyArrow versions?
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
