potiuk commented on PR #36755: URL: https://github.com/apache/airflow/pull/36755#issuecomment-1954688181
> As I said if packages are always installed as wheel is would work out of the box, libev should be baked into the wheel. > If not, and you can enforce libev install before install from pip, it would work as well, see https://docs.datastax.com/en/developer/python-driver/3.14/installation/#libev-support for more details of need thing to be able to compile from source. @fruch . Just double checked it and for Python 3.12, whells for linux do not seem to be compiled with libev support. That basically makes python 3.12 cassandra-driver whl useless as it falls back to asyncore that is missing in Python 3.12 When I install it from PyPI with wheels, our test fail: ``` root@d71d7cdeb0d9:/opt/airflow# apt list libev4 Listing... Done libev4/stable,now 1:4.33-1 amd64 [installed] root@d71d7cdeb0d9:/opt/airflow# apt list libev-dev Listing... Done libev-dev/stable,now 1:4.33-1 amd64 [installed] .... Downloading cassandra_driver-3.29.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.6/19.6 MB 12.3 MB/s eta 0:00:00 Installing collected packages: cassandra-driver Successfully installed cassandra-driver-3.29.0 .... pytest tests/providers/google/cloud/transfers/test_cassandra_to_gcs.py _________________________________________________________________________________________________________ ERROR collecting tests/providers/google/cloud/transfers/test_cassandra_to_gcs.py _________________________________________________________________________________________________________ tests/providers/google/cloud/transfers/test_cassandra_to_gcs.py:27: in <module> from airflow.providers.google.cloud.transfers.cassandra_to_gcs import CassandraToGCSOperator # noqa: E402 airflow/providers/google/cloud/transfers/cassandra_to_gcs.py:34: in <module> from airflow.providers.apache.cassandra.hooks.cassandra import CassandraHook airflow/providers/apache/cassandra/hooks/cassandra.py:25: in <module> from cassandra.cluster import Cluster, Session cassandra/cluster.py:173: in init cassandra.cluster ??? E cassandra.DependencyException: Unable to load a default connection class E The following exceptions were observed: E - The C extension needed to use libev was not found. This probably means that you didn't have the required build dependencies when installing the driver. See http://datastax.github.io/python-driver/installation.html#c-extensions for instructions on installing build dependencies and building the C extension. E - Unable to import asyncore module. Note that this module has been removed in Python 3.12 so when using the driver with this version (or anything newer) you will need to use one of the other event loop implementations. ``` I tested that when I build cassandra driver from sdist, the test start to work (so libev is used) ``` oot@d71d7cdeb0d9:/opt/airflow# pip uninstall cassandra-driver Found existing installation: cassandra-driver 3.29.0 Uninstalling cassandra-driver-3.29.0: Would remove: /usr/local/lib/python3.12/site-packages/cassandra/* /usr/local/lib/python3.12/site-packages/cassandra_driver-3.29.0.dist-info/* Proceed (Y/n)? y Successfully uninstalled cassandra-driver-3.29.0 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv root@d71d7cdeb0d9:/opt/airflow# CASS_DRIVER_BUILD_CONCURRENCY=8 pip install cassandra-driver --no-binary ":all:" Collecting cassandra-driver Downloading cassandra-driver-3.29.0.tar.gz (292 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 292.7/292.7 kB 4.5 MB/s eta 0:00:00 Preparing metadata (setup.py) ... done Requirement already satisfied: geomet<0.3,>=0.1 in /usr/local/lib/python3.12/site-packages (from cassandra-driver) (0.2.1.post1) Requirement already satisfied: click in /usr/local/lib/python3.12/site-packages (from geomet<0.3,>=0.1->cassandra-driver) (8.1.7) Requirement already satisfied: six in /usr/local/lib/python3.12/site-packages (from geomet<0.3,>=0.1->cassandra-driver) (1.16.0) Building wheels for collected packages: cassandra-driver Building wheel for cassandra-driver (setup.py) ... done Created wheel for cassandra-driver: filename=cassandra_driver-3.29.0-cp312-cp312-linux_x86_64.whl size=19788637 sha256=303432506fb53a17c48f6fee738a235848f32d1b2bfe929d1cb72254695c1edb Stored in directory: /tmp/pip-ephem-wheel-cache-8gq19r54/wheels/f1/6b/0e/cd552adf492c2ef8392e2567f8f829be39ca14f473359e4279 Successfully built cassandra-driver Installing collected packages: cassandra-driver Successfully installed cassandra-driver-3.29.0 tests/providers/google/cloud/transfers/test_cassandra_to_gcs.py::TestCassandraToGCS::test_execute PASSED [ 50%] tests/providers/google/cloud/transfers/test_cassandra_to_gcs.py::TestCassandraToGCS::test_convert_value PASSED ``` Since it takes quite some time to build cassandra driver and it would complicate our CI builds (I'd have to specifically uninstall/install the driver from sdist - and our users will have to go through the same hoops, I will disable cassandra for Python 3.12 and open an issue in cassandra for the whl not supporting libev. I will re-enable it when the whl is compied with libev support. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
