fruch commented on PR #36755: URL: https://github.com/apache/airflow/pull/36755#issuecomment-1954846781
> > As I said if packages are always installed as wheel is would work out of the box, libev should be baked into the wheel. > > > If not, and you can enforce libev install before install from pip, it would work as well, see > https://docs.datastax.com/en/developer/python-driver/3.14/installation/#libev-support for more details of need thing to be able to compile from source. > > @fruch . Just double checked it and for Python 3.12, whells for linux do not seem to be compiled with libev support. That basically makes python 3.12 cassandra-driver whl useless as it falls back to asyncore that is missing in Python 3.12 > > When I install it from PyPI with wheels, our test fail: > > ``` > root@d71d7cdeb0d9:/opt/airflow# apt list libev4 > Listing... Done > libev4/stable,now 1:4.33-1 amd64 [installed] > root@d71d7cdeb0d9:/opt/airflow# apt list libev-dev > Listing... Done > libev-dev/stable,now 1:4.33-1 amd64 [installed] > > .... > > Downloading cassandra_driver-3.29.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.6 MB) > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.6/19.6 MB 12.3 MB/s eta 0:00:00 > Installing collected packages: cassandra-driver > Successfully installed cassandra-driver-3.29.0 > > .... > pytest tests/providers/google/cloud/transfers/test_cassandra_to_gcs.py > > _________________________________________________________________________________________________________ ERROR collecting tests/providers/google/cloud/transfers/test_cassandra_to_gcs.py _________________________________________________________________________________________________________ > tests/providers/google/cloud/transfers/test_cassandra_to_gcs.py:27: in <module> > from airflow.providers.google.cloud.transfers.cassandra_to_gcs import CassandraToGCSOperator # noqa: E402 > airflow/providers/google/cloud/transfers/cassandra_to_gcs.py:34: in <module> > from airflow.providers.apache.cassandra.hooks.cassandra import CassandraHook > airflow/providers/apache/cassandra/hooks/cassandra.py:25: in <module> > from cassandra.cluster import Cluster, Session > cassandra/cluster.py:173: in init cassandra.cluster > ??? > E cassandra.DependencyException: Unable to load a default connection class > E The following exceptions were observed: > E - The C extension needed to use libev was not found. This probably means that you didn't have the required build dependencies when installing the driver. See http://datastax.github.io/python-driver/installation.html#c-extensions for instructions on installing build dependencies and building the C extension. > E - Unable to import asyncore module. Note that this module has been removed in Python 3.12 so when using the driver with this version (or anything newer) you will need to use one of the other event loop implementations. > ``` > > I tested that when I build cassandra driver from sdist, the test start to work (so libev is used) > > > ``` > oot@d71d7cdeb0d9:/opt/airflow# pip uninstall cassandra-driver > Found existing installation: cassandra-driver 3.29.0 > Uninstalling cassandra-driver-3.29.0: > Would remove: > /usr/local/lib/python3.12/site-packages/cassandra/* > /usr/local/lib/python3.12/site-packages/cassandra_driver-3.29.0.dist-info/* > Proceed (Y/n)? y > Successfully uninstalled cassandra-driver-3.29.0 > WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv > root@d71d7cdeb0d9:/opt/airflow# CASS_DRIVER_BUILD_CONCURRENCY=8 pip install cassandra-driver --no-binary ":all:" > Collecting cassandra-driver > Downloading cassandra-driver-3.29.0.tar.gz (292 kB) > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 292.7/292.7 kB 4.5 MB/s eta 0:00:00 > Preparing metadata (setup.py) ... done > Requirement already satisfied: geomet<0.3,>=0.1 in /usr/local/lib/python3.12/site-packages (from cassandra-driver) (0.2.1.post1) > Requirement already satisfied: click in /usr/local/lib/python3.12/site-packages (from geomet<0.3,>=0.1->cassandra-driver) (8.1.7) > Requirement already satisfied: six in /usr/local/lib/python3.12/site-packages (from geomet<0.3,>=0.1->cassandra-driver) (1.16.0) > Building wheels for collected packages: cassandra-driver > Building wheel for cassandra-driver (setup.py) ... done > Created wheel for cassandra-driver: filename=cassandra_driver-3.29.0-cp312-cp312-linux_x86_64.whl size=19788637 sha256=303432506fb53a17c48f6fee738a235848f32d1b2bfe929d1cb72254695c1edb > Stored in directory: /tmp/pip-ephem-wheel-cache-8gq19r54/wheels/f1/6b/0e/cd552adf492c2ef8392e2567f8f829be39ca14f473359e4279 > Successfully built cassandra-driver > Installing collected packages: cassandra-driver > Successfully installed cassandra-driver-3.29.0 > > > tests/providers/google/cloud/transfers/test_cassandra_to_gcs.py::TestCassandraToGCS::test_execute PASSED [ 50%] > tests/providers/google/cloud/transfers/test_cassandra_to_gcs.py::TestCassandraToGCS::test_convert_value PASSED > ``` > > Since it takes quite some time to build cassandra driver and it would complicate our CI builds (I'd have to specifically uninstall/install the driver from sdist - and our users will have to go through the same hoops, I will disable cassandra for Python 3.12 and open an issue in cassandra for the whl not supporting libev. I will re-enable it when the whl is compied with libev support. > Can you try out the `scylla-driver`, it's the one I maintain, and should be working out of the box with libev by default (as far as I know), it's a drop-in replacement and works with Cassandra and with scylladb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
