Hello friends, I am reaching out from Apache Airflow and I would like to kindly ask you if you could help with gently nagging the grpcio team to release 1.78.1 release of it and then rather quickly release python beam release which would free the grpc limitation it brings ?
I raised the issue almost exactly a year ago [1] and it's been blocked because of grpcio / proto upgrade caused stability issues for Apache Beam (so we understood it could not be addressed) but a number of people worked on it - and diagnosed it - including some beam committers and this seems to be finally very very close to be solved, all we need is grpcio 1.78.1 [2] that was discussed 3 weeks ago as "will be released soon" and we attempted to nag grpcio team a bit, but so far to no avail. And even if they do, I understand beam will have to release a new python release. Unfortunately we cannot wait any longer. Basically that limit blocks us from adding some new providers we want to release before Airflow 3.2 is out (in a month) - and those are important new `pydantic-ai` providers that will provide our users with new capabilities of using LLMs for data analysis etc. Those are impossible to be installed and run alongside Apache Beam provider (and this is the only blocker) - in fact, beam - because of that blocks us from upgrading to a number of other dependencies (see the list below). We cannot wait for a long time, so we decided to start discussion, and suspend the beam provider from Apache Airflow (following the suspension process we have [3] foreseen for such cases). This will result in skipping beam from being built and tested with latest airflow and other providers (for example with google provider that has dataproc integration - and it means that only already released versions of beam provider will be used - we will not check for compatibility, not run tests for it and we will not release new version of beam provider. This will not have short term consequences, but if it lasts longer, it might happen that users who will want to use beam and google provider together and some other providers (say the new AI providers) will not be able to do it. Also the longer it will last, the more difficult it will be to bring beam provider back from the suspension, and our general approach is that it's on the stakeholders (we start to name them stewards) of the provider to make it happen - so once we suspend beam, we will expect that someone from Apache Beam will revert the PR suspending it and will fix all the issues / conflicts etc. in order to unsuspend it. This happened already a few times in the past with other providers (yandexcloud for example) and they successfully "unsuspended" it after several months of suspension, so it's not difficult, but it might also be more difficult taking into account the "difficult" dependencies beam provider has. It would really be great if we could avoid this - but for that, we need a gentle push for the grpcio team and subsequent apache.beam release (which we also do not know how fast it can happen) - so I would love to hear from you if you can help with that and preventing this suspension ? Discussion about the suspension started today at airflow devlist [4] - we will need a vote to pass to do it, but given the need we have, it is likely to happen if we do not have other options in a week or two. Can you help somehow, please? J. [1] The original issue where I raised Airflow's problem: https://github.com/apache/beam/issues/34081 [2] Grpc team comment about upcoming 1.78.1: https://github.com/grpc/grpc/issues/37710#issuecomment-3809046144 [3] Provider suspension process: https://github.com/apache/airflow/blob/main/PROVIDERS.rst#suspending-releases-for-providers [4] "Suspend Beam" Airflow discussion https://lists.apache.org/thread/s0y610cdn2t1hq2mg6y3xws02bskp89w ----- Those dependencies are currently blocked by this grpc/protobuf limitation beam holds (suspending beam frees all of those). 80d79 < apache-beam==2.71.0 130d128 < beartype==0.22.9 132d129 < betterproto==1.2.5 172c169 < cryptography==42.0.8 --- > cryptography==44.0.3 201d197 < envoy_data_plane==0.1.0 212d207 < fasteners==0.20 234c229 < google-auth-httplib2==0.2.1 --- > google-auth-httplib2==0.3.0 275c270 < google-cloud-storage==2.19.0 --- > google-cloud-storage==3.4.1 294,296c289,291 < grpcio-status==1.62.3 < grpcio==1.65.5 < grpclib==0.4.9 --- > grpcio-status==1.71.2 > grpcio-tools==1.71.2 > grpcio==1.78.0 308c303 < httplib2==0.22.0 --- > httplib2==0.31.2 325,326c320,321 < immutabledict==4.3.0 < importlib_metadata==8.4.0 --- > immutabledict==4.3.1 > importlib_metadata==8.7.1 351d345 < jsonpickle==3.4.2 427d420 < objsize==0.7.1 437,447c430,441 < opensearch-py==3.0.0 < opentelemetry-api==1.27.0 < opentelemetry-exporter-otlp-proto-common==1.27.0 < opentelemetry-exporter-otlp-proto-grpc==1.27.0 < opentelemetry-exporter-otlp-proto-http==1.27.0 < opentelemetry-exporter-otlp==1.27.0 < opentelemetry-exporter-prometheus==0.48b0 < opentelemetry-proto==1.27.0 < opentelemetry-resourcedetector-gcp==1.9.0a0 < opentelemetry-sdk==1.27.0 < opentelemetry-semantic-conventions==0.48b0 --- > opensearch-protobufs==0.19.0 > opensearch-py==3.1.0 > opentelemetry-api==1.39.1 > opentelemetry-exporter-otlp-proto-common==1.39.1 > opentelemetry-exporter-otlp-proto-grpc==1.39.1 > opentelemetry-exporter-otlp-proto-http==1.39.1 > opentelemetry-exporter-otlp==1.39.1 > opentelemetry-exporter-prometheus==0.60b1 > opentelemetry-proto==1.39.1 > opentelemetry-resourcedetector-gcp==1.11.0a0 > opentelemetry-sdk==1.39.1 > opentelemetry-semantic-conventions==0.60b1 493c487 < protobuf==4.25.8 --- > protobuf==5.29.6 503,504c497 < pyarrow-hotfix==0.7 < pyarrow==18.1.0 --- > pyarrow==23.0.0 512c505 < pydantic-settings==2.12.0 --- > pydantic-settings==2.13.0 549c542 < python-keycloak==7.1.0 --- > python-keycloak==7.1.1 561c554 < ray==2.47.1 --- > ray==2.53.0 609c602 < snowflake-connector-python==4.0.0 --- > snowflake-connector-python==4.3.0 642d634 < stringcase==1.2.0 726c718 < yandexcloud==0.328.0 --- > yandexcloud==0.377.0 J.
