stankiewicz commented on issue #39667:
URL: https://github.com/apache/beam/issues/39667#issuecomment-5887578009
### Why the Regression Happens on 3.10 vs. 3.11
On **Python 3.10**, `tensorflow-metadata` enforces an environment-specific
ceiling: `protobuf<=6.32;python_version<"3.11"`. Because Beam master
(`20260831`) bumped Bigtable to `>=2.42.0` (which requires `protobuf>=6.33.5`),
these mutually exclusive ranges force pip into an aggressive backtracking loop
on every worker node. Even on standard default containers without any custom
image, pip resolves the conflict by downgrading Beam to `2.75.0` and `pyarrow`
to `23.0.1`, followed by compiling Beam master from source on the worker VM. On
**Python 3.11**, this regression vanishes completely because
`tensorflow-metadata` drops the upper bound
(`protobuf>=4.25.2;python_version>="3.11"`), allowing the container’s
pre-installed `protobuf 6.33.6` to satisfy both libraries and resolve
dependencies cleanly in seconds without touching Beam or PyArrow.
---
### How 3.10 Eventually Still Works on Head/Master (Just Slower)
The pipeline on Python 3.10 **does not actually fail and still runs on
master/head code**. This is because Gradle's `tftTests` task automatically
stages local Beam checkout as an `--extra_package` (`apache-beam.tar.gz`).
After pip finishes downgrading to `2.75.0` during the requirements phase, the
worker boot daemon immediately runs `pip install --force-reinstall --no-deps
apache-beam.tar.gz`. This successfully replaces `2.75.0` with local master
build (`2.78.0.dev0`). However, it runs much slower because:
1. Every worker node spends **over 4 minutes of 100% CPU compiling Beam’s
Cython/C++ extensions from source** during boot.
2. The transient `2.75.0` resolution permanently leaves behind older
companion packages in the final environment, notably **`pyarrow 23.0.1`**
instead of `25.0.1`.
---
### Proposed Solution: Bump `gradle.properties` to 3.11
We should update
[sdks/python/test-suites/gradle.properties](file:///usr/local/google/home/radoslaws/repos/beam/sdks/python/test-suites/gradle.properties#L34-L35):
```diff
-# TFX_BSL is not yet supported on Python 3.10.
-dataflow_cloudml_benchmark_tests_py_versions=3.10
+# Run cloudml TFT benchmark tests on Python 3.11 to avoid Python 3.10
protobuf<=6.32 cap
+dataflow_cloudml_benchmark_tests_py_versions=3.11
```
#### How Gradle Passes It to the Test:
1. **Task Delegation:** When you run `./gradlew
:sdks:python:test-suites:dataflow:tftTests`, `dataflow/build.gradle` iterates
over `dataflow_cloudml_benchmark_tests_py_versions` and adds
`:sdks:python:test-suites:dataflow:py311:tftTests` to its `dependsOn` list.
2. **Environment Configuration:** The `py311` subproject sets `pythonVersion
= '3.11'`, which configures Gradle to:
* Build the manylinux Python 3.11 Beam wheel (`bdistPy311linux`).
* Provision a local Python 3.11 virtual environment for the test runner.
* Automatically tell DataflowRunner to use the official Python 3.11
worker harness image (`beam_python3.11_sdk:beam-master-20260831`), bypassing
the protobuf clash and avoiding the 4-minute on-worker compilation entirely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
