shunping opened a new pull request, #38572:
URL: https://github.com/apache/beam/pull/38572
We are seeing the following test timeout when expansion service is being
started.
```
=================================== FAILURES
===================================
______________________ MLTest.test_ml_preprocessing_yaml
_______________________
...
/opt/hostedtoolcache/Python/3.14.5/x64/lib/python3.14/threading.py:373:
Failed
----------------------------- Captured stdout call
-----------------------------
Cloning dev environment from
/runner/_work/beam/beam/sdks/python/test-suites/tox/py314/build/srcs/sdks/python/target/.tox-py314/py314
Requirement already satisfied: numpy in
./target/.tox-py314/test-venv-cache-_ujxtw1x/31094b9cfa8383d00947c0b043fad03c1b780a2756271f73ea8e0c51b4845142/lib/python3.14/site-packages
(1.26.4)
------------------------------ Captured log call
-------------------------------
WARNING apache_beam.yaml.yaml_provider:yaml_provider.py:1410 WARNING:
Apache Beam is installing Python packages from PyPI at runtime.
This may pose security risks or cause instability due to repository
availability.
Packages: numpy
Consider pre-staging dependencies or using a private repository mirror.
For more information, see:
https://beam.apache.org/documentation/sdks/python-dependencies/
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
WARNING root:subprocess_server.py:219 Waiting for grpc channel to be ready
at localhost:55955.
ERROR apache_beam.utils.subprocess_server:subprocess_server.py:225 Error
bringing up service
Traceback (most recent call last):
File
"/runner/_work/beam/beam/sdks/python/test-suites/tox/py314/build/srcs/sdks/python/target/.tox-py314/py314/lib/python3.14/site-packages/apache_beam/utils/subprocess_server.py",
line 215, in start
channel_ready.result(timeout=wait_secs)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
File
"/runner/_work/beam/beam/sdks/python/test-suites/tox/py314/build/srcs/sdks/python/target/.tox-py314/py314/lib/python3.14/site-packages/grpc/_utilities.py",
line 160, in result
self._block(timeout)
~~~~~~~~~~~^^^^^^^^^
File
"/runner/_work/beam/beam/sdks/python/test-suites/tox/py314/build/srcs/sdks/python/target/.tox-py314/py314/lib/python3.14/site-packages/grpc/_utilities.py",
line 106, in _block
self._condition.wait(timeout=remaining)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.14.5/x64/lib/python3.14/threading.py",
line 373, in wait
gotit = waiter.acquire(True, timeout)
File
"/runner/_work/beam/beam/sdks/python/test-suites/tox/py314/build/srcs/sdks/python/target/.tox-py314/py314/lib/python3.14/site-packages/pytest_timeout.py",
line 317, in handler
timeout_sigalrm(item, settings)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
File
"/runner/_work/beam/beam/sdks/python/test-suites/tox/py314/build/srcs/sdks/python/target/.tox-py314/py314/lib/python3.14/site-packages/pytest_timeout.py",
line 502, in timeout_sigalrm
pytest.fail(PYTEST_FAILURE_MESSAGE % settings.timeout)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/runner/_work/beam/beam/sdks/python/test-suites/tox/py314/build/srcs/sdks/python/target/.tox-py314/py314/lib/python3.14/site-packages/_pytest/outcomes.py",
line 163, in __call__
raise Failed(msg=reason, pytrace=pytrace)
Failed: Timeout (>600.0s) from pytest-timeout.
```
If the expansion service failed to bind to its port, `add_insecure_port`
returned `0` but was ignored.
https://github.com/apache/beam/blob/290e3721b503faf84bd83bf0b99644ccf9c87491/sdks/python/apache_beam/runners/portability/expansion_service_main.py#L74
The subprocess remained alive, causing the parent to wait indefinitely for
the gRPC channel to become ready until pytest timeout.
Additionally, binding to '0.0.0.0' caused connection timeouts on dual-stack
hosts when the client connected via 'localhost' (resolving to IPv6 loopback
`::1`).
In this PR, we address the above problems by:
- Raising a `RuntimeError` in `expansion_service_main` if `add_insecure_port`
fails, allowing the parent to instantly detect the crash, fail fast, and
retry.
- Changing the bind address from '0.0.0.0' to 'localhost'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
