aIbrahiim commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3025192582
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV:
"DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5
--reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5
--reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
re: 1 --The issue ties to YAML external providers: they clone a venv in
yaml_provider via clonevirtualenv so source is either a fresh mini venv, or in
.dev the whole tox venv (path from base_python) and in CI that tree is live, so
paths like tmp/ can disappear mid-copy, fits what we saw so I suggesst ignore
tmp/, __pycache__/, .pytest_cache/, pip/build temps (or replace clonevirtualenv
with a copy that supports ignore / only needed dirs) and for .dev/CI, prefer
_create_venv_from_scratch (or a template venv) instead of cloning the full tox
env, slower, stabler. Optional unit test with a fake venv + volatile tmp/ to
guard regressions.
re: 2 -- subprocess_server waits on gRPC channel ready for
expansion_service_main until pytest-timeout. faulthandler showed xdist/execnet
sometimes, xdist off for one Python version didn’t fix other legs, parallelism
isnt the whole story so I suggesst on child exit or long wait, surface exit
code + stdout/stderr (and maybe fail faster with a clear error).
For ML YAML, preinstall/pin heavy deps in tox so slow pip/import/model
startup doesn’t look like a hang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
