aIbrahiim commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3024022664
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV:
"DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5
--reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5
--reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
I added faulthandler in process (periodic tracebacks + SIGUSR1) and tried
pystack remote in the background with artifact uploads and what happened:
pystack pretty much always failed, Operation not permitted / no CAP_SYS_PTRACE
on the runner, so relying on attach from outside isn’t realistic there.
faulthandler helped though, I saw xdist/execnet stuck in some dumps, so
parallelism is part of the story but not everything and I also tried turning
off xdist for one Python version, stuff still failed elsewhere while concrete
examples: 3.11 hit clonevirtualenv / shutil copying the live tox venv (junk
under tmp/ etc.), 3.12/3.13 sat in subprocess_server until pytest-timeout
waiting for “grpc channel ready” on the Python expansion subprocess. so not one
root bug for all flakes, but two places worth real fixes: (1) venv clone for
YAML expansion and (2) expansion subprocess actually coming up + better errors
when it doesn’t. xdist might still make it worse sometimes. @tvalentyn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
