Yicong-Huang opened a new issue, #7056: URL: https://github.com/apache/texera/issues/7056
### What happened? On the macOS `amber-integration` job, a `LoopIntegrationSpec` workflow intermittently stops making progress mid-run with no error logged, burns the harness completion deadline, and is re-run by `withRetry` (failing the suite when both attempts hang). Observed in 3 of 4 attempts across two runs on 2026-07-29; never observed on ubuntu. Log signature (job log, WARN backstop level): steady Python-worker spawns every ~2s, then total silence for ~the full deadline, then the retry's spawns resume. In the [failed run](https://github.com/apache/texera/actions/runs/30479437487/job/90673007976) an `EndHandler` `worker still has unprocessed messages` race (recovered ~23s before the silence) appears nearby, but the same race also self-heals in passing runs and locally, so it is not confirmed as the cause. `RegionExecutionManager` region-termination retries never log a second attempt during the silence, pointing at an await with no timeout between iterations (e.g. a termination or scheduling future that never resolves). Expected: a healthy loop workflow completes in well under a minute; a broken one fails with a diagnosable error instead of silently idling. ### How to reproduce? Re-run `build / amber-integration (macos-latest, 17)` on any PR until it exceeds ~13 min; the suite hangs most runs (e.g. runs 30479437487 and 30478651425 attempt 2 on 2026-07-29). Does not reproduce locally on an M-series Mac (JDK 17, venv with amber requirements, regenerated proto bindings): 4/4 pass in ~1.5 min with zero hangs, so the window is likely specific to the 3-core CI runner's timing. #7053 trims the suite and deadline so each hang costs ~2 min instead of ~4. ### Version/Branch 1.3.0-incubating-SNAPSHOT (main) ### Commit Hash (Optional) _No response_ ### What browsers are you seeing the problem on? _No response_ ### Relevant log output ```shell 18:45:19 WARN [WF6-LimitOpDesc-...-main-0] Received EndHandler before all messages are processed. Unprocessed: ControlInvocation(queryStatistics, ...) 18:45:19 ERROR [COORDINATOR] AsyncRPCClient - received error from Worker:WF6-LimitOpDesc-... 18:45:19 WARN [COORDINATOR] RegionExecutionManager - Failed to terminate region 1 on attempt 1 of 150. Retrying in 200 ms. 18:45:42 <last Python worker spawn; total silence for ~171s until the deadline fires and the test is retried> ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
