sshiv012 opened a new pull request, #8668:
URL: https://github.com/apache/texera/pull/8668
### What changes were proposed in this PR?
A worker failing to initialize in a region's second (non-dependee) phase,
e.g. a Python UDF after a HashJoin, used to leave the execution `RUNNING`
forever with no error. See the issue for the full chain.
- `WorkflowExecutionManager.advanceRegionExecutions`: keep the
`syncStatusAndTransitionRegionExecutionPhase()` futures instead of dropping
them in a `foreach`. The advance future now fails if any sync fails, so the
existing `.onFailure → FatalError` in `AdvanceRegionExecutionsHandler` reports
it. The rest of the method moved unchanged into `advanceToNextRegions`.
- `AsyncRPCClient.logControlReply`: log the `ControlError` message, details
and stack trace. It isn't a `Throwable`, so the logger silently dropped it.
| | Before | After |
|---|---|---|
| UDF missing `from pytexera import *` after a join | hangs, all 0 tuples,
no error | fails in ~1s with `name 'UDFTableOperator' is not defined` |
| Log line | `received error from …` | `received error from … (controlID:
N): <message>` |
Behavior note: the future returned by `advanceRegionExecutions` now also
waits on, and fails with, the syncs it triggers.
`AdvanceRegionExecutionsHandler` doesn't wait on it. `StartWorkflowHandler` has
no unfinished regions at start, so nothing changes there.
### Any related issues, documentation, discussions?
Closes #8667
### How was this PR tested?
Added tests. Each new failing-case test fails before the fix and passes
after it.
- `WorkflowExecutionManagerSpec`, 2 new tests: a region whose probe op has a
dependee port, followed by a UDF op.
- The UDF's `initializeExecutor` returns a `ControlError` → the advance
fails with that message.
- The UDF initializes fine → it is started.
- `DataProcessingSpec`, 2 new e2e tests with a real Python worker, on csv,
csv → join → Python UDF.
- UDF with the import → completes.
- UDF without the import → before the fix, times out after 1 min; after,
fails in ~1s with `UDFTableOperator` in the error.
- `RegionExecutionManagerTestSupport`: `CoordinatorRpcProbe` takes an
optional `initializeExecutorResponse`.
```
sbt 'WorkflowExecutionService/testOnly
org.apache.texera.amber.engine.architecture.scheduling.WorkflowExecutionManagerSpec'
sbt 'WorkflowExecutionService/testOnly
org.apache.texera.amber.engine.e2e.DataProcessingSpec -- -z "python-udf
workflow normally" -z "fail, not hang"'
```
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
