ahmedabu98 opened a new pull request, #41457:
URL: https://github.com/apache/beam/pull/41457

   Canceling a DirectRunner pipeline while another thread waits in 
`waitUntilFinish()` sometimes throws instead of returning `CANCELLED`. Beam 
SQL's `JdbcDriverTest.testCancelUnboundedQuery` hits it every now and then:
   
   ```
   Error while executing SQL "INSERT INTO sink SELECT sequence FROM seq": Tried 
to get an evaluator for a finished TransformEvaluatorRegistry
   ```
   
   `ExecutorServiceParallelExecutor.stop()` tore the pipeline down first and 
posted the `CANCELLED` update last. Shutting down the worker pool doesn't drop 
tasks that are already queued, so a bundle queued before the cancel could start 
after the evaluator registry was cleaned up, and fail. If that failure reached 
`waitUntilFinish()` before the `CANCELLED` update, it got thrown.
   
   This PR makes two changes:
   
   * `stop()` marks the pipeline canceled and posts the `CANCELLED` update 
before it tears anything down. Failures caused by the teardown now arrive after 
it. Failures from before the cancel still come first and are still thrown.
   * The parallel and serial `TransformExecutorService`s skip work that was 
still waiting for a thread when they were shut down, so stale bundles no longer 
run against a torn-down pipeline.
   
   One behavior change on the cancel path: a `waitUntilFinish()` call in 
another thread now returns `CANCELLED` as soon as the cancel starts, instead of 
after the teardown. If a DoFn's `@Teardown` throws during the cancel, 
`cancel()` still throws, but that other `waitUntilFinish()` call returns 
`CANCELLED` instead of rethrowing it.
   
   The new `DirectRunnerTest` case runs one worker thread, with one bundle 
blocked until teardown and another queued behind it. It fails every time 
without this change.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to