tkaymak commented on PR #40103: URL: https://github.com/apache/beam/pull/40103#issuecomment-5661126733
Thanks, that framing is clearer than mine and I will follow it. No waits anywhere. Shape of the rework: - Job group cancellation stays as it is. - `cancel()` requests, marks CANCELLED and returns. No join. `waitUntilFinish()` reports CANCELLED once the execution ended after a cancel request. - `SparkSessionFactory` keeps a count of pipelines per session the runner created, the last one to release stops it, on the execution thread after evaluation. A pipeline that starts while another still holds the session shares it and logs that its conf is not applied, which is today's behavior. Sessions the runner did not create are never stopped. Only the map updates are synchronized, nothing waits. - Tests cut to two unit and three live cases, one of them starts a pipeline right after a cancel without waiting. Your point on the state after an asynchronous `cancel()` is #40120, follow up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
