tkaymak commented on PR #40103: URL: https://github.com/apache/beam/pull/40103#issuecomment-5644225006
Agreed on both, the synchronous cancel and the reference count go. Before I push the rework, the shape I have in mind: 1. `cancel()` asynchronous again: stop the evaluation context, cancel the pipeline's job group, return CANCELLED. No join. `waitUntilFinish()` after a cancel reports CANCELLED once the execution thread ends. 2. The session stop moves onto the execution thread, in its `finally` after evaluation, so it never runs under a live thread, and it applies only to sessions the runner created. A session it adopted through `getOrCreate` (a test rule, a notebook) is left alone. No counting. 3. The race you describe is between the next pipeline's `getOrCreate` and the previous thread's stop. Spark's `getOrCreate` adopts a context that is mid stop, and creating a new one while the old is not fully stopped throws. So `acquire` waits, bounded, for a session the runner created for a still running pipeline to be stopped and then creates a fresh one, which keeps the previous behavior of a fresh session with the pipeline's own conf. Parallel pipelines in one JVM stay a Spark limitation, documented. Tests halved to two unit plus two live cases, plus one that starts a pipeline right after a cancel and asserts it got a new SparkContext, which is the CI failure shape. A short rationale block goes into `SparkSessionFactory` as you asked. Does that match what you have in mind? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
