tkaymak commented on PR #40090: URL: https://github.com/apache/beam/pull/40090#issuecomment-5637566571
You are right on both counts, it is a production bug and the 60 s wait was a mitigation. I reverted it here (8bad262), this PR is the translator slice again, and filed #40101 as the blocker. What is established, from the Spark 4.0.2 sources: - `cancel()` interrupts the execution thread. An interrupt does not cancel a Spark job, `DAGScheduler.runJob` waits on the `JobWaiter` until it is cancelled explicitly. Today only the session stop ends batch work, so with `useActiveSparkSession=true` a batch pipeline is never cancelled and `cancel()` still reports CANCELLED. - `SparkSessionFactory.getOrCreateSession` uses `Builder.getOrCreate`, which adopts a usable default session, so two pipelines in one JVM share it and cancelling one stops the SparkContext under the other. The legacy runner has ownership guards, this one has none. - Reproduced locally: after `cancel()` the execution thread keeps translating and fails with `Cannot call methods on a stopped SparkContext`. Why that turned into the `PipelineTranslatorBatch` class init failure in CI is not proven, the primary trace was swallowed by a test that never reads its result. Nothing in that static chain calls the SparkContext. Fix in a separate PR today: job group cancellation in `EvaluationContext`, session ownership with a reference count in `SparkSessionFactory`, session stop on the execution thread after evaluation, and an unbounded join in `cancel()` without `Future.cancel(true)`. Until it lands this PR can hit the race in the Spark Versions job, one failure in three runs so far. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
