tkaymak opened a new pull request, #40098: URL: https://github.com/apache/beam/pull/40098
`SparkStructuredStreamingPipelineResult.cancel()` interrupts the execution thread with `Future.cancel(true)` and then runs the terminal state callback at once, which stops the SparkSession unless `useActiveSparkSession` is set. The interrupt does not wait for the thread. It keeps translating or evaluating on a stopped SparkContext and dies with `IllegalStateException: Cannot call methods on a stopped SparkContext`. When that happens during the first pipeline of a JVM the stop can land inside the static initialization of `PipelineTranslatorBatch` on the execution thread. The class is then poisoned for the JVM and every later batch pipeline fails with `NoClassDefFoundError: Could not initialize class PipelineTranslatorBatch`. That is the failure of `beam_PreCommit_Java_Spark_Versions` on #40090 (run 34484375079, 12 batch tests in one fork), the first run of the Spark 4 module since #40093 restored it. `StructuredStreamingPipelineStateTest` cancels right after `run()` in three of its tests, and with a diagnostic log on the execution thread all three show the stopped context locally. Fix: `runAsync` hands the result its single thread executor, already shut down after submit, and `cancel()` waits for it to terminate, bounded at 60 seconds, before the terminal state callback runs. Streaming pipelines end the thread through `StreamingEvaluationContext.stop()`, batch pipelines through the interrupt, so the bound is not reached in normal use. New `SparkStructuredStreamingPipelineResultTest` checks that the callback runs only after the execution thread has ended. Shared code, compiled for Spark 3 and Spark 4. Tested locally with the two targeted test classes on both modules plus five runs of `StructuredStreamingPipelineStateTest` in the CI fork mode. R: @Abacn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
