tkaymak opened a new issue, #40427: URL: https://github.com/apache/beam/issues/40427
### What happened? Since #40324 (merged 2026-10-01), every Spark ValidatesRunner PostCommit on master is red. Last green runs were 2026-10-01 16:47 UTC, first red 2026-10-01 22:46 UTC. Affected: beam_PostCommit_Java_ValidatesRunner_Spark, beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming, beam_PostCommit_Java_ValidatesRunner_Spark4, beam_PostCommit_Java_PVR_Spark_Batch, beam_PostCommit_Java_PVR_Spark3_Streaming, beam_PostCommit_Java_PVR_Spark4_Batch, beam_PostCommit_Java_PVR_Spark4_Streaming, beam_PostCommit_Java_PVR_Spark4_StructuredStreaming. Flink and Direct stay green. Symptoms: 100 to 200 tests per run fail with `Expected 1 successful assertions, but found 4` (or 0), the failing set differs between runs. The PVR suites fail with `Only one SparkContext should be running in this JVM`. Cause: #40324 sets `beam.test.parallelThreads=8` for every Test task. `BeamParallelJunit4Runner.selectExecutor` falls back from `beam.validatesRunner.parallelThreads` to that value, so every runner's ValidatesRunner suite became parallel without an opt in. Dataflow opts in explicitly in its build.gradle. The Spark runners keep metrics in a JVM wide `MetricsAccumulator` singleton, cleared on every `run()`, and allow one SparkContext per JVM, so concurrent pipelines mix or reset each other's PAssert counters. Fix: set `beam.validatesRunner.parallelThreads=1` for the Spark runner and Spark job server test tasks, as the Dataflow runner already sets its own value. Per pipeline metrics in the Spark runners would allow parallel tests later. A default of 1 for `beam.validatesRunner.parallelThreads` in BeamParallelJunit4Runner would protect other runners too. ### Issue Priority Priority: 2 (default / most bugs should be filed as P2) ### Issue Components - [x] Component: Java SDK - [x] Component: Spark Runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
