tkaymak opened a new pull request, #40967: URL: https://github.com/apache/beam/pull/40967
Part of #36841. This adds the streaming ValidatesRunner suite for the Spark 4 Structured Streaming runner, `:runners:spark:4:validatesStructuredStreamingRunnerStreaming`, and its PostCommit `beam_PostCommit_Java_ValidatesRunner_Spark4_StructuredStreaming_Streaming` on Java 17. The suite runs with `--streaming` and stops each pipeline after 5 idle batches. The batch and streaming tasks now share one setup in `spark_runner.gradle`, the effective configuration of the batch task on Spark 3 and 4 is unchanged. In streaming mode, a Flatten of no inputs is replaced by `EmptyFlattenAsCreateFactory`, as Dataflow does. It has to come before the Impulse override from #40766, because the replacement expands to an Impulse. Exclusions are by category where one exists, grouped by the runner gap. Tests that fail on a gap without a category are excluded by name and cite their issue: ParDo with additional outputs #40949, Redistribute #40950, merging windows, side inputs and splittable DoFn under #36841, bounded source metrics under #32021. Locally the suite passes with 191 tests, 0 failures and 8 skipped in about 22 minutes. Four Flatten and multiple apply tests that were excluded before now pass with the empty Flatten override and #40932. Full `:runners:spark:4:test` and `:runners:spark:3:test` pass. The new trigger file runs the suite on this PR. The CHANGES.md entry for Spark 4 streaming follows once this is approved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
