tkaymak opened a new pull request, #40130:
URL: https://github.com/apache/beam/pull/40130
Part of #36841. Cleans up obsolete DStream test remnants in the Structured
Streaming test suite and adds streaming lifecycle coverage for timeout handling.
**Context**
The shared StructuredStreamingPipelineStateTest retained four tests marked
with @Ignore("TODO: Reactivate with streaming."). They used CreateStream, which
is a legacy DStream concept not supported by Structured Streaming. Spark 4 now
has dedicated streaming lifecycle tests in StreamingPipelineLifecycleTest
covering RUNNING to DONE on idle, cancellation, and sibling query failure
propagation.
**What this changes**
- runners/spark/src/test/.../StructuredStreamingPipelineStateTest.java:
Removed the four obsolete @Ignore streaming test cases. Inlined
Create.of("one", "two") at call sites and removed the obsolete getValues() and
getStreamingOptions() helpers and CreateStream imports.
- runners/spark/src/test/.../translation/streaming/SimpleSourceTest.java:
Deleted this obsolete, ignored file.
-
runners/spark/4/src/test/.../translation/streaming/StreamingPipelineLifecycleTest.java:
Added timeoutKeepsRunningState to assert that
waitUntilFinish(Duration.millis(1)) preserves RUNNING state while the query
remains active, with try, finally query cancellation to prevent leaks in the
shared session.
**Tests**
- :runners:spark:3:test passed locally
(StructuredStreamingPipelineStateTest).
- :runners:spark:4:test passed locally (StreamingPipelineLifecycleTest, 4
passed).
- checkstyle and spotless verified clean on Spark 3 and Spark 4.
R: @Abacn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
