uros-b commented on code in PR #56588:
URL: https://github.com/apache/spark/pull/56588#discussion_r3461641548
##########
sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/graph/TriggeredGraphExecutionSuite.scala:
##########
@@ -462,13 +462,26 @@ class TriggeredGraphExecutionSuite extends ExecutionTest
with SharedSparkSession
updateContext2.pipelineExecution.runPipeline()
updateContext2.pipelineExecution.awaitCompletion()
+ // A streaming source change is unrecoverable without a full refresh, so
the flow must not be
+ // retried: we should see exactly one failure rather than
maxFlowRetryAttempts + 1 of them.
assertFlowProgressEvent(
eventBuffer = updateContext2.eventBuffer,
identifier = fullyQualifiedIdentifier("input_table"),
expectedFlowStatus = FlowStatus.FAILED,
expectedEventLevel = EventLevel.ERROR,
msgChecker = _.contains(
s"Flow '${eventLogName("input_table")}' had streaming sources added or
removed."
+ ),
+ expectedNumOfEvents = Option(1)
+ )
+
+ // The run should fail because of the source change, not because the flow
exhausted its retries.
+ assertRunProgressEvent(
+ eventBuffer = updateContext2.eventBuffer,
+ state = RunState.FAILED,
+ expectedEventLevel = EventLevel.ERROR,
+ msgChecker = _.contains(
+ s"flow '${eventLogName("input_table")}' has failed."
)
Review Comment:
Nit: suggested to add a focused unit test for the decision logic. The
end-to-end suite does cover it, but a small PlanTest-level test asserting
determineFlowExecutionActionFromError(...) returns StreamingSourcesChanged (and
streamingSourcesChanged matches a wrapped AssertionError but not, say, a
MaxRetryExceeded-style error) would lock in the centralized behavior cheaply
and protect ordering (source-change is now checked before the retry-count
check).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
