LuciferYang commented on code in PR #56588:
URL: https://github.com/apache/spark/pull/56588#discussion_r3464455392
##########
sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/graph/TriggeredGraphExecutionSuite.scala:
##########
@@ -462,13 +462,26 @@ class TriggeredGraphExecutionSuite extends ExecutionTest
with SharedSparkSession
updateContext2.pipelineExecution.runPipeline()
updateContext2.pipelineExecution.awaitCompletion()
+ // A streaming source change is unrecoverable without a full refresh, so
the flow must not be
+ // retried: we should see exactly one failure rather than
maxFlowRetryAttempts + 1 of them.
assertFlowProgressEvent(
eventBuffer = updateContext2.eventBuffer,
identifier = fullyQualifiedIdentifier("input_table"),
expectedFlowStatus = FlowStatus.FAILED,
expectedEventLevel = EventLevel.ERROR,
msgChecker = _.contains(
s"Flow '${eventLogName("input_table")}' had streaming sources added or
removed."
+ ),
+ expectedNumOfEvents = Option(1)
+ )
+
+ // The run should fail because of the source change, not because the flow
exhausted its retries.
+ assertRunProgressEvent(
+ eventBuffer = updateContext2.eventBuffer,
+ state = RunState.FAILED,
+ expectedEventLevel = EventLevel.ERROR,
+ msgChecker = _.contains(
+ s"flow '${eventLogName("input_table")}' has failed."
)
Review Comment:
Done. Added three focused unit tests in `TriggeredGraphExecutionSuite`:
- `streamingSourcesChanged` matches a wrapped (and bare) source-set-change
`AssertionError`, but not unrelated errors or an `AssertionError` with a
different message;
- `determineFlowExecutionActionFromError` returns the source-change stop
reason even when retries remain (`currentNumTries < maxAllowedRetries`), which
pins the ordering - source change is checked before the retry-count;
- the same call still retries other errors and only stops them once the
budget is exhausted (a max-retries stop, not a source-change one).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
