juliuszsompolski commented on code in PR #56597:
URL: https://github.com/apache/spark/pull/56597#discussion_r3460799081
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala:
##########
@@ -342,12 +342,12 @@ abstract class UpdateTableSuiteBase extends
RowLevelOperationSuiteBase {
}
test("metric values are stable across stage retries") {
- // Force a shuffle in the UPDATE plan via an IN-subquery (with broadcast
disabled), then
- // have the DAGScheduler corrupt the first attempt of every upstream
shuffle map stage.
- // Note: the current fetch-failure injection does not retry the writer
stage, so this
- // test passes equally well with plain SQLMetric — it only exercises the
SLAM-aware
- // read path. Follow-up #55738 will add infra to actually retry the writer
stage and
- // exercise the SLAM behavior end-to-end for UPDATE.
+ // INJECT_SHUFFLE_FETCH_FAILURES corrupts the partition-0 task of the
first successful
Review Comment:
yes, unfortunately it's still impossible to assert for overcounts in result
stage metrics, because Spark does not support any retries in this stage. But
the infra PR at least added more "interesting" scenarios of restarts, and it
added coverage for Merge with metadata.
The new infra-level AQE tests checks that retry fires - it asserts overcount
in stage1 and stage2, but not stage3 (again, because there can't be restarts in
result stage).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
