anew commented on code in PR #57584:
URL: https://github.com/apache/spark/pull/57584#discussion_r3667668147
##########
sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/graph/AutoCdcOutOfOrderConvergenceSuite.scala:
##########
@@ -197,11 +219,18 @@ class AutoCdcScd1OutOfOrderConvergenceSuite
runPipeline(outOfOrderCtx)
}
+ // Only the user-visible target must converge. The auxiliary tables
legitimately differ by
Review Comment:
Confirmed. findAffectedRowsFromAuxiliaryTable bounds aux-row participation
per key to recordStartAt >= minSequenceInMicrobatch plus a single "anchor" —
the aux row with the largest recordStartAt strictly below the batch's min
sequence. There's no unbounded look-back (no "N+1000" case): an aux row from
version N only affects microbatches adjacent to N, so once ingestion advances
past it (with a newer anchor present) it stops mattering. That's what makes the
target-only convergence comparison valid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
