AnishMahto commented on PR #58209: URL: https://github.com/apache/spark/pull/58209#issuecomment-5376014348
@szehon-ho Your understanding is exactly correct. The primary fix here is using a single anchor, across both the auxiliary and target tables. Rather than computing one anchor per table, which could lead to breaking the invariant that reconciliation needs in some scenarios: if a row is considered affected, _all_ other existing rows that order after it (by sequence) must also be considered affected. In the two-anchor row approach, it was possible we select a hidden upsert row in the auxiliary table without selecting its matching and later occuring visible tail row in the target table. But this fix requires that each of the aux and target table merge operations read both the aux and target tables. Merging into these tables is not a single atomic operation, and we instead merge into the aux table first. That means when the target table begins its reconciliation and merge process, it will now read the updated auxiliary table, in addition to the same microbatch. In some cases that could lead to a row in the microbatch also arising as a row in the auxiliary table, and we need a stable sort order (in conjunction with `dropRedundantRowsPostDecomposition`) that can correctly dedup them. ---- Regarding your open question, yes we can materialize the aux table once before starting the entire foreachBatch operation, and require the target table's reconciliation reads from that materialized aux table rather than the updated aux table. Alternatively, for table formats that support it, we can require the target table's reconciliation reads from the pre-merge version of the aux table. Let's leave that as a meaningful follow-up, to improve performance and make the algorithm correct by construction for table formats that support efficient materialization or versioned reads. Otherwise the current implementation is a reasonable fallback for all table types. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
