zhuxiangyi commented on code in PR #9667:
URL: https://github.com/apache/paimon/pull/9667#discussion_r4002577067
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/commands/PaimonSparkWriter.scala:
##########
@@ -454,6 +464,16 @@ case class PaimonSparkWriter(
writeBuilder.asInstanceOf[BatchWriteBuilderImpl].rowIdCheckConflict(rowIdCheckFromSnapshot)
}
+ /**
+ * The commit identifier to deduplicate on, present only for a streaming
write that has both a
+ * batch id and a commit user that is stable across restarts.
+ */
+ private def idempotentCommitIdentifier: Option[Long] =
+ for {
+ identifier <- batchId
+ _ <- commitUser
+ } yield identifier
Review Comment:
Reproduced through the sink (`delta-commits = 3`, four batches:
`compacted-full` read 103 instead of 102) and fixed in f9995d459 with your
first option: the identifier is `batchId + 1`, defined once and used for both
the published commit and the full compaction schedule, so the schedule shares
the scanner's rule. The schedule itself is unchanged.
The regression test asserts the identifiers of the full compaction (3) and
the batch after it (4) and reads the full compaction through `compacted-full`;
a zero-based identifier fails it and the existing schedule test, nothing else.
Both behaviour changes — the identifier numbering and the visibility wait —
are in a release notes section of the description now, with the bound on replay
recognition by retained snapshot history.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
