JingsongLi commented on code in PR #9667:
URL: https://github.com/apache/paimon/pull/9667#discussion_r4001789266
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/commands/PaimonSparkWriter.scala:
##########
@@ -454,6 +464,16 @@ case class PaimonSparkWriter(
writeBuilder.asInstanceOf[BatchWriteBuilderImpl].rowIdCheckConflict(rowIdCheckFromSnapshot)
}
+ /**
+ * The commit identifier to deduplicate on, present only for a streaming
write that has both a
+ * batch id and a commit user that is stable across restarts.
+ */
+ private def idempotentCommitIdentifier: Option[Long] =
+ for {
+ identifier <- batchId
+ _ <- commitUser
+ } yield identifier
Review Comment:
[P2] Align streaming commit IDs with full-compaction numbering
Publishing the raw zero-based batchId breaks FullCompactedStartingScanner's
recognition of Spark's scheduled full compactions. DataWrite.needFullCompaction
triggers when (batchId + 1) % deltaCommits == 0, while the scanner accepts
commitIdentifier % deltaCommits == 0 (or the old Long.MAX_VALUE sentinel). With
deltaCommits=3, every scheduled full compaction now has an identifier that the
scanner rejects.
I reproduced this with four real core writes using the exact Spark
compaction schedule and filtered commit sequence: batch 2 publishes COMPACT
snapshot 4 with identifier 2; batch 3 publishes APPEND snapshot 5. A
compacted-full scan falls back to snapshot 5 and reads value 103, instead of
snapshot 4/value 102. The previous sentinel sequence and changing only the
published identifier to batchId + 1 both select the correct full snapshot.
Please use numbering consistent with the existing scanner contract, or
explicitly preserve full-compaction recognition, and add a Spark
write-to-compacted-full-read regression with a delta count greater than one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
