JingsongLi commented on code in PR #9667:
URL: https://github.com/apache/paimon/pull/9667#discussion_r3986215205
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/commands/PaimonSparkWriter.scala:
##########
@@ -472,14 +498,32 @@ case class PaimonSparkWriter(
postCommit(finalMessages)
return
}
- val activeWriteBuilder =
- Option(directPostponeWriteBuilder).getOrElse(writeBuilder)
- val tableCommit = activeWriteBuilder.newCommit()
+ val tableCommit: InnerTableCommit =
+ if (directPostponeWriteBuilder != null) {
+ directPostponeWriteBuilder.newCommit()
Review Comment:
[P2] Define replay handling for the direct postpone committer too
With bucket=-2, postpone.default-bucket-num=1, and complete output mode,
each overwrite takes this direct branch and never reaches the staged-path
limitation warning. The actual PostponeFixedBucketWriteBuilder has its own
random commit user, bypassing the stable user configured on writeBuilder. A
replay therefore commits another overwrite snapshot instead of being a no-op. I
reproduced snapshot 1 becoming snapshot 2; the ordinary stable-user overwrite
control stayed at snapshot 1. This does not imply duplicate visible rows in
complete mode, but it repeats a commit and its side effects.
Propagating the user alone is insufficient: this builder also resets
commit.strict-mode.last-safe-snapshot to the current latest snapshot on each
attempt, and filterCommitted searches only after that bound. The second probe
with a stable user still committed snapshot 2. Handle replay lookup
independently of that conflict bound and pass the stable identity to the actual
committer, or explicitly exclude and warn for the direct path alongside the
staged path. Please cover complete-mode replay with this configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
