JingsongLi commented on code in PR #10105:
URL: https://github.com/apache/paimon/pull/10105#discussion_r4492733726


##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/write/StreamingWriteContext.scala:
##########
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.spark.write
+
+import org.apache.paimon.io.{CompactIncrement, DataIncrement}
+import org.apache.paimon.table.FileStoreTable
+import org.apache.paimon.table.sink.{CommitMessage, CommitMessageImpl, 
InnerTableCommit}
+
+import org.apache.spark.internal.Logging
+
+import java.io.FileNotFoundException
+import java.util.Collections
+
+import scala.collection.JavaConverters._
+
+/**
+ * The committer of one run of a streaming query. It is created with the first 
micro-batch of the
+ * run and kept until the query terminates, the way a Flink committer lives 
across checkpoints: the
+ * maintenance a commit starts (tag creation, partition and snapshot 
expiration) can run on while
+ * the next micro-batch is written, and only a micro-batch that may be a 
replay has to look up what
+ * a previous run committed.
+ *
+ * @param commitUser
+ *   the commit user of the query, stable across its runs, under which every 
micro-batch is
+ *   committed so that a replayed one can be recognised.
+ * @param marker
+ *   where the query records what it is about to commit, when it has a 
checkpoint location to keep
+ *   it in.
+ */
+class StreamingWriteContext(val commitUser: String, @transient marker: 
Option[CommitMarker])
+  extends AutoCloseable
+  with Serializable
+  with Logging {
+
+  /** Driver-side state; the context itself travels with the write closures. */
+  @transient private var committer: InnerTableCommit = _
+
+  /** The identifier of the last micro-batch this run committed, none before 
the first one. */
+  @transient private var lastCommitted: Option[Long] = None
+
+  /** The committer of the run, created by `create` on first use and kept 
afterwards. */
+  def openCommitter(create: => InnerTableCommit): InnerTableCommit =
+    synchronized {
+      if (committer == null) {
+        committer = create
+      }
+      committer
+    }
+
+  /**
+   * Commits a micro-batch under its identifier.
+   *
+   * Structured Streaming replays a micro-batch with its original batch id 
when a query is restarted
+   * after failing between the sink returning and Spark recording the batch as 
completed, so the
+   * first micro-batch of a run may already have been committed by a previous 
run. A batch that
+   * follows one this run committed cannot be a replay of it, and is committed 
without looking the
+   * history of the commit user up.
+   *
+   * An overwrite, the commit of a query in complete mode, publishes the 
complete result of its
+   * micro-batch: committing a replay of it again leaves the table as its 
first commit did, and runs
+   * the callbacks of an overwrite again, as the cleanup of the snapshot 
branch of a chain table
+   * needs. Every micro-batch of such a query is committed, as it always was.
+   */
+  def commit(
+      write: StreamingWrite,
+      messages: Seq[CommitMessage],
+      table: FileStoreTable,
+      overwrite: Boolean): Unit =
+    synchronized {
+      require(committer != null, "The committer of the streaming query has not 
been created.")
+      val identifier = write.commitIdentifier
+      if (overwrite) {
+        committer.commit(identifier, messages.asJava)
+      } else if (lastCommitted.exists(identifier > _)) {
+        writeMarker(write, table)
+        committer.commit(identifier, messages.asJava)
+      } else {
+        commitFirstOfRun(write, messages, table)
+      }
+      lastCommitted = Some(lastCommitted.fold(identifier)(Math.max(_, 
identifier)))
+    }
+
+  private def writeMarker(write: StreamingWrite, table: FileStoreTable): Unit =
+    marker.foreach(
+      _.write(
+        commitUser,
+        write.batchId,
+        
Option(table.snapshotManager().latestSnapshotId()).map(_.longValue()).getOrElse(0L)))
+
+  /**
+   * The first micro-batch of a run, which may be a replay. Whether its commit 
user already
+   * committed it is looked up here first, and only then is the lookup 
validated against snapshot
+   * expiration, so that the validation covers what the lookup saw.
+   *
+   * A micro-batch that was not committed is committed directly. One that was 
is not committed
+   * again, but is handed to the commit callbacks, through the committer's 
filtered commit, to retry
+   * what may have failed after its first commit, such as registering its 
partitions. It is handed
+   * over without its files: the filtered commit only compares with the latest 
snapshot of the
+   * commit user, and should that one not show the first commit, what gets 
published is an empty
+   * commit, not the batch a second time.
+   */
+  private def commitFirstOfRun(
+      write: StreamingWrite,
+      messages: Seq[CommitMessage],
+      table: FileStoreTable): Unit = {
+    val identifier = write.commitIdentifier
+    val previousAttempt = marker.flatMap(_.latestSnapshotIdBefore(commitUser, 
write.batchId))
+    writeMarker(write, table)
+    if (previousAttempt.exists(committedAfter(_, identifier, table))) {
+      committer.filterAndCommit(
+        Collections.singletonMap(Long.box(identifier), 
withoutFiles(messages).asJava))
+      // Expected of a replay; the only trace of a misconfigured identity 
otherwise.
+      logInfo(
+        s"Micro-batch $identifier was already committed to ${table.name()} 
under commit user " +
+          s"'$commitUser' and is skipped as a replay.")
+    } else {
+      checkReplayIsRecognisable(previousAttempt, identifier, table)
+      committer.commit(identifier, messages.asJava)
+    }
+  }
+
+  /**
+   * Whether the commit user published `identifier`, or a later one, in a 
snapshot after `before`,
+   * the latest snapshot when the previous attempt of the micro-batch was 
about to commit. Every
+   * snapshot since is looked at, not only the latest one of the commit user: 
the maintenance a
+   * commit starts, such as the expiration of partitions, publishes under the 
identifier of the
+   * micro-batch that started it, and when it runs asynchronously it can do so 
after the next
+   * micro-batch committed. A snapshot expired during the lookup ends it, and 
the validation of the
+   * lookup tells whether it may have missed a commit.
+   */
+  private def committedAfter(before: Long, identifier: Long, table: 
FileStoreTable): Boolean = {
+    val snapshots = table.snapshotManager()
+    var id = 
Option(snapshots.latestSnapshotId()).map(_.longValue()).getOrElse(before)
+    var committed = false
+    var retained = true
+    while (!committed && retained && id > before) {
+      try {
+        val snapshot = snapshots.tryGetSnapshot(id)
+        committed = snapshot.commitUser() == commitUser && 
snapshot.commitIdentifier() >= identifier
+      } catch {
+        case _: FileNotFoundException => retained = false
+      }
+      id -= 1
+    }
+    committed
+  }
+
+  /** The partitions and buckets a micro-batch wrote to, without anything it 
wrote there. */
+  private def withoutFiles(messages: Seq[CommitMessage]): Seq[CommitMessage] =
+    messages.map(
+      message =>
+        new CommitMessageImpl(
+          message.partition(),
+          message.bucket(),
+          message.totalBuckets(),
+          DataIncrement.emptyIncrement(),
+          CompactIncrement.emptyIncrement()))
+
+  /**
+   * Validates a lookup that found no commit of this micro-batch. A 
micro-batch whose previous
+   * attempt left no marker never reached its commit, and is not looked up. 
One that did was
+   * committed, if at all, after the snapshot its marker recorded; while the 
snapshot after that one
+   * is still retained, every commit since was retained when it was looked up, 
as expiration only
+   * removes the oldest snapshots, so the lookup is exact. Otherwise the 
lookup may have missed an
+   * expired commit, and committing again could write the micro-batch twice.
+   */
+  private def checkReplayIsRecognisable(
+      previousAttempt: Option[Long],
+      identifier: Long,
+      table: FileStoreTable): Unit = {
+    val snapshots = table.snapshotManager()
+    previousAttempt.foreach {
+      before =>
+        // Check the file itself: expiration deletes snapshots before updating 
the earliest hint.
+        if (
+          Option(snapshots.latestSnapshotId()).exists(_.longValue() > before) 
&&
+          !snapshots.snapshotExists(before + 1)

Review Comment:
   [P1] Do not trust a surviving boundary file after an incomplete history scan
   
   A `before + 1` file can survive normal expiration even after later commits 
have been expired: `SnapshotManager.deleteSnapshot()` uses 
`FileIO.deleteQuietly()`, so one deletion IOException is swallowed and 
`ExpireSnapshotsImpl` continues deleting later snapshots and advances EARLIEST. 
Consequently, existence of that old file does not prove the negative 
`committedAfter` lookup was complete.
   
   I reproduced this on a real ordinary append table: the durable marker 
records boundary 0, another writer commits snapshot 1, this micro-batch commits 
snapshot 2, and other writers publish 3/4. Run the real expiration API 
retaining only snapshot 4, injecting one delete IOException for snapshot 1. 
EARLIEST becomes 4, snapshot 1 remains, and the actual micro-batch snapshot 2 
is gone. A fresh context's scan stops at missing snapshot 3; this check trusts 
the stranded snapshot 1 and republishes the batch. The standard Spark read then 
returns `(1, 'stream')` twice instead of once. No rollback, manual metadata 
deletion or stale read is involved. Without the injected deletion failure, the 
replay is refused as intended.
   
   Please retain an incomplete/unknown result when the scan encounters missing 
metadata, and refuse publication unless positive commit evidence or a complete 
negative scan establishes safety. Preserving that flag makes the same 
fault-path test retain one row.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to