JeonDaehong commented on code in PR #1417:
URL: https://github.com/apache/paimon-rust/pull/1417#discussion_r4599702945
##########
crates/paimon/src/table/table_commit.rs:
##########
@@ -1423,16 +1423,37 @@ impl TableCommit {
// Once publication starts its outcome may be unknown. These files must
// remain available even if the response is lost or a later retry
fails.
let base_snapshot_uuid =
latest_snapshot.as_ref().and_then(Snapshot::uuid);
- let publication_error = match self
+ let publication = self
.snapshot_commit
.commit(
base_snapshot_uuid,
&snapshot,
self.table.branch(),
&statistics,
)
- .await
- {
+ .await;
+ // Storage may replay a successful conditional write after losing its
+ // response. Establish ownership before treating false as a rejection.
+ let publication = match publication {
+ Ok(false) => match
self.snapshot_manager.get_snapshot_uncached(snapshot.id()).await {
+ Ok(current) if current.uuid() == snapshot.uuid() => Ok(true),
Review Comment:
`prepare_snapshot` always sets a UUID on the new snapshot, so a legacy
snapshot without one can't match here, and falls through to the manifest list
check.
##########
crates/paimon/src/table/table_commit.rs:
##########
@@ -1423,16 +1423,37 @@ impl TableCommit {
// Once publication starts its outcome may be unknown. These files must
// remain available even if the response is lost or a later retry
fails.
let base_snapshot_uuid =
latest_snapshot.as_ref().and_then(Snapshot::uuid);
- let publication_error = match self
+ let publication = self
.snapshot_commit
.commit(
base_snapshot_uuid,
&snapshot,
self.table.branch(),
&statistics,
)
- .await
- {
+ .await;
+ // Storage may replay a successful conditional write after losing its
+ // response. Establish ownership before treating false as a rejection.
+ let publication = match publication {
+ Ok(false) => match
self.snapshot_manager.get_snapshot_uncached(snapshot.id()).await {
Review Comment:
For `RESTSnapshotCommit`, `false` is the server rejecting the commit, and
`SnapshotManager` otherwise reads REST snapshots through `load_snapshot` rather
than from storage. If a server doesn't keep snapshot files in the table
location, a normal conflict would come back here as `SnapshotNotExist`, be
treated as an unknown outcome, and skip the DV cleanup below. Should this check
only run for `RenamingSnapshotCommit`, or go through the catalog for REST
tables?
##########
crates/paimon/src/table/table_commit.rs:
##########
@@ -1423,16 +1423,37 @@ impl TableCommit {
// Once publication starts its outcome may be unknown. These files must
// remain available even if the response is lost or a later retry
fails.
let base_snapshot_uuid =
latest_snapshot.as_ref().and_then(Snapshot::uuid);
- let publication_error = match self
+ let publication = self
.snapshot_commit
.commit(
base_snapshot_uuid,
&snapshot,
self.table.branch(),
&statistics,
)
- .await
- {
+ .await;
+ // Storage may replay a successful conditional write after losing its
+ // response. Establish ownership before treating false as a rejection.
+ let publication = match publication {
+ Ok(false) => match
self.snapshot_manager.get_snapshot_uncached(snapshot.id()).await {
+ Ok(current) if current.uuid() == snapshot.uuid() => Ok(true),
+ // Older writers can omit the UUID. Each attempt writes a
+ // unique delta manifest list before publication.
+ Ok(current) if current.delta_manifest_list() !=
snapshot.delta_manifest_list() => {
Review Comment:
Comparing the delta manifest list works for legacy writers, since every
attempt writes a new one. Making this arm return `false` for equal lists fails
`compact_unverified_publication_preserves_dv_for_recovery`, so the error branch
is covered.
##########
crates/paimon/src/table/snapshot_manager.rs:
##########
@@ -225,8 +225,20 @@ impl SnapshotManager {
/// Get a snapshot by id.
pub async fn get_snapshot(&self, snapshot_id: i64) ->
crate::Result<Snapshot> {
- let snapshot_path = self.snapshot_path(snapshot_id);
- let snap_input = self.file_io.new_input(&snapshot_path)?;
+ let input = self.file_io.new_input(&self.snapshot_path(snapshot_id))?;
+ Self::read_snapshot(snapshot_id, input).await
+ }
+
+ /// Read snapshot identity directly from storage for publication checks.
+ /// An external rollback can reuse an ID whose old snapshot is cached.
+ /// Invalidate that entry so subsequent ordinary reads see the verified
state.
+ pub(super) async fn get_snapshot_uncached(&self, snapshot_id: i64) ->
crate::Result<Snapshot> {
+ let input = self.file_io.new_input(&self.snapshot_path(snapshot_id))?;
+ input.invalidate_cache().await?;
Review Comment:
Without this invalidation (reading through the cache),
`compact_replayed_publication_ignores_snapshot_cached_before_rollback` fails,
so the rollback case depends on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]