JingsongLi commented on code in PR #11097:
URL: https://github.com/apache/paimon/pull/11097#discussion_r4561510740


##########
paimon-python/pypaimon/execution/python/write/file_store_commit.py:
##########
@@ -361,7 +369,19 @@ def _try_commit(self, commit_kind, commit_identifier, 
commit_entries_plan,
                     commit_identifier,
                     commit_kind,
                     notify_callbacks=True):
-                break
+                return True
+            if self.commit_condition is not None:
+                try:
+                    needed = self.commit_condition(latest_snapshot)
+                except BaseException:
+                    if not commit_result_may_be_uncertain and 
prepared_messages is not None:
+                        self.abort(prepared_messages)
+                    raise
+                if not needed:
+                    # An uncertain response may have published these files.
+                    if not commit_result_may_be_uncertain and 
prepared_messages is not None:
+                        self.abort(prepared_messages)

Review Comment:
   [P1] Preserve published files when a commit callback fails
   
   A snapshot can already be published while `commit_outcome_uncertain` remains 
false: `_try_commit_once` calls `_notify_commit_callbacks` after the 
publication try/except, so a callback exception escapes without marking the 
prepared messages. Retrying the same conditional stream commit then sees its 
condition become false and aborts these files. I reproduced this through the 
public API: prepare one row, install a `CommitCallback` that raises `OSError`, 
use condition `snapshot is None`, commit identifier 1, then retry the same 
identifier/messages. Snapshot 1 exists after the first exception; the retry 
returns false and deletes the data file referenced by snapshot 1. Reads are 
subsequently broken. Record published ownership before invoking callbacks, or 
recognize this commit's publication before aborting on a false condition; 
checking only publication-response uncertainty is insufficient.



##########
paimon-python/pypaimon/multimodal/lerobot/dataset.py:
##########
@@ -1116,7 +1121,14 @@ def _load_dataset(table, tag_name):
     for feature in info.get("features", {}).values():
         feature["shape"] = tuple(feature["shape"])
     if source_local_layout:
-        stats = _merged_source_stats(source_rows, episodes_table)
+        episode_snapshot = 
episodes_table.snapshot_manager().get_latest_snapshot()
+        recompute_controls = (any(row.get(_COORDINATED_COLUMN) for row in 
source_rows)
+                              or episode_snapshot is not None
+                              and episode_snapshot.next_row_id != 
len(episodes))
+        stats = _merged_source_stats(
+            source_rows, episodes_table,
+            frames if recompute_controls else None,

Review Comment:
   [P2] Keep Episode statistics on the checked snapshot
   
   During the first coordinated append to an ordinary imported group, the 
reader can observe the old Episode snapshot here and choose 
`recompute_controls=False`, then `_merged_source_stats` scans a newer, unpinned 
Episode snapshot. That fallback does not filter statistics by the captured 
published Dataset IDs. A deterministic interleaving that appends 
Frames/Episodes after this check and fails before Dataset publication gives a 
successful reader with 5 published frames but `meta.stats['index']['count'] == 
[12]`; control moments include the unpublished 7-frame source as well. No 
published coordinated marker exists yet to force the safe path. Pin/reuse the 
checked Episode snapshot, or filter/rebuild statistics strictly from the 
captured published sources.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to