JingsongLi commented on code in PR #11097:
URL: https://github.com/apache/paimon/pull/11097#discussion_r4561510740
##########
paimon-python/pypaimon/execution/python/write/file_store_commit.py:
##########
@@ -361,7 +369,19 @@ def _try_commit(self, commit_kind, commit_identifier,
commit_entries_plan,
commit_identifier,
commit_kind,
notify_callbacks=True):
- break
+ return True
+ if self.commit_condition is not None:
+ try:
+ needed = self.commit_condition(latest_snapshot)
+ except BaseException:
+ if not commit_result_may_be_uncertain and
prepared_messages is not None:
+ self.abort(prepared_messages)
+ raise
+ if not needed:
+ # An uncertain response may have published these files.
+ if not commit_result_may_be_uncertain and
prepared_messages is not None:
+ self.abort(prepared_messages)
Review Comment:
[P1] Preserve published files when a commit callback fails
A snapshot can already be published while `commit_outcome_uncertain` remains
false: `_try_commit_once` calls `_notify_commit_callbacks` after the
publication try/except, so a callback exception escapes without marking the
prepared messages. Retrying the same conditional stream commit then sees its
condition become false and aborts these files. I reproduced this through the
public API: prepare one row, install a `CommitCallback` that raises `OSError`,
use condition `snapshot is None`, commit identifier 1, then retry the same
identifier/messages. Snapshot 1 exists after the first exception; the retry
returns false and deletes the data file referenced by snapshot 1. Reads are
subsequently broken. Record published ownership before invoking callbacks, or
recognize this commit's publication before aborting on a false condition;
checking only publication-response uncertainty is insufficient.
##########
paimon-python/pypaimon/multimodal/lerobot/dataset.py:
##########
@@ -1116,7 +1121,14 @@ def _load_dataset(table, tag_name):
for feature in info.get("features", {}).values():
feature["shape"] = tuple(feature["shape"])
if source_local_layout:
- stats = _merged_source_stats(source_rows, episodes_table)
+ episode_snapshot =
episodes_table.snapshot_manager().get_latest_snapshot()
+ recompute_controls = (any(row.get(_COORDINATED_COLUMN) for row in
source_rows)
+ or episode_snapshot is not None
+ and episode_snapshot.next_row_id !=
len(episodes))
+ stats = _merged_source_stats(
+ source_rows, episodes_table,
+ frames if recompute_controls else None,
Review Comment:
[P2] Keep Episode statistics on the checked snapshot
During the first coordinated append to an ordinary imported group, the
reader can observe the old Episode snapshot here and choose
`recompute_controls=False`, then `_merged_source_stats` scans a newer, unpinned
Episode snapshot. That fallback does not filter statistics by the captured
published Dataset IDs. A deterministic interleaving that appends
Frames/Episodes after this check and fails before Dataset publication gives a
successful reader with 5 published frames but `meta.stats['index']['count'] ==
[12]`; control moments include the unpublished 7-frame source as well. No
published coordinated marker exists yet to force the safe path. Pin/reuse the
checked Episode snapshot, or filter/rebuild statistics strictly from the
captured published sources.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]