JingsongLi commented on code in PR #11097:
URL: https://github.com/apache/paimon/pull/11097#discussion_r4579132770
##########
paimon-python/pypaimon/multimodal/lerobot/aggregate.py:
##########
@@ -593,13 +773,81 @@ def _merged_episodes(plans, schema):
raise ValueError("LeRobot Episode metadata changed after
validation.")
+def _backfill_legacy_control_counts(frames_table, datasets_table):
+ """Persist old sources' histograms once, before coordinated publication."""
+ from pypaimon.multimodal.lerobot.dataset import _frame_control_counts,
_read_arrow
+
+ baseline = datasets_table.snapshot_manager().get_latest_snapshot()
+ rows = _read_arrow(datasets_table, [
+ "id", "_ROW_ID", _CONTROL_COUNTS_COLUMN]).to_pylist()
+ missing = sorted((row for row in rows if row[_CONTROL_COUNTS_COLUMN] is
None),
+ key=lambda row: row["_ROW_ID"])
+ if not missing:
+ return
+ fields = [name for name in ("task_index", "subtask_index")
+ if name in _target_schema(frames_table).names]
+ updates = [(row["_ROW_ID"], json.dumps(
+ _frame_control_counts(frames_table, row["id"], fields),
+ sort_keys=True, separators=(",", ":"))) for row in missing]
+ builder = datasets_table.new_batch_write_builder()
+ commit = builder.new_commit()
+ writers, messages = [], []
+ commit_started = False
+ try:
+ # A partial Data Evolution write preserves every other Dataset column.
+ for row_id, value in updates:
+ writer =
builder.new_write().with_write_type([_CONTROL_COUNTS_COLUMN])
+ writers.append(writer)
+ writer.write_arrow(pa.Table.from_pydict(
+ {_CONTROL_COUNTS_COLUMN: [value]},
+ schema=pa.schema([(_CONTROL_COUNTS_COLUMN, pa.string())])))
Review Comment:
[P1] Align count backfills with the existing Dataset file ranges
This writes one normal partial-column file per Dataset row and then assigns
that row's `_ROW_ID` as `first_row_id`. An existing group initially imported
from `[source_a, source_b]` stores both Dataset rows in one two-row normal
file, so these singleton backfill files violate Data Evolution's requirement
that overlapping normal files cover the same complete row-ID range. I
reproduced this by importing two sources together, dropping
`__paimon_control_counts` to emulate a legacy group, and appending a third
source with `coordinated=True`. The backfill publishes an invalid Dataset
snapshot: Python reads fail with a field-merge row-count error, and native
reads fail with a row-ID-range error. The existing published group becomes
unreadable and subsequent retries also fail. Please use the existing
range-aware row-ID update path, or write the complete original file ranges, and
cover a legacy Dataset file containing multiple rows.
##########
paimon-python/pypaimon/execution/python/write/file_store_commit.py:
##########
@@ -361,7 +369,19 @@ def _try_commit(self, commit_kind, commit_identifier,
commit_entries_plan,
commit_identifier,
commit_kind,
notify_callbacks=True):
- break
+ return True
+ if self.commit_condition is not None:
+ try:
+ needed = self.commit_condition(latest_snapshot)
+ except BaseException:
+ if not commit_result_may_be_uncertain and
prepared_messages is not None:
+ self.abort(prepared_messages)
+ raise
+ if not needed:
+ # An uncertain response may have published these files.
+ if not commit_result_may_be_uncertain and
prepared_messages is not None:
+ self.abort(prepared_messages)
Review Comment:
[P1] Preserve published files when recovering a durable checkpoint
The ownership marker only protects the in-memory messages that observed the
commit attempt. If prepared messages are checkpointed before publication, then
the snapshot is published but its response is lost and the process restarts,
the restored messages still have `commit_outcome_uncertain=False`. I reproduced
this with a real filesystem table: pickle the messages immediately after
`prepare_commit(1)`, use condition `snapshot is None` and
`commit.max-retries=0`, publish the snapshot and raise `OSError`, then restore
the checkpoint into a new stream committer with the same commit user and
identifier. This branch returns False and deletes the file referenced by
snapshot 1; reads subsequently fail with `FileNotFoundError`. The duplicate
check above cannot help because `retry_result` is local to the new invocation.
Please resolve publication through stable commit identity or file references
before aborting recovered messages; absence of the in-memory marker does not
establish that the
y are uncommitted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]