JingsongLi commented on code in PR #11097:
URL: https://github.com/apache/paimon/pull/11097#discussion_r4579132770


##########
paimon-python/pypaimon/multimodal/lerobot/aggregate.py:
##########
@@ -593,13 +773,81 @@ def _merged_episodes(plans, schema):
             raise ValueError("LeRobot Episode metadata changed after 
validation.")
 
 
+def _backfill_legacy_control_counts(frames_table, datasets_table):
+    """Persist old sources' histograms once, before coordinated publication."""
+    from pypaimon.multimodal.lerobot.dataset import _frame_control_counts, 
_read_arrow
+
+    baseline = datasets_table.snapshot_manager().get_latest_snapshot()
+    rows = _read_arrow(datasets_table, [
+        "id", "_ROW_ID", _CONTROL_COUNTS_COLUMN]).to_pylist()
+    missing = sorted((row for row in rows if row[_CONTROL_COUNTS_COLUMN] is 
None),
+                     key=lambda row: row["_ROW_ID"])
+    if not missing:
+        return
+    fields = [name for name in ("task_index", "subtask_index")
+              if name in _target_schema(frames_table).names]
+    updates = [(row["_ROW_ID"], json.dumps(
+        _frame_control_counts(frames_table, row["id"], fields),
+        sort_keys=True, separators=(",", ":"))) for row in missing]
+    builder = datasets_table.new_batch_write_builder()
+    commit = builder.new_commit()
+    writers, messages = [], []
+    commit_started = False
+    try:
+        # A partial Data Evolution write preserves every other Dataset column.
+        for row_id, value in updates:
+            writer = 
builder.new_write().with_write_type([_CONTROL_COUNTS_COLUMN])
+            writers.append(writer)
+            writer.write_arrow(pa.Table.from_pydict(
+                {_CONTROL_COUNTS_COLUMN: [value]},
+                schema=pa.schema([(_CONTROL_COUNTS_COLUMN, pa.string())])))

Review Comment:
   [P1] Align count backfills with the existing Dataset file ranges
   
   This writes one normal partial-column file per Dataset row and then assigns 
that row's `_ROW_ID` as `first_row_id`. An existing group initially imported 
from `[source_a, source_b]` stores both Dataset rows in one two-row normal 
file, so these singleton backfill files violate Data Evolution's requirement 
that overlapping normal files cover the same complete row-ID range. I 
reproduced this by importing two sources together, dropping 
`__paimon_control_counts` to emulate a legacy group, and appending a third 
source with `coordinated=True`. The backfill publishes an invalid Dataset 
snapshot: Python reads fail with a field-merge row-count error, and native 
reads fail with a row-ID-range error. The existing published group becomes 
unreadable and subsequent retries also fail. Please use the existing 
range-aware row-ID update path, or write the complete original file ranges, and 
cover a legacy Dataset file containing multiple rows.



##########
paimon-python/pypaimon/execution/python/write/file_store_commit.py:
##########
@@ -361,7 +369,19 @@ def _try_commit(self, commit_kind, commit_identifier, 
commit_entries_plan,
                     commit_identifier,
                     commit_kind,
                     notify_callbacks=True):
-                break
+                return True
+            if self.commit_condition is not None:
+                try:
+                    needed = self.commit_condition(latest_snapshot)
+                except BaseException:
+                    if not commit_result_may_be_uncertain and 
prepared_messages is not None:
+                        self.abort(prepared_messages)
+                    raise
+                if not needed:
+                    # An uncertain response may have published these files.
+                    if not commit_result_may_be_uncertain and 
prepared_messages is not None:
+                        self.abort(prepared_messages)

Review Comment:
   [P1] Preserve published files when recovering a durable checkpoint
   
   The ownership marker only protects the in-memory messages that observed the 
commit attempt. If prepared messages are checkpointed before publication, then 
the snapshot is published but its response is lost and the process restarts, 
the restored messages still have `commit_outcome_uncertain=False`. I reproduced 
this with a real filesystem table: pickle the messages immediately after 
`prepare_commit(1)`, use condition `snapshot is None` and 
`commit.max-retries=0`, publish the snapshot and raise `OSError`, then restore 
the checkpoint into a new stream committer with the same commit user and 
identifier. This branch returns False and deletes the file referenced by 
snapshot 1; reads subsequently fail with `FileNotFoundError`. The duplicate 
check above cannot help because `retry_result` is local to the new invocation. 
Please resolve publication through stable commit identity or file references 
before aborting recovered messages; absence of the in-memory marker does not 
establish that the
 y are uncommitted.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to