Zouxxyy commented on code in PR #11079:
URL: https://github.com/apache/paimon/pull/11079#discussion_r4529951838


##########
paimon-python/pypaimon/multimodal/lerobot/aggregate.py:
##########
@@ -494,15 +494,93 @@ def _import_sources(connection, table_name, sources, 
batch_size, options,
                           "schema": schema, "videos": videos,
                           "extra_dataset_fields": _read_text_fields(resolved, 
extra_dataset_fields),
                           "metadata": _load_dataset_metadata(dataset, info, 
resolved)})
+        existing = None
+        if append:
+            if options:
+                raise ValueError("LeRobot append cannot change table options.")
+            from pypaimon.multimodal.lerobot.dataset import (
+                _read_arrow, _source_dataset_ranges,
+            )
+            frames_name = _companion_identifier(
+                connection._identifier(table_name), "_frames")
+            table = connection.get_table(frames_name)
+            identifiers = _companion_table_identifiers(table.raw_table)
+            tables = {name: connection.catalog.get_table(identifier)
+                      for name, identifier in identifiers.items()}
+            if tag_name is not None:
+                for component in (table.raw_table, *tables.values()):
+                    if _tag_snapshot_id(connection.catalog, 
component.identifier,
+                                        tag_name) is not None:
+                        raise ValueError("LeRobot tag %s already exists on %s."
+                                         % (tag_name, component.identifier))
+            if _DATASET_INDEX_COLUMN not in 
_target_schema(tables["episodes"]).names:
+                raise ValueError("LeRobot append requires a source-local table 
group.")
+            source_rows = _read_arrow(tables["datasets"], [

Review Comment:
   [P2] Include existing vocabularies when planning appended control statistics
   
   `_plan_merge(plans, ...)` builds task/subtask mappings using only the newly 
appended sources, so their mapped statistics start from that append batch's 
vocabulary. The adjustment below adds frame and dataset offsets but does not 
reconcile those mappings with the existing group. For example, with existing 
tasks `pick/place` and appended tasks `place/stack`, the appended Episode 
statistics describe IDs `0/1`, whereas `_source_vocabulary()` over the complete 
Dataset table makes the training reader return `1/2`.
   
   In a reproduction with five frames per source and local task IDs 
`[0,0,1,1,1]`, `_merged_source_stats()` reports task mean `0.6`, while the 
actual training IDs have mean `1.1` (and max `2`, rather than `1`). The same 
mismatch affects subtask statistics. Please load the existing task/subtask 
vocabularies and seed the mappings used for appended Episode statistics from 
the complete group, or recompute the summaries in the reader's global ID space.



##########
paimon-python/pypaimon/multimodal/lerobot/aggregate.py:
##########
@@ -545,13 +623,24 @@ def _import_sources(connection, table_name, sources, 
batch_size, options,
                     episode_write.abort()
                 else:
                     episode_commit.abort(episode_messages)
-        if stats is not None:
+        if existing is None and stats is not None:
             for name in _CONTROLS:
                 parts = [s[name] for p in plans for s in 
p["control_stats"].values() if name in s]
                 if parts:
                     stats[name] = _merge_stats(parts)
                 else:
                     stats.pop(name, None)
             metadata["stats_table"] = _metadata_table(stats)
-        _commit_metadata(connection, tables, tag_name, metadata, 
table.identifier,
-                         frames_snapshot, episodes_snapshot)
+        if existing is None:
+            _commit_metadata(connection, tables, tag_name, metadata, 
table.identifier,
+                             frames_snapshot, episodes_snapshot)
+        else:
+            for column in ("dataset_index", "index"):
+                if table.raw_table.create_global_index(column, 
index_type="btree") <= 0:
+                    raise RuntimeError("LeRobot append did not refresh %s 
index."
+                                       % column)
+            if _append_arrow(tables["datasets"], metadata["datasets_table"]) 
is None:

Review Comment:
   [P2] Update FPS mode when an appended source introduces a different rate
   
   Importing `[source_a]` at 10 FPS sets the frame table's managed 
`pypaimon.lerobot.fps-mode` option to `dataset`. Appending a compatible 20 FPS 
source passes the checks above and commits successfully, but this branch never 
updates that option. On the next `PaimonLeRobotDataset` construction, 
`_merged_source_info()` sees the two rates and produces `fps=None`, while 
retaining dataset mode; `_episode_fps()` then raises `ValueError: LeRobot 
metadata is missing fps`. Thus a successful append makes the entire current 
group unreadable for training, even though mixed FPS is documented as automatic.
   
   Please derive the effective FPS mode from both existing and appended sources 
and enable source mode when necessary, or reject the rate mismatch before 
committing Frames/Episodes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to