Zouxxyy commented on code in PR #11079:
URL: https://github.com/apache/paimon/pull/11079#discussion_r4529951838
##########
paimon-python/pypaimon/multimodal/lerobot/aggregate.py:
##########
@@ -494,15 +494,93 @@ def _import_sources(connection, table_name, sources,
batch_size, options,
"schema": schema, "videos": videos,
"extra_dataset_fields": _read_text_fields(resolved,
extra_dataset_fields),
"metadata": _load_dataset_metadata(dataset, info,
resolved)})
+ existing = None
+ if append:
+ if options:
+ raise ValueError("LeRobot append cannot change table options.")
+ from pypaimon.multimodal.lerobot.dataset import (
+ _read_arrow, _source_dataset_ranges,
+ )
+ frames_name = _companion_identifier(
+ connection._identifier(table_name), "_frames")
+ table = connection.get_table(frames_name)
+ identifiers = _companion_table_identifiers(table.raw_table)
+ tables = {name: connection.catalog.get_table(identifier)
+ for name, identifier in identifiers.items()}
+ if tag_name is not None:
+ for component in (table.raw_table, *tables.values()):
+ if _tag_snapshot_id(connection.catalog,
component.identifier,
+ tag_name) is not None:
+ raise ValueError("LeRobot tag %s already exists on %s."
+ % (tag_name, component.identifier))
+ if _DATASET_INDEX_COLUMN not in
_target_schema(tables["episodes"]).names:
+ raise ValueError("LeRobot append requires a source-local table
group.")
+ source_rows = _read_arrow(tables["datasets"], [
Review Comment:
[P2] Include existing vocabularies when planning appended control statistics
`_plan_merge(plans, ...)` builds task/subtask mappings using only the newly
appended sources, so their mapped statistics start from that append batch's
vocabulary. The adjustment below adds frame and dataset offsets but does not
reconcile those mappings with the existing group. For example, with existing
tasks `pick/place` and appended tasks `place/stack`, the appended Episode
statistics describe IDs `0/1`, whereas `_source_vocabulary()` over the complete
Dataset table makes the training reader return `1/2`.
In a reproduction with five frames per source and local task IDs
`[0,0,1,1,1]`, `_merged_source_stats()` reports task mean `0.6`, while the
actual training IDs have mean `1.1` (and max `2`, rather than `1`). The same
mismatch affects subtask statistics. Please load the existing task/subtask
vocabularies and seed the mappings used for appended Episode statistics from
the complete group, or recompute the summaries in the reader's global ID space.
##########
paimon-python/pypaimon/multimodal/lerobot/aggregate.py:
##########
@@ -545,13 +623,24 @@ def _import_sources(connection, table_name, sources,
batch_size, options,
episode_write.abort()
else:
episode_commit.abort(episode_messages)
- if stats is not None:
+ if existing is None and stats is not None:
for name in _CONTROLS:
parts = [s[name] for p in plans for s in
p["control_stats"].values() if name in s]
if parts:
stats[name] = _merge_stats(parts)
else:
stats.pop(name, None)
metadata["stats_table"] = _metadata_table(stats)
- _commit_metadata(connection, tables, tag_name, metadata,
table.identifier,
- frames_snapshot, episodes_snapshot)
+ if existing is None:
+ _commit_metadata(connection, tables, tag_name, metadata,
table.identifier,
+ frames_snapshot, episodes_snapshot)
+ else:
+ for column in ("dataset_index", "index"):
+ if table.raw_table.create_global_index(column,
index_type="btree") <= 0:
+ raise RuntimeError("LeRobot append did not refresh %s
index."
+ % column)
+ if _append_arrow(tables["datasets"], metadata["datasets_table"])
is None:
Review Comment:
[P2] Update FPS mode when an appended source introduces a different rate
Importing `[source_a]` at 10 FPS sets the frame table's managed
`pypaimon.lerobot.fps-mode` option to `dataset`. Appending a compatible 20 FPS
source passes the checks above and commits successfully, but this branch never
updates that option. On the next `PaimonLeRobotDataset` construction,
`_merged_source_info()` sees the two rates and produces `fps=None`, while
retaining dataset mode; `_episode_fps()` then raises `ValueError: LeRobot
metadata is missing fps`. Thus a successful append makes the entire current
group unreadable for training, even though mixed FPS is documented as automatic.
Please derive the effective FPS mode from both existing and appended sources
and enable source mode when necessary, or reject the rate mismatch before
committing Frames/Episodes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]