JingsongLi commented on code in PR #11097:
URL: https://github.com/apache/paimon/pull/11097#discussion_r4546099854
##########
paimon-python/pypaimon/multimodal/lerobot/aggregate.py:
##########
@@ -618,30 +808,72 @@ def _import_sources(connection, table_name, sources,
batch_size, options,
% (tag_name, component.identifier))
if _DATASET_INDEX_COLUMN not in
_target_schema(tables["episodes"]).names:
raise ValueError("LeRobot append requires a source-local table
group.")
- source_rows = _read_arrow(tables["datasets"], [
- "index", "info", "tasks", "subtasks", "stats",
- ]).sort_by("index").to_pylist()
+ source_columns = ["index", "info", "tasks", "subtasks", "stats"]
+ dataset_columns = _target_schema(tables["datasets"]).names
+ for name in ("dataset_id", "source_hash"):
+ if name in dataset_columns:
+ source_columns.append(name)
+ source_rows = _read_arrow(tables["datasets"],
+
source_columns).sort_by("index").to_pylist()
if not source_rows:
raise ValueError("LeRobot append requires an initialized table
group.")
+ has_coordinated_rows = any(row.get("dataset_id") is not None
+ for row in source_rows)
Review Comment:
[P2] Distinguish ordinary idempotent appends from coordinated writes
A non-null `dataset_id` does not imply `coordinated=True`: the API and new
tests explicitly support supplying an ID for an ordinary single-writer append,
and `_plan_merge` saves that ID in `_datasets` in either mode. I reproduced an
initial list import followed by `append_to_lerobot(..., dataset_id="b")`; that
append succeeds, but the next ordinary append, with either a new ID or no ID,
raises `LeRobot group has coordinated Datasets`. This prevents continuing the
supported single-writer workflow after its first idempotent append. Please
detect the actual coordinated storage mode, for example through an explicit
marker or its physical-key allocation, instead of treating every saved ID as
evidence of coordination.
##########
paimon-python/pypaimon/multimodal/lerobot/aggregate.py:
##########
@@ -618,30 +808,72 @@ def _import_sources(connection, table_name, sources,
batch_size, options,
% (tag_name, component.identifier))
if _DATASET_INDEX_COLUMN not in
_target_schema(tables["episodes"]).names:
raise ValueError("LeRobot append requires a source-local table
group.")
- source_rows = _read_arrow(tables["datasets"], [
- "index", "info", "tasks", "subtasks", "stats",
- ]).sort_by("index").to_pylist()
+ source_columns = ["index", "info", "tasks", "subtasks", "stats"]
+ dataset_columns = _target_schema(tables["datasets"]).names
+ for name in ("dataset_id", "source_hash"):
+ if name in dataset_columns:
+ source_columns.append(name)
+ source_rows = _read_arrow(tables["datasets"],
+
source_columns).sort_by("index").to_pylist()
if not source_rows:
raise ValueError("LeRobot append requires an initialized table
group.")
+ has_coordinated_rows = any(row.get("dataset_id") is not None
+ for row in source_rows)
ranges = _source_dataset_ranges(source_rows)
expected_counts = {"frames": ranges[-1][4],
"episodes": ranges[-1][2],
"datasets": len(source_rows)}
- for name, component in (("frames", table.raw_table),
- ("episodes", tables["episodes"]),
- ("datasets", tables["datasets"])):
- snapshot = component.snapshot_manager().get_latest_snapshot()
- if snapshot is None or snapshot.next_row_id !=
expected_counts[name]:
- raise ValueError(
- "LeRobot group has incomplete %s data; call "
- "recover_lerobot_append before appending." % name)
+ if not coordinated:
+ for name, component in (("frames", table.raw_table),
+ ("episodes", tables["episodes"]),
+ ("datasets", tables["datasets"])):
+ snapshot =
component.snapshot_manager().get_latest_snapshot()
+ if snapshot is None or snapshot.next_row_id !=
expected_counts[name]:
+ raise ValueError(
+ "LeRobot group has incomplete %s data; call "
+ "recover_lerobot_append before appending." % name)
+ if dataset_id is not None:
+ matches = [row for row in source_rows
+ if row.get("dataset_id") == dataset_id]
+ if matches:
+ if len(matches) != len(sources) or any(
+ row.get("source_hash") != source_hash
+ for row in matches):
+ raise ValueError(
+ "LeRobot dataset_id %r was used for different
sources."
+ % dataset_id)
+ return
+ if has_coordinated_rows and not coordinated:
+ raise ValueError(
+ "LeRobot group has coordinated Datasets; append with
coordinated=True.")
existing = (table, tables, source_rows, ranges)
metadata, stats = _plan_merge(
plans, extra_dataset_fields, schema_evolution if existing is None
else True,
- source_rows if existing is not None else None)
+ source_rows if existing is not None else None,
+ dataset_id, source_hash)
target_schema = metadata["frame_schema"]
target_schema = target_schema.append(
pa.field(_DATASET_INDEX_COLUMN, pa.int64(), nullable=False))
+ for name in ("dataset_id", "source_hash"):
+ if existing is None or coordinated or name in
_target_schema(table.raw_table).names:
+ target_schema = target_schema.append(pa.field(name,
pa.string()))
Review Comment:
[P2] Preserve custom features that share the new identity-column names
For a new group, these identity columns are appended even when no
`dataset_id` is supplied and `coordinated=False`. A previously valid source
feature named `dataset_id` or `source_hash` therefore produces duplicate schema
fields and makes an ordinary import fail. Using the existing test helpers, I
added an int64 `dataset_id` feature and called `load_from_lerobot("robot",
[uri])`: base `40a386e12f` imports and preserves the values, while this commit
raises `ValueError ... must not contain duplicate fields. Found:
['dataset_id']`. Please use collision-safe internal identity storage or
otherwise preserve ordinary imports whose source schema already owns these
names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]