JingsongLi commented on code in PR #11097:
URL: https://github.com/apache/paimon/pull/11097#discussion_r4546099854


##########
paimon-python/pypaimon/multimodal/lerobot/aggregate.py:
##########
@@ -618,30 +808,72 @@ def _import_sources(connection, table_name, sources, 
batch_size, options,
                                          % (tag_name, component.identifier))
             if _DATASET_INDEX_COLUMN not in 
_target_schema(tables["episodes"]).names:
                 raise ValueError("LeRobot append requires a source-local table 
group.")
-            source_rows = _read_arrow(tables["datasets"], [
-                "index", "info", "tasks", "subtasks", "stats",
-            ]).sort_by("index").to_pylist()
+            source_columns = ["index", "info", "tasks", "subtasks", "stats"]
+            dataset_columns = _target_schema(tables["datasets"]).names
+            for name in ("dataset_id", "source_hash"):
+                if name in dataset_columns:
+                    source_columns.append(name)
+            source_rows = _read_arrow(tables["datasets"],
+                                      
source_columns).sort_by("index").to_pylist()
             if not source_rows:
                 raise ValueError("LeRobot append requires an initialized table 
group.")
+            has_coordinated_rows = any(row.get("dataset_id") is not None
+                                       for row in source_rows)

Review Comment:
   [P2] Distinguish ordinary idempotent appends from coordinated writes
   
   A non-null `dataset_id` does not imply `coordinated=True`: the API and new 
tests explicitly support supplying an ID for an ordinary single-writer append, 
and `_plan_merge` saves that ID in `_datasets` in either mode. I reproduced an 
initial list import followed by `append_to_lerobot(..., dataset_id="b")`; that 
append succeeds, but the next ordinary append, with either a new ID or no ID, 
raises `LeRobot group has coordinated Datasets`. This prevents continuing the 
supported single-writer workflow after its first idempotent append. Please 
detect the actual coordinated storage mode, for example through an explicit 
marker or its physical-key allocation, instead of treating every saved ID as 
evidence of coordination.



##########
paimon-python/pypaimon/multimodal/lerobot/aggregate.py:
##########
@@ -618,30 +808,72 @@ def _import_sources(connection, table_name, sources, 
batch_size, options,
                                          % (tag_name, component.identifier))
             if _DATASET_INDEX_COLUMN not in 
_target_schema(tables["episodes"]).names:
                 raise ValueError("LeRobot append requires a source-local table 
group.")
-            source_rows = _read_arrow(tables["datasets"], [
-                "index", "info", "tasks", "subtasks", "stats",
-            ]).sort_by("index").to_pylist()
+            source_columns = ["index", "info", "tasks", "subtasks", "stats"]
+            dataset_columns = _target_schema(tables["datasets"]).names
+            for name in ("dataset_id", "source_hash"):
+                if name in dataset_columns:
+                    source_columns.append(name)
+            source_rows = _read_arrow(tables["datasets"],
+                                      
source_columns).sort_by("index").to_pylist()
             if not source_rows:
                 raise ValueError("LeRobot append requires an initialized table 
group.")
+            has_coordinated_rows = any(row.get("dataset_id") is not None
+                                       for row in source_rows)
             ranges = _source_dataset_ranges(source_rows)
             expected_counts = {"frames": ranges[-1][4],
                                "episodes": ranges[-1][2],
                                "datasets": len(source_rows)}
-            for name, component in (("frames", table.raw_table),
-                                    ("episodes", tables["episodes"]),
-                                    ("datasets", tables["datasets"])):
-                snapshot = component.snapshot_manager().get_latest_snapshot()
-                if snapshot is None or snapshot.next_row_id != 
expected_counts[name]:
-                    raise ValueError(
-                        "LeRobot group has incomplete %s data; call "
-                        "recover_lerobot_append before appending." % name)
+            if not coordinated:
+                for name, component in (("frames", table.raw_table),
+                                        ("episodes", tables["episodes"]),
+                                        ("datasets", tables["datasets"])):
+                    snapshot = 
component.snapshot_manager().get_latest_snapshot()
+                    if snapshot is None or snapshot.next_row_id != 
expected_counts[name]:
+                        raise ValueError(
+                            "LeRobot group has incomplete %s data; call "
+                            "recover_lerobot_append before appending." % name)
+            if dataset_id is not None:
+                matches = [row for row in source_rows
+                           if row.get("dataset_id") == dataset_id]
+                if matches:
+                    if len(matches) != len(sources) or any(
+                            row.get("source_hash") != source_hash
+                            for row in matches):
+                        raise ValueError(
+                            "LeRobot dataset_id %r was used for different 
sources."
+                            % dataset_id)
+                    return
+            if has_coordinated_rows and not coordinated:
+                raise ValueError(
+                    "LeRobot group has coordinated Datasets; append with 
coordinated=True.")
             existing = (table, tables, source_rows, ranges)
         metadata, stats = _plan_merge(
             plans, extra_dataset_fields, schema_evolution if existing is None 
else True,
-            source_rows if existing is not None else None)
+            source_rows if existing is not None else None,
+            dataset_id, source_hash)
         target_schema = metadata["frame_schema"]
         target_schema = target_schema.append(
             pa.field(_DATASET_INDEX_COLUMN, pa.int64(), nullable=False))
+        for name in ("dataset_id", "source_hash"):
+            if existing is None or coordinated or name in 
_target_schema(table.raw_table).names:
+                target_schema = target_schema.append(pa.field(name, 
pa.string()))

Review Comment:
   [P2] Preserve custom features that share the new identity-column names
   
   For a new group, these identity columns are appended even when no 
`dataset_id` is supplied and `coordinated=False`. A previously valid source 
feature named `dataset_id` or `source_hash` therefore produces duplicate schema 
fields and makes an ordinary import fail. Using the existing test helpers, I 
added an int64 `dataset_id` feature and called `load_from_lerobot("robot", 
[uri])`: base `40a386e12f` imports and preserves the values, while this commit 
raises `ValueError ... must not contain duplicate fields. Found: 
['dataset_id']`. Please use collision-safe internal identity storage or 
otherwise preserve ordinary imports whose source schema already owns these 
names.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to