Zouxxyy commented on code in PR #11077:
URL: https://github.com/apache/paimon/pull/11077#discussion_r4529418726


##########
paimon-python/pypaimon/multimodal/lerobot/dataset.py:
##########
@@ -1031,45 +1065,170 @@ def _load_dataset(table, tag_name):
     identifiers = _companion_table_identifiers(raw_table)
     catalog = table.catalog
     frames = _component_table(catalog, raw_table, tag_name)
-    datasets = None
-    if "datasets" in identifiers:
-        datasets_table = _component_table(
-            catalog, catalog.get_table(identifiers["datasets"]), tag_name)
-        projection = [name for name in _target_schema(datasets_table).names
-                      if name not in ("tasks", "subtasks", "source_tasks", 
"source_subtasks", "stats",
-                                      "episodes_schema_json", "tasks_json", 
"subtasks_json")
-                      and not name.endswith("_ipc")]
-        dataset_rows = _read_arrow(datasets_table, projection)
-        datasets = dataset_rows.sort_by(
-            "index" if "index" in dataset_rows.column_names else 
"dataset_index").to_pylist()
+    source_local_layout = "dataset_index" in _target_schema(frames).names

Review Comment:
   [P2] Avoid inferring the source-local layout from a user feature name
   
   A single-source import can legitimately declare an additional 
`dataset_index` feature (`dtype="int64", shape=[1]`): the feature/schema 
validation accepts it, and the single-source import path preserves it in Frames 
without adding `dataset_index` to Episodes. This check then misclassifies that 
group as a merged/source-local layout, passes non-null `datasets` to 
`_episode_dataset()`, and fails when it sorts Episodes by the missing 
`dataset_index` column (`ArrowInvalid`). The imported group consequently cannot 
be opened with `PaimonLeRobotDataset`.
   
   Could we use an explicit layout marker or validate the companion Episode 
layout instead? Alternatively, if this name is reserved for every import mode, 
reject it before creating the target tables. I reproduced the mismatch using 
the PR's schema-validation and reader functions in isolation; full integration 
tests could not start because the locally installed PyPaimon lacks the current 
`pypaimon.schema.arrow_schema` module.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to