Zouxxyy commented on code in PR #11077:
URL: https://github.com/apache/paimon/pull/11077#discussion_r4529418726
##########
paimon-python/pypaimon/multimodal/lerobot/dataset.py:
##########
@@ -1031,45 +1065,170 @@ def _load_dataset(table, tag_name):
identifiers = _companion_table_identifiers(raw_table)
catalog = table.catalog
frames = _component_table(catalog, raw_table, tag_name)
- datasets = None
- if "datasets" in identifiers:
- datasets_table = _component_table(
- catalog, catalog.get_table(identifiers["datasets"]), tag_name)
- projection = [name for name in _target_schema(datasets_table).names
- if name not in ("tasks", "subtasks", "source_tasks",
"source_subtasks", "stats",
- "episodes_schema_json", "tasks_json",
"subtasks_json")
- and not name.endswith("_ipc")]
- dataset_rows = _read_arrow(datasets_table, projection)
- datasets = dataset_rows.sort_by(
- "index" if "index" in dataset_rows.column_names else
"dataset_index").to_pylist()
+ source_local_layout = "dataset_index" in _target_schema(frames).names
Review Comment:
[P2] Avoid inferring the source-local layout from a user feature name
A single-source import can legitimately declare an additional
`dataset_index` feature (`dtype="int64", shape=[1]`): the feature/schema
validation accepts it, and the single-source import path preserves it in Frames
without adding `dataset_index` to Episodes. This check then misclassifies that
group as a merged/source-local layout, passes non-null `datasets` to
`_episode_dataset()`, and fails when it sorts Episodes by the missing
`dataset_index` column (`ArrowInvalid`). The imported group consequently cannot
be opened with `PaimonLeRobotDataset`.
Could we use an explicit layout marker or validate the companion Episode
layout instead? Alternatively, if this name is reserved for every import mode,
reject it before creating the target tables. I reproduced the mismatch using
the PR's schema-validation and reader functions in isolation; full integration
tests could not start because the locally installed PyPaimon lacks the current
`pypaimon.schema.arrow_schema` module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]