JingsongLi commented on PR #10754: URL: https://github.com/apache/paimon/pull/10754#issuecomment-6360111580
**[P2] Preserve hidden optional columns in formatted episode metadata** In [`_episode_rows`, lines 1024–1026](https://github.com/apache/paimon/blob/19e469480f2788bfffa9b79cccb98fae62baf4f1/paimon-python/pypaimon/multimodal/lerobot/dataset.py#L1024-L1026), `episodes.column_names` describes the physical columns, which can differ from the columns exposed by the Dataset's format. For example, a custom `PaimonDatasetReader` can receive episode metadata configured as follows: ```python episodes = Dataset.from_list(valid_episode_rows).with_format( "numpy", # "torch" and None also reproduce this columns=[ "episode_index", "dataset_from_index", "dataset_to_index", "length" ], ) ``` If the underlying rows also contain `tasks`, the default `output_all_columns=False` hides that field from formatted access. `_episode_ranges` succeeds, but `_episode_tasks` then projects the physically present `tasks` column, receives an empty formatted batch, and `batch["tasks"]` raises `KeyError` during Reader initialization. Previously, `row.get("tasks")` treated the hidden optional field as absent, and the same metadata could initialize and read frames successfully. I reproduced the base-versus-head difference through the public Reader API with `datasets` 4.0.0 and 4.8.5. This affects custom readers using filtered HF metadata; the default table-backed path is unaffected. Please preserve the existing handling of hidden optional fields, for example by retaining row access for Datasets with configured column filtering, and add a regression test for this case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
