JingsongLi commented on PR #9498: URL: https://github.com/apache/paimon/pull/9498#issuecomment-5487595281
I think the imported dataset should be self-contained in Paimon, but separate typed tables would be clearer than either duplicating metadata in every frame or storing all metadata in a generic `kind + payload` table. A possible logical dataset bundle is: - `frames`: the existing per-frame data, with a stable `dataset_id`. - `datasets`: one row per dataset metadata version, containing the LeRobot format version, FPS, feature schema, global statistics, total counts, source information, and the associated frame-table snapshot/version. - `episodes`: one row per episode, keyed by `dataset_id`, `metadata_version`, and `episode_index`, with its frame range, length, and any episode-level statistics. - `tasks`: the `task_index` to task-text mapping, keyed by `dataset_id` and `metadata_version`. `features` and global `stats` can remain JSON initially because their structures are format-dependent. Episodes and tasks have stable identities and relationships, so explicit columns and tables provide a much clearer contract and make them independently queryable. Video metadata could be added as another typed table when video-backed datasets are supported. All component tables should use the same `dataset_id` and `metadata_version`. Since the tables cannot be committed atomically, `datasets` can act as the manifest and publication point: write the frame, episode, and task records first, then publish a completed dataset manifest which pins their corresponding versions. A reader must resolve exactly the versions referenced by that manifest and fail if any component is missing or mismatched. This avoids silently combining frame data with stale metadata. With this contract, `PaimonLeRobotDataset(table, dataset_id)` becomes a thin adapter which reconstructs a LeRobot-compatible view from Paimon. The original LeRobot `meta/` directory is only an import-time input and is no longer required during training. The current PR leaves the source of truth split between Paimon frames and an external metadata directory. I suggest defining and implementing the self-contained multi-table import contract first, then keeping `PaimonLeRobotDataset` focused on reading that contract. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
