JingsongLi commented on PR #9498: URL: https://github.com/apache/paimon/pull/9498#issuecomment-5487633345
To clarify the business meaning of the three proposed metadata tables, they represent three different levels of the dataset rather than merely splitting one metadata blob by file type. ### 1. `datasets`: the published dataset manifest One row represents one complete, reproducible dataset version. It should contain dataset-level information such as: ```text dataset_id metadata_version format / format_version fps features_json global_stats_json total_frames / total_episodes / total_tasks frames_snapshot_id episodes_snapshot_id tasks_snapshot_id source_uri checksum status ``` This table defines the global training contract. For example, `fps` determines how delta timestamps are converted to frame offsets, `features_json` defines the dtype and shape of every feature, and global statistics are used for state/action normalization. More importantly, this row is the manifest for a reproducible release. A training job should be able to record something like `aloha_pick_cube@3` and later reopen exactly the same frame, episode, and task versions. The importer can write the component tables first and publish the `datasets` row with `status = READY` only after validation succeeds. Readers should ignore incomplete versions. ### 2. `episodes`: the trajectory index and sampling boundaries One row represents one robot rollout/trajectory: ```text dataset_id metadata_version episode_index dataset_from_index dataset_to_index length task_indices split episode_stats_json ``` This is not only redundant aggregation over the frame table. It is the authoritative trajectory directory used to: - select or shuffle complete episodes without scanning all frames; - prevent delta-window reads from crossing an episode boundary; - apply padding correctly at the beginning and end of a rollout; - create train/validation splits at episode granularity; - attach episode-level properties such as success, duration, quality, or statistics. Although some values could be derived with `GROUP BY episode_index`, recomputing them whenever a dataset is opened would be expensive and would not provide a stable, versioned contract. ### 3. `tasks`: the task semantic dictionary One row maps a stable task identifier to its human-readable instruction: ```text dataset_id metadata_version task_index task task_metadata_json ``` For example: ```text 0 -> "Pick up the red cube" 1 -> "Place the cube in the drawer" ``` Frame rows can store only `task_index`, avoiding repeated task strings while still supporting filtering, balancing, and statistics by task. The task definition is also pinned to the same metadata version as the frames. If one episode contains multiple tasks, `episodes.task_indices` can preserve that relationship; a separate relation table is only needed if the model becomes more complex later. The resulting mapping to the LeRobot API is straightforward: ```text meta.info <- datasets meta.stats <- datasets meta.episodes <- episodes meta.tasks <- tasks dataset[i] <- frames ``` Therefore, the tables have distinct business granularity and lifecycle: ```text datasets: one row per published dataset version episodes: one row per trajectory tasks: one row per task definition frames: one row per observation/frame ``` This makes the schema, keys, query behavior, and version relationships much clearer than a single generic metadata table, while keeping `PaimonLeRobotDataset` as a thin reconstruction layer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
