YannByron commented on PR #9580: URL: https://github.com/apache/paimon/pull/9580#issuecomment-5527387796
> I suggest defining RoboMIND's native Paimon storage contract early, following the approach in #9529: > > * Preserve original `master`/`puppet` fields, dtypes and image/depth payloads in `frames`; keep trajectory metadata in `episodes`. > * Add `tasks`, `annotations` and `calibrations` when provided by the source. > * Publish components under the same `BIGINT version_id` tag, with a `versions` manifest and `READY` written last. > * Treat canonical state/action and normalization statistics as explicitly derived data, not as the native schema. > > Start with AgileX without assuming other RoboMIND variants share its layout or sampling axis. This can be a separate schema/ingestion change; the window API should stay generic and be tested against that native contract. [robomind_agilex](https://github.com/apache/paimon/blob/master/paimon-python/pypaimon/sample/robomind_agilex.py) already defines an explicit Paimon storage contract for the RoboMIND AgileX dataset through `episode_schema`, `frame_schema`, and `feature_stats_schema`. It covers dataset metadata, ordered frame data, image/depth payloads, and the normalization data required for training. The current AgileX contract does not yet model `tasks`, `annotations`, or `calibrations`. These can be added when they are present in the AgileX source and required by a concrete consumer. I definitely agree that different datasets and variants should define their own table groups and table schemas according to their native contracts. The current `contiguous-window` dataset API addresses the requirements of the HDF5-based RoboMIND AgileX dataset while remaining extensible for future sources and modalities. I will first address the comments above. Afterwards, I will use PR https://github.com/apache/paimon/pull/9529 as a reference for adding a multi-table publication contract. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
