YannByron opened a new pull request, #9445: URL: https://github.com/apache/paimon/pull/9445
## Summary Add a RoboMIND AgileX sample that uses the same HDF5 transform contract for local and Ray ingestion. The pipeline writes episode and frame tables, materializes canonical actions, and refreshes train-split statistics independently. ## Changes - Add pypaimon.ray.load_from_hdf5 with shared HDF5 discovery, transform validation, and a single coordinated Paimon commit. - Add separate RoboMIND AgileX episode and frame transforms for local and Ray ingestion. - Keep deletion vectors and Vortex vector storage enabled while storing image and depth payloads as raw bytes. - Split canonical action materialization from statistics refresh and process frame updates by Paimon split. - Generate multi-episode AgileX HDF5 fixtures in pytest, with an optional flag for downloaded customer data. - Document the dataset layout, split semantics, table options, local usage, and Ray usage. ## Testing - [x] python -m pytest -q pypaimon/tests/multimodal_hdf5_test.py pypaimon/tests/ray_hdf5_test.py pypaimon/tests/robomind_agilex_pipeline_test.py (29 passed, 1 skipped) - [x] Ruff on all changed Python files - [x] git diff --check origin/master...HEAD - [ ] Customer dataset test with --robomind-agilex-input (requires a downloaded RoboMIND dataset) ## Notes This builds on the HDF5 DataSource ingestion merged in #9411. Repeating ingestion appends duplicate rows by design; a future merge-into mode can provide episode-level upserts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
