YannByron opened a new pull request, #9530: URL: https://github.com/apache/paimon/pull/9530
## Summary Add transform-based ROSBag ingestion to PyPaimon for ROS1 and ROS2 recordings, including remote FileIO sources and Ray distributed execution. Source, transform, and Arrow schema validation finish before a Paimon writer is created. ## Changes - support ROS1 `.bag`, ROS2 SQLite3 and MCAP recording directories, and standalone `.mcap` files - allow standalone `.db3` only through the explicit `allow_storage_fragment=True` mode - stage OSS, S3, HDFS, ViewFS, and GCS sources locally with byte limits, free-space checks, path containment, and source mutation detection - scan recordings to EOF, validate declared message counts, and spool validated serial output to Arrow IPC before one Paimon commit - add Ray ingestion with driver-side discovery, worker-side validation, a `materialize()` barrier, and one coordinator commit - add the optional `pypaimon[rosbag]` dependency, public API documentation, and CI coverage ## Testing - [x] `python -m pytest pypaimon/tests/multimodal_rosbag_test.py pypaimon/tests/ray_rosbag_test.py pypaimon/tests/multimodal_hdf5_test.py pypaimon/tests/multimodal_lerobot_test.py pypaimon/tests/ray_hdf5_test.py pypaimon/tests/ray_sink_test.py -q` (102 passed) - [x] Ruff checks for changed Python sources and tests - [x] `python setup.py check` - [x] `git diff origin/master...HEAD --check` ## Notes `rosbags` requires Python 3.10 or newer and remains an optional dependency. Remote recordings are copied to attempt-scoped local storage because `rosbags.AnyReader` requires local paths. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
