XiaoHongbo-Hope opened a new pull request, #11097: URL: https://github.com/apache/paimon/pull/11097
Independent Ray jobs could read the same Dataset count, allocate the same `dataset_index`, and interleave commits to the Frames, Episodes, and Datasets tables. A failed race can leave the group unreadable. This follow-up to #11095 adds opt-in `coordinated=True` for `append_to_lerobot`. A named Ray actor per table group executes append requests sequentially while each request can still prepare data with multiple Ray workers. A stable `request_id` and source fingerprint in `_datasets` make a completed request safe to retry; the actor checks and recovers an interrupted append before the next request. Existing direct appends remain available for a single writer. All writers to a group must use the coordinated path on the same Ray cluster. Publication still spans three tables, so readers that need a stable training view should use a completed group tag. This PR does not claim throughput scaling from concurrent commits. Validation: LeRobot aggregate tests: 161 passed, 75 skipped in the training-dependency environment; 131 passed, 105 skipped in the Ray environment. Focused coordinator/recovery tests passed on supported Ray 2.59.0. Two independent Ray 2.59.0 Python jobs appended simultaneously to one group; readback showed 3 Datasets, 6 Episodes, and 15 Frames. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
