XiaoHongbo-Hope opened a new pull request, #11097:
URL: https://github.com/apache/paimon/pull/11097

   Independent Ray jobs could read the same Dataset count, allocate the same 
`dataset_index`, and interleave commits to the Frames, Episodes, and Datasets 
tables. A failed race can leave the group unreadable.
   
   This follow-up to #11095 adds opt-in `coordinated=True` for 
`append_to_lerobot`. A named Ray actor per table group executes append requests 
sequentially while each request can still prepare data with multiple Ray 
workers. A stable `request_id` and source fingerprint in `_datasets` make a 
completed request safe to retry; the actor checks and recovers an interrupted 
append before the next request. Existing direct appends remain available for a 
single writer.
   
   All writers to a group must use the coordinated path on the same Ray 
cluster. Publication still spans three tables, so readers that need a stable 
training view should use a completed group tag. This PR does not claim 
throughput scaling from concurrent commits.
   
   Validation: LeRobot aggregate tests: 161 passed, 75 skipped in the 
training-dependency environment; 131 passed, 105 skipped in the Ray 
environment. Focused coordinator/recovery tests passed on supported Ray 2.59.0. 
Two independent Ray 2.59.0 Python jobs appended simultaneously to one group; 
readback showed 3 Datasets, 6 Episodes, and 15 Frames.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to