YannByron opened a new pull request, #9708: URL: https://github.com/apache/paimon/pull/9708
## Summary Follow up on #9665 by preserving native LeRobot recording metadata when frames are captured directly into Paimon. The writer now publishes a managed table group containing frames, episodes, tasks, dataset information, and statistics without creating an intermediate LeRobot dataset directory. ## Changes - Create and validate `__episodes`, `__tasks`, `__info`, and `__stats` companion tables for direct capture. - Compute native LeRobot episode statistics when an episode is accepted and incrementally maintain global statistics across commits. - Store task labels once in the tasks table while frame rows retain `task_index`. - Resume frame, episode, task, and statistics state from snapshot properties and projected companion metadata without scanning frame data. - Commit companion metadata before frames and reject inconsistent or legacy frame-only table groups instead of silently continuing from partial state. - Document the direct-capture table group, statistics, resume behavior, and multi-table commit boundary. ## Testing - [x] Run all LeRobot loader and writer unit tests (74 tests). - [x] Verify numeric and image episode/global statistics against native LeRobot implementations. - [x] Verify multiple flushes, resume, schema validation, task deduplication, and inconsistent companion state handling. - [x] Run Ruff on all changed Python files. - [x] Run `git diff --check`. ## Notes - Direct capture still stores image frames as PNG blobs; video and subtasks remain unsupported. - Existing frame-only writer tables are not implicitly migrated because they do not contain the metadata required to reconstruct complete native LeRobot recording state. - Paimon does not provide a transaction across the table group, so reopening validates component state and rejects a partial batch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
