XiaoHongbo-Hope commented on PR #9498: URL: https://github.com/apache/paimon/pull/9498#issuecomment-5549355401
Added two focused follow-up commits without changing the public `PaimonLeRobotDataset` constructor API: 1. `ad4d78963d` vectorizes semantic-index validation over Arrow record batches. On the 23,408,858-frame DROID-derived Paimon table used for validation, the complete semantic mapping phase dropped from 253.02 s to 16.99 s. The implementation still validates non-identity physical ordering, episode/frame controls, timestamps, task/subtask metadata, cross-batch duplicates, and only returns `range` after proving an identity layout. 2. `d48655aa0c` retries BLOB fetch and image decode as one operation. This restores the original BLOB descriptors before retrying, so a transient truncated response is fetched again instead of retrying decode against the same invalid bytes. A fault-injection A10 smoke test forced an invalid first JPEG response and succeeded on the second fetch. Validation on the updated head: - `py_compile`: passed - flake8 for the changed files: passed - `multimodal_lerobot_test.py`: 32 passed, 28 skipped, 32 subtests passed (the skipped cases require the optional LeRobot integration environment) - `git diff --check`: passed The separate per-rank Arrow descriptor materialization used in the scale benchmark is intentionally not included here. It requires a generic `TorchDataset`/`TableRead` design for bounded caching and sampler interaction; exposing rank/cache policy through LeRobot-specific constructor switches would be premature. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
