XiaoHongbo-Hope opened a new pull request, #11034:
URL: https://github.com/apache/paimon/pull/11034

   ### Purpose
   
   Sampling frames throughout a small remote video can spend more time on 
repeated range requests than decoding. Add explicit, bounded whole-payload 
prefetch to `VideoFrameCollator`, exposed to LeRobot through 
`read.video.prefetch-max-bytes`.
   
   The default is **0 (disabled)**. Prefetch admits a payload only when it fits 
the remaining encoded-byte budget across cached decoders in that collator. 
Otherwise it preserves range reads. The budget is per video feature and worker; 
decoded frames, codec buffers and temporary copies are additional, and 
independent decoder replicas consume it separately. Existing decoder 
eviction/close releases the buffered payload. Indexed range callbacks use the 
prefetched bytes as well, rather than fetching the same media again through 
FileIO.
   
   This changes the read policy only; no video format, re-encoding or metadata 
migration is required. Documentation covers the bandwidth/memory tradeoff and 
the public collator argument. It does not automatically select a policy based 
on access density.
   
   ### Tests
   
   - Real MP4 indexed decoding matches range reads, including reordered and 
duplicate frames; prefetch performs one media read and repeated batches perform 
no further media reads.
   - Added coverage for cached-decoder reuse, aggregate encoded-payload 
admission, disabled/oversized fallback, budget release on close, truncated 
payload cleanup, invalid arguments, and LeRobot option propagation.
   - Related video/collator/LeRobot/TorchCodec suites: **118 passed, 63 
skipped, 1 deselected** in the local optional-dependency environment.
   - The deselected 
`test_video_batches_include_delta_frames_and_preserve_backends` reproduces 
unchanged on base `b000c3945f` (combined-call expectation versus per-window 
batching).
   - Flake8 with the repository configuration and `git diff --check` pass.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to