YannByron opened a new pull request, #10979: URL: https://github.com/apache/paimon/pull/10979
### Purpose Indexed PyAV reads currently reopen the MP4 container and parse its metadata whenever a batch misses the frame cache. The outer decoder LRU therefore reuses the Python wrapper but not the demuxer. Keep the container and stream until decoder eviction or close, and discard a failed session before retrying. Release the Python Range byte cache after every batch, rather than retaining media payloads across batches. The existing per-read cache bound and decoder LRU still apply. Public APIs, file formats, codec settings and sample ordering are unchanged. Measurements use the same four RoboMIND AgileX front-camera videos, 640x480, identical MP4 bytes and random sample plans. Baseline and candidate outputs match exactly. Runs alternate baseline/candidate in isolated processes on an Apple M2 Pro, Python 3.13, Torch 2.10 and PyAV 15.1. | Read workload | Baseline median samples/s | Candidate median samples/s | Median paired gain | | --- | ---: | ---: | ---: | | Local indexed GOP2, 256 KiB experimental cache cap, 3 pairs | 473.75 | 537.17 | +13.4% | | Local CWD + DataLoader, GOP2, history [-1,0], batch32, 2 spawn workers, 3 pairs | 646.42 | 730.65 | +11.5% | The DataLoader test uses 10 warmup and 32 measured batches, default PyAV codec threads, persistent workers and prefetch_factor=1. Its paired gains range from +7.2% to +26.3%. The small cache cap in the direct-reader test cannot hold a whole input video; no new cache option is introduced. With actual OSS Range GETs and the unchanged Paimon range-coalescing algorithm, a separate 12-batch test reduces requests from 96 to 64 and transferred media bytes from 2,714,160 to 2,417,192. All five pairs produce those counts. Index preparation is excluded, and no disk/object cache or simulated latency is used. Public-endpoint timings are variable and are not a GPU-training measurement. Limits and costs: the working set is four videos. GOP16 direct-read paired results cross zero (-2.1% to +3.5%), so this is not a general long-GOP speedup. Retained FFmpeg contexts also retain resources: in the batch32 test, the worker-inclusive RSS snapshot is about 1.81 -> 2.04 GB, and post-loop threads are 24 -> 105. These are snapshots, not peak measurements. The existing decoder LRU bounds the number of sessions. Large working sets with little decoder reuse may see little benefit. No model/GPU throughput or advantage over HDF5/LeRobot is claimed. ### Tests - Added real H.264/B-frame regression coverage for unordered and duplicate frames across batches, one container open, transient buffer release and resource closure. - Added failed-session cleanup and successful retry coverage. - Updated the codec-thread assertion to check the requested value before FFmpeg opens the codec; FFmpeg can resolve a different runtime value on the retained stream. - 204 tests passed across `multimodal_lerobot_test.py`, `contiguous_window_dataset_test.py`, `multimodal_video_test.py`, `video_keyframe_index_test.py` and `video_decode_parallelism_test.py`. This includes existing H.265, fragmented MP4, bounded-cache, large-index, backend fallback and parallel-decoding checks. - Changed-file flake8 and `git diff --check` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
