XiaoHongbo-Hope opened a new pull request, #10794: URL: https://github.com/apache/paimon/pull/10794
### Purpose Reduce repeated video seek-index parsing during random frame reads. Deserialize bounded entries with NumPy comparisons and construct Python tuples once, instead of validating and normalizing each entry twice in Python. The wire format, tuple API and count limits are unchanged. Decompressed entry bytes are capped at 1.5 MiB plus one overflow-detection byte; malformed ordering, payload bounds, truncation and trailing data are still rejected. Writer-side `validate()` remains streaming. Performance evidence: | Measurement | Before | After | | --- | ---: | ---: | | This patch: deserialize a real DROID index (45,804 entries, 319 KB), median of 5 rounds × 100 calls | 28.0 ms | 10.1 ms | | Peak Python allocation for that call | 7.59 MiB | 8.42 MiB | | Earlier parser-only prototype: full local DROID DataLoader throughput | 52.1 samples/s | 75.4 samples/s | The DataLoader result used 8 workers, batch 32, three cameras, 8-frame windows, 10 warmup batches and 100 timed batches (3,200 samples), with matching sampled tensors and sample order. It used the experimental ARRAY/seek-index setup; it is motivation, not a full-training measurement of this PR head. No remote-storage throughput claim. ### Tests - 65 video index/format/read/reassignment/parallelism tests and 20 LeRobot video tests passed. - Added parity checks against streaming validation, signed 64-bit boundaries, metadata merging, maximum entry count and bounded decompression of malformed input. - Python 3.6 compilation, flake8 and `git diff --check` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
