XiaoHongbo-Hope opened a new pull request, #10891: URL: https://github.com/apache/paimon/pull/10891
### Purpose Videos with more than 65,536 keyframes currently lose their generated seek index and fall back to scanning. Raising that limit alone would expand Python tuple/int memory. Keep keyframes as packed int64 triples with sequence access, and use compact packet mappings and decoder anchors. Java and Python now allow up to 16 MiB of decoded entries (699,050 keyframes). Existing compressed-block/file limits and validation remain; generation beyond the budget logs a warning before the existing no-index fallback. The wire layout is unchanged. Readers enforcing the old count limit still reject larger indexes, so both readers and writers must be upgraded. ### Tests - Real 65,537-keyframe H.264 MP4: index generation/round-trip and indexed first/middle/last frame pixels match full decoding. - Python: 65 related tests + 6 reader tests passed. Java: 16 video-format tests passed. Flake8, syntax and diff checks passed. - Allocation microbenchmark at 65,536 entries, including decoder anchors: retained memory **11.0 → 2.0 MiB**; 500,000 entries retain **15.5 MiB**. This measures index memory, not training throughput. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
