XiaoHongbo-Hope opened a new pull request, #10905: URL: https://github.com/apache/paimon/pull/10905
### Purpose Reduce video-index storage and seek I/O while keeping large indexes readable with bounded memory. - Add a codec and page directory to index v1. Pages support `zlib` or no compression; Java and Python validate both. - Default newly generated indexes to `video.index.compression=zlib`; `none` disables compression. Copied indexes retain their encoding, which readers detect automatically. - Load small indexes once into compact memory; search large indexes through the page directory with bounded caches. Check page endpoints without fetching neighboring data pages. Three-round index-only OSS measurements: 65,536 keyframes took about 80 ms versus 97 ms for the old whole-index compression; one million keyframes took 3.3 s versus 10.2 s for current paged reads. These are index microbenchmarks, not training throughput. ### Tests - Python: 49 passed, 62 subtests, including real PyAV decoding, corruption checks, mixed codecs and rewrite. - Java: 23 passed, including shared compressed/uncompressed `.video` fixtures and bounded decompression. - Maven checks, flake8, Python 3.6 syntax and `git diff --check` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
