JingsongLi commented on PR #10905: URL: https://github.com/apache/paimon/pull/10905#issuecomment-6556946597
Requirement fit: SUPPORTED. The page directory is connected to actual indexed seeking and provides a credible reduction in remote index I/O. I found no introduced correctness defect in the codec, directory bounds/CRC checks, Java/Python format handling, or mixed-version byte-preserving rewrites. There is a production rollout condition that should be explicit in the video documentation: upgrade every video reader and rewrite/compaction worker before producing v2 indexes. I tested the exact `release-2.1.0` implementation against bytes produced by this head: the new reader reads released v1 successfully, but the released reader rejects both `compression=none` and `delta-zlib` output with `Invalid video keyframe index version or magic`. Both options write version 2; `none` is not a legacy-format switch. The old collator propagates that error, and released Java rewrite validation also rejects v2. Once v2 is written, rolling back those workers to 2.1.0 is not safe without avoiding v2 data or providing a legacy/no-index output path. The intermediate unreleased paged-v1 baseline was already incompatible, so I am identifying a release/rollback condition, not attributing that entire incompatibility to this PR. The current Native CI job also needs a green result or a demonstrated same-baseline/dependency explanation before treating the head as production-ready: https://github.com/apache/paimon/actions/runs/41286128293/job/124788039362. It reports 335 failures across vector/index/search suites with `Unknown vindex option(s)` errors. Those paths are outside this video diff, so I have not classified them as a video regression. Local verification on this exact head: 21 Java format tests with normal JDK 8 Maven `verify`; 2 real Core/Flink integration tests for video rolling and Data Evolution BLOB updates with normal `flink1` Maven packaging; and 68 Python video/index/format/collator/table cases, all passing with no skips after installing PyAV/Torch. The Python run includes a real 65,537-keyframe MP4 with sparse pixel-checked reads, both compression options through table writes, mixed-version rewrites and fallback/corruption cases. Configured Flake8 on all changed Python files and `git diff --check` pass. I did not rerun the OSS or DROID benchmarks; the supplied DROID figures do not establish a training-throughput improvement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
