zhengguangzhuo commented on PR #10319: URL: https://github.com/apache/paimon/pull/10319#issuecomment-6075314761
Thanks for the detailed cross-version results and OSS measurements. I agree this PR is not ready to merge, and I will hold further code changes until we align on the compatibility scope and performance trade-off. On the current head (`0e1ac11`), CI run [37882502997](https://github.com/apache/paimon/actions/runs/37882502997) completed successfully. All Python jobs passed; Native CI reported 9,586 passed and 112 skipped. The standalone Java job was skipped. However, the Python 3.10 job did run the mixed Java-Python suite: the Java composite DATE write, Python composite DATE read and append, and Java read-after-write/rollback tests passed for both partition naming modes. This validates those current-version combinations, but not the full historical-client matrix or every historical Python layout. I appreciate your reported 256-table cross-version mixed-writer matrix; I have not independently reproduced it. The historical-layout test currently in this PR constructs the old directory layout synthetically rather than running each supported historical client version. The support boundary for Java readers of historical Python layouts also remains unresolved; your report indicates that some of those combinations are not currently supported. I also need to correct the performance statement in the PR description. The current change adds canonical-path `exists_batch` checks when the canonical and legacy bucket paths differ, followed by a historical-path lookup when the canonical candidates are absent. `exists_batch` batches paths at the FileIO API, but that alone does not establish the number or cost of metadata requests made by an object-store backend. I have not independently reproduced your OSS benchmark, and local-filesystem timings are not comparable evidence. I will not use them to dismiss the reported regressions of +31% and +68%. Before I make further code changes, could we agree on: 1. The exact Java/Python versions and writer-to-reader combinations in scope, for both `partition.legacy-name` modes, including mixed-writer append and rollback scenarios—and which combinations are explicitly out of scope. 2. Which historical Python layouts must be readable, and by which readers, especially whether Java reading those layouts is a requirement. 3. An acceptable object-store metadata-lookup/latency budget and a reproducible benchmark setup. In particular, should reads of an already-canonical layout incur any existence probes? Once we agree on these boundaries and the performance criterion, I can update the compatibility tests and performance description to match. I will also keep new comments and docstrings in English, consistent with the surrounding code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
