zhengguangzhuo commented on PR #10319: URL: https://github.com/apache/paimon/pull/10319#issuecomment-6121201997
Thanks for the latest validation. I updated the PR description to distinguish the maintainer-reported OSS latency measurements on 1156834035 (compared with 80c5ab9593) from the source-free metadata-call comparison on the current 0e1ac11 head. Before further code changes, could we choose and document one of these strategies? 1. **Metadata-directed reads:** Keep the canonical-layout fast path free of new metadata probes, using a reliable layout signal only if it can also represent historical and canonical files coexisting in the same partition/bucket. Please confirm whether suitable table/partition metadata already exists, and how mixed layouts and migration would be represented. 2. **Compatibility probing:** Retain canonical/historical existence probes for mixed-layout compatibility, but explicitly accept a quantified object-store cost. On 0e1ac11, the source-free call-count comparison observed 128 additional backend `get_file_info` calls for 128 already-canonical DATE-composite files, and the same count for escaped STRING files without DATE; this isolates calls, not latency. The separate OSS report on 1156834035 measured +31% and +68% median planning-plus-full-read time for those workloads. For acceptance, I propose rerunning baseline and candidate on the same frozen OSS data, with the same warm-up and three measured runs, reporting both planning-plus-full-read latency and backend metadata-call counts against an agreed budget. Could you confirm the supported Java/Python version floor and writer→reader combinations for both `partition.legacy-name` modes, including mixed-writer append and rollback; which historical Python layouts/readers are in scope (especially Java reading those layouts); and whether strategy 1 or 2 is acceptable? I will hold further implementation changes until we agree on the compatibility boundary and performance criterion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
