wangzhigang1999 commented on PR #10319: URL: https://github.com/apache/paimon/pull/10319#issuecomment-5908109284
Thanks for working on this issue. I reported it and also attempted a fix in #10045, which I later closed because of compatibility concerns. I think this implementation still misses some of the difficulties I encountered. There is a read-after-write regression with composite partitions. With `partition.legacy-name=true`, `day=1970-01-02`, and `region=a/b`, this version writes to `day=1/region=a/b/...`, but the Python reader cannot resolve that mixed-format path and fails with `FileNotFoundError`. The added composite-partition test checks the path without verifying readback. The historical-directory test also writes with the new code and then renames the entire directory. That doesn’t cover historical and new paths coexisting in the same logical partition/bucket, or compatibility between client versions. We need a compatibility matrix covering Java and Python writers/readers, old and new versions, and both partition naming modes, with the full Cartesian product of supported combinations. This should include different writers appending to the same partition/bucket, verifying that readers return all expected rows, and checking reads after rollback. I previously tried adding compatibility handling for different path formats, but the additional lookups caused performance regressions. We haven’t settled on how to handle that tradeoff, so this change needs performance evaluation alongside the compatibility tests. Could you address these gaps before we proceed with the write-path change? Preserving reads of historical ISO directories alone isn’t enough to establish compatibility. Please also use English for the added comments and docstrings, consistent with the surrounding code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
