JingsongLi commented on PR #10319: URL: https://github.com/apache/paimon/pull/10319#issuecomment-5934382634
Confirmed the composite-partition read-after-write issue at `7557c7e144` with an actual table write, commit and read. The DATE compatibility problem in #10034 is worth fixing, but this version should be blocked from production. **[P1] Make the writer and reader agree on the new composite partition layout** `pypaimon/utils/file_store_path_factory.py:247-273` converts only DATE components, creating a third layout that neither reader path resolves. With default `partition.legacy-name=true`, `day=1970-01-02` and `region=a/b`: ```text New writer: day=1/region=a/b/bucket-0/data-....parquet Historical reader path: day=1970-01-02/region=a/b/bucket-0/data-....parquet Canonical reader path: day=1/region=a%2Fb/bucket-0/data-....parquet ``` The commit succeeds, then readback raises `FileNotFoundError`. An in-process control retaining the previous writer path reads the same row successfully; `partition.legacy-name=false` also reads back successfully. This confirms a newly introduced failure rather than an existing unreadable table. I ran the added DATE tests plus file-store-commit and Mosaic writer regressions: 39 tests and 2 subtests passed; changed files passed Flake8. The current composite test asserts only the path and misses the failing readback. Please make planning resolve the actual new layout, or use a fully canonical layout with historical compatibility, and add write → commit → read assertions for composite DATE partitions with values requiring escaping. Preserve historical/new-file coexistence while doing so. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
