JingsongLi commented on PR #10894: URL: https://github.com/apache/paimon/pull/10894#issuecomment-6538233218
**[P1] Preserve nanosecond precision in the returned partition specs** The new `_canonical_partition()` call in [catalog_utils.py:101–103](https://github.com/apache/paimon/blob/895438beffeef69a2629f099e4c934682170c17b/paimon-python/pypaimon/catalog/catalog_utils.py#L101-L103) exposes a precision-loss bug in the existing formatter. With the default `partition.legacy-name=true`, its TIMESTAMP branch uses `isoformat(timespec='seconds'/'microseconds')`, which drops the nanosecond remainder of a pandas `Timestamp`. The serialized grouping keys remain distinct, but the returned specs can identify a different partition. I reproduced this on head `895438beff` using an ordinary append table partitioned by a `TIMESTAMP(9)` column, with Python writing and committing: | Partition value | Row count | | --- | ---: | | `2024-01-02 03:04:05` | 1 | | `2024-01-02 03:04:05.000000001` | 2 | | `2024-01-02 03:04:05.000000002` | 3 | All three entries from `list_partitions_paged()` return the same spec: `{'dt': '2024-01-02T03:04:05'}`. Passing the spec of the two-row partition to `drop_partitions()`: ```python partitions = catalog.list_partitions_paged('db.t').elements target = next(p for p in partitions if p.record_count == 2) catalog.drop_partitions('db.t', [target.spec]) ``` **deletes the one-row, whole-second partition and leaves the intended two-row partition untouched.** I verified the manifest entries before and after deletion. The baseline listing retains the nanoseconds, and Java's legacy `Timestamp.toString()` also retains them. Could we fix the canonical TIMESTAMP formatter to preserve nanoseconds before reusing it here, and add a regression test with whole-second, +1 ns, and +2 ns partitions that verifies deleting a listed partition removes the intended one? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
