zhang-arvin commented on PR #9497:
URL: https://github.com/apache/paimon/pull/9497#issuecomment-5573549322
Thanks for the review @JingsongLi! I've addressed all three P1 comments:
1. **content header** (C3): `IcebergManifestFile` now builds per-writer
metadata from the `Content` being written — `create()` iterates
`Content.values()`, puts `content = data`/`deletes` into each writer's options,
and `createWriter()` picks the factory by `Content`. Both `Content.DATA` and
`Content.DELETES` manifests get the header.
2. **partition-spec** (C2): the manifest `partition-spec` metadata is now
the JSON array of partition fields via `JsonSerdeUtil.toJson(partitionFields)`
(equivalent to Iceberg's `PartitionSpecParser.toJsonFields`).
`partition-spec-id` remains a separate key. No more `{"spec-id":0,"fields":[]}`
object.
3. **schema field IDs** (C1): added `withPositiveFieldIds` which rebuilds
the schema with top-level field IDs assigned from 1 (preserving
name/required/type/dataType/doc), so manifests no longer carry `"id": 0`.
Being transparent on the remaining follow-ups, which I'll continue working
on in this PR:
- Nested field IDs inside struct/map/list types are not yet remapped to
positive, monotonically-increasing IDs.
- Metrics maps (lower/upper-bound keyed by field ID) still reference the
original Paimon column IDs.
- The P2 one-time rewrite/migration of already-committed legacy headerless
manifests (C4) plus the corresponding upgrade tests.
I'll post updates here as these land. Feedback welcome in the meantime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
