zhuxiangyi opened a new pull request, #8531: URL: https://github.com/apache/paimon/pull/8531
## Purpose This is the first refactoring PR toward moving the data-evolution read/write path from column **names** to **field ids**, as discussed in #8334 (so that top-level fields and nested fields are treated the same, per @JingsongLi's suggestion). It records the columns written in a data file by **field id** (`writtenFieldIds`) in addition to the existing name-based `writeCols`. Field ids are stable across column renames and can address nested fields uniformly (every nested field has a globally unique id), which later lets the read/write/commit/compact path drop name-based resolution. This PR is groundwork only — **behavior is unchanged**. ## Changes - **`DataFileMeta`**: append a nullable `_WRITTEN_FIELD_IDS ARRAY<INT>` to `SCHEMA`; add `writtenFieldIds()` (default `null`); thread it through `PojoDataFileMeta` and the `forAppend`/`create` factories. - **`DataFileMetaSerializer`**: serialize/deserialize the new field, `isNullAt`-guarded so old manifests read it as `null`. - **Version dispatch**: add `DataFileMetaWriteColsLegacySerializer` freezing the previous 20-field layout, and bump `DataSplit` (8→9) and `CommitMessageSerializer` (11→12) to dispatch old binary streams to it. - **Writers**: dual-write `writtenFieldIds` (derived from the `writeCols` field ids) next to `writeCols`, so old readers keep working via `writeCols`. - **`DataEvolutionUtils.writtenFieldIds(file, schemaFetcher)`**: resolve a file's written columns to ids (`writtenFieldIds` if present, else `writeCols` names → ids). Consumers switch to this in the follow-up PR. ## Compatibility - *Backward* (new reader, old data): the new field is a nullable appended field, so old manifests read it as `null`; old binary `DataSplit`/`CommitMessage` streams are dispatched to the frozen legacy serializer. - *Forward* (old reader, new data): manifests are read by name (Avro), so an old reader drops the unknown field and keeps working via the dual-written `writeCols`. `writeCols` will only be dropped later, once all supported versions understand `writtenFieldIds`. ## Tests `DataFileMetaWrittenFieldIdsCompatibilityTest`: round-trip with/without ids, the frozen legacy layout still deserializes, new-stream/old-serializer forward read, and name→id resolution (incl. row-tracking system fields). Existing `DataSplitCompatibleTest` (v1–v8 fixtures), `DataEvolutionTableTest` and serializer tests remain green. Part of the plan in #8334. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
