RussellSpitzer opened a new pull request, #18935: URL: https://github.com/apache/iceberg/pull/18935
Part 2 of #17919, following #18252. A file column's six nested fields take the IDs immediately after its own, so the type must stay in step with the field that holds it and those IDs must never go to another column. This PR enforces that in schema construction and evolution. `UpdateSchema.addColumn` now gives a file column, at any depth, a base ID that matches its field and reserves the derived block so later columns cannot collide. Replacing a table reserves the block when a column becomes a file. Committing file columns still needs schema JSON serialization, which comes in Part 3. **API changes** (no new public methods) - `Schema` rejects a file column whose type's base ID differs from its field ID. A column that takes a reserved ID is rejected by the existing duplicate-ID check. - `CheckCompatibility` accepts reading a file as a file and reports any other type as incompatible. **Core changes** - `UpdateSchema` rejects renaming, deleting, promoting, or changing the requirement, doc, or default of a file's nested fields. - `MetricsConfig` and `SchemaOrder` visit a file like a struct. Table metadata creation reaches both visitors after #18269 and #18331. ### Series - Part 1 (merged): #18252 - Part 2: this PR The following parts are draft branches in my fork. They are not open PRs and will change as this one is reviewed. Each link shows only that part's diff against the previous part. - Part 3: [Core: Serialize the file type in schema JSON](https://github.com/RussellSpitzer/iceberg/compare/file-type-2-id-block...file-type-3-schema-json) - Part 4: [API, Core, Data: Handle the file type in type switches and struct views](https://github.com/RussellSpitzer/iceberg/compare/file-type-3-schema-json...file-type-4-type-switches) - Part 5: [Core: Read and write file columns in Avro](https://github.com/RussellSpitzer/iceberg/compare/file-type-4-type-switches...file-type-5-avro) - Part 6: [Parquet: Map the file type to a group](https://github.com/RussellSpitzer/iceberg/compare/file-type-5-avro...file-type-6-parquet) - Part 7: [ORC: Support the file type in schema conversion and visitors](https://github.com/RussellSpitzer/iceberg/compare/file-type-6-parquet...file-type-7-orc) - Part 8: [Hive, Kafka Connect, AWS, Arrow: Render a file column as a struct](https://github.com/RussellSpitzer/iceberg/compare/file-type-7-orc...file-type-8-peripheral) --- **AI Disclosure** - Model: Claude Opus 5.5 - Platform/Tool: Cursor - Human Oversight: Reviewed and Guided - Prompt Summary: Split the file type implementation into a reviewable series; this part reserves the derived field ID block and guards schema evolution for file columns. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
