jamespfaulkner opened a new pull request, #18913: URL: https://github.com/apache/iceberg/pull/18913
Glue stores column names lowercased ([Lake Formation](https://docs.aws.amazon.com/lake-formation/latest/dg/notes-s3-catalog.html), [Athena](https://docs.aws.amazon.com/athena/latest/ug/tables-databases-columns-names.html)), but `IcebergToGlueConverter` dedupes columns across schemas by exact name. After a case-only rename or drop-and-add (`data` → `Data`), the historical column is still emitted and Glue stores two `data` columns, breaking clients that key Glue columns by name. #16853 fixed the resulting comment-map crash but kept the duplicate to preserve historical names; once lowercased, though, the historical name is identical to the current one. This dedupes on the lowercased name so the current column wins, and matches preserved comments the same way. Two current fields differing only by case can't both exist in Glue either; only the first is written. Test plan: - A case-only drop-and-add and a case-only rename each emit only the current column. - A comment stored under Glue's lowercased name is kept for a mixed-case field. - All three fail before the change. `TestIcebergToGlueConverter`, `TestGlueCatalog`, `TestGlueToIcebergConverter`, spotless and checkstyle pass; `TestGlueCatalogTable` integration tests not run (need AWS). --- **AI Disclosure** - Model: Claude Opus 5.5 - Platform/Tool: Claude Code - Human Oversight: fully reviewed - Prompt Summary: Fix IcebergToGlueConverter emitting columns that collide after Glue lowercases names, with tests and AWS doc links. The converter change, tests and this description are generated; the root-cause diagnosis is the author's. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
