xanderbailey opened a new issue, #3411: URL: https://github.com/apache/iceberg-rust/issues/3411
#1619 added a check that a new schema field name must not collide with a partition field name. The check looks at every partition spec the table has ever had: https://github.com/apache/iceberg-rust/blob/693339072/crates/iceberg/src/spec/table_metadata.rs#L158 Iceberg keeps specs forever, so this reserves a name permanently. Partition by `bucket(x)` named `x_bucket`, later drop that partitioning, and the table can never gain an `x_bucket` column — even though nothing reads that name as a partition any more. Java has no schema-side check at all. The only name-conflict check is on the spec side, against the current schema: https://github.com/apache/iceberg/blob/4eb9cb7c3d53cb8080724eebf6e6898871ee23b0/api/src/main/java/org/apache/iceberg/PartitionSpec.java#L407-L418 And it treats old specs as free to be invalid against the current schema. On every schema change it rebuilds them with `buildUnchecked()`, with a comment saying why: https://github.com/apache/iceberg/blob/4eb9cb7c3d53cb8080724eebf6e6898871ee23b0/core/src/main/java/org/apache/iceberg/TableMetadata.java#L760-L773 So we reject schema changes Java accepts. It also blocks `REPLACE TABLE` (#3056), where the old spec is superseded by definition. Repro — fails today with `Cannot add schema field 'x_bucket' because it conflicts with existing partition field name`: 1. v2 table, schema `x: long`, partitioned by `bucket(x)` as `x_bucket` 2. set the default spec to unpartitioned 3. add a schema with columns `x: long`, `x_bucket: string` I think the fix is to scope the check to the current spec and leave the schema side as it is — a name any schema already has stays allowed, so evolution can still re-add a dropped column. @liurenjie1024 @fvaleye does that match what you intended in #1619? The review there discussed checking across all *schemas*; I don't think the spec side came up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
