mbutrovich commented on code in PR #3057:
URL: https://github.com/apache/iceberg-rust/pull/3057#discussion_r4517088595


##########
crates/iceberg/src/spec/table_metadata_builder.rs:
##########
@@ -192,6 +193,86 @@ impl TableMetadataBuilder {
         )
     }
 
+    /// Replaces the table's schema, partition spec and sort order, leaving it 
with no
+    /// current snapshot. Backs `REPLACE TABLE`.
+    ///
+    /// The table keeps its uuid, properties and snapshots, so earlier 
snapshots stay readable.
+    /// Columns the table already has keep their field ids, so data written 
before the replacement
+    /// stays readable through the columns both schemas share; new columns get 
ids above
+    /// `last_column_id`, never a dropped column's id.
+    ///
+    /// Partition fields reuse the highest id an equivalent field has in any 
of the table's specs,
+    /// and other fields get ids above `last_partition_id`.
+    ///
+    /// `location` moves the table, `None` leaves it in place. `properties` 
are applied on top of
+    /// the existing ones. `format-version` upgrades the table without being 
persisted; other
+    /// reserved properties are ignored.
+    ///
+    /// # Errors
+    /// - The table is format version 1, which is not supported.
+    /// - `format-version` is invalid or would downgrade the table.
+    /// - The partition spec or sort order names a column missing from 
`schema`.
+    /// - `schema` takes the name of a partition field in one of the table's 
specs without being
+    ///   that field's source column.
+    pub fn build_replacement(

Review Comment:
   > @mbutrovich asked for a rust doc example, would you prefer to have this 
pub(crate) and drop that for now and add it later or keep it pub and have the 
example?
   
   I'd go with `pub(crate)` for now. Nothing outside the tests calls it yet, 
and the RTAS consumer may want a different signature. Once it ships as `pub`, 
changing it is a breaking change. Marking it `#[allow(dead_code)]` with a TODO 
pointing at #3056, as #3066 did for `assign_fresh_ids`, builds without 
warnings. `make generate-public-api` then drops it from `public-api.txt`.
   
   A doctest compiles as an external crate, so it can't call a `pub(crate)` 
function. Could you move the example's flow into a unit test? The doctest is 
the only place that adds a snapshot and sets `main` after `build_replacement`, 
which is how the RTAS commit will use it. The doctest can come back in the PR 
that makes it `pub`.



##########
crates/iceberg/src/spec/table_metadata_builder.rs:
##########
@@ -1235,6 +1316,32 @@ impl TableMetadataBuilder {
             .collect()
     }
 
+    /// Gives each field of `spec` the highest id an equivalent field has in 
the table's specs, or
+    /// the next id above `last_partition_id`.
+    fn reassign_partition_field_ids(&self, spec: PartitionSpec) -> 
Result<UnboundPartitionSpec> {
+        let mut field_ids = Self::highest_equivalent_field_ids(
+            self.metadata
+                .partition_specs
+                .values()
+                .flat_map(|spec| spec.fields()),
+        );
+        let mut last_partition_id = self.metadata.last_partition_id;
+
+        let fields = spec.fields().iter().map(|field| {
+            let field_id = *field_ids
+                .entry((vec![field.source_id], field.transform))
+                .or_insert_with(|| {
+                    last_partition_id += 1;

Review Comment:
   > The copies already differ at the edge: `+= 1` here panics in debug at 
`i32::MAX`, while 
[`set_field_ids`](https://github.com/apache/iceberg-rust/blob/9bf99975aaca1320111cfe166e583406795087fd/crates/iceberg/src/spec/partition.rs#L694-L710)
 returns an error.
   
   Keeping the assignment here makes sense given how `AddSpec` records ids. The 
overflow is still there at this head, though. With `last_partition_id` set to 
`i32::MAX` and a replacement spec that adds `bucket(8, z)`, `build_replacement` 
panics with `attempt to add with overflow` at [line 
1408](https://github.com/apache/iceberg-rust/blob/66baae31a6e46cf0016c6cbcb5808c357d7e11dd/crates/iceberg/src/spec/table_metadata_builder.rs#L1404-L1412)
 in a test build. A release build without overflow checks wraps to `i32::MIN` 
and gives the new field that id. The spec assigns new ids [based on the last 
assigned partition 
ID](https://github.com/apache/iceberg/blob/52fc764127402ade8170b1b53ddd95fde13cb75c/format/spec.md?plain=1#L707)
 so that [partition fields are always assigned an unused 
ID](https://github.com/apache/iceberg/blob/52fc764127402ade8170b1b53ddd95fde13cb75c/format/spec.md?plain=1#L1501),
 and a wrapped id has no such guarantee.
   
   Could this return an error when no id is left, with a test that sets 
`last_partition_id` to `i32::MAX`? One option also drops the second copy of the 
id rule. `reuse_partition_field_ids` could fill in only the reused ids, then 
`PartitionSpecBuilder::new_from_unbound(...).with_last_assigned_field_id(self.metadata.last_partition_id)`
 assigns the rest. That goes through `set_field_ids`, which already returns 
`Cannot assign more partition ids. Overflow.`. Calling `into_unbound()` on the 
bound spec keeps every id, so `AddSpec` still carries them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to