laskoviymishka commented on code in PR #3057:
URL: https://github.com/apache/iceberg-rust/pull/3057#discussion_r4490455410


##########
crates/iceberg/src/spec/table_metadata_builder.rs:
##########
@@ -192,6 +193,86 @@ impl TableMetadataBuilder {
         )
     }
 
+    /// Replaces the table's schema, partition spec and sort order, leaving it 
with no
+    /// current snapshot. Backs `REPLACE TABLE`.
+    ///
+    /// The table keeps its uuid, properties and snapshots, so earlier 
snapshots stay readable.
+    /// Columns the table already has keep their field ids, so data written 
before the replacement
+    /// stays readable through the columns both schemas share; new columns get 
ids above
+    /// `last_column_id`, never a dropped column's id.
+    ///
+    /// Partition fields reuse the highest id an equivalent field has in any 
of the table's specs,
+    /// and other fields get ids above `last_partition_id`.
+    ///
+    /// `location` moves the table, `None` leaves it in place. `properties` 
are applied on top of
+    /// the existing ones. `format-version` upgrades the table without being 
persisted; other
+    /// reserved properties are ignored.
+    ///
+    /// # Errors
+    /// - The table is format version 1, which is not supported.
+    /// - `format-version` is invalid or would downgrade the table.
+    /// - The partition spec or sort order names a column missing from 
`schema`.
+    /// - `schema` takes the name of a partition field in one of the table's 
specs without being
+    ///   that field's source column.
+    pub fn build_replacement(

Review Comment:
   Could this be `pub(crate)` until the RTAS consumer lands? At this head only 
tests call it, and #3056 doesn't link a Transaction or catalog PR yet. The 
signature is still moving in the `format_version` thread, and once it's in 
[`public-api.txt`](https://github.com/apache/iceberg-rust/blob/9bf99975aaca1320111cfe166e583406795087fd/crates/iceberg/public-api.txt#L2977)
 the stability policy keeps it. #3066 landed `assign_fresh_ids` the same way, 
with `#[allow(dead_code)]` and a TODO pointing at #3056.



##########
crates/iceberg/src/spec/table_metadata_builder.rs:
##########
@@ -1235,6 +1316,32 @@ impl TableMetadataBuilder {
             .collect()
     }
 
+    /// Gives each field of `spec` the highest id an equivalent field has in 
the table's specs, or
+    /// the next id above `last_partition_id`.
+    fn reassign_partition_field_ids(&self, spec: PartitionSpec) -> 
Result<UnboundPartitionSpec> {
+        let mut field_ids = Self::highest_equivalent_field_ids(
+            self.metadata
+                .partition_specs
+                .values()
+                .flat_map(|spec| spec.fields()),
+        );
+        let mut last_partition_id = self.metadata.last_partition_id;
+
+        let fields = spec.fields().iter().map(|field| {
+            let field_id = *field_ids
+                .entry((vec![field.source_id], field.transform))
+                .or_insert_with(|| {
+                    last_partition_id += 1;

Review Comment:
   Could this hand id-less fields to `add_default_partition_spec` and drop the 
second copy of the id rule? As I read it, 
[`add_partition_spec`](https://github.com/apache/iceberg-rust/blob/9bf99975aaca1320111cfe166e583406795087fd/crates/iceberg/src/spec/table_metadata_builder.rs#L877-L888)
 reuses through the same `highest_equivalent_field_ids` over all specs and 
allocates above `last_partition_id`, so the ids should match. The copies 
already differ at the edge: `+= 1` here panics in debug at `i32::MAX`, while 
[`set_field_ids`](https://github.com/apache/iceberg-rust/blob/9bf99975aaca1320111cfe166e583406795087fd/crates/iceberg/src/spec/partition.rs#L694-L710)
 returns an error.



##########
crates/iceberg/src/spec/table_metadata_builder.rs:
##########
@@ -192,6 +193,86 @@ impl TableMetadataBuilder {
         )
     }
 
+    /// Replaces the table's schema, partition spec and sort order, leaving it 
with no
+    /// current snapshot. Backs `REPLACE TABLE`.
+    ///
+    /// The table keeps its uuid, properties and snapshots, so earlier 
snapshots stay readable.
+    /// Columns the table already has keep their field ids, so data written 
before the replacement
+    /// stays readable through the columns both schemas share; new columns get 
ids above
+    /// `last_column_id`, never a dropped column's id.

Review Comment:
   Should this say ids are matched by full name only? 
[`resolve_or_assign_id`](https://github.com/apache/iceberg-rust/blob/9bf99975aaca1320111cfe166e583406795087fd/crates/iceberg/src/spec/schema/assign_fresh_ids.rs#L80-L102)
 doesn't look at the type, so a replacement that turns `x: long` into `x: 
string` keeps `x`'s id and the readability promise doesn't hold for it. Java's 
[`AssignFreshIds`](https://github.com/apache/iceberg/blob/428fe57eb9f8cff72a46390da8a9d35bc819308e/api/src/main/java/org/apache/iceberg/types/AssignFreshIds.java#L51-L60)
 matches by name too, so this is the doc, not the code.
   
   ```suggestion
       /// Columns are matched to the current schema by full name and keep 
their field ids, even
       /// when their type changes; new columns get ids above `last_column_id`, 
never a dropped
       /// column's id.
   ```



##########
crates/iceberg/src/spec/table_metadata_builder.rs:
##########
@@ -192,6 +193,86 @@ impl TableMetadataBuilder {
         )
     }
 
+    /// Replaces the table's schema, partition spec and sort order, leaving it 
with no
+    /// current snapshot. Backs `REPLACE TABLE`.
+    ///
+    /// The table keeps its uuid, properties and snapshots, so earlier 
snapshots stay readable.
+    /// Columns the table already has keep their field ids, so data written 
before the replacement
+    /// stays readable through the columns both schemas share; new columns get 
ids above
+    /// `last_column_id`, never a dropped column's id.
+    ///
+    /// Partition fields reuse the highest id an equivalent field has in any 
of the table's specs,
+    /// and other fields get ids above `last_partition_id`.
+    ///
+    /// `location` moves the table, `None` leaves it in place. `properties` 
are applied on top of
+    /// the existing ones. `format-version` upgrades the table without being 
persisted; other
+    /// reserved properties are ignored.
+    ///
+    /// # Errors
+    /// - The table is format version 1, which is not supported.
+    /// - `format-version` is invalid or would downgrade the table.
+    /// - The partition spec or sort order names a column missing from 
`schema`.
+    /// - `schema` takes the name of a partition field in one of the table's 
specs without being
+    ///   that field's source column.
+    pub fn build_replacement(
+        self,
+        schema: Schema,
+        spec: impl Into<UnboundPartitionSpec>,
+        sort_order: SortOrder,
+        location: Option<String>,
+        properties: HashMap<String, String>,
+    ) -> Result<Self> {
+        if self.metadata.format_version == FormatVersion::V1 {
+            return Err(Error::new(
+                ErrorKind::FeatureUnsupported,
+                "Table replacement is not supported for format version 1",
+            ));
+        }
+
+        let format_version = properties
+            .get(TableProperties::PROPERTY_FORMAT_VERSION)
+            .map(|value| {
+                serde_json::from_str::<FormatVersion>(value).map_err(|err| {
+                    invalid_data!("Unsupported format version: 
{value}").with_source(err)
+                })
+            })
+            .transpose()?
+            .unwrap_or(self.metadata.format_version);
+        let properties = properties
+            .into_iter()
+            .filter(|(key, _)| 
!TableProperties::RESERVED_PROPERTIES.contains(&key.as_str()))
+            .collect();
+        let spec = spec.into();
+
+        let previous_id_to_name = schema.field_id_to_name_map().clone();
+        let fresh_schema = assign_fresh_ids(
+            schema,
+            self.get_current_schema()?,
+            self.metadata.last_column_id,
+        )?;
+
+        let (fresh_spec, fresh_sort_order) =
+            Self::rebind_to_schema(&previous_id_to_name, &fresh_schema, spec, 
sort_order)?;
+        let fresh_spec = self.reassign_partition_field_ids(fresh_spec)?;
+
+        let mut builder = self.upgrade_format_version(format_version)?;
+        if builder.metadata.refs.contains_key(MAIN_BRANCH) {

Review Comment:
   Could we add a test where the table has a second branch or tag, and say in 
the rustdoc that refs other than `main` are kept? Only `main` is removed here, 
as in Java's 
[`buildReplacement`](https://github.com/apache/iceberg/blob/428fe57eb9f8cff72a46390da8a9d35bc819308e/core/src/main/java/org/apache/iceberg/TableMetadata.java#L742-L743),
 but 
[`populated_builder`](https://github.com/apache/iceberg-rust/blob/9bf99975aaca1320111cfe166e583406795087fd/crates/iceberg/src/spec/table_metadata_builder.rs#L4206-L4234)
 sets only `MAIN_BRANCH`, so nothing pins the rest.



##########
crates/iceberg/src/spec/table_metadata_builder.rs:
##########
@@ -192,6 +193,86 @@ impl TableMetadataBuilder {
         )
     }
 
+    /// Replaces the table's schema, partition spec and sort order, leaving it 
with no
+    /// current snapshot. Backs `REPLACE TABLE`.
+    ///
+    /// The table keeps its uuid, properties and snapshots, so earlier 
snapshots stay readable.
+    /// Columns the table already has keep their field ids, so data written 
before the replacement
+    /// stays readable through the columns both schemas share; new columns get 
ids above
+    /// `last_column_id`, never a dropped column's id.
+    ///
+    /// Partition fields reuse the highest id an equivalent field has in any 
of the table's specs,
+    /// and other fields get ids above `last_partition_id`.
+    ///
+    /// `location` moves the table, `None` leaves it in place. `properties` 
are applied on top of
+    /// the existing ones. `format-version` upgrades the table without being 
persisted; other
+    /// reserved properties are ignored.
+    ///
+    /// # Errors
+    /// - The table is format version 1, which is not supported.
+    /// - `format-version` is invalid or would downgrade the table.
+    /// - The partition spec or sort order names a column missing from 
`schema`.
+    /// - `schema` takes the name of a partition field in one of the table's 
specs without being
+    ///   that field's source column.
+    pub fn build_replacement(
+        self,
+        schema: Schema,
+        spec: impl Into<UnboundPartitionSpec>,
+        sort_order: SortOrder,
+        location: Option<String>,
+        properties: HashMap<String, String>,
+    ) -> Result<Self> {
+        if self.metadata.format_version == FormatVersion::V1 {
+            return Err(Error::new(
+                ErrorKind::FeatureUnsupported,
+                "Table replacement is not supported for format version 1",
+            ));
+        }
+
+        let format_version = properties
+            .get(TableProperties::PROPERTY_FORMAT_VERSION)
+            .map(|value| {
+                serde_json::from_str::<FormatVersion>(value).map_err(|err| {
+                    invalid_data!("Unsupported format version: 
{value}").with_source(err)
+                })
+            })
+            .transpose()?
+            .unwrap_or(self.metadata.format_version);
+        let properties = properties
+            .into_iter()
+            .filter(|(key, _)| 
!TableProperties::RESERVED_PROPERTIES.contains(&key.as_str()))
+            .collect();

Review Comment:
   +1 on a typed `format_version`. Java splits the same way: the map-based 
[`buildReplacement`](https://github.com/apache/iceberg/blob/428fe57eb9f8cff72a46390da8a9d35bc819308e/core/src/main/java/org/apache/iceberg/TableMetadata.java#L709-L750)
 and `newTableMetadata(…, Map)` read `format-version` from the map and drop 
reserved keys in 
[`persistedProperties`](https://github.com/apache/iceberg/blob/428fe57eb9f8cff72a46390da8a9d35bc819308e/core/src/main/java/org/apache/iceberg/TableMetadata.java#L94-L107),
 while the internal [typed 
overload](https://github.com/apache/iceberg/blob/428fe57eb9f8cff72a46390da8a9d35bc819308e/core/src/main/java/org/apache/iceberg/TableMetadata.java#L109-L118)
 rejects them. `new` here already follows the typed side, so a typed argument 
plus `set_properties` erroring on reserved keys would line up with it.
   
   One parity question on properties: `persistedProperties` also writes 
`write.parquet.compression-codec=zstd` before the caller's keys, so a Java 
REPLACE turns a `gzip` table into `zstd`, while here `gzip` survives. Keeping 
the user's codec seems right to me, and `new` doesn't write that default 
either. Should the rustdoc say the existing codec is kept?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to