szehon-ho commented on code in PR #3662:
URL: https://github.com/apache/iceberg-rust/pull/3662#discussion_r4520160548
##########
crates/iceberg/src/transaction/snapshot/simple.rs:
##########
@@ -128,24 +126,24 @@ impl SimpleSnapshotProducer {
.await
}
- /// Checks that every file is written under the table's default partition
- /// spec with a compatible partition value.
+ /// Checks that every file uses a known partition spec with a compatible
+ /// partition value.
///
/// Runs against the table the manifests are written for; a later rebase
/// does not invalidate manifests already written under a spec the table
/// still has.
fn validate_added_data_files(&self, table: &Table) -> Result<()> {
let metadata = table.metadata();
for data_file in &self.intent.data_files {
- // TODO(#3500): accept any spec known to the table and group the
- // files by spec when writing manifests, instead of requiring the
- // default spec.
- if metadata.default_partition_spec_id() !=
data_file.partition_spec_id {
- return Err(invalid_data!(
- "Data file partition spec id does not match table default
partition spec id"
- ));
- }
- validate_partition_value(data_file.partition(),
metadata.default_partition_type())?;
+ let spec = metadata
+ .partition_spec_by_id(data_file.partition_spec_id)
+ .ok_or_else(|| {
+ invalid_data!("Unknown partition spec id {}",
data_file.partition_spec_id)
+ })?;
+ validate_partition_value(
+ data_file.partition(),
+ &spec.partition_type(metadata.current_schema())?,
Review Comment:
The rejection described is real, but two points on scope:
- This is not a regression: before this PR, appending under a nondefault
spec was rejected altogether. This is an additional edge case in the
retained-spec support.
- The proposed historical-schema lookup would go beyond directly mirroring
Java FastAppend. Java [rebinds retained specs to the current
schema](https://github.com/apache/iceberg/blob/52fc764127402ade8170b1b53ddd95fde13cb75c/core/src/main/java/org/apache/iceberg/TableMetadata.java#L1105-L1109),
and [uses `Unknown` when a source column has been
dropped](https://github.com/apache/iceberg/blob/52fc764127402ade8170b1b53ddd95fde13cb75c/api/src/main/java/org/apache/iceberg/PartitionSpec.java#L200-L208).
Resolving an older schema would follow the existing Rust removed-file summary
path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
