CTTY opened a new issue, #3500:
URL: https://github.com/apache/iceberg-rust/issues/3500
### Is your feature request related to a problem or challenge?
`FastAppendAction` rejects any data file whose `partition_spec_id` is not
the table's current default spec:
```rust
if metadata.default_partition_spec_id() != data_file.partition_spec_id {
return Err(invalid_data!(
"Data file partition spec id does not match table default partition
spec id"
));
}
```
This is stricter than the format allows: a snapshot may reference manifests
written under any of the table's partition specs, and iceberg-java's
`FastAppend` groups new files by spec (`newDataFilesBySpec`) and writes one
manifest per spec.
The check also makes a legitimate retry fail. A writer produces files under
spec 0, hits a commit conflict, and in the meantime another transaction changes
the default spec to spec 1. Spec 0 is still a valid spec on the table and the
files are fine, but the retry is rejected. This was raised in review of #3474
and predates that PR (the same check existed in the old `SnapshotProducer`).
### Describe the solution you'd like
In `SimpleSnapshotProducer`:
1. **Validate against the file's own spec.** Replace the default-spec
equality check with
`metadata.partition_spec_by_id(data_file.partition_spec_id)`; fail only if the
spec is unknown to the table. Validate the partition value against that spec's
partition type (`spec.partition_type(schema)`), not `default_partition_type()`.
2. **Group by spec when writing.** Bucket the added files by
`partition_spec_id` (a `BTreeMap` keeps manifest-list order deterministic) and
write one manifest per bucket under that spec. The resulting
`Vec<ManifestFile>` goes into `AddedManifests.manifests`, whose shape already
allows several manifests per append.
3. **Summary uses the file's spec.** `snapshot_helpers::build_summary`
currently passes `default_partition_spec()` for added files; use
`partition_spec_by_id(file.partition_spec_id)` as the removed-files branch
already does.
Tests: an append mixing files from two specs yields two manifests, each
under its own spec, and the duplicate-file / summary paths still work; a file
under a non-default but valid spec commits successfully.
Related: epic #3266, milestone-2 PR #3474, rolling manifests #3499.
### Willingness to contribute
I can contribute to this feature independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
