ethan-tyler opened a new issue, #1923:
URL: https://github.com/apache/iceberg-rust/issues/1923
## Summary
DELETE, UPDATE, and MERGE operations on tables that have undergone partition
evolution may fail with errors like:
```
Partition value is not compatible with partition type
```
This occurs when DML operations span data files written under different
partition specs.
## Current State
### Infrastructure in Place
The DataFusion integration has the serialization infrastructure to handle
partition evolution:
- `SerializedFileWithSpec` wrapper preserves `spec_id` during file
serialization
- `build_partition_type_map()` builds a map from spec ID to partition struct
type
- `position_delete_schema()` creates the spec-defined position delete schema
with correct field IDs
- Commit executors (`delete_commit.rs`, `update_commit.rs`,
`merge_commit.rs`) use the partition type map for deserialization
### What's Missing
The underlying iceberg-rust core doesn't fully support DML operations that
span files from multiple partition specs. The writer infrastructure needs to
handle:
1. Reading files from any partition spec (scan side)
2. Writing position deletes that reference files from any spec
3. Partition value compatibility during manifest generation
## Watchdog Tests
Three ignored integration tests serve as watchdogs for this feature:
```rust
// crates/integrations/datafusion/tests/integration_datafusion_test.rs
#[ignore] // Partition evolution DML not yet supported
async fn test_delete_with_partition_evolution() { ... }
#[ignore] // Partition evolution DML not yet supported
async fn test_update_with_partition_evolution() { ... }
#[ignore] // Partition evolution DML not yet supported
async fn test_merge_with_partition_evolution() { ... }
```
These tests:
1. Create an unpartitioned table
2. Insert initial data
3. Evolve partition spec to add partitioning
4. Insert more data under new spec
5. Attempt DML spanning both specs
6. Verify correct rows affected
## Acceptance Criteria
- [ ] DELETE works on tables with evolved partition specs
- [ ] UPDATE works on tables with evolved partition specs
- [ ] MERGE works on tables with evolved partition specs
- [ ] All three watchdog tests pass (remove `#[ignore]`)
## Related
- `EvolvePartitionAction` API added for testing partition evolution via
transactions
- MERGE currently has an explicit guard that blocks partition-evolved tables
(can be removed once supported)
## References
- Iceberg Spec: [Partition
Evolution](https://iceberg.apache.org/spec/#partition-evolution)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
