abrarsher23 opened a new issue, #18355:
URL: https://github.com/apache/iceberg/issues/18355
**Apache Iceberg version:** 1.11.0 (reproduced). On `main`, the class check
in `PartitionData.get(pos, javaClass)` is the same and `PartitionSummary` and
`SnapshotSummary` are unchanged; I have not run the repro on `main`.
**Query engine:** None (core API). The same failure shows up when a Flink
`DynamicIcebergSink` checkpoint spans the promotion.
**Please describe the bug 🐞**
### Summary
`UpdateSchema.updateColumn` allows `int` → `long` and `float` → `double` on
a column that is the source of an `identity` partition field (`truncate` on
`int` / `long` should behave the same; I did not test it). The partition spec
keeps its ID, and `TableMetadata` rebinds it to the new schema, so its
partition type becomes `long` (or `double`). `PartitionData.get(pos,
javaClass)` then rejects a value of the other class. As a result:
1. A data file built before the promotion, whose partition holds an
`Integer`, cannot be committed after it.
2. A data file built after the promotion cannot be added to a manifest
written with a `PartitionSpec` obtained before the promotion.
3. One commit that contains files from both sides fails in either case. A
Flink dynamic sink checkpoint that spans the promotion produces exactly that.
It surfaced while looking at the non-key partition cases of #18350.
### Steps to reproduce
```java
Schema schema = new Schema(
Types.NestedField.required(1, "id", Types.IntegerType.get()),
Types.NestedField.optional(2, "region", Types.IntegerType.get()));
PartitionSpec spec =
PartitionSpec.builderFor(schema).identity("region").build();
Table table = /* any catalog */ createTable(schema, spec);
PartitionSpec specBeforePromotion = table.spec();
PartitionData partition = new PartitionData(table.spec().partitionType());
partition.set(0, 5); // Integer, built before the promotion
DataFile before = DataFiles.builder(table.spec())
.withPath("/before.parquet")
.withFileSizeInBytes(10)
.withRecordCount(1)
.withPartition(partition)
.build();
table.updateSchema().updateColumn("region", Types.LongType.get()).commit();
table.newAppend().appendFile(before).commit(); // case 1
```
For case 2, build a file the same way after the promotion with the partition
value `5L`, and add it to a manifest created with `ManifestFiles.write(2,
specBeforePromotion, outputFile, 1L)`.
Control: the `5L` file built after the promotion, appended to the table with
the current spec, commits fine.
### Expected behavior
Both files commit. A partition value written before the promotion is
accepted for the promoted partition type, and a file with a promoted partition
value can be written to a manifest from an older bound spec of the same spec ID.
### Actual behavior
Case 1, from `SnapshotSummary.Builder.updatePartitions` →
`PartitionSpec.partitionToPath` → `PartitionData.get`:
```
java.lang.IllegalArgumentException: Wrong class, expected java.lang.Long,
but was java.lang.Integer, for object: 5
```
Case 2, from `ManifestWriter.addEntry` → `PartitionSummary.updateFields`:
```
java.lang.IllegalArgumentException: Wrong class, expected java.lang.Integer,
but was java.lang.Long, for object: 5
```
Case 3, in a Flink dynamic sink checkpoint with files from both sides: the
same two messages, depending on which spec object is used. I reproduced this
only by passing both kinds of files through the sink's manifest utilities and a
`RowDelta` in a test, not in a running job. All 5 variants I tried failed,
including 3 controls with no re-write at all.
### Notes
- Likely impact in Flink `DynamicIcebergSink`:
`DynamicWriteResultAggregator` caches `table.specs()` per table and looks specs
up by ID, which the promotion keeps. An aggregator that saw the table before
the promotion would then fail on the first post-promotion checkpoint, even with
one writer. I inferred this from reading the code; I did not run a job.
- Bucket partitions are not affected: their values are `Integer` at both
widths.
- `float` → `double` on an identity partition column fails the same way in a
sink test.
- I haven't worked out the fix. Options to discuss: make `PartitionData`
accept the narrower numeric class for a promoted type, or normalize partition
values when a file or manifest entry meets the promoted spec.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
