mattfaltyn opened a new issue, #1987:
URL: https://github.com/apache/iceberg-go/issues/1987
### Apache Iceberg version
main (development), `aa76a28c34787f23f8eee1c5648271fc0ee6042f`
### Please describe the bug 🐞
`ParseMetadataBytes` accepts v2 metadata whose `last-partition-id` is lower
than an explicitly assigned field ID in partition-spec history. A subsequent
`UpdateSpec` then allocates that historical ID to a different transform.
The metadata contract says `LastPartitionSpecID` is the highest assigned
partition field ID across all specs, so this creates an invalid cross-spec ID
collision.
Minimal metadata shape:
```json
{
"format-version": 2,
"partition-specs": [
{"spec-id": 0, "fields": [
{"source-id": 1, "field-id": 1000, "name": "id_identity", "transform":
"identity"}
]},
{"spec-id": 1, "fields": []}
],
"default-spec-id": 1,
"last-partition-id": 999
}
```
After parsing otherwise-valid table metadata with that shape, adding
`bucket[16](id)` produces:
```text
historical=id_identity:1000/identity fresh=id_bucket:1000/bucket[16]
reused=true
```
Repeating the same operation is deterministic. A control with
`"last-partition-id": 1000` correctly allocates `id_bucket:1001`.
Expected behavior: metadata preflight should normalize the counter to at
least the greatest of `999` and every explicit partition field ID, including
when no field IDs are missing. Alternatively, parsing should reject the
inconsistent metadata. It must not leave the stale counter available to
`UpdateSpec`.
Root cause: `assignMissingPartitionFieldIDsFromMetadata` scans explicit IDs
and computes the maximum, but returns the original JSON whenever
`missingFields` is empty. `commonMetadata.validate` only checks that
`last-partition-id` exists, and `UpdateSpec` trusts the stale value.
Relevant prior work does not cover this case:
- #1514 assigns omitted field IDs, but the all-IDs-present early return
preserves a stale counter.
- #1641 reuses an ID when re-adding the same historical source/transform;
this report concerns allocating that ID to a different transform.
- #1651 concerns staged transaction metadata.
- #1834 concerns concurrent spec-update fencing.
- #1981 was a closed performance draft and did not add the invariant.
Suggested regression coverage: parse metadata with explicit field ID `1000`
and `last-partition-id` `999`, then assert the parsed counter is `1000` and the
next distinct partition field receives `1001`.
### Contribution
I will submit a focused PR with the parser normalization and regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
