hsdfat opened a new pull request, #3284:
URL: https://github.com/apache/iceberg-rust/pull/3284
## Which issue does this PR close?
- Closes #2365.
## What changes are included in this PR?
The manifest Avro metadata key `partition-spec` is defined by the spec as
the JSON partition *fields array* (the spec id travels separately in
`partition-spec-id`). iceberg-rust, Java and PyIceberg all write that form, but
some writers emit the full spec object `{"spec-id": N, "fields": [...]}` — e.g.
iceberg-cpp (`src/iceberg/manifest/v2_metadata.cc`,
`ToJsonString(*partition_spec_)`). `ManifestMetadata::parse` only accepted the
array, so reading such a manifest (e.g. the parent-manifest load in
`fast_append`) failed with `invalid type: map, expected a sequence`.
This builds on #2366 by @123digits and addresses the review there:
- The reader accepts both shapes via a single-pass `#[serde(untagged)]`
enum. The array is the spec form; the object is accepted defensively.
- The object form's embedded `spec-id` is no longer ignored: if it disagrees
with `partition-spec-id`, parsing fails with `DataInvalid`. If
`partition-spec-id` is absent, the embedded id is used.
- The writer is unchanged and keeps emitting the bare fields array (Java
rejects the object form on read).
## Are these changes tested?
Yes, new unit tests in `spec/manifest/metadata.rs`:
- array form parses (spec form)
- object form parses (with and without `partition-spec-id`, with and without
embedded `spec-id`)
- object form with mismatched `spec-id` -> `DataInvalid`
- malformed payload -> `DataInvalid`
- `ManifestWriter` output: the raw `partition-spec` metadata is a JSON array
and round-trips
The two object-form tests fail on `main` with the error reported in the
issue. I also labelled `test_parse_snappy_manifest_v2` as the spec-form test.
## AI Disclosure
I used AI assistants (Claude and OpenAI models) to help write and review
this change. I reviewed the code, checked the behaviour against the spec and
the #2366 review, ran the tests myself, and take responsibility for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
