leaves12138 opened a new pull request, #9816:
URL: https://github.com/apache/paimon/pull/9816
### Purpose
Fix compatibility when reading manifests whose writer schema declares
`_FILE` as a nullable record, such as `["null", {"type": "record", ...}]`.
The specialized manifest reader currently rejects these files with
`Unexpected Manifest Avro type for field _FILE: expected RECORD but found
UNION`, although the field decoder already supports nullable unions. This
prevents reading existing manifests that the generic Avro reader can decode.
This change resolves the non-null branch only when validating the record
field. It preserves the original writer schema for reading and skipping the
union discriminator, and keeps strict validation for primitive fields that are
decoded directly. Non-nullable unions, multi-branch unions, and incorrect field
types remain rejected. No writer or on-disk format change is involved.
### Tests
- Extend manifest compatibility tests to cover nullable `_FILE` with
reordered top-level and nested fields, full and projected reads, field
skipping, filtering, and decoded raw blocks.
- Verify cold and warm manifest-cache reads and rejection of invalid field
types.
- Verify ordinary records and nullable records with either union branch
order in the Avro decoder, including skip behavior and invalid-union rejection.
- 108 tests passed across `ManifestFileTest`,
`ManifestFileMetaSerializerTest`, and `AvroFileFormatTest`.
- An existing manifest also passed local checks for uncached, cold-cache,
warm-cache, and `_FILE`-omitted projection reads.
- Spotless and Checkstyle passed for `paimon-core` and `paimon-format`.
Commands:
```shell
mvn -B -pl paimon-core -am -Pfast-build \
-DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false \
-DwildcardSuites=none \
-Dtest=ManifestFileTest,ManifestFileMetaSerializerTest,AvroFileFormatTest
package
mvn -B -pl paimon-core,paimon-format spotless:check checkstyle:check
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
