yihua opened a new pull request, #803:
URL: https://github.com/apache/hudi-rs/pull/803
## Description
Closes #802.
Two merge-on-read log block reads fail.
- **V2 Avro data blocks.** Writers before table version 6 (Hudi 0.13 and
earlier) stamp log blocks V2, and a table upgraded to version 6 keeps them
until compaction. The reader accepted only V3 (`Only support log block version
3 but got: V2`). V2 and V3 Avro data blocks share one layout (version, record
count, records), so Avro data blocks now accept V2. A V2 delete block is
Kryo-serialized, which this reader cannot decode, so it stays refused with an
error naming the block and version.
- **Avro enum columns.** arrow-avro decodes an enum as `Dictionary(Int32,
Utf8)`, while the table schema and parquet base files hold it as `Utf8`, so
merging a log update or insert on an enum column failed in reconcile. The Avro
block decoder now unpacks dictionaries at any depth, and
`AvroBlockDecoder::schema()` reports the unpacked schema to match.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
A new integration test builds a version 6 MOR table whose log block updates
and inserts rows on an enum column, at block versions V2 and V3, and reads it
through `Table::read`; both fail without the fix. A decoder unit test covers an
enum at the top level, in a nullable union, a record, an array and a map.
Content tests cover a V2 Avro block decoded, a V2 delete block refused and a V1
block refused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
