lokeshj1703 opened a new pull request, #19560:
URL: https://github.com/apache/hudi/pull/19560
### Describe the issue this Pull Request addresses
Closes #19559
When a Hudi 1.x reader reads a Metadata Table (MDT) that was written by an
older Hudi release, merging log-file delta records against base-file records
throws an `ArrayIndexOutOfBoundsException`:
```
Caused by: java.lang.ArrayIndexOutOfBoundsException: 6
at org.apache.avro.generic.GenericData$Record.put(GenericData.java:273)
at
org.apache.hudi.metadata.HoodieMetadataPayload.getInsertValue(HoodieMetadataPayload.java:422)
at
org.apache.hudi.metadata.HoodieMetadataPayload.combineAndGetUpdateValue(HoodieMetadataPayload.java:399)
at
org.apache.hudi.common.model.HoodieAvroRecordMerger.merge(HoodieAvroRecordMerger.java:67)
at
org.apache.hudi.common.table.read.BufferedRecordMergerFactory$CustomPayloadRecordMerger.deltaMergeRecords(BufferedRecordMergerFactory.java:404)
at
org.apache.hudi.common.table.read.BufferedRecordMergerFactory$BaseCustomMerger.deltaMerge(BufferedRecordMergerFactory.java:439)
at
org.apache.hudi.common.table.read.buffer.KeyBasedFileGroupRecordBuffer.processNextDataRecord(...)
```
### Summary and Changelog
**Root cause:** `SecondaryIndexMetadata` was added to `HoodieMetadataRecord`
in a later Hudi release, growing the schema from 6 to 7 fields. MDT log blocks
written by older releases carry the 6-field writer schema in their block
header. When merging, `HoodieAvroRecordMerger` retrieves the writer schema from
the buffered record and passes it to
`HoodieMetadataPayload.getInsertValue(schema)`.
Inside `getInsertValue`, the fast path uses an object-identity check:
```java
if (schema == null || schema == HOODIE_METADATA_AVRO_SCHEMA) {
// safe path
} else {
// assumes HOODIE_META_COLUMNS are prepended; uses TYPE_FIELD_OFFSET=6
record.put(TYPE_FIELD_OFFSET, type); // index 6 on a 6-field schema →
ArrayIndexOutOfBoundsException
}
```
The old 6-field schema is a different `Schema` object from
`HOODIE_METADATA_AVRO_SCHEMA` (7 fields), so it fails the identity check and
falls into the `else` branch designed for schemas with `HOODIE_META_COLUMNS`
prepended. Writing to index 6 on a 6-field schema throws
`ArrayIndexOutOfBoundsException`.
**Fix:** Add `isHoodieMetadataRecordSchema(Schema)` that recognises any
version of `HoodieMetadataRecord` by Avro record name and namespace. Route
these older-version schemas to the same fast path as the current schema,
returning a `HoodieMetadataRecord` with the current schema
(`SecondaryIndexMetadata` defaults to `null`).
**Changes:**
- `HoodieMetadataPayload.java`: add `isHoodieMetadataRecordSchema()` helper;
extend the fast-path guard in `getInsertValue()`
- `TestHoodieMetadataPayload.java`: add
`testGetInsertValueWithOldMetadataSchema` regression test that constructs the
old 6-field schema and verifies no exception is thrown
### Impact
No user-facing API or config change. Tables written by older Hudi releases
can be read safely by a 1.x reader without requiring a table upgrade.
### Risk Level
Low. The change is confined to `getInsertValue()` in
`HoodieMetadataPayload`. The added condition only triggers when the schema is a
`HoodieMetadataRecord` schema that is not object-identical to
`HOODIE_METADATA_AVRO_SCHEMA` (i.e. an older-version schema read from a log
block); all other callers are unaffected.
### Documentation Update
None.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
