ravjot28 commented on PR #19560:
URL: https://github.com/apache/hudi/pull/19560#issuecomment-6785269939
We hit the same `ArrayIndexOutOfBoundsException` in
`HoodieMetadataPayload#getInsertValue` from a different trigger: a schema that
equals `HoodieMetadataRecord.getClassSchema()` but is a different instance.
The metadata write config carries the schema as a string.
`HoodieWriteHandle` parses it and interns it through `HoodieSchemaCache`, so
whether `HoodieAppendHandle` ends up passing the class schema instance depends
on whether `HoodieMetadataPayload`'s static initializer interned its own
wrapper first, and whether that entry is still in the cache. On a two-executor
Spark job on 1.2.1, this failed on an executor in 1 of 14 runs. On current
master it reproduces at unit level:
```java
HoodieMetadataPayload payload =
HoodieMetadataPayload.createPartitionFilesRecord(
"2022/10/01", Collections.singletonMap("file.parquet", 10L),
Collections.emptyList()).getData();
Schema equalCopy = new
Schema.Parser().parse(HoodieMetadataRecord.getClassSchema().toString());
payload.getInsertValue(equalCopy); // ArrayIndexOutOfBoundsException: Index
7 out of bounds for length 7
```
The name-and-namespace check in this PR covers that case too, so I won't
open a competing PR. Two points for the rebase:
- #19809 has since added
`TestHoodieMetadataPayload#testInsertValueFastPathOnlyForClassSchema`. It
asserts that an equal copy throws, and that the class schema with the Hudi meta
fields gets a `GenericData.Record`. The equal-copy assertion has to flip with
this change.
- `HoodieSchemaUtils.addMetadataFields` keeps the record name and namespace,
so the schema with meta fields would also take the fast path here and get a
record without them. That breaks the second assertion; the bot review on this
PR pointed at the same thing. Keeping the meta-field schema on the other path
would look something like:
```java
if (schema == null || schema == HOODIE_METADATA_AVRO_SCHEMA
|| (isHoodieMetadataRecordSchema(schema) &&
schema.getField(HoodieRecord.COMMIT_TIME_METADATA_FIELD) == null)) {
```
If it helps, I can send a PR with this condition and an equal-copy test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]