ravjot28 commented on PR #19560:
URL: https://github.com/apache/hudi/pull/19560#issuecomment-6785269939

   We hit the same `ArrayIndexOutOfBoundsException` in 
`HoodieMetadataPayload#getInsertValue` from a different trigger: a schema that 
equals `HoodieMetadataRecord.getClassSchema()` but is a different instance.
   
   The metadata write config carries the schema as a string. 
`HoodieWriteHandle` parses it and interns it through `HoodieSchemaCache`, so 
whether `HoodieAppendHandle` ends up passing the class schema instance depends 
on whether `HoodieMetadataPayload`'s static initializer interned its own 
wrapper first, and whether that entry is still in the cache. On a two-executor 
Spark job on 1.2.1, this failed on an executor in 1 of 14 runs. On current 
master it reproduces at unit level:
   
   ```java
   HoodieMetadataPayload payload = 
HoodieMetadataPayload.createPartitionFilesRecord(
       "2022/10/01", Collections.singletonMap("file.parquet", 10L), 
Collections.emptyList()).getData();
   Schema equalCopy = new 
Schema.Parser().parse(HoodieMetadataRecord.getClassSchema().toString());
   payload.getInsertValue(equalCopy); // ArrayIndexOutOfBoundsException: Index 
7 out of bounds for length 7
   ```
   
   The name-and-namespace check in this PR covers that case too, so I won't 
open a competing PR. Two points for the rebase:
   
   - #19809 has since added 
`TestHoodieMetadataPayload#testInsertValueFastPathOnlyForClassSchema`. It 
asserts that an equal copy throws, and that the class schema with the Hudi meta 
fields gets a `GenericData.Record`. The equal-copy assertion has to flip with 
this change.
   - `HoodieSchemaUtils.addMetadataFields` keeps the record name and namespace, 
so the schema with meta fields would also take the fast path here and get a 
record without them. That breaks the second assertion; the bot review on this 
PR pointed at the same thing. Keeping the meta-field schema on the other path 
would look something like:
   
   ```java
   if (schema == null || schema == HOODIE_METADATA_AVRO_SCHEMA
       || (isHoodieMetadataRecordSchema(schema) && 
schema.getField(HoodieRecord.COMMIT_TIME_METADATA_FIELD) == null)) {
   ```
   
   If it helps, I can send a PR with this condition and an equal-copy test.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to