yihua opened a new pull request, #20269: URL: https://github.com/apache/hudi/pull/20269
### Describe the issue this Pull Request addresses closes #20268 Merging a partial update into a full record built the merged schema by converting the Spark struct type back to a `HoodieSchema`, which drops field defaults (for example the `-1` default of the position-merging row index column). That schema is unequal to the reader schema but has the same Avro hash, so every schema-keyed lookup on the merge path (`LocalHoodieSchemaCache`, `HoodieInternalRowUtils.getCachedSchema`, the merged-schema cache, `isPartial`) compared the two field by field once per merged record. #12949 removed this class of cost by interning schemas; interning does not help here because the two schemas are not equal. ### Summary and Changelog When the merged fields are all the reader fields, `SparkRecordMergingUtils.getCachedMergedSchema` now returns the reader schema itself; partial-only merges are unchanged. `HoodieSchema.hashCode` delegates to the Avro schema's cached hash instead of `Objects.hash`. New `TestSparkRecordMergingUtils` covers the full-plus-partial, full-update and partial-plus-partial merges (two cases fail on master). ### Impact Lower CPU and allocation when reading or compacting MOR file groups with partial-update log records on Spark. Single-thread merge benchmark (31-field reader schema, 2-column partial records): 2367 to 486 ns and 13.8 KB to 0.7 KB allocated per merged record. Merged values are unchanged. ### Risk Level low. A full newer record is now returned as is instead of copied, so in event-time ordering `deltaMerge` returns empty (keep the buffered record) when the buffered full record wins; `TestBufferedRecordMerger` is updated for that. The kept record has the same values, ordering value and delete flag, and keeps its own operation and key. `TestPartialUpdateForMergeInto` passes. ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
