yihua opened a new issue, #19989: URL: https://github.com/apache/hudi/issues/19989
`HoodieRecordUtils.createHoodieRecord` decides whether to build a payload-carrying `HoodieAvroRecord` or a payload-free `HoodieAvroIndexedRecord` from `requiresPayload`, which is `isChangingRecords(op) && !HoodieWriteConfig.isFileGroupReaderBasedMergeHandle(props)`. The second half is a subclass test against `FileGroupReaderBasedMergeHandle` on the class named in `hoodie.write.merge.handle.class`. That is the wrong question in three ways: - It classifies the handle by inheritance, not by whether it consumes `HoodieRecordPayload` APIs. A payload-free handle that does not extend `FileGroupReaderBasedMergeHandle` is treated as needing a payload. - `HoodieMergeHandleFactory.getMergeHandleClassesWrite` overrides the configured class in most of its branches (sorted records, concat on duplicate inserts, CDC, which always uses the FileGroupReader handle), so the config being tested often is not the handle that runs. - The decision is made once on the driver for the whole write, while the handle is instantiated per file group and falls back at runtime on `HoodieNotSupportedException`. The observable consequence is that the same record on the same table gets a different ordering value depending on which merge handle is *configured*: `HoodieAvroIndexedRecord` re-derives it from the record data, `HoodieAvroRecord` returns the payload's stored value, which is the `Integer` default whenever the write path did not compute one. On a table with a `Long` ordering column that surfaces as a `ClassCastException` at merge time on prepped Spark SQL writes and on upserts with `hoodie.combine.before.upsert=false`. No handle on the changing-records path needs the incoming record to carry a payload. `FileGroupReaderBasedMergeHandle`, `HoodieWriteMergeHandle` and `HoodieSortedMergeHandle` all merge through `BufferedRecords.fromHoodieRecord` and a `HoodieRecordMerger`; custom merge modes rebuild the payload from buffered data in `HoodieAvroRecordMerger`. Every caller of `createHoodieRecord` outside the Spark write path and HoodieStreamer already passes a literal `false` for `requiresPayload`. The representation should be decided by `isPayloadClassDeprecated(payloadClass)` alone, and `requiresPayload` and `isFileGroupReaderBasedMergeHandle` removed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
