cshuo opened a new pull request, #20544: URL: https://github.com/apache/hudi/pull/20544
### Describe the issue this Pull Request addresses Spark SQL `UPDATE` and COW upsert can fail with a `ClassCastException` when a table uses a logical type such as `DATE`, `TIMESTAMP`, or `DECIMAL` as its ordering field. The Avro file-group reader and incoming `HoodieAvroIndexedRecord` can extract different Java representations for the same ordering field, such as `Integer` and `LocalDate` for a date, or `GenericData.Fixed` and `BigDecimal` for a decimal. This is an alternative to https://github.com/apache/hudi/pull/20529. ### Summary and Changelog - Share ordering-value extraction between `AvroRecordContext` and `HoodieAvroIndexedRecord` using `HoodieAvroUtils.getNestedFieldVal`, so logical types are normalized consistently. - Pass the effective reader configuration to `AvroRecordContext`, including `hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled`. - Add Avro record-context tests and Spark SQL tests for COW upsert and UPDATE with logical-type ordering fields. The tests cover both values of the consistent logical timestamp setting at the record-context level and in timestamp upsert cases. This change intentionally retains the existing difference for null components in multiple ordering fields: `AvroRecordContext` uses `OrderingValues.getDefault()`, while `HoodieAvroIndexedRecord` retains `null`. Null ordering semantics are being discussed in https://github.com/apache/hudi/pull/19637 and https://github.com/apache/hudi/issues/19636. ### Impact Aligns Avro engine-record ordering values with incoming Avro record ordering values during merges. No configuration defaults or on-disk formats are changed. ### Risk Level Medium: the change affects ordering-value extraction for Avro records on file-group reader merge paths. Targeted Spark 3.5 COW upsert and UPDATE tests passed during development, as did `TestAvroRecordContext`. The SQL tests were run before the final helper refactor; the Avro unit tests were rerun after that refactor. Spark 4.1 has not been tested. ### Documentation Update None. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
