cshuo opened a new pull request, #20544:
URL: https://github.com/apache/hudi/pull/20544

   ### Describe the issue this Pull Request addresses
   
   Spark SQL `UPDATE` and COW upsert can fail with a `ClassCastException` when 
a table uses a logical type such as `DATE`, `TIMESTAMP`, or `DECIMAL` as its 
ordering field. The Avro file-group reader and incoming 
`HoodieAvroIndexedRecord` can extract different Java representations for the 
same ordering field, such as `Integer` and `LocalDate` for a date, or 
`GenericData.Fixed` and `BigDecimal` for a decimal.
   
   This is an alternative to https://github.com/apache/hudi/pull/20529.
   
   ### Summary and Changelog
   
   - Share ordering-value extraction between `AvroRecordContext` and 
`HoodieAvroIndexedRecord` using `HoodieAvroUtils.getNestedFieldVal`, so logical 
types are normalized consistently.
   - Pass the effective reader configuration to `AvroRecordContext`, including 
`hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled`.
   - Add Avro record-context tests and Spark SQL tests for COW upsert and 
UPDATE with logical-type ordering fields. The tests cover both values of the 
consistent logical timestamp setting at the record-context level and in 
timestamp upsert cases.
   
   This change intentionally retains the existing difference for null 
components in multiple ordering fields: `AvroRecordContext` uses 
`OrderingValues.getDefault()`, while `HoodieAvroIndexedRecord` retains `null`. 
Null ordering semantics are being discussed in 
https://github.com/apache/hudi/pull/19637 and 
https://github.com/apache/hudi/issues/19636.
   
   ### Impact
   
   Aligns Avro engine-record ordering values with incoming Avro record ordering 
values during merges. No configuration defaults or on-disk formats are changed.
   
   ### Risk Level
   
   Medium: the change affects ordering-value extraction for Avro records on 
file-group reader merge paths. Targeted Spark 3.5 COW upsert and UPDATE tests 
passed during development, as did `TestAvroRecordContext`. The SQL tests were 
run before the final helper refactor; the Avro unit tests were rerun after that 
refactor. Spark 4.1 has not been tested.
   
   ### Documentation Update
   
   None.
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to