nsivabalan opened a new pull request, #20529:
URL: https://github.com/apache/hudi/pull/20529

   ### Describe the issue this Pull Request addresses
   
   Spark SQL `UPDATE` fails with a `ClassCastException` on tables whose 
ordering field (`preCombineField` / `orderingFields`) is a `TIMESTAMP`, `DATE` 
or `DECIMAL` column. `MERGE INTO` on the same tables works. Repro, on released 
1.2.0 with Spark 4.1.3 and also on master:
   
   ```sql
   CREATE TABLE t (id INT, v INT, ts TIMESTAMP) USING hudi
     TBLPROPERTIES (type='cow', primaryKey='id', preCombineField='ts');
   INSERT INTO t VALUES (1, 1, TIMESTAMP'2026-10-01 00:00:00'), (2, 2, 
TIMESTAMP'2026-10-01 00:00:00');
   UPDATE t SET v = v * 10 WHERE id = 1;
   -- java.lang.ClassCastException: class java.time.Instant cannot be cast to 
class java.lang.Long
   --   at java.lang.Long.compareTo
   --   at 
org.apache.hudi.common.table.read.BufferedRecordMergerFactory.shouldKeepNewerRecord
   --   at ...EventTimeRecordMerger.finalMerge
   --   at org.apache.hudi.io.FileGroupReaderBasedMergeHandle.doMerge
   ```
   
   Found while testing the Spark Connect endpoint (#20234), but it isn't 
Connect-specific: it reproduces in plain `spark-sql`.
   
   ### Summary and Changelog
   
   **Root cause.** With the default (Avro) record merger, the COW merge handle 
reads the base file through `AvroRecordContext`, and the file group reader 
compares two ordering values that were built differently:
   - **Incoming record:** `HoodieAvroIndexedRecord#doGetOrderingValue` (and 
`HoodieCreateRecordUtils`) normalize logical types through 
`HoodieAvroUtils.convertValueForAvroLogicalTypes`: date becomes `LocalDate`, 
`timestamp-*` an epoch `Long`, decimal a `BigDecimal`.
   - **Base or log record:** `RecordContext#getOrderingValue` used 
`AvroRecordContext#getValue`, which returns the **raw** Avro value: `Integer` 
or `LocalDate` for date, `Long` or `Instant` for timestamp (Avro 1.12.1 
materializes `java.time` values), and `GenericData.Fixed` for decimal.
   
   Which types fail therefore depends on the Avro version:
   
   | Profile | Fails before this PR |
   |---|---|
   | Spark 3.5 (Avro 1.11) | DATE, DECIMAL |
   | Spark 4.1 (Avro 1.12.1) | TIMESTAMP, DECIMAL |
   
   **Why MERGE INTO and BIGINT aren't affected:** MERGE INTO merges through 
`ExpressionPayload` and doesn't compare these values. The Spark record merger 
compares Catalyst values on both sides. BIGINT is a `Long` on both sides.
   
   **Changes:**
   - `RecordContext`: a new protected hook, `getOrderingFieldValue(record, 
schema, field)`, which by default calls `getValue`. Both `getOrderingValue` 
overloads use it. The javadoc states the contract: the value must be in the 
same form that `HoodieRecord#getOrderingValue` returns.
   - `AvroRecordContext`: overrides the hook and normalizes the leaf value 
through `convertValueForAvroLogicalTypes`, which already accepts both the 
primitive and the `java.time` forms. It unwraps the nullable union with the 
non-allocating `unwrapNullable`, and skips fields without a logical type, the 
common case for ordering fields, because this runs once per merged record. 
`getValue` and raw field access are unchanged.
   - The fix is in the values being compared, not in `compareTo`.
   
   **Tests:**
   - `TestUpdateTable`: UPDATE with a TIMESTAMP, DATE or DECIMAL(10,2) ordering 
field, across COW/MOR × partitioned/non-partitioned × optimized writes on/off. 
Each test updates a value column, then moves the ordering field forward, and 
checks exact rows each time. There is also a MERGE INTO regression test per 
type.
   - `TestAvroRecordContext`: the engine-record ordering value equals the 
`HoodieAvroIndexedRecord` ordering value and compares both ways. It covers 
date, timestamp-micros, fixed decimal and a nested date field, with both 
primitive and `java.time` inputs.
   
   **Local results:**
   - `TestUpdateTable` on Spark 3.5: 15 passed, 2 failed before the fix (DATE, 
DECIMAL); 17/17 after.
   - `TestUpdateTable` on Spark 4.1: 17/17 with the fix.
   - `TestAvroRecordContext`: 18/18, with checkstyle passing.
   
   ### Impact
   
   UPDATE and other merges on the Avro record path now work for tables with 
logical-type ordering fields. There's no API or config change. The per-record 
overhead of the extra check is negligible for ordering fields without a logical 
type.
   
   ### Risk Level
   
   Low. The change only affects how ordering values are extracted from Avro 
engine records, and makes them consistent with how incoming records already 
compute them. Other `getValue` callers are untouched.
   
   **Follow-ups, not in this PR:**
   - With 
`hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled=true`,
 incoming timestamps become `java.sql.Timestamp`, while this path normalizes to 
`Long`. That case was already broken before this change and needs separate 
handling.
   - Audit `HiveRecordContext` and `FlinkRecordContext` for the same mismatch.
   - The Spark 4.1 CI jobs are currently disabled, and `TestUpdateTable` 
previously only used BIGINT ordering fields, so CI would not have caught this.
   
   ### Documentation Update
   
   None.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to