nsivabalan opened a new pull request, #20529:
URL: https://github.com/apache/hudi/pull/20529
### Describe the issue this Pull Request addresses
Spark SQL `UPDATE` fails with a `ClassCastException` on tables whose
ordering field (`preCombineField` / `orderingFields`) is a `TIMESTAMP`, `DATE`
or `DECIMAL` column. `MERGE INTO` on the same tables works. Repro, on released
1.2.0 with Spark 4.1.3 and also on master:
```sql
CREATE TABLE t (id INT, v INT, ts TIMESTAMP) USING hudi
TBLPROPERTIES (type='cow', primaryKey='id', preCombineField='ts');
INSERT INTO t VALUES (1, 1, TIMESTAMP'2026-10-01 00:00:00'), (2, 2,
TIMESTAMP'2026-10-01 00:00:00');
UPDATE t SET v = v * 10 WHERE id = 1;
-- java.lang.ClassCastException: class java.time.Instant cannot be cast to
class java.lang.Long
-- at java.lang.Long.compareTo
-- at
org.apache.hudi.common.table.read.BufferedRecordMergerFactory.shouldKeepNewerRecord
-- at ...EventTimeRecordMerger.finalMerge
-- at org.apache.hudi.io.FileGroupReaderBasedMergeHandle.doMerge
```
Found while testing the Spark Connect endpoint (#20234), but it isn't
Connect-specific: it reproduces in plain `spark-sql`.
### Summary and Changelog
**Root cause.** With the default (Avro) record merger, the COW merge handle
reads the base file through `AvroRecordContext`, and the file group reader
compares two ordering values that were built differently:
- **Incoming record:** `HoodieAvroIndexedRecord#doGetOrderingValue` (and
`HoodieCreateRecordUtils`) normalize logical types through
`HoodieAvroUtils.convertValueForAvroLogicalTypes`: date becomes `LocalDate`,
`timestamp-*` an epoch `Long`, decimal a `BigDecimal`.
- **Base or log record:** `RecordContext#getOrderingValue` used
`AvroRecordContext#getValue`, which returns the **raw** Avro value: `Integer`
or `LocalDate` for date, `Long` or `Instant` for timestamp (Avro 1.12.1
materializes `java.time` values), and `GenericData.Fixed` for decimal.
Which types fail therefore depends on the Avro version:
| Profile | Fails before this PR |
|---|---|
| Spark 3.5 (Avro 1.11) | DATE, DECIMAL |
| Spark 4.1 (Avro 1.12.1) | TIMESTAMP, DECIMAL |
**Why MERGE INTO and BIGINT aren't affected:** MERGE INTO merges through
`ExpressionPayload` and doesn't compare these values. The Spark record merger
compares Catalyst values on both sides. BIGINT is a `Long` on both sides.
**Changes:**
- `RecordContext`: a new protected hook, `getOrderingFieldValue(record,
schema, field)`, which by default calls `getValue`. Both `getOrderingValue`
overloads use it. The javadoc states the contract: the value must be in the
same form that `HoodieRecord#getOrderingValue` returns.
- `AvroRecordContext`: overrides the hook and normalizes the leaf value
through `convertValueForAvroLogicalTypes`, which already accepts both the
primitive and the `java.time` forms. It unwraps the nullable union with the
non-allocating `unwrapNullable`, and skips fields without a logical type, the
common case for ordering fields, because this runs once per merged record.
`getValue` and raw field access are unchanged.
- The fix is in the values being compared, not in `compareTo`.
**Tests:**
- `TestUpdateTable`: UPDATE with a TIMESTAMP, DATE or DECIMAL(10,2) ordering
field, across COW/MOR × partitioned/non-partitioned × optimized writes on/off.
Each test updates a value column, then moves the ordering field forward, and
checks exact rows each time. There is also a MERGE INTO regression test per
type.
- `TestAvroRecordContext`: the engine-record ordering value equals the
`HoodieAvroIndexedRecord` ordering value and compares both ways. It covers
date, timestamp-micros, fixed decimal and a nested date field, with both
primitive and `java.time` inputs.
**Local results:**
- `TestUpdateTable` on Spark 3.5: 15 passed, 2 failed before the fix (DATE,
DECIMAL); 17/17 after.
- `TestUpdateTable` on Spark 4.1: 17/17 with the fix.
- `TestAvroRecordContext`: 18/18, with checkstyle passing.
### Impact
UPDATE and other merges on the Avro record path now work for tables with
logical-type ordering fields. There's no API or config change. The per-record
overhead of the extra check is negligible for ordering fields without a logical
type.
### Risk Level
Low. The change only affects how ordering values are extracted from Avro
engine records, and makes them consistent with how incoming records already
compute them. Other `getValue` callers are untouched.
**Follow-ups, not in this PR:**
- With
`hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled=true`,
incoming timestamps become `java.sql.Timestamp`, while this path normalizes to
`Long`. That case was already broken before this change and needs separate
handling.
- Audit `HiveRecordContext` and `FlinkRecordContext` for the same mismatch.
- The Spark 4.1 CI jobs are currently disabled, and `TestUpdateTable`
previously only used BIGINT ordering fields, so CI would not have caught this.
### Documentation Update
None.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]