linliu-code opened a new issue, #19712:
URL: https://github.com/apache/hudi/issues/19712
On a Merge-on-Read table using a global bloom or global simple index, a
`MERGE INTO` with a partial `UPDATE SET` fails while serializing the merged
payload:
```
org.apache.avro.UnresolvedUnionException: Not in union ["null","long"]: 15.0
(field=ts)
at
org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:247)
at
org.apache.hudi.common.avro.HoodieAvroUtils.avroToBytes(HoodieAvroUtils.java:182)
at
org.apache.hudi.common.model.BaseAvroPayload.getRecordBytes(BaseAvroPayload.java:111)
at
org.apache.hudi.common.model.DefaultHoodieRecordPayload.write(DefaultHoodieRecordPayload.java:214)
at com.esotericsoftware.kryo...KryoSerializableSerializer.write
at org.apache.hudi.common.model.HoodieRecord.write(HoodieRecord.java:406)
```
`amount`'s value is landing in `ts`'s slot. The record is malformed during
index tagging and the failure surfaces the moment Spark serializes it to
shuffle the tagged records, before any log block is written.
**To reproduce**
```sql
-- hoodie.index.type = GLOBAL_BLOOM
-- hoodie.bloom.index.update.partition.path = false
CREATE TABLE t (id BIGINT, name STRING, amount DOUBLE, ts BIGINT, dt STRING)
USING hudi PARTITIONED BY (dt)
TBLPROPERTIES (type = 'mor', primaryKey = 'id', preCombineField = 'ts');
INSERT INTO t VALUES (1, 'a', 10.0, 1, '2026-08-11');
MERGE INTO t AS t
USING (SELECT 1L AS id, 15.0 AS amount, 200L AS ts, '2026-08-11' AS dt) AS s
ON t.id = s.id
WHEN MATCHED THEN UPDATE SET t.amount = s.amount, t.ts = s.ts;
```
Expected: the row updates in place, `name` and `dt` keeping their existing
values, one row, still in `dt=2026-08-11`.
Not assigning the record key is the ordinary shape for a partial update:
`hoodie.spark.sql.merge.into.partial.updates` defaults to `true`, and
`isPartialUpdateActionForMOR` deliberately waives the record-key assignment
requirement.
**Scope**
| Configuration | Result |
| --- | --- |
| MOR, `GLOBAL_BLOOM` or `GLOBAL_SIMPLE` | fails |
| MOR, `BLOOM` or `SIMPLE` (non-global) | works |
| MOR, `RECORD_INDEX` or `GLOBAL_RECORD_LEVEL_INDEX` | works |
| Copy-on-Write, any of the above | works |
| same statement with the record key assigned | works |
Those two index types are the only ones that both pre-merge at tagging time
and then inspect the merged record: `mayContainDuplicateLookup` is `tableType
== MERGE_ON_READ` for global bloom and simple, while the record-index
implementations pass `false` and short-circuit, and CoW defers the merge to the
file rewrite. Assigning the record key works because it makes
`areAllFieldsUpdated` true, switching partial updates off.
Verified present on released 1.1.1, on 1.2.0-SNAPSHOT, and on a bundle built
from the 1.2.1 release-staging branch.
**Cause**
Under partial updates the merge produces a record carrying only the assigned
columns, which is intended: `HoodieAppendHandle` takes the partial schema as
its writer schema, and `BaseWriteHelper` does the same for dedup. But
`HoodieIndexUtils.mergeIncomingWithExistingRecordWithExpressionPayload` wraps
that record against the full write schema, and
`HoodieAvroIndexedRecord#prependMetaFields` infers the meta-field count as
`targetSchema.size() - record.size()`. For a two-field record against a
ten-field target that is 8, so `JoinedGenericRecord` treats eight slots as meta
and the data lands at indices 8 and 9 instead of 7 and 8.
**Also on the same path**
`inferPartitionPath` derives the partition from that same merged record, and
`KeyGenUtils#getPartitionPath` substitutes the default partition for a field it
cannot find, so the payload already carries `dt=__HIVE_DEFAULT_PARTITION__`
while both the incoming and existing records are in the real partition. It is
harmless today only because the write fails first: fixing the schema alone
would turn a loud failure into a silently mis-partitioned row.
**Note**
This is only reachable once #19708 is fixed. Before that, the same statement
fails earlier with `HoodieKeyException: recordKey value: "null" for field:
"id"` from `SqlKeyGenerator.getPartitionPath`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
