linliu-code opened a new pull request, #19610:
URL: https://github.com/apache/hudi/pull/19610
### Change Logs
**Draft / placeholder** — opened to track the issue upstream and to serve as
the companion PR for an internal one. **The fix approach in this branch is
known to be incomplete; see "Status" below.**
`MERGE INTO` against a **partitioned** table whose source query does not
project the partition column **reports success while writing an empty commit**
— 0 bytes, 0 files, 0 records, 0 errors. The target data is left untouched,
with no error and no warning.
Reproduced on Spark 3.5, on `hoodie.table.version` both 6 and 9.
```sql
CREATE TABLE t (id BIGINT, name STRING, amount DOUBLE, ts BIGINT, dt STRING)
USING hudi PARTITIONED BY (dt)
TBLPROPERTIES (type='cow', primaryKey='id', preCombineField='ts');
INSERT INTO t VALUES (1,'a',10.0,1,'2026-08-11');
-- source projects the record key and ordering field, but NOT `dt`
MERGE INTO t AS tgt
USING (SELECT 1L AS id, 15.0 AS amount, 200L AS ts) AS s
ON tgt.id = s.id
WHEN MATCHED THEN UPDATE SET tgt.amount = s.amount, tgt.ts = s.ts;
-- statement succeeds; id=1 is still amount=10.0 / ts=1
```
`show_commits` afterwards:
```
commit_time action total_bytes files_added files_updated
partitions records errors
20260811073208749 commit 0 0 0 0
0 0
```
Adding `dt` to the **source projection** alone — leaving the `ON` clause
matching on the record key only — makes the merge apply correctly. The same
statement on a **non-partitioned** table works unmodified. So the trigger is
the source projection, not the join condition.
### Impact
Data-correctness / silent-no-op. A user's `MERGE INTO` reports success and
changes nothing; there is no diagnostic distinguishing it from a legitimate
no-match.
Note the asymmetry that makes this surprising: omitting the **ordering**
field from the source raises a clear `MergeIntoFieldResolutionException`,
whereas omitting the **partition** field is silent. In
`MergeIntoHoodieTableCommand.checkSchemaMergeIntoCompatibility` the
partition-column resolution is wrapped in a `try/catch` that discards
`MergeIntoFieldResolutionException`, while the identical failure for the
primary key is allowed to propagate.
### Status — why this is a draft
The change in this branch makes the partition column a hard error, mirroring
the primary key. Internal CI shows that is **too strict**: it also rejects the
*documented, supported* partial-update pattern, where the source omits the
partition column and the update clause does not assign it, e.g. the existing
test `Test merge into Allowed-patterns of assignment clauses` (table
partitioned by `value`, `USING (SELECT 1 AS id, 1003 AS ts)`, `UPDATE SET h0.id
= s0.id`) which runs with
`hoodie.datasource.write.merge.into.partial.updates=true`.
So there are evidently (at least) two paths through this code — one where a
missing partition column is handled by taking the partition from the matched
target record, and one where it silently produces an empty commit. **Which
condition separates them is still being investigated**; the partial-updates
flag is the leading candidate, since the passing upstream test enables it and
the failing production case did not.
Repointing this PR once that is settled. The likely shape is either
narrowing the validation to the configuration where the no-op actually occurs,
or fixing the write path so the partition is resolved from the target in that
configuration too.
### Impact
Behaviour change for `MERGE INTO` on partitioned tables — currently silent
data loss becomes either a clear error or a correct write, depending on the
final approach.
### Risk level: medium
Touches `MergeIntoHoodieTableCommand` analysis-time validation on both the
`hudi-spark3-common` and `hudi-spark4-common` copies. Existing
`TestMergeIntoTable*` suites are the guard; internal CI has already surfaced
one over-strictness regression, which is being addressed.
### Documentation Update
None required.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable — tests added
(`TestMergeIntoPartitionFieldResolution`, CoW + MOR) but **not yet green**; see
Status
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
