PavithranRick opened a new pull request, #17604:
URL: https://github.com/apache/hudi/pull/17604
### Describe the issue this Pull Request addresses
This PR unifies Partial Update Modes defined via table properties with
partial update encoding used by Spark SQL `MERGE INTO`, and cleans up
inconsistencies in how partial updates are handled across the write path.
Specifically, prior to this change:
- Partial update handling logic was fragmented.
- Merge Into partial update encoding was only supported for Spark records.
- Table property–driven partial update modes and Merge Into encoding were
not unified.
This work was originally based on PR #13540 from Lin, with additional fixes
and refinements applied.
### Summary and Changelog
This PR introduces a revised and unified design for partial update handling
across all record formats and write paths.
**Key changes:**
- Unifying Partial Update Modes from table properties and Merge Into partial
update encoding.
- Fixed all partial update handling within `BufferedRecordMergerFactory`.
- Extended Partial update encoding (Merge Into support) across all record
formats (previously only Spark record was supported).
**Revised PartialUpdateMode design:**
Possible values for table property `hoodie.table.partial.update.mode`:
- `IGNORE_DEFAULTS`
- `FILL_UNAVAILABLE`
- `KEEP_VALUES`
There is **no default value** for `PartialUpdateMode`. For tables without
partial update requirements, this property may be absent.
**Note on `KEEP_VALUES`:**
- This value may not be explicitly present in table properties.
- It will take effect when partial columns are encoded via Spark SQL `MERGE
INTO`.
- When invoking `BufferedRecordMerger` for merging, this mode is expected to
be set accordingly.
**BufferedRecordMergerFactory changes:**
`BufferedRecordMergerFactory.create()` now accepts:
- `enablePartialEncoding` (boolean): indicates whether partial (vs full)
record merging is used.
- `Option<PartialUpdateMode>`: defines the merge semantics when partial
encoding is enabled.
If `enablePartialEncoding` is `true`, the provided `PartialUpdateMode` will
be honored.
**Interaction between table property and Merge Into encoding:**
| Merge Mode | Table Property: hoodie.table.partial.update.mode | Writer
| BufferedRecordMerger: enablePartialEncoding | BufferedRecordMerger:
Option<PartialUpdateMode> |
|------------|--------------------------------------------------|-------------|---------------------------------------------|--------------------------------------------------|
| event time | not set | spark-ds
| false | Option.empty()
|
| event time | not set | Merge Into
| true | KEEP_VALUES
|
| event time | IGNORE_DEFAULTS | spark-ds
| true | IGNORE_DEFAULTS
|
| event time | IGNORE_DEFAULTS | Merge Into
| true | IGNORE_DEFAULTS
|
**Pending:**
- Unless we land get the fix landed for
https://issues.apache.org/jira/browse/HUDI-9638 these new PartialUpdateModes
are not taking effect for end to end functional tests.
- Also, we need to add more tests directly against BufferedRecordMerger
class.
### Impact
Cleaning up PartialUpdateMode feature and unifying MergeInto partial update
encoding along with other PartialUpdateModes from table config.
### Risk Level
medium
### Documentation Update
The config description must be updated if new configs are added or the
default value of the configs are changed.
Any new feature or user-facing change requires updating the Hudi website.
Please create a Jira ticket, attach the ticket number here and follow the
instruction to make changes to the website.
### Contributor's checklist
- [ ] Read through contributor's guide
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
