This is an automated email from the ASF dual-hosted git repository.
sivabalan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new c02706ed4c1f [HUDI-9798][RFC-101] Updates to the HoodieRecordMerger
Interface (#13865)
c02706ed4c1f is described below
commit c02706ed4c1f45c57330a8d6742c819222d5ab6d
Author: Tim Brown <[email protected]>
AuthorDate: Sat Sep 13 12:39:36 2025 -0400
[HUDI-9798][RFC-101] Updates to the HoodieRecordMerger Interface (#13865)
---
rfc/README.md | 2 +-
rfc/rfc-101/rfc-101.md | 103 +++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 104 insertions(+), 1 deletion(-)
diff --git a/rfc/README.md b/rfc/README.md
index 1f37521008ae..1867381ce1e1 100644
--- a/rfc/README.md
+++ b/rfc/README.md
@@ -136,4 +136,4 @@ The list of all RFCs can be found here.
| 98 | [Spark Datasource V2 Read](./rfc-98/rfc-98.md)
| :eyes:
`UNDER REVIEW` |
| 99 | [Hudi Type System Redesign](./rfc-99/rfc-99.md)
| :eyes:
`UNDER REVIEW` |
| 100 | [Unstructured Data Storage in Hudi](./rfc-100/rfc-100.md)
| :eyes:
`UNDER REVIEW` |
-| 100 | Updates to the HoodieRecordMerger API
|
:hammer_and_wrench: `IN PROGRESS` |
\ No newline at end of file
+| 100 | [Updates to the HoodieRecordMerger API](./rfc-101/rfc-101.md)
|
:hammer_and_wrench: `IN PROGRESS` |
\ No newline at end of file
diff --git a/rfc/rfc-101/rfc-101.md b/rfc/rfc-101/rfc-101.md
new file mode 100644
index 000000000000..276a0e83d5cc
--- /dev/null
+++ b/rfc/rfc-101/rfc-101.md
@@ -0,0 +1,103 @@
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+# RFC-101: Updates to the HoodieRecordMerger API
+
+## Proposers
+
+- @the-other-tim-brown
+
+## Approvers
+ - @<approver1 github username>
+ - @<approver2 github username>
+
+## Status
+
+JIRA: https://issues.apache.org/jira/browse/HUDI-9798
+
+> Please keep the status updated in `rfc/README.md`.
+
+## Abstract
+The HoodieRecordMerger interface allows users to implement custom merging
logic for records. The interface does not provide clear instructions on how to
handle deletions and can currently lead to differences in behavior based on
execution engine.
+This RFC proposes updates to the interface to clarify the expectations so that
out of order updates and deletes are handled properly across implementations.
+
+## Background
+Currently, the
[HoodieRecordMerger](https://github.com/apache/hudi/blob/9ca72f9d932b14b414ca605560d1145de7e525a6/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java)
interface implementations within the Hudi codebase are not consistent when it
comes to how deletions are handled and can lead to differences in the results
when merging records as part of the writer or reader path. Also, it is unclear
from the current documentation of the interface for how the impl [...]
+
+As part of the migration to the new
[HoodieFileGroupReader](https://github.com/apache/hudi/blob/9ca72f9d932b14b414ca605560d1145de7e525a6/hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java),
the merging logic has been refactored for the common use cases by creating a
[BufferedRecord](https://github.com/apache/hudi/blob/9ca72f9d932b14b414ca605560d1145de7e525a6/hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecord.java),
which wraps t [...]
+
+## Implementation
+The HoodieRecordMerger interface will be updated to no longer rely on the
HoodieRecord class, and instead use the `BufferedRecord` and `RecordContext`
classes.
+The `BufferedRecord` class contains the data as well as the record key,
ordering value, schema identifier, and the `HoodieOperation`.
+
+The existing interface methods are listed here for reference:
+
+```java
+import java.io.IOException;
+
+Option<Pair<HoodieRecord, Schema>> partialMerge(HoodieRecord older, Schema
oldSchema, HoodieRecord newer,
+ Schema newSchema, Schema
readerSchema, TypedProperties props) throws IOException;
+
+Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema,
HoodieRecord newer, Schema newSchema,
+ TypedProperties props) throws
IOException;
+
+ist<Pair<HoodieRecord, Schema>> fullOuterMerge(HoodieRecord older, Schema
oldSchema, HoodieRecord newer,
+ Schema newSchema,
TypedProperties props) throws IOException;
+
+boolean shouldFlush(HoodieRecord record, Schema schema, TypedProperties props)
throws IOException;
+```
+
+The methods in the interface will be updated as follows:
+```java
+<T> BufferedRecord<T> partialMerge(BufferedRecord<T> older, BufferedRecord<T>
newer, Schema readerSchema,
+ RecordContext<T> recordContext,
MergerConfig config) throws IOException;
+
+<T> BufferedRecord merge(BufferedRecord<T> older, BufferedRecord<T> newer,
RecordContext<T> recordContext,
+ MergerConfig config) throws IOException;
+---
+
+/**
+ * MergerConfig is a configuration class used to pass all the required
properties for merging records.
+ * The record key, partition, and ordering field configurations along with
options required by the
+ * DeleteContext will be present in this object. Any custom options provided
to the reader or writer
+ * will also be passed to this object to ensure custom mergers and payloads
have access to the properties
+ * set by the user.
+ */
+public class MergerConfig extends HoodieConfig {
+ ...
+}
+```
+For both the `merge` and `partialMerge` methods, the implementation will mark
the resulting `BufferedRecord` as a delete operation if the resulting record
should result in a deletion of the row matching that record key. The
implementation should also return the data in the `BufferedRecord` as non-null
whenever possible to allow for the logic in future merge operations to
reference the previous value of the data, even if that value results in a
deletion. The ordering values must always be [...]
+
+Using the new `BufferedRecord` class will allow for easier integration with
the `BufferedRecordMerger` interface used by the `HoodieFileGroupReader` and
avoid converting to and from the `HoodieRecord` class while merging.
+
+In addition to the updates to the existing methods, two methods will be
removed. The `shouldFlush` method will be removed from the interface as part of
simplifying the interface. This method served as another way a user could
decide to not write the output to a file, but now any record that should not be
written to the file should simply be marked as a delete operation. The
`fullOuterMerge` method will also be removed since it is not used in the
current codebase. It can be re-added if th [...]
+
+## Rollout/Adoption Plan
+
+ - What impact (if any) will there be on existing users?
+ - This change is a breaking change in the API. For all users relying simply
on the merging logic that the project already provides, they will not need to
make any changes. For other users that have a custom implementation, it will
cause failures. They will need to update their implementation to comply with
the new interface and expectations.
+ - If we are changing behavior how will we phase out the older behavior?
+ - The change will force users to update their implementation as part of
upgrading to Apache Hudi 1.1.0 or above. The usage of the `shouldFlush` API
will be removed from the codebase and the updates to the existing merger
implementations will be handled before the Apache Hudi 1.1.0 release.
+ - If we need special migration tools, describe them here.
+ - When will we remove the existing behavior
+ - As part of the Apache Hudi 1.1.0 release.
+
+## Test Plan
+
+1. Ensure that all existing tests pass with the new interface and
implementation updates.
+2. Add new unit tests for the usage of custom mergers and payloads within the
HoodieFileGroupReader path to ensure that a custom ordering is respected across
updates and deletes.
+3. Add functional tests to ensure the end-to-end writer flows work as expected
when using a custom merger implementation.
\ No newline at end of file
