This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 9d7e7fd66da7 docs(record-merger): document projection compatibility
and mandatory merge fields (#19586)
9d7e7fd66da7 is described below
commit 9d7e7fd66da7dfedae8693e193c5cdfec9cfd104
Author: Ranga Reddy <[email protected]>
AuthorDate: Mon Aug 31 15:19:38 2026 +0530
docs(record-merger): document projection compatibility and mandatory merge
fields (#19586)
The custom-merger section sketched HoodieRecordMerger as merge,
partialMerge, getRecordType
and getMergingStrategy, and its Implementation Guidelines listed five
rules. Neither mentioned
isProjectionCompatible() or getMandatoryFieldsForMerging(), even though the
two decide whether
a MOR read is allowed to project columns and, if so, which columns the
merger still needs.
Someone implementing a merger against these docs therefore had no way to
learn that:
- isProjectionCompatible() defaults to false, which makes the file-group
reader fall back to
the full table schema for a CUSTOM merger (FileGroupReaderSchemaHandler
returns tableSchema
when the merger is not projection compatible). Correct, but no column
pruning.
- once it returns true, the reader reads the query's columns plus whatever
getMandatoryFieldsForMerging() names, and for CUSTOM mode that method
is the only source of
the extra columns.
- its default is the record key field plus the ordering fields, so a
merger reading anything
else has to say so.
Adds both methods to the interface sketch, a sixth guideline, and a
"Projection compatibility"
subsection covering the pairing, the trade-off and the failure mode. The
worked example mirrors
MaxRankRecordMerger in hudi-trino's tests, which is real compiling code,
rather than an invented
snippet: it keeps the interface defaults via HoodieRecordMerger.super and
adds its own column.
Also corrects the partialMerge signature in the sketch, which still showed
Avro Schema for the
reader schema. Master and 1.2.0 take HoodieSchema - release-1.2.md already
lists partialMerge
among the source-breaking signature changes.
Applied to three copies, and the signature genuinely differs between them,
so they are not
identical: docs/ and version-1.2.0 use HoodieSchema, while version-1.1.1
keeps Avro Schema in
both partialMerge and getMandatoryFieldsForMerging, which is what
release-1.1.1 shipped.
version-1.0.x carries the interface sketch but no Implementation Guidelines
section, so it is
left alone.
Closes #16697
---
website/docs/record_merger.md | 63 +++++++++++++++++++++-
.../versioned_docs/version-1.1.1/record_merger.md | 61 +++++++++++++++++++++
.../versioned_docs/version-1.2.0/record_merger.md | 63 +++++++++++++++++++++-
3 files changed, 185 insertions(+), 2 deletions(-)
diff --git a/website/docs/record_merger.md b/website/docs/record_merger.md
index 1bbd44117578..4970b18d646b 100644
--- a/website/docs/record_merger.md
+++ b/website/docs/record_merger.md
@@ -93,7 +93,7 @@ interface HoodieRecordMerger {
}
<T> BufferedRecord<T> partialMerge(BufferedRecord<T> older,
BufferedRecord<T> newer,
- Schema readerSchema, RecordContext<T>
recordContext,
+ HoodieSchema readerSchema,
RecordContext<T> recordContext,
TypedProperties props) throws
IOException {
// Merges records which can contain partial updates.
// Returns a non-null BufferedRecord with only changed fields included.
@@ -102,6 +102,20 @@ interface HoodieRecordMerger {
...
}
+ boolean isProjectionCompatible() {
+ // Whether this merger can work on a projection of the table schema
rather than every column.
+ // Defaults to false, which makes MOR reads fetch all columns. See
"Projection compatibility" below.
+ ...
+ }
+
+ String[] getMandatoryFieldsForMerging(HoodieSchema dataSchema,
HoodieTableConfig cfg,
+ TypedProperties properties) {
+ // The columns merge() and partialMerge() read, beyond those the query
itself requests.
+ // Only consulted when isProjectionCompatible() returns true.
+ // Defaults to the record key field plus the ordering fields.
+ ...
+ }
+
HoodieRecordType getRecordType() {...}
String getMergingStrategy() {...}
@@ -122,8 +136,55 @@ When implementing the `HoodieRecordMerger` interface,
follow these guidelines to
5. **Associative property**: The `merge()` method should be associative:
`merge(a, merge(b, c))` should yield the same result as `merge(merge(a, b), c)`
for any three versions A, B, C of the same record.
+6. **Declare every column your merge logic reads**: if you make the merger
projection compatible, `getMandatoryFieldsForMerging()` must list every column
`merge()` and `partialMerge()` touch that the query does not already request.
See below.
+
For more details on the implementation, see [RFC
101](https://github.com/apache/hudi/blob/master/rfc/rfc-101/rfc-101.md).
+#### Projection compatibility
+
+On a Merge-on-Read table the reader would rather fetch only the columns a
query asks for. It cannot do that
+safely for a custom merger, because it has no way to know which columns your
merge logic reads. Two methods
+resolve that, and they work as a pair:
+
+* `isProjectionCompatible()` — defaults to **`false`**, which tells the reader
to fall back to reading the
+ **full table schema**. Merging is always correct in this mode, but a query
selecting two columns still
+ reads every column of every log block, so it is the slower option.
`COMMIT_TIME_ORDERING` and
+ `EVENT_TIME_ORDERING` are projection compatible; a `CUSTOM` merger is not
until you say so.
+* `getMandatoryFieldsForMerging()` — only consulted once
`isProjectionCompatible()` returns `true`. The
+ reader then reads the columns the query requested **plus** the ones this
method names. Its default is the
+ record key field and the ordering fields, which is what a merger comparing
only an ordering value needs.
+
+The two must agree, and this is where it goes wrong: if you return `true` from
+`isProjectionCompatible()` and your merge logic reads a column that is neither
requested by the query nor
+returned by `getMandatoryFieldsForMerging()`, that column is absent from the
records your merger receives.
+What happens next is up to your code — reading the missing field typically
fails with a
+`NullPointerException`, and a merger that tolerates a null there will instead
make its decision on
+incomplete data and return a silently wrong result. Either way it is
query-dependent, showing up only for
+queries that do not happen to select the column themselves, which makes it
easy to miss in testing.
+
+So if your merger compares, say, a `priority` column to decide a winner,
either declare it:
+
+```Java
+@Override
+public boolean isProjectionCompatible() {
+ return true;
+}
+
+@Override
+public String[] getMandatoryFieldsForMerging(HoodieSchema dataSchema,
HoodieTableConfig cfg,
+ TypedProperties properties) {
+ // Keep the defaults - record key and ordering fields - and add the column
merge() reads, so the
+ // reader includes it even when a query does not project it.
+ LinkedHashSet<String> fields = new LinkedHashSet<>(
+
Arrays.asList(HoodieRecordMerger.super.getMandatoryFieldsForMerging(dataSchema,
cfg, properties)));
+ fields.add("priority");
+ return fields.toArray(new String[0]);
+}
+```
+
+or leave `isProjectionCompatible()` at its default `false` and accept the
full-schema read. Both are correct;
+only the half-way position, projection compatible without declaring the
fields, is not.
+
### Merge Mode Configs
The record merge mode and optional record merge strategy ID and custom merge
implementation classes can be specified using the below configs.
diff --git a/website/versioned_docs/version-1.1.1/record_merger.md
b/website/versioned_docs/version-1.1.1/record_merger.md
index 1bbd44117578..bf6ac125ffc8 100644
--- a/website/versioned_docs/version-1.1.1/record_merger.md
+++ b/website/versioned_docs/version-1.1.1/record_merger.md
@@ -102,6 +102,20 @@ interface HoodieRecordMerger {
...
}
+ boolean isProjectionCompatible() {
+ // Whether this merger can work on a projection of the table schema
rather than every column.
+ // Defaults to false, which makes MOR reads fetch all columns. See
"Projection compatibility" below.
+ ...
+ }
+
+ String[] getMandatoryFieldsForMerging(Schema dataSchema, HoodieTableConfig
cfg,
+ TypedProperties properties) {
+ // The columns merge() and partialMerge() read, beyond those the query
itself requests.
+ // Only consulted when isProjectionCompatible() returns true.
+ // Defaults to the record key field plus the ordering fields.
+ ...
+ }
+
HoodieRecordType getRecordType() {...}
String getMergingStrategy() {...}
@@ -122,8 +136,55 @@ When implementing the `HoodieRecordMerger` interface,
follow these guidelines to
5. **Associative property**: The `merge()` method should be associative:
`merge(a, merge(b, c))` should yield the same result as `merge(merge(a, b), c)`
for any three versions A, B, C of the same record.
+6. **Declare every column your merge logic reads**: if you make the merger
projection compatible, `getMandatoryFieldsForMerging()` must list every column
`merge()` and `partialMerge()` touch that the query does not already request.
See below.
+
For more details on the implementation, see [RFC
101](https://github.com/apache/hudi/blob/master/rfc/rfc-101/rfc-101.md).
+#### Projection compatibility
+
+On a Merge-on-Read table the reader would rather fetch only the columns a
query asks for. It cannot do that
+safely for a custom merger, because it has no way to know which columns your
merge logic reads. Two methods
+resolve that, and they work as a pair:
+
+* `isProjectionCompatible()` — defaults to **`false`**, which tells the reader
to fall back to reading the
+ **full table schema**. Merging is always correct in this mode, but a query
selecting two columns still
+ reads every column of every log block, so it is the slower option.
`COMMIT_TIME_ORDERING` and
+ `EVENT_TIME_ORDERING` are projection compatible; a `CUSTOM` merger is not
until you say so.
+* `getMandatoryFieldsForMerging()` — only consulted once
`isProjectionCompatible()` returns `true`. The
+ reader then reads the columns the query requested **plus** the ones this
method names. Its default is the
+ record key field and the ordering fields, which is what a merger comparing
only an ordering value needs.
+
+The two must agree, and this is where it goes wrong: if you return `true` from
+`isProjectionCompatible()` and your merge logic reads a column that is neither
requested by the query nor
+returned by `getMandatoryFieldsForMerging()`, that column is absent from the
records your merger receives.
+What happens next is up to your code — reading the missing field typically
fails with a
+`NullPointerException`, and a merger that tolerates a null there will instead
make its decision on
+incomplete data and return a silently wrong result. Either way it is
query-dependent, showing up only for
+queries that do not happen to select the column themselves, which makes it
easy to miss in testing.
+
+So if your merger compares, say, a `priority` column to decide a winner,
either declare it:
+
+```Java
+@Override
+public boolean isProjectionCompatible() {
+ return true;
+}
+
+@Override
+public String[] getMandatoryFieldsForMerging(Schema dataSchema,
HoodieTableConfig cfg,
+ TypedProperties properties) {
+ // Keep the defaults - record key and ordering fields - and add the column
merge() reads, so the
+ // reader includes it even when a query does not project it.
+ LinkedHashSet<String> fields = new LinkedHashSet<>(
+
Arrays.asList(HoodieRecordMerger.super.getMandatoryFieldsForMerging(dataSchema,
cfg, properties)));
+ fields.add("priority");
+ return fields.toArray(new String[0]);
+}
+```
+
+or leave `isProjectionCompatible()` at its default `false` and accept the
full-schema read. Both are correct;
+only the half-way position, projection compatible without declaring the
fields, is not.
+
### Merge Mode Configs
The record merge mode and optional record merge strategy ID and custom merge
implementation classes can be specified using the below configs.
diff --git a/website/versioned_docs/version-1.2.0/record_merger.md
b/website/versioned_docs/version-1.2.0/record_merger.md
index 1bbd44117578..4970b18d646b 100644
--- a/website/versioned_docs/version-1.2.0/record_merger.md
+++ b/website/versioned_docs/version-1.2.0/record_merger.md
@@ -93,7 +93,7 @@ interface HoodieRecordMerger {
}
<T> BufferedRecord<T> partialMerge(BufferedRecord<T> older,
BufferedRecord<T> newer,
- Schema readerSchema, RecordContext<T>
recordContext,
+ HoodieSchema readerSchema,
RecordContext<T> recordContext,
TypedProperties props) throws
IOException {
// Merges records which can contain partial updates.
// Returns a non-null BufferedRecord with only changed fields included.
@@ -102,6 +102,20 @@ interface HoodieRecordMerger {
...
}
+ boolean isProjectionCompatible() {
+ // Whether this merger can work on a projection of the table schema
rather than every column.
+ // Defaults to false, which makes MOR reads fetch all columns. See
"Projection compatibility" below.
+ ...
+ }
+
+ String[] getMandatoryFieldsForMerging(HoodieSchema dataSchema,
HoodieTableConfig cfg,
+ TypedProperties properties) {
+ // The columns merge() and partialMerge() read, beyond those the query
itself requests.
+ // Only consulted when isProjectionCompatible() returns true.
+ // Defaults to the record key field plus the ordering fields.
+ ...
+ }
+
HoodieRecordType getRecordType() {...}
String getMergingStrategy() {...}
@@ -122,8 +136,55 @@ When implementing the `HoodieRecordMerger` interface,
follow these guidelines to
5. **Associative property**: The `merge()` method should be associative:
`merge(a, merge(b, c))` should yield the same result as `merge(merge(a, b), c)`
for any three versions A, B, C of the same record.
+6. **Declare every column your merge logic reads**: if you make the merger
projection compatible, `getMandatoryFieldsForMerging()` must list every column
`merge()` and `partialMerge()` touch that the query does not already request.
See below.
+
For more details on the implementation, see [RFC
101](https://github.com/apache/hudi/blob/master/rfc/rfc-101/rfc-101.md).
+#### Projection compatibility
+
+On a Merge-on-Read table the reader would rather fetch only the columns a
query asks for. It cannot do that
+safely for a custom merger, because it has no way to know which columns your
merge logic reads. Two methods
+resolve that, and they work as a pair:
+
+* `isProjectionCompatible()` — defaults to **`false`**, which tells the reader
to fall back to reading the
+ **full table schema**. Merging is always correct in this mode, but a query
selecting two columns still
+ reads every column of every log block, so it is the slower option.
`COMMIT_TIME_ORDERING` and
+ `EVENT_TIME_ORDERING` are projection compatible; a `CUSTOM` merger is not
until you say so.
+* `getMandatoryFieldsForMerging()` — only consulted once
`isProjectionCompatible()` returns `true`. The
+ reader then reads the columns the query requested **plus** the ones this
method names. Its default is the
+ record key field and the ordering fields, which is what a merger comparing
only an ordering value needs.
+
+The two must agree, and this is where it goes wrong: if you return `true` from
+`isProjectionCompatible()` and your merge logic reads a column that is neither
requested by the query nor
+returned by `getMandatoryFieldsForMerging()`, that column is absent from the
records your merger receives.
+What happens next is up to your code — reading the missing field typically
fails with a
+`NullPointerException`, and a merger that tolerates a null there will instead
make its decision on
+incomplete data and return a silently wrong result. Either way it is
query-dependent, showing up only for
+queries that do not happen to select the column themselves, which makes it
easy to miss in testing.
+
+So if your merger compares, say, a `priority` column to decide a winner,
either declare it:
+
+```Java
+@Override
+public boolean isProjectionCompatible() {
+ return true;
+}
+
+@Override
+public String[] getMandatoryFieldsForMerging(HoodieSchema dataSchema,
HoodieTableConfig cfg,
+ TypedProperties properties) {
+ // Keep the defaults - record key and ordering fields - and add the column
merge() reads, so the
+ // reader includes it even when a query does not project it.
+ LinkedHashSet<String> fields = new LinkedHashSet<>(
+
Arrays.asList(HoodieRecordMerger.super.getMandatoryFieldsForMerging(dataSchema,
cfg, properties)));
+ fields.add("priority");
+ return fields.toArray(new String[0]);
+}
+```
+
+or leave `isProjectionCompatible()` at its default `false` and accept the
full-schema read. Both are correct;
+only the half-way position, projection compatible without declaring the
fields, is not.
+
### Merge Mode Configs
The record merge mode and optional record merge strategy ID and custom merge
implementation classes can be specified using the below configs.
