voonhous commented on issue #20630:
URL: https://github.com/apache/hudi/issues/20630#issuecomment-6976637404

   ## Design note: Trino-native records for the MOR merge
   
   **Proposal.** Give the connector its own `RecordContext<TrinoRow>`, where a 
`TrinoRow` is a view of one position in a Trino page. The file-group reader 
then merges Trino rows directly, the way Spark (`InternalRow`), Flink 
(`RowData`) and Hive (`ArrayWritable`) already do. Avro stays only where 
hudi-common hands us Avro: classic Avro log blocks, and payload classes for the 
two records being merged. A base row with no log update becomes one small view 
object plus a position copy per column, which meets the second exit criterion 
of this issue by construction.
   
   **Depends on** typing columns from `HoodieSchema` 
(`hudi.schema-column-types.enabled`, branch `trino-mapper-column-types` on top 
of #20680). Native rows need each value's Trino type to match its schema field. 
On metastore types (bigint for a timestamp, varbinary for a uuid), the record 
context would have to guess value shapes the way `HudiAvroSerializer` does 
today. This lands on that gated path first; the legacy path stays as it is 
until the default flips.
   
   ### Today
   
   <!-- diagram: old-path.png -->
   
   - `HudiTrinoReaderContext` extends `HoodieReaderContext<IndexedRecord>` and 
uses hudi-common's `AvroRecordContext` (`HudiTrinoReaderContext.java:67-68, 
:122`).
   - Every base row goes through `HudiAvroSerializer.serialize()` (page row to 
`GenericData.Record`) and back through `buildRecordInPage()`, including rows no 
log record touches.
   - The values `serialize()` puts in the record are Trino objects 
(`SqlTimestamp`, `SqlDecimal`, `SqlVarbinary`), not Avro-native ones. So 
`appendTo` has to work out each value's shape with `instanceof`, and anything 
that Avro-encodes these records breaks on them.
   
   ### Proposed
   
   <!-- diagram: new-path.png -->
   
   <details>
   <summary>Why this fits hudi-common's contract (line references against 
branch trino-mapper-column-types)</summary>
   
   - **Commit-time and event-time ordering need no merger class.** 
`BufferedRecordMergerFactory` returns `CommitTimeRecordMerger` / 
`EventTimeRecordMerger` for them, which only compare ordering values read 
through the `RecordContext` (`BufferedRecordMergerFactory.java:78-87, 
:179-209`).
   - **Mergers are generic over the record type.** 
`HoodieRecordMerger.merge(BufferedRecord<T>, BufferedRecord<T>, 
RecordContext<T>, ...)` (`HoodieRecordMerger.java:73`). 
`HoodieAvroRecordMerger` converts just the two records it merges through 
`convertToAvroRecord` / `convertAvroRecord` 
(`HoodieAvroRecordMerger.java:58-60, :79`). `PayloadUpdateProcessor` does the 
same for log-only rows (`UpdateProcessor.java:136-147`). So payload tables work 
on `TrinoRow` and pay for Avro only on merged rows.
   - **`seal()` runs on every base row** (`FileGroupRecordBuffer.java:212`). 
Spark copies there because `UnsafeRow` reuses its buffer 
(`BaseSparkInternalRecordContext.java:222-224`). Trino blocks are immutable, so 
`seal()` can return the view as is.
   - **`toBinaryRow()` runs only when a record is retained.** That's insertion 
into the log record buffer (`BufferedRecord.java:68-77` via 
`KeyBasedFileGroupRecordBuffer.java:130`) and the LSM spill iterator 
(`SpillableLsmRecordIterator.java:88`). It is the one place to copy: compact 
the view into a single-position row so buffered log records don't pin whole log 
pages.
   - **Avro log blocks arrive through `convertAvroRecord`** 
(`HoodieAvroDataBlock.java:173`). The Avro-to-block half of 
`HoodieSchemaAvroSerializer` becomes that conversion.
   
   </details>
   
   <details>
   <summary>RecordContext&lt;TrinoRow&gt; method mapping</summary>
   
   | Method | TrinoRow implementation |
   |---|---|
   | `getValue(row, schema, field)` | Channel by name (cached per schema, 
dotted paths walk row blocks). Returns plain Java values (`Long`, `Integer`, 
`String`, `BigDecimal`, `Boolean`), so ordering values compare with the ones 
delete blocks carry, and `castToBoolean` / delete-marker `toString()` work. |
   | `getMetaFieldValue(row, pos)` | `VARCHAR` slice of the meta column, 
decoded to `String` (the record key the buffer map is keyed by). |
   | `convertValueToEngineType` | Identity, since `getValue` already returns 
Java values. |
   | `seal(schema, row)` | Returns `row` (blocks are immutable). |
   | `toBinaryRow(schema, row)` | Single-position copy 
(`Block.getSingleValueBlock` per channel). |
   | `projectRecord(from, to, renames)` | Reorders channels by name; a view 
over the same blocks, no copy. |
   | `constructEngineRecord(schema, values)` / `mergeWithEngineRecord` | Writes 
Java values into one-position blocks through the per-`HoodieSchemaType` writers 
(partial-update modes). |
   | `getDeleteRow(key)` | One-position row with the key fields set, other 
channels null. |
   | `convertAvroRecord(avro)` | Avro-to-block writers, keyed by 
`HoodieSchemaType`. |
   | `convertToAvroRecord(row, schema)` | Block-to-Avro writers, keyed by 
`HoodieSchemaType` (Avro-native values). |
   | `constructHoodieRecord` | Needed only by custom-merger paths; see the 
first open question. |
   | `getEngineRecordType` | `HoodieRecordType` has no TRINO 
(`HoodieRecord.java:574-575`); see the second open question. |
   
   Output: `HudiPageSource` writes each channel with `Type.appendTo(block, 
position, builder)`, by name as today. Partition and hidden columns still come 
from `PrefilledColumnValues`.
   
   </details>
   
   <details>
   <summary>Scope of the first cut</summary>
   
   - **Native `TrinoRow` context** when `hudi.schema-column-types.enabled` is 
on and the table uses commit-time or event-time ordering (including 
partial-update modes) or a payload class.
   - **Today's Avro context** (schema-typed `HoodieSchemaAvroSerializer`) for 
tables with a custom `HoodieRecordMerger`, because a user merger may cast 
`getRecord()` to `IndexedRecord`. The repo's test mergers read only through 
`recordContext.getValue`, so an opt-in for well-behaved mergers is a later step.
   - **Unchanged:** everything with the property off, and the base-file-only 
path, which already passes pages through.
   - **Spill:** `HudiRecordSerializer` from #20648 gets a `TrinoRow` variant; 
size estimation stays with #20627.
   
   </details>
   
   <details>
   <summary>Open questions</summary>
   
   1. **`constructHoodieRecord`** for a custom merger: implement it over 
`TrinoRow` (no `HoodieRecord` subclass exists for Trino), or keep custom-merger 
tables on the Avro context for good?
   2. **`getEngineRecordType`:** there is no TRINO value and no caller in 
hudi-common's read path (it feeds the native log delete writer). Throw from 
this read-only context, or add the enum value?
   3. **KEEP_VALUES partial merging** goes through `recordMerger.partialMerge` 
(`BufferedRecordMergerFactory.java:72-75`); confirm the ordering-mode mergers 
can run it on `TrinoRow` (today's Avro path has the same gap, #19413).
   4. **Batch pass-through:** runs of untouched base rows could be copied as a 
range (`copyPositions`) instead of row by row. That needs a hook in the 
file-group reader's iterator, so it's a follow-up.
   
   </details>
   
   <details>
   <summary>Plan and measurement</summary>
   
   1. `TrinoRow` and `TrinoRecordContext`, with unit tests per method, reusing 
the `HoodieSchemaType` writers.
   2. Route the native context from `HudiPageSourceProvider` by merge mode; 
`HudiPageSource` copies positions.
   3. `TrinoRow` spill serializer (on top of #20648).
   4. `HudiMorReadBenchmark` scan with `-Dbench.s3.latency.enabled=0`, measured 
on top of #20627, against this issue's `_rt`/`_ro` CPU ratio of 3x or less. 
Plus an allocation profile showing no per-row record object for base rows 
without a log entry. The merge-semantics suites (`TestHudiMor*`, custom-merger 
tests) run with the property on.
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to