yihua commented on PR #9593:
URL: https://github.com/apache/hudi/pull/9593#issuecomment-1731899702
Pasting my deep-diving of the code based on release 0.12.2 (pre-landing of
RFC-46 PR #7345 ) here. Here is how different APIs, `#preCombine`,
`#combineAndGetUpdateValue`, and `#getInsertValue`, of `HoodieRecordPayload`,
responsible for handling record-level inserts, updates, and deletes, are used:
- `T preCombine(T oldValue, Properties props)`: deduplicate the records
within the same batch, or merge records from the log files in the same file
slice, for the same record key
- `HoodieWriteHelper#deduplicateRecords`: deduplicate records of the same
record key within the same batch from the data source (similar for Flink and
Java write helpers)
- `HoodieMergedLogRecordScanner#processNextRecord`: when scanning records
from the log files in the same file slice, merge the records of the same record
key from the log files and save the result in a spillable map containing the
mapping of the record key to record mapping
- `HoodieBackedTableMetadata#readFromBaseAndMergeWithLogRecords`: merging
records of the same record key from log files for the metadata table
- `Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord
currentValue, Schema schema, Properties properties)`: merge the update record
with the record from base file
- `HoodieMergeHandle.write`: Merge new record to the one on storage
- `HoodieMergeOnReadRDD.ecordMergingFileIterator.merge`: merge record from
the base file with the record from the log files (results from the merged log
scanner) for MOR in spark reader
- `MergeOnReadInputFormat.MergeIterator.mergeRowWithLog` (Flink): merge
record from base file with the record from log files (after merged log scanner)
- `Option<IndexedRecord> getInsertValue(Schema schema, Properties
properties)`: get record to insert when there is no existing record on storage
to combine with
- `HoodieFileSliceReader.getFileSliceReader`: transform records from the
log files with getInsertValue
- `HoodieLazyInsertIterable , HoodieAppendHandle , HoodieCreateHandle ,
HoodieMergeHandle , HoodieSortedMergeHandle`: get the insert value before
writing
- `ColumnStatsIndices.readColumnStatsIndexByColumns ,
ColumnStatsIndexSupport`: get insert value of col stats
- `RealtimeCompactedRecordReader , RealtimeUnmergedRecordReader`: get
records from the log files
- `HoodieMergeOnReadRDD`: transform records read from the log files with
getInsertValue
- Comparable<?> getOrderingValue() : returns ordering value
- `HoodieAppendHandle.writeToBuffer`: get the ordering value for delete
records
- `HoodieMergedLogRecordScanner.processNextDeletedRecord`: get the
ordering value for processing deletes
- `Option<Map<String, String>> getMetadata()`: returns record metadata, used
by `HoodieAppendHandle`, `HoodieCreateHandle`, `HoodieMergeHandle`, and
`HoodieWriteHandle`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
