[
https://issues.apache.org/jira/browse/HUDI-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232896#comment-17232896
]
Gary Li commented on HUDI-1397:
-------------------------------
[~advancedxy] thanks for reporting. We need a serializer here to convert
arraywritable to avro and then merge them, just like the Spark implementation.
We already have avroToArrayWritable and we need to implement
arrayWritableToAvro.
I am a bit busy recently but I can take a look when I have some time. This is
also a good task if you are interested in contributing.
> Different behavior between RealtimeCompactedRecordReader and
> HoodieMergeOnReadRDD
> ---------------------------------------------------------------------------------
>
> Key: HUDI-1397
> URL: https://issues.apache.org/jira/browse/HUDI-1397
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Xianjin YE
> Priority: Major
>
> Hi, We were writing our internal payload class and found that Hudi's
> RealtimeCompactedRecordReader's behavior doesn't match HoodieMergeOnReadRDD.
>
> To be specifically, when reading a delta record with merging log and base
> enabled, the expected behavior would be merge base + delta record. It's
> correctly handled by `HoodieMergeOnReadRDD`'s `mergeRowWithLog` method
> {code:java}
> private def mergeRowWithLog(curRow: InternalRow, curKey: String) = {
> val historyAvroRecord =
> serializer.serialize(curRow).asInstanceOf[GenericRecord]
> logRecords.get(curKey).getData.combineAndGetUpdateValue(historyAvroRecord,
> tableAvroSchema)
> }
> {code}
> However the similar logic cannot be found in
> `RealtimeCompactedRecordReader's`, it just assumes the record in delta log is
> the latest.
>
> cc [~garyli1019] since you wrote the `HoodieMergeOnReadRDD` code. It would be
> wonderful to merge base and delta record in the
> `RealtimeCompactedRecordReader`
>
> Also cc [~wayblink].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
