n3nash commented on a change in pull request #1704:
URL: https://github.com/apache/hudi/pull/1704#discussion_r439755465
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java
##########
@@ -50,8 +50,25 @@
* @param schema Schema used for record
* @return new combined/merged value to be written back to storage. EMPTY to
skip writing this record.
*/
+ @Deprecated
Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue,
Schema schema) throws IOException;
+ /**
+ * This methods lets you write custom merging/combining logic to produce new
values as a function of current value on
+ * storage and whats contained in this object.
+ * <p>
+ * eg: 1) You are updating counters, you may want to add counts to
currentValue and write back updated counts 2) You
+ * may be reading DB redo logs, and merge them with current image for a
database row on storage
+ *
+ * @param currentValue Current value in storage, to merge/combine this
payload with
+ * @param schema Schema used for record
+ * @param props Payload related properties. For example pass the ordering
field(s) name to extract from value in storage.
+ * @return new combined/merged value to be written back to storage. EMPTY to
skip writing this record.
+ */
+ default Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord
currentValue, Schema schema, Map<String, String> props) throws IOException {
Review comment:
@bhasudha thanks for explaining, although my metapoint is still not
resolved. If you look at the log scanner ->
https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java#L114,
the preCombine method is used to combine records written to the log (over
multiple batches). Ideally, according to your use-case, since during logging we
don't know what is the order of updates, we simply write to disk and leave the
resolution for a later time. In this scenario, since `combineAndGetUpdateValue`
is not called, how will `preCombine` honor the `orderingVal` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
