[GitHub] [incubator-hudi] n3nash edited a comment on issue #1549: Potential issue when using Deltastreamer with DMS
n3nash edited a comment on issue #1549: URL: https://github.com/apache/incubator-hudi/issues/1549#issuecomment-619654653 @PhatakN1 COW & MOR both support all the operations, choosing which one to use is based on your use-case rather than feature set. Are you overriding the method `combineAndGetUpdateValue` in your custom payload implementation ? This part of the code -> https://github.com/apache/incubator-hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/model/OverwriteWithLatestAvroPayload.java#L70 basically takes care of that in the `OverwriteWithLatestAvroPayload` payload implementation. But if you override that and make use of your own class and method impl, then you'll need to ensure you do the same in your code. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-hudi] n3nash edited a comment on issue #1549: Potential issue when using Deltastreamer with DMS
n3nash edited a comment on issue #1549: URL: https://github.com/apache/incubator-hudi/issues/1549#issuecomment-618811796 @vinothchandar We do invoke the same payload when combining records during merge/compaction. For deletes, the payload has to be an empty payload and then the record should be skipped -> https://github.com/apache/incubator-hudi/blob/master/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeCompactedRecordReader.java#L94 @PhatakN1 when you try deletes, is that any empty payload ? Or is this something you just drive through configs in deltastreamer ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-hudi] n3nash edited a comment on issue #1549: Potential issue when using Deltastreamer with DMS
n3nash edited a comment on issue #1549: URL: https://github.com/apache/incubator-hudi/issues/1549#issuecomment-618811796 @vinothchandar We do invoke the same payload when combining records during merge/compaction. For deletes, the payload has to be an empty payload and then the record should be skipped -> https://github.com/apache/incubator-hudi/blob/master/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeCompactedRecordReader.java#L94 @PhatakN1 when you try deletes, is that any empty payload ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
