xushiyan commented on a change in pull request #1819:
URL: https://github.com/apache/hudi/pull/1819#discussion_r453220218
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/model/OverwriteWithLatestAvroPayload.java
##########
@@ -36,6 +36,9 @@
public class OverwriteWithLatestAvroPayload extends BaseAvroPayload
implements HoodieRecordPayload<OverwriteWithLatestAvroPayload> {
+ public static final String DEFAULT_DELETE_FIELD = "_hoodie_is_deleted";
+ private String userDefineDeleteField = null;
Review comment:
typo: it should be `userDefinedXXX`
##########
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java
##########
@@ -202,6 +202,10 @@ public Operation convert(String value) throws
ParameterException {
+ " to break ties between records with same key in input data.
Default: 'ts' holding unix timestamp of record")
public String sourceOrderingField = "ts";
+ @Parameter(names = {"--source-delete-field"}, description = "Field within
source record to decide"
+ + " is this record is delete record. Default: " +
OverwriteWithLatestAvroPayload.DEFAULT_DELETE_FIELD)
+ public String sourceDeleteField =
OverwriteWithLatestAvroPayload.DEFAULT_DELETE_FIELD;
+
Review comment:
not necessary if config goes to `HoodieWriteConfig`
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/model/OverwriteWithLatestAvroPayload.java
##########
@@ -67,7 +74,8 @@ public OverwriteWithLatestAvroPayload
preCombine(OverwriteWithLatestAvroPayload
GenericRecord genericRecord = (GenericRecord) recordOption.get();
// combining strategy here trivially ignores currentValue on disk and
writes this record
- Object deleteMarker = genericRecord.get("_hoodie_is_deleted");
+ String deleteField = userDefineDeleteField == null ? DEFAULT_DELETE_FIELD
: userDefineDeleteField;
+ Object deleteMarker = genericRecord.get(deleteField);
Review comment:
The only caller to this method is
`org.apache.hudi.io.HoodieMergeHandle#write`, where you have access to
`HoodieWriteConfig config`.
If you make the configuration `delete.field` accessible from
`HoodieWriteConfig` then make this method
`combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema,
Option<String> deleteField)`, the overall logic will be simiplified a lot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
