nsivabalan commented on a change in pull request #1819:
URL: https://github.com/apache/hudi/pull/1819#discussion_r455700501
##########
File path: hudi-spark/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -184,6 +184,13 @@ object DataSourceWriteOptions {
val PAYLOAD_CLASS_OPT_KEY = "hoodie.datasource.write.payload.class"
val DEFAULT_PAYLOAD_OPT_VAL = classOf[OverwriteWithLatestAvroPayload].getName
+ /**
+ * Field used in OverwriteWithLatestAvroPayload combineAndGetUpdateValue,
When two records have the same
+ * key value, we will check if the new record is deleted by the delete field.
+ */
+ val DELETE_FIELD_OPT_KEY = "hoodie.datasource.write.delete.field"
+ val DEFAULT_DELETE_FIELD_OPT_VAL =
OverwriteWithLatestAvroPayload.DEFAULT_DELETE_FIELD
Review comment:
you might as well define the default var in this class rather than
OverwriteWithLatestAvroPayload.
##########
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java
##########
@@ -202,6 +202,10 @@ public Operation convert(String value) throws
ParameterException {
+ " to break ties between records with same key in input data.
Default: 'ts' holding unix timestamp of record")
public String sourceOrderingField = "ts";
+ @Parameter(names = {"--source-delete-field"}, description = "Field within
source record to decide"
+ + " is this record is deleted. Default: " +
OverwriteWithLatestAvroPayload.DEFAULT_DELETE_FIELD)
Review comment:
this leakage is what I wish to avoid. Why would these classes access
OverwriteWithLatestAvroPayload. moving it to DatasourceUtils or some config
classes makes sense.
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/model/OverwriteWithLatestAvroPayload.java
##########
@@ -36,6 +36,9 @@
public class OverwriteWithLatestAvroPayload extends BaseAvroPayload
implements HoodieRecordPayload<OverwriteWithLatestAvroPayload> {
+ public static final String DEFAULT_DELETE_FIELD = "_hoodie_is_deleted";
+ private String userDefinedDeleteField = DEFAULT_DELETE_FIELD;
Review comment:
can we name it as "isDeletedField" or "isDeletedMarkerField". may or may
not be user defined and hence.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
