danny0405 commented on code in PR #18061:
URL: https://github.com/apache/hudi/pull/18061#discussion_r3376957138
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieCreateRecordUtils.scala:
##########
@@ -282,4 +281,39 @@ object HoodieCreateRecordUtils {
(new HoodieKey(recordKey, partitionPath), recordLocation)
}
+
+ /**
+ * Gets the ordering value from the ordering fields of an Avro record.
+ * For payload classes that don't require ordering (e.g.,
OverwriteWithLatestAvroPayload)
+ * or COMMIT_TIME_ORDERING merge mode, null values are allowed and a default
ordering value is used.
+ * For other cases, throws IllegalArgumentException if any ordering field
has a null value.
+ */
+ private def getOrderingValue(orderingFields: java.util.List[String],
+ avroRec: GenericRecord,
+ recordKey: String,
+ consistentLogicalTimestampEnabled: Boolean,
+ recordMergeMode: RecordMergeMode,
+ payloadClass: String): Comparable[_] = {
+ // Ordering values are not required for COMMIT_TIME_ORDERING or
OverwriteWithLatestAvroPayload
+ val requiresOrderingValue = recordMergeMode !=
RecordMergeMode.COMMIT_TIME_ORDERING &&
+ !classOf[OverwriteWithLatestAvroPayload].getName.equals(payloadClass)
Review Comment:
the merge mode should be inferred as `COMMIT_TIME_ORDERING` if the payload
is `OverwriteWithLatestAvroPayload`, we can remove the check for the payload
class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
