danny0405 commented on code in PR #19769:
URL: https://github.com/apache/hudi/pull/19769#discussion_r3871495666
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -512,6 +512,16 @@ private static <T> boolean
shouldKeepNewerRecord(BufferedRecord<T> oldRecord, Bu
// The orderingVal is constant 0 (int) and not guaranteed to match the
type of the old or new record's ordering value.
return true;
}
- return
newRecord.getOrderingValue().compareTo(oldRecord.getOrderingValue()) >= 0;
+ Comparable oldOrderingVal = oldRecord.getOrderingValue();
+ Comparable newOrderingVal = newRecord.getOrderingValue();
+ // Checks the ordering value does not equal to 0
+ // because we use 0 as the default value which means natural order.
+ // OrderingValues#create coerces a null ordering field value to that
default, so its class need
Review Comment:
should we even allows null ordering values for non-deletes records with
event time merging, for records from `DELETE` statement, it is clear since it
is indeed natural order or commit time order, but for regular records, we can
not decide the ordering with a null ordering value which is ambiguous.
I see some code paths keep the null values(Flink) while some code paths
reset nulls as 0(spark), should we just abandon the reset?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
