yihua commented on code in PR #12772:
URL: https://github.com/apache/hudi/pull/12772#discussion_r2074097726
##########
hudi-cli/pom.xml:
##########
@@ -302,6 +306,10 @@
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
</exclusion>
+ <exclusion>
+ <groupId>ch.qos.logback</groupId>
Review Comment:
I'm curious about what this is used for. Is this introduced in Hadoop 3?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/DefaultSparkRecordMerger.java:
##########
@@ -110,7 +112,7 @@ public Option<Pair<HoodieRecord, Schema>>
partialMerge(HoodieRecord older, Schem
return Option.of(Pair.of(newer, newSchema));
}
}
- if (older.getOrderingValue(oldSchema,
props).compareTo(newer.getOrderingValue(newSchema, props)) > 0) {
+ if (compareValues(older.getOrderingValue(oldSchema, props),
newer.getOrderingValue(newSchema, props)) > 0) {
Review Comment:
Same here.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/DefaultSparkRecordMerger.java:
##########
@@ -73,7 +75,7 @@ public Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord
older, Schema oldSc
return Option.of(Pair.of(newer, newSchema));
}
}
- if (older.getOrderingValue(oldSchema,
props).compareTo(newer.getOrderingValue(newSchema, props)) > 0) {
+ if (compareValues(older.getOrderingValue(oldSchema, props),
newer.getOrderingValue(newSchema, props)) > 0) {
Review Comment:
`compareValues` incurs `SparkAdapterSupport$.MODULE$.sparkAdapter()` which
incurs reflection and new instance of the adapter class per record which has
too much overhead. Let's revert it. If this is for new variant type, let's
thrown an error for such a type in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
