prashantwason commented on code in PR #18061:
URL: https://github.com/apache/hudi/pull/18061#discussion_r2756386207
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieCreateRecordUtils.scala:
##########
@@ -153,8 +153,17 @@ object HoodieCreateRecordUtils {
val orderingVal = OrderingValues.create(
orderingFields,
JFunction.toJavaFunction[String, Comparable[_]](
- field => HoodieAvroUtils.getNestedFieldVal(avroRec, field,
false,
-
consistentLogicalTimestampEnabled).asInstanceOf[Comparable[_]]))
+ field => {
+ val fieldVal = HoodieAvroUtils.getNestedFieldVal(avroRec,
field, false,
+ consistentLogicalTimestampEnabled)
+ if (fieldVal == null) {
+ throw new IllegalArgumentException(
+ s"Precombine/ordering field '$field' has null value
for record key '${hoodieKey.getRecordKey}'. " +
+ s"Please ensure all records have non-null values for
the precombine field, " +
+ s"or use a payload class that doesn't require
ordering (e.g., OverwriteWithLatestAvroPayload).")
Review Comment:
Thanks for the feedback @nsivabalan, you're correct.
I'll update the PR to check the merge mode and payload class before throwing
the exception. The null check should only throw when ordering is actually
required, specifically:
- Skip the exception for `RecordMergeMode.COMMIT_TIME_ORDERING`
- Skip the exception for `OverwriteWithLatestAvroPayload`
For these cases, I'll use `OrderingValues.getDefault()` as a fallback
instead of throwing, which aligns with how `OverwriteWithLatestAvroPayload`
handles missing ordering values in its constructor.
Will push an updated commit shortly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
