shangxinli commented on code in PR #18792:
URL: https://github.com/apache/hudi/pull/18792#discussion_r4371338450
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -253,6 +253,41 @@ private static class ColumnStats {
}
}
+ // Package-private for testing.
+ static boolean isFloatingPointNaN(Object value) {
+ if (value instanceof Double) {
+ return Double.isNaN((Double) value);
+ }
+ if (value instanceof Float) {
+ return Float.isNaN((Float) value);
+ }
+ return false;
+ }
+
+ /**
+ * Returns true if the given min/max value is null in a way that indicates
stats are
+ * unreliable (rather than the column being legitimately all-null). When a
column is
+ * truly all-null, nullCount == valueCount and a null min/max is the correct
answer.
+ * When stats are unreliable (NaN poisoning, value-size truncation),
nullCount is
+ * lower than valueCount but min/max are still null — that's the case
partition-stats
+ * aggregation must NOT silently drop.
+ */
+ static boolean isStatsUnreliable(HoodieMetadataColumnStats stats, Comparable
unwrappedValue) {
+ if (unwrappedValue != null) {
Review Comment:
This only catches null min/max, so a column already poisoned to NaN before
this fix landed has a non-null unwrapped value, passes as reliable, and keeps
poisoning every future merge since nothing here nulls out pre-existing
corrupted stats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
