thswlsqls commented on code in PR #10403:
URL: https://github.com/apache/paimon/pull/10403#discussion_r4214519737
##########
paimon-hive/paimon-hive-connector-common/src/main/java/org/apache/paimon/hive/SearchArgumentToPredicateConverter.java:
##########
@@ -182,6 +184,33 @@ private Object toLiteral(DataType literalType, Object o) {
if (o instanceof HiveDecimalWritable) {
o = ((HiveDecimalWritable) o).getHiveDecimal().bigDecimalValue();
}
- return convertJavaObject(literalType, o);
+ Object literal = convertJavaObject(literalType, o);
+ // Hive does not cast literals to the column type. A literal that
cannot be represented
+ // exactly in the column type (e.g. 1000 for DECIMAL(5, 2), 1.005 for
DECIMAL(5, 2) or 200
+ // for TINYINT) would be altered here and prune data files wrongly, so
leave this
+ // conjunct to Hive's residual filter instead.
+ if (o != null && !isExact(literalType, o, literal)) {
+ throw new UnsupportedOperationException(
+ "Literal "
+ + o
+ + " cannot be represented exactly in column type "
+ + literalType
+ + ".");
+ }
+ return literal;
+ }
+
+ private static boolean isExact(DataType literalType, Object original,
Object literal) {
Review Comment:
@Akash3121 Thanks for the careful review. I checked this, and I don't think
FLOAT can lose rows here.
Hive narrows the constant to the column type before the SARG is built: for a
comparison between a FLOAT column and a numeric constant,
`TypeCheckProcFactory` rewrites the constant as a `Float`. So Hive itself
evaluates `f > 0.2` as `f > 0.2f`, and the SARG literal is already a float
value widened to `Double`, which `floatValue()` round-trips exactly.
To confirm, I ran a probe in the MR and Tez ITCases on Hive 2.3, writing
`0.2f` both in its own file (prunable) and next to `0.5f` (not prunable). For
`>`, `>=`, `=`, `NOT (f <= 0.2)`, `BETWEEN` and `IN`, both copies always got
the same result — e.g. `f > 0.2` returns neither and `f = 0.2` returns both.
If you'd still like a defensive FLOAT check in `isExact`, I'm happy to add
it; it would not change results on Hive 2.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
