JackieTien97 commented on code in PR #19277:
URL: https://github.com/apache/iotdb/pull/19277#discussion_r4581225260
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/analyzer/predicate/ConvertPredicateToFilterVisitor.java:
##########
@@ -137,10 +138,25 @@ private <T extends Comparable<T>> Filter
constructInFilter(
return ValueFilterApi.in(measurementIndex, inSet, dataType);
}
+ @SuppressWarnings("unchecked")
private <T extends Comparable<T>> Set<T> constructInSet(List<Literal>
literals, Type dataType) {
+ TSDataType tsDataType = InternalTypeManager.getTSDataType(dataType);
Set<T> values = new HashSet<>();
for (Literal literal : literals) {
- values.add(getValue(literal, dataType));
+ if (tsDataType != INT32 && tsDataType != INT64) {
+ values.add(getValue(literal, dataType));
+ continue;
+ }
+ // Like "=", drop literals that can never equal an INT32/INT64 value
instead of narrowing them
+ Long value = getIntegralInListValue(literal, dataType);
+ if (value == null) {
+ continue;
Review Comment:
When every literal is dropped here, `constructInFilter` passes an empty set
to TsFile's `ValueIn`. Its `allSatisfy` treats a page whose target column is
entirely NULL as a match for an empty candidate set, so the table page reader
can skip row-level filtering.
I reproduced this with an INT32 field `i` that is NULL in two rows with a
populated payload field. `WHERE i IN (4294967298, 2.5)` returned zero rows
before `FLUSH` and both NULL rows afterward. The same issue occurs for INT64.
Please handle the empty set explicitly and add a flushed-data regression
case. Keep the `NOT IN` behavior that excludes NULLs.
##########
iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/utils/TypeServices.java:
##########
@@ -185,6 +186,32 @@ private static <T> TypeService<T> cached(final
TypeService<T> service) {
return type -> cache.computeIfAbsent(type.getTypeEnum(), ignored ->
service.call(type));
}
+ /**
+ * Converts a numeric IN-list literal for an INT32 or INT64 column. Returns
{@code null} when the
+ * literal can never equal a value of the column (out of range or not an
integer), which matches
+ * how {@code =} treats such literals.
+ */
+ public static Long getIntegralInListValue(final Literal value, final Type
type) {
+ final boolean isInt32 = type.getTypeEnum() == TypeEnum.INT32;
+ final long min = isInt32 ? Integer.MIN_VALUE : Long.MIN_VALUE;
+ final long max = isInt32 ? Integer.MAX_VALUE : Long.MAX_VALUE;
+ if (value instanceof LongLiteral) {
+ final long v = ((LongLiteral) value).getParsedValue();
+ return v >= min && v <= max ? v : null;
+ }
+ final double v;
+ if (value instanceof DoubleLiteral) {
+ v = ((DoubleLiteral) value).getValue();
+ } else if (value instanceof FloatLiteral) {
+ v = ((FloatLiteral) value).getValue();
+ } else {
+ throw new SemanticException(
+ String.format(CalcMessages.CANNOT_CAST_TO_TYPE, value,
type.getTypeEnum()));
+ }
+ // (double) max + 1 is exact for INT32 and rounds to 2^63 for INT64, so
the bound stays exact.
+ return v == Math.floor(v) && v >= min && v < (double) max + 1 ? (long) v :
null;
Review Comment:
The optimizer can deduplicate an IN list and rewrite it to `=`, bypassing
this bound check. With an INT64 field `l` set to `9223372036854775807`, I
verified that `l IN (9223372036854775808.0, 9223372036854775808.0)` matches in
`WHERE` and evaluates to `true` in a projection. The list
`(9223372036854775808.0, 0)` correctly misses the same row.
In the pushed-down equality path, converting `Long.MAX_VALUE` to double
rounds it to `2^63`, and converting the out-of-range literal back to long
saturates to `Long.MAX_VALUE`. The projected equality also rounds the field to
`2^63` when comparing as doubles. Please make the INT64 boundary behavior
consistent after deduplication and cover both `WHERE` and `SELECT` in the
regression tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]