Karman580 opened a new pull request, #19277:
URL: https://github.com/apache/iotdb/pull/19277
## Description
Closes #19276
### Bug
A multi-value `IN` list on an INT32 or INT64 field converted its literals
without the range and decimal handling that `=` already has. The bug only shows
up when the list has two or more values; a single value is rewritten to `=`.
On a field `i INT32` holding 1, 2, 3, INT32_MIN, INT32_MAX and NULL:
| Query | Before | After |
|---|---|---|
| `where i in (1, 4294967298)` | rows 1, **2** | row 1 |
| `where i in (1, -4294967293)` | rows 1, **3** | row 1 |
| `where i in (1, 2147483648)` | rows 1, **INT32_MIN** | row 1 |
| `where i = 1 or i = 4294967298` (normalized to IN) | rows 1, **2** | row 1
|
| `where i not in (1, 4294967298)` | row 2 missing | rows 2, 3, INT32_MIN,
INT32_MAX |
| `where i in (2, 2.5)`, `where l in (2, 2.5)` (INT64) | `301: DoubleLiteral
cannot be cast to LongLiteral` | row 2 |
| `select i in (2, 2.5)`, `select l in (2, 2.5)` | same `ClassCastException`
| `true` for 2, `null` for NULL |
| `where i not in (2.5, 4294967298)` | `ClassCastException` | every non-NULL
row, NULL row excluded |
The "Before" column comes from a standalone server built from master. The
"After" column comes from a standalone server built from this branch, running
the same statements as #19276.
### Cause
There were two paths with the same problem:
- **Filter pushdown:** `ConvertPredicateToFilterVisitor.constructInSet`
converted every literal through
`RELATIONAL_CONVERT_PREDICATE_VALUE_PARSER_SERVICE`. That is `(int)
((LongLiteral) v).getParsedValue()` for INT32. Out-of-range values wrapped, and
any `DoubleLiteral` failed the cast.
- **Projection:** the column-transformer path, `calc-commons`
`TypeServices.IN_MULTI_COLUMN_TRANSFORMER_SERVICE`, already skipped
out-of-range INT32 values. It still cast every literal to `LongLiteral`,
though, so decimals failed for both INT32 and INT64.
The analyzer does not insert casts for IN lists, so these literals reach
both paths unchanged.
### Fix
- **Shared conversion:** a new `TypeServices.getIntegralInListValue(Literal,
Type)` in calc-commons converts each literal for an INT32/INT64 column. It
keeps the value when it is an integer within the column range, or an integral
decimal within that range. It returns `null` when the literal can never match.
This is the same rule `constructCompareFilter` uses for `=`.
- **Filter pushdown:** `constructInSet` uses it for INT32/INT64 columns and
drops literals that can never match. Other column types keep the existing
conversion.
- **Projection:** the INT32 and INT64 branches of
`IN_MULTI_COLUMN_TRANSFORMER_SERVICE` use the same helper. The INT32 `try/catch
IllegalArgumentException` around `getParsedValue()` is removed; that method
cannot throw, so the block was dead code.
- **Impossible lists and NULLs:** if no literal can match, the result is an
empty IN set rather than `FalseLiteralFilter`. `NOT IN` then still goes through
the normal value filter and excludes NULL rows; the IT covers this case.
- **INT64 upper bound:** `(double) Long.MAX_VALUE` rounds up to 2^63, so the
bound is checked as `v < (double) max + 1`. That keeps `9223372036854775808.0`
from saturating onto `Long.MAX_VALUE`.
- **Non-numeric literals:** these still fail, now with the existing
`CANNOT_CAST_TO_TYPE` message instead of a `ClassCastException`.
### Tests
- **`TablePredicateConversionTest`** (new tests:
`testMultiValueInPredicateAndInt32Column`,
`testMultiValueInPredicateAndInt64Column`):
- INT32 out-of-range literals (positive, negative, and `2147483648` at the
boundary);
- integral and non-integral decimals, and duplicate values;
- NOT IN, and a list where no literal can match;
- INT64 decimals and the 2^63 boundary.
- **calc-commons `TypeServicesTest.testIntegralInListValue`:** the shared
conversion used by the projection path.
- **`IoTDBPredicateConversionTableIT`** (new tests:
`testMultiValueInPredicateAndInt32Column`,
`testMultiValueInPredicateAndInt64Column`):
- filter pushdown, and the OR form normalized into IN;
- NOT IN, including that NULL rows stay excluded;
- duplicates;
- projection for INT32 and INT64.
Local runs (JDK 25, macOS arm64):
```
./mvnw test -pl iotdb-core/calc-commons,iotdb-core/datanode \
-Dtest='TypeServicesTest,TablePredicateConversionTest'
-Dsurefire.failIfNoSpecifiedTests=false
Tests run: 13, Failures: 0, Errors: 0 -- calc-commons TypeServicesTest
Tests run: 9, Failures: 0, Errors: 0 -- TablePredicateConversionTest
Tests run: 11, Failures: 0, Errors: 0 -- datanode TypeServicesTest
BUILD SUCCESS
./mvnw verify -DskipUTs -pl integration-test -am -PTableSimpleIT -P
with-integration-tests \
-Dit.test=IoTDBPredicateConversionTableIT
-Dfailsafe.failIfNoSpecifiedTests=false
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0 --
IoTDBPredicateConversionTableIT
BUILD SUCCESS
```
Without the main-code change, both new `TablePredicateConversionTest` cases
fail with `ClassCastException: ... DoubleLiteral cannot be cast to ...
LongLiteral`.
Not run locally: `TableClusterIT` (I relied on the CI run for the cluster
environment) and the full unit-test suites of the two modules.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for code coverage.
- [x] added integration tests.
<hr>
##### Key changed/added classes (or packages if there are too many classes)
in this PR
- `org.apache.iotdb.calc.utils.TypeServices`
-
`org.apache.iotdb.db.queryengine.plan.relational.analyzer.predicate.ConvertPredicateToFilterVisitor`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]