This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new b358e2d57c7 [fix](be) Keep null tests over Variant casts in the scan
expression (#70365)
b358e2d57c7 is described below
commit b358e2d57c764f13003c55587a294f12921351bb
Author: lihangyu <[email protected]>
AuthorDate: Sun Nov 15 18:59:11 2026 -0800
[fix](be) Keep null tests over Variant casts in the scan expression (#70365)
### What problem does this PR solve?
A null test over a Variant cast silently drops rows whose Variant value
is not convertible to the
target type. For a `VARIANT NOT NULL` column `v` holding `1`, `{"a":1}`,
`"x"`, `[1,2]`:
```sql
SELECT id, CAST(v AS INT) IS NULL FROM t ORDER BY id; -- 0,1,1,1
SELECT id FROM t WHERE CAST(v AS INT) IS NULL ORDER BY id; -- empty,
expected 2,3,4
```
`_normalize_is_null_predicate` rewrites the null test on the cast into a
`NullPredicate` on the
scan slot (`v IS NULL`), but `CAST(v AS INT)` returns NULL for the
non-NULL object/string/array
values, so the pushed predicate removes rows that the expression would
keep. `IS NOT NULL` is
unaffected (`v IS NOT NULL` is a superset and the expression is still
evaluated), and
`<=> NULL` is wrong in the same way as `IS NULL` because it is rewritten
to the same null test.
`is_valid_push_down_cast` only admits casts over Variant slots, so the
rewrite is reachable
exactly for Variant casts, where `CAST(v AS T) IS NULL` is not
equivalent to `v IS NULL`.
### Fix
When the predicate's argument is a cast whose source is a Variant slot,
keep the null test in the
expression tree instead of pushing a `NullPredicate` down. The predicate
then filters with the
cast's own NULL semantics. No other pushdown is affected: the guard only
triggers for the
Variant-cast shape that `is_valid_push_down_cast` already admits.
### Tests
New regression `variant_p0/test_variant_cast_isnull_pushdown` (golden
generated by the
regression runner): INT cast values plus `IS NULL`, `IS NOT NULL` and
`<=> NULL` filters over a
`VARIANT NOT NULL` column.
Red/green was verified locally with a 4.2 build of the same code path:
- before the fix: `WHERE CAST(v AS INT) IS NULL` returns no rows, while
the projection returns
`0,1,1,1`;
- after the fix: the filter returns `2,3,4`, `<=> NULL` returns `2,3,4`,
`IS NOT NULL` returns `1`.
### Release note
Fix `CAST(variant_col AS T) IS NULL` / `<=> NULL` returning wrong
(empty) results.
### Check List (For Author)
- Test: Regression test (`test_variant_cast_isnull_pushdown`) + manual
end-to-end red/green
- Behavior changed: Yes (the filters now return the rows whose cast
result is NULL)
- Does this need documentation: No
---
be/src/exec/operator/scan_operator.cpp | 9 +++++
.../test_variant_cast_isnull_pushdown.out | 17 +++++++++
.../test_variant_cast_isnull_pushdown.groovy | 42 ++++++++++++++++++++++
3 files changed, 68 insertions(+)
diff --git a/be/src/exec/operator/scan_operator.cpp
b/be/src/exec/operator/scan_operator.cpp
index 0874c525c98..51471ac93c0 100644
--- a/be/src/exec/operator/scan_operator.cpp
+++ b/be/src/exec/operator/scan_operator.cpp
@@ -964,6 +964,15 @@ Status ScanLocalStateBase::_normalize_is_null_predicate(
return Status::OK();
}
+ // A Variant cast can introduce NULLs for non-NULL stored values, so
`CAST(v AS T) IS NULL`
+ // is not equivalent to `v IS NULL`. Keep the null test in the expression
tree instead of
+ // pushing a NullPredicate down on the source Variant column.
+ if (root->children()[0]->node_type() == TExprNodeType::CAST_EXPR &&
+
VExpr::expr_without_cast(root->children()[0])->data_type()->get_primitive_type()
==
+ PrimitiveType::TYPE_VARIANT) {
+ return Status::OK();
+ }
+
auto fn_call = assert_cast<VectorizedFnCall*>(root.get());
if (fn_call->fn().name.function_name == "is_null_pred") {
pred = NullPredicate::create_shared(
diff --git
a/regression-test/data/variant_p0/test_variant_cast_isnull_pushdown.out
b/regression-test/data/variant_p0/test_variant_cast_isnull_pushdown.out
new file mode 100644
index 00000000000..0ca199f6de2
--- /dev/null
+++ b/regression-test/data/variant_p0/test_variant_cast_isnull_pushdown.out
@@ -0,0 +1,17 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !cast_int --
+1 1
+2 \N
+3 \N
+
+-- !is_null --
+2
+3
+
+-- !is_not_null --
+1
+
+-- !null_safe --
+2
+3
+
diff --git
a/regression-test/suites/variant_p0/test_variant_cast_isnull_pushdown.groovy
b/regression-test/suites/variant_p0/test_variant_cast_isnull_pushdown.groovy
new file mode 100644
index 00000000000..20287c01ac8
--- /dev/null
+++ b/regression-test/suites/variant_p0/test_variant_cast_isnull_pushdown.groovy
@@ -0,0 +1,42 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_variant_cast_isnull_pushdown", "p0") {
+ // A Variant cast can turn a non-NULL Variant value into NULL, so `CAST(v
AS INT) IS NULL`
+ // is not equivalent to `v IS NULL`. The scan node must keep this null
test as an expression
+ // instead of pushing a null predicate down on the source Variant column.
+ sql "DROP TABLE IF EXISTS variant_cast_isnull_pushdown"
+ sql """
+ CREATE TABLE variant_cast_isnull_pushdown (
+ id INT NOT NULL,
+ v VARIANT NOT NULL
+ ) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES("replication_num" = "1")
+ """
+ sql """
+ INSERT INTO variant_cast_isnull_pushdown VALUES
+ (1, '1'),
+ (2, '"x"'),
+ (3, '[1,2]')
+ """
+
+ order_qt_cast_int "SELECT id, CAST(v AS INT) FROM
variant_cast_isnull_pushdown ORDER BY id"
+ order_qt_is_null "SELECT id FROM variant_cast_isnull_pushdown WHERE CAST(v
AS INT) IS NULL ORDER BY id"
+ order_qt_is_not_null "SELECT id FROM variant_cast_isnull_pushdown WHERE
CAST(v AS INT) IS NOT NULL ORDER BY id"
+ order_qt_null_safe "SELECT id FROM variant_cast_isnull_pushdown WHERE
CAST(v AS INT) <=> NULL ORDER BY id"
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]