This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new b05dc8ab39d branch-4.1: [fix](cast) Declare Variant casts to
non-Variant targets nullable (#70363)
b05dc8ab39d is described below
commit b05dc8ab39dbeb1e518477a6f18247b99497bd67
Author: lihangyu <[email protected]>
AuthorDate: Sun Nov 15 18:33:54 2026 -0800
branch-4.1: [fix](cast) Declare Variant casts to non-Variant targets
nullable (#70363)
### What problem does this PR solve?
Cherry-pick of #70359 (adapted for `branch-4.1`) plus the BE scan fix
proposed on master in #70365.
Related PR: #70359, #70365
Problem Summary: backport the Variant-to-non-Variant nullable fix to
branch-4.1 and retain the resulting null tests during scan
normalization. A NOT NULL Variant column can contain Variant nulls or
values that cannot be converted to a scalar. The existing BE CAST can
return SQL NULL, but `Cast.castNullable` only accounted for Variant V2
and legacy casts to JSON.
This causes both wrong optimizer rewrites and execution contract
failures: `<=> NULL` becomes FALSE, COALESCE/NVL lose their fallback,
and direct CAST queries can fail with a declared non-nullable / actual
nullable mismatch.
End-to-end validation exposed a second issue: after correcting FE
metadata, scan normalization still replaced `CAST(v AS INT) IS NULL`
with a null test on the source Variant column. This loses NULLs
introduced by conversion; IS NOT NULL can retain the wrong rows.
### Fix
- Reuse the existing `Cast.castNullable` rule to declare
Variant-to-non-Variant casts nullable for both representations. Identity
casts retain input nullability; no optimizer-specific nullable rule is
added.
- Keep NULL tests over casts as expressions through the existing scan
pushdown rejection path. The scan eligibility check only admits casts
from Variant here, so ordinary slot NULL predicates keep their existing
path.
- Add focused FE optimizer coverage and legacy-Variant SQL regression
for projections, filters, COALESCE/NVL, nullable inputs, incompatible
values, identity casts, an aliased CTE, and all-Variant-null roots. The
representation-specific SQL golden is explicitly skipped on V2; V2
nullable inference remains covered by CastTest.
### Validation
- Before the fix: FE tests reproduce the wrong nullable flag, FALSE
folding of `<=> NULL`, and removal of conditional fallbacks. SQL fails
with a nullable column mismatch. With the FE fix alone, scan filters
still incorrectly return no rows; the captured projection results
identify the rows that should survive.
- After the fix: 59 FE unit tests passed (CastTest, CheckCastTest,
NullSafeEqualToEqualTest, SimplifyConditionalFunctionTest,
VariantCastNullableTest).
- FE build with Checkstyle and BE ASAN build passed using `build.sh`.
- Regression goldens were generated by the repository runner and
inspected. Normal comparison: test_variant_cast_nullable and existing
test_variant_cast passed (2 suites).
- C++ formatting: repository wrapper with clang-format 16. Static
analysis: attempted after the ASAN build; after correcting the toolchain
resource directory, analysis is still blocked by the pre-existing
unmatched NOLINTEND in be/src/core/types.h:577. Reported source warnings
are outside the added lines; this is not a clean clang-tidy pass.
### Compatibility and performance
BE CAST conversion semantics, storage formats, and Thrift definitions
are unchanged. The fix corrects metadata, optimizer simplification, and
filtering against the existing conversion result. Existing legacy
Variant conversion quirks are preserved by the golden tests.
The FE change adds no per-row conversion work. Unsafe NULL-predicate
pushdown over a Variant cast is replaced by expression evaluation, which
may scan/evaluate more rows for those predicates; ordinary source-column
NULL pushdown is unchanged. No performance benchmark is claimed. No new
public API or dependency is added.
### Release note
Fix incorrect NULL predicates, conditional-function simplification, and
nullable result type mismatches when casting Variant values to
non-Variant types with legacy Variant execution.
### Check List (For Author)
- Test: FE Unit Test: 59 passed; SQL regression: 2 suites passed; FE/BE
ASAN builds passed
- Behavior changed: Yes. Preserve NULL/fallback results and filter rows
according to the CAST result; conversion semantics are unchanged.
- Does this need documentation: No
### Update: explicit Variant check + NOT NULL-column regression
- Related BE fix on master: #70365 (this branch carries the same change,
adapted).
- The scan guard now checks the cast source type explicitly (Variant)
instead of skipping every `CAST_EXPR`. Behaviour is unchanged for the
reachable shape (only Variant casts pass `is_valid_push_down_cast`), but
the condition is self-documenting and no longer relies on that implicit
precondition.
- Added `variant_p0/test_variant_cast_isnull_pushdown`: `CAST(v AS INT)
IS NULL`, `IS NOT NULL` and `<=> NULL` on a `VARIANT NOT NULL` column.
The existing `is_null_filter` only covered `CAST(v AS STRING)`, where
every row is non-null, so it could not catch a removed guard. The golden
was generated by the regression runner. Without the guard the new test
returns no rows for `IS NULL` / `<=> NULL`.
---
be/src/exec/operator/scan_operator.cpp | 9 ++
.../doris/nereids/trees/expressions/Cast.java | 9 +-
.../expression/rules/VariantCastNullableTest.java | 67 +++++++++++++++
.../doris/nereids/trees/expressions/CastTest.java | 8 +-
.../test_variant_cast_isnull_pushdown.out | 17 ++++
.../data/variant_p0/test_variant_cast_nullable.out | 53 ++++++++++++
.../test_variant_cast_isnull_pushdown.groovy | 42 ++++++++++
.../variant_p0/test_variant_cast_nullable.groovy | 98 ++++++++++++++++++++++
8 files changed, 292 insertions(+), 11 deletions(-)
diff --git a/be/src/exec/operator/scan_operator.cpp
b/be/src/exec/operator/scan_operator.cpp
index 7eb6c8c2489..7555787dc5f 100644
--- a/be/src/exec/operator/scan_operator.cpp
+++ b/be/src/exec/operator/scan_operator.cpp
@@ -943,6 +943,15 @@ Status ScanLocalStateBase::_normalize_is_null_predicate(
return Status::OK();
}
+ // A Variant cast can introduce NULLs for non-NULL stored values, so
`CAST(v AS T) IS NULL`
+ // is not equivalent to `v IS NULL`. Keep the null test in the expression
tree instead of
+ // pushing a NullPredicate down on the source Variant column.
+ if (root->children()[0]->node_type() == TExprNodeType::CAST_EXPR &&
+
VExpr::expr_without_cast(root->children()[0])->data_type()->get_primitive_type()
==
+ PrimitiveType::TYPE_VARIANT) {
+ return Status::OK();
+ }
+
auto fn_call = assert_cast<VectorizedFnCall*>(root.get());
if (fn_call->fn().name.function_name == "is_null_pred") {
pred = NullPredicate::create_shared(
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Cast.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Cast.java
index 3ea2573e36b..161db55ad7f 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Cast.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Cast.java
@@ -34,7 +34,6 @@ import org.apache.doris.nereids.types.LargeIntType;
import org.apache.doris.nereids.types.SmallIntType;
import org.apache.doris.nereids.types.TimeStampTzType;
import org.apache.doris.nereids.types.TinyIntType;
-import org.apache.doris.nereids.types.VariantType;
import org.apache.doris.nereids.types.coercion.DateLikeType;
import org.apache.doris.nereids.util.DateUtils;
@@ -207,12 +206,8 @@ public class Cast extends Expression implements
UnaryExpression, Monotonic {
} else if (childDataType.isJsonType() && !targetType.isJsonType()) {
// Json to other type is always nullable
return true;
- } else if (childDataType instanceof VariantType
- && ((VariantType) childDataType).isExecutionV2() &&
!targetType.isVariantType()) {
- // Variant V2 values can have a shape that is incompatible with
the target type.
- return true;
- } else if (childDataType.isVariantType() && targetType.isJsonType()) {
- // Legacy Variant to Json is always nullable.
+ } else if (childDataType.isVariantType() &&
!targetType.isVariantType()) {
+ // Variant nulls and incompatible values can produce SQL NULL in
both execution representations.
return true;
}
return false;
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/rules/VariantCastNullableTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/rules/VariantCastNullableTest.java
new file mode 100644
index 00000000000..60635154237
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/rules/VariantCastNullableTest.java
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rules;
+
+import org.apache.doris.common.Config;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteTestHelper;
+import org.apache.doris.nereids.rules.expression.ExpressionRuleExecutor;
+import org.apache.doris.nereids.trees.expressions.Cast;
+import org.apache.doris.nereids.trees.expressions.IsNull;
+import org.apache.doris.nereids.trees.expressions.NullSafeEqual;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.Coalesce;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.Nvl;
+import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.StringLiteral;
+import org.apache.doris.nereids.types.StringType;
+import org.apache.doris.nereids.types.VariantType;
+
+import com.google.common.collect.ImmutableList;
+import org.junit.jupiter.api.Test;
+
+public class VariantCastNullableTest extends ExpressionRewriteTestHelper {
+ @Test
+ public void testNullPredicatesRetainVariantCast() {
+ boolean original = Config.enable_variant_v2;
+ try {
+ Config.enable_variant_v2 = false;
+ executor = new ExpressionRuleExecutor(ImmutableList.of(
+ bottomUp(NullSafeEqualToEqual.INSTANCE)));
+ Cast cast = new Cast(new SlotReference("v", VariantType.INSTANCE,
false), StringType.INSTANCE);
+ assertRewrite(new IsNull(cast), new IsNull(cast));
+ assertRewrite(new NullSafeEqual(cast, new
NullLiteral(StringType.INSTANCE)), new IsNull(cast));
+ } finally {
+ Config.enable_variant_v2 = original;
+ }
+ }
+
+ @Test
+ public void testConditionalFunctionsRetainVariantCastFallback() {
+ boolean original = Config.enable_variant_v2;
+ try {
+ Config.enable_variant_v2 = false;
+ executor = new
ExpressionRuleExecutor(ImmutableList.of(bottomUp(SimplifyConditionalFunction.INSTANCE)));
+ Cast cast = new Cast(new SlotReference("v", VariantType.INSTANCE,
false), StringType.INSTANCE);
+ StringLiteral fallback = new StringLiteral("fallback");
+ assertRewrite(new Coalesce(cast, fallback), new Coalesce(cast,
fallback));
+ assertRewrite(new Nvl(cast, fallback), new Nvl(cast, fallback));
+ } finally {
+ Config.enable_variant_v2 = original;
+ }
+ }
+}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/CastTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/CastTest.java
index 6acb32dc279..d443ff58847 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/CastTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/CastTest.java
@@ -641,11 +641,11 @@ public class CastTest {
boolean originalEnableVariantV2 = Config.enable_variant_v2;
try {
Config.enable_variant_v2 = false;
- Assertions.assertFalse(castNullable(false, VariantType.INSTANCE,
BooleanType.INSTANCE));
- Assertions.assertFalse(castNullable(false, VariantType.INSTANCE,
IntegerType.INSTANCE));
- Assertions.assertFalse(castNullable(false, VariantType.INSTANCE,
StringType.INSTANCE));
+ Assertions.assertTrue(castNullable(false, VariantType.INSTANCE,
BooleanType.INSTANCE));
+ Assertions.assertTrue(castNullable(false, VariantType.INSTANCE,
IntegerType.INSTANCE));
+ Assertions.assertTrue(castNullable(false, VariantType.INSTANCE,
StringType.INSTANCE));
Assertions.assertTrue(castNullable(false, VariantType.INSTANCE,
JsonType.INSTANCE));
- Assertions.assertFalse(castNullable(false, VariantType.INSTANCE,
ArrayType.of(IntegerType.INSTANCE)));
+ Assertions.assertTrue(castNullable(false, VariantType.INSTANCE,
ArrayType.of(IntegerType.INSTANCE)));
Assertions.assertFalse(castNullable(false, VariantType.INSTANCE,
VariantType.INSTANCE));
Assertions.assertTrue(castNullable(true, VariantType.INSTANCE,
VariantType.INSTANCE));
diff --git
a/regression-test/data/variant_p0/test_variant_cast_isnull_pushdown.out
b/regression-test/data/variant_p0/test_variant_cast_isnull_pushdown.out
new file mode 100644
index 00000000000..0ca199f6de2
--- /dev/null
+++ b/regression-test/data/variant_p0/test_variant_cast_isnull_pushdown.out
@@ -0,0 +1,17 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !cast_int --
+1 1
+2 \N
+3 \N
+
+-- !is_null --
+2
+3
+
+-- !is_not_null --
+1
+
+-- !null_safe --
+2
+3
+
diff --git a/regression-test/data/variant_p0/test_variant_cast_nullable.out
b/regression-test/data/variant_p0/test_variant_cast_nullable.out
new file mode 100644
index 00000000000..98c27a1a7d4
--- /dev/null
+++ b/regression-test/data/variant_p0/test_variant_cast_nullable.out
@@ -0,0 +1,53 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !casts --
+1 {} \N \N
+2 42 42 42
+3 {"a":1} 0 0
+4 "invalid" \N \N
+5 [1,2] \N \N
+
+-- !null_predicates --
+1 false false true true false
+2 false false false false false
+3 false false false false false
+4 false false true true false
+5 false false true true false
+
+-- !is_null_filter --
+
+-- !is_not_null_filter --
+2
+3
+
+-- !nullable_input_filter --
+1
+4
+5
+
+-- !null_safe_filter --
+1
+4
+5
+
+-- !fallbacks --
+1 {} {} 99 99 99 99
+2 42 42 42 42 42 42
+3 {"a":1} {"a":1} 0 0 0 0
+4 "invalid" "invalid" 99 99 99 99
+5 [1,2] [1,2] 99 99 99 99
+
+-- !alias_null_predicate --
+1 \N 99
+4 \N 99
+5 \N 99
+
+-- !all_null_filter --
+1
+2
+
+-- !all_not_null_filter --
+
+-- !all_null_roots --
+1 \N true true fallback fallback 99 99
+2 \N true true fallback fallback 99 99
+
diff --git
a/regression-test/suites/variant_p0/test_variant_cast_isnull_pushdown.groovy
b/regression-test/suites/variant_p0/test_variant_cast_isnull_pushdown.groovy
new file mode 100644
index 00000000000..20287c01ac8
--- /dev/null
+++ b/regression-test/suites/variant_p0/test_variant_cast_isnull_pushdown.groovy
@@ -0,0 +1,42 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_variant_cast_isnull_pushdown", "p0") {
+ // A Variant cast can turn a non-NULL Variant value into NULL, so `CAST(v
AS INT) IS NULL`
+ // is not equivalent to `v IS NULL`. The scan node must keep this null
test as an expression
+ // instead of pushing a null predicate down on the source Variant column.
+ sql "DROP TABLE IF EXISTS variant_cast_isnull_pushdown"
+ sql """
+ CREATE TABLE variant_cast_isnull_pushdown (
+ id INT NOT NULL,
+ v VARIANT NOT NULL
+ ) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES("replication_num" = "1")
+ """
+ sql """
+ INSERT INTO variant_cast_isnull_pushdown VALUES
+ (1, '1'),
+ (2, '"x"'),
+ (3, '[1,2]')
+ """
+
+ order_qt_cast_int "SELECT id, CAST(v AS INT) FROM
variant_cast_isnull_pushdown ORDER BY id"
+ order_qt_is_null "SELECT id FROM variant_cast_isnull_pushdown WHERE CAST(v
AS INT) IS NULL ORDER BY id"
+ order_qt_is_not_null "SELECT id FROM variant_cast_isnull_pushdown WHERE
CAST(v AS INT) IS NOT NULL ORDER BY id"
+ order_qt_null_safe "SELECT id FROM variant_cast_isnull_pushdown WHERE
CAST(v AS INT) <=> NULL ORDER BY id"
+}
diff --git
a/regression-test/suites/variant_p0/test_variant_cast_nullable.groovy
b/regression-test/suites/variant_p0/test_variant_cast_nullable.groovy
new file mode 100644
index 00000000000..affed0bb9ec
--- /dev/null
+++ b/regression-test/suites/variant_p0/test_variant_cast_nullable.groovy
@@ -0,0 +1,98 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_variant_cast_nullable", "p0") {
+ // This regression covers legacy Variant conversion results; V2 has
different conversion semantics.
+ if (getFeConfig("enable_variant_v2").toBoolean()) {
+ return
+ }
+ sql "DROP TABLE IF EXISTS variant_cast_nullable"
+ sql """
+ CREATE TABLE variant_cast_nullable (
+ id INT NOT NULL,
+ v VARIANT NOT NULL,
+ nullable_v VARIANT NULL
+ ) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES("replication_num" = "1")
+ """
+ sql """
+ INSERT INTO variant_cast_nullable VALUES
+ (1, 'null', NULL),
+ (2, '42', '42'),
+ (3, '{"a":1}', '{"a":1}'),
+ (4, '"invalid"', '"invalid"'),
+ (5, '[1,2]', '[1,2]')
+ """
+
+ qt_casts """
+ SELECT id, CAST(v AS STRING), CAST(v AS INT), CAST(nullable_v AS INT)
+ FROM variant_cast_nullable ORDER BY id
+ """
+ qt_null_predicates """
+ SELECT id, CAST(v AS STRING) IS NULL, CAST(v AS STRING) <=> NULL,
+ CAST(v AS INT) IS NULL, CAST(v AS INT) <=> NULL,
+ CAST(v AS VARIANT) IS NULL
+ FROM variant_cast_nullable ORDER BY id
+ """
+ qt_is_null_filter """
+ SELECT id FROM variant_cast_nullable WHERE CAST(v AS STRING) IS NULL
ORDER BY id
+ """
+ qt_is_not_null_filter """
+ SELECT id FROM variant_cast_nullable WHERE CAST(v AS INT) IS NOT NULL
ORDER BY id
+ """
+ qt_nullable_input_filter """
+ SELECT id FROM variant_cast_nullable WHERE CAST(nullable_v AS INT) IS
NULL ORDER BY id
+ """
+ qt_null_safe_filter """
+ SELECT id FROM variant_cast_nullable WHERE CAST(v AS INT) <=> NULL
ORDER BY id
+ """
+ qt_fallbacks """
+ SELECT id, COALESCE(CAST(v AS STRING), 'fallback'), NVL(CAST(v AS
STRING), 'fallback'),
+ COALESCE(CAST(v AS INT), 99), NVL(CAST(v AS INT), 99),
+ COALESCE(CAST(nullable_v AS INT), 99), NVL(CAST(nullable_v AS
INT), 99)
+ FROM variant_cast_nullable ORDER BY id
+ """
+ qt_alias_null_predicate """
+ WITH casts AS (SELECT id, CAST(v AS INT) AS value FROM
variant_cast_nullable)
+ SELECT id, value, COALESCE(value, 99) FROM casts WHERE value IS NULL
ORDER BY id
+ """
+
+ // Keep an all-Variant-null root to exercise the legacy all-default-values
cast path.
+ sql "DROP TABLE IF EXISTS variant_cast_nullable_nulls"
+ sql """
+ CREATE TABLE variant_cast_nullable_nulls (
+ id INT NOT NULL,
+ v VARIANT NOT NULL
+ ) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES("replication_num" = "1")
+ """
+ sql "INSERT INTO variant_cast_nullable_nulls VALUES (1, 'null'), (2,
'null')"
+ qt_all_null_filter """
+ SELECT id FROM variant_cast_nullable_nulls WHERE CAST(v AS STRING) IS
NULL ORDER BY id
+ """
+ qt_all_not_null_filter """
+ SELECT id FROM variant_cast_nullable_nulls WHERE CAST(v AS STRING) IS
NOT NULL ORDER BY id
+ """
+ qt_all_null_roots """
+ SELECT id, CAST(v AS STRING), CAST(v AS STRING) IS NULL, CAST(v AS
STRING) <=> NULL,
+ COALESCE(CAST(v AS STRING), 'fallback'), NVL(CAST(v AS STRING),
'fallback'),
+ COALESCE(CAST(v AS INT), 99), NVL(CAST(v AS INT), 99)
+ FROM variant_cast_nullable_nulls ORDER BY id
+ """
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]