This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-4.2 by this push:
     new 71d452d628c branch-4.2: [fix](be) Preserve shared columns in distinct 
streaming pass-through (#68880) (#70533)
71d452d628c is described below

commit 71d452d628c4c4f87d0325da4a1edc435b45323b
Author: TengJianPing <[email protected]>
AuthorDate: Fri Nov 20 09:23:16 2026 +0800

    branch-4.2: [fix](be) Preserve shared columns in distinct streaming 
pass-through (#68880) (#70533)
    
    ### What problem does this PR solve?
    
    Issue Number: N/A
    
    Related PR: #68880
    
    Backport of #68880 to `branch-4.2`, cherry-picked from
    `0f987b6d24314a28a968449c718a008c8709a0fd` with source attribution
    (`-x`). This branch is based directly on `apache/doris:branch-4.2` and
    is independent of the other backports.
    
    Repeated grouping keys can refer to the same input column. On 4.2,
    moving a column out of the input block while constructing pass-through
    output can leave later references without the original column. Share the
    evaluated immutable key columns in both pass-through paths, preserving
    repeated keys and retained output ownership.
    
    ### Branch-4.2 adaptations
    
    Applied the shared-column fix to the 4.2 IColumn::mutate/move
    implementation. Added only the repeated-key tests from this PR, without
    importing unrelated master-only memory-limit tests. Removed the
    regression setup for the master-only distinct memory-budget behavior:
    4.2 uses the reduction ratio/low-memory mode, and the unit tests force
    pass-through explicitly.
    
    ### Release note
    
    Fix incorrect handling of repeated grouping keys in distinct streaming
    aggregation pass-through.
    
    ### Validation
    
    - Reviewed both fresh-output and reused-output paths against the 4.2
    column ownership behavior.
    - Verified that the added parameterized unit tests cover nullable
    output, retained output ownership, repeated keys, and transition from
    deduplication.
    - Whitespace/conflict-marker checks passed (allowing the standard
    trailing blank line in regression `.out` files).
    - Exactly one backport commit on top of
    `7523405f6aa8e9df3d110eb432fed1ff6e1c3418`; clean worktree.
    - Full builds, unit tests and SQL regression suites were not run for
    this backport on this macOS host. The local environment lacks the
    required JDK 17/full Doris build dependencies; the offline Maven attempt
    could not resolve the required BOMs. Submodules remain uninitialized.
    
    BE repeated-key unit tests and EXCEPT/INTERSECT/UNION/GROUP BY
    regression coverage from the original PR are included.
    
    ### Check List (For Author)
    
    - Test: existing upstream unit/regression coverage included; local
    checks and execution limits are listed above.
    - Behavior changed: Yes; restores correct behavior as described above.
    - Does this need documentation: No.
    
    ### Check List (For Reviewer who merge this PR)
    
    - [ ] Confirm the release note
    - [ ] Confirm test cases
    - [ ] Confirm document
    - [ ] Add branch pick label
---
 .../distinct_streaming_aggregation_operator.cpp    |  15 +--
 ...istinct_streaming_aggregation_operator_test.cpp |  68 ++++++++++
 .../distinct_streaming_repeated_keys.out           |  13 ++
 .../distinct_streaming_repeated_keys.groovy        | 146 +++++++++++++++++++++
 4 files changed, 233 insertions(+), 9 deletions(-)

diff --git a/be/src/exec/operator/distinct_streaming_aggregation_operator.cpp 
b/be/src/exec/operator/distinct_streaming_aggregation_operator.cpp
index b89f00fdd5a..da2d28deada 100644
--- a/be/src/exec/operator/distinct_streaming_aggregation_operator.cpp
+++ b/be/src/exec/operator/distinct_streaming_aggregation_operator.cpp
@@ -205,12 +205,11 @@ Status 
DistinctStreamingAggLocalState::_distinct_pre_agg_with_serialized_key(
         DCHECK_EQ(out_block->columns(), key_size);
         if (_stop_emplace_flag && _distinct_row.empty()) {
             // If _stop_emplace_flag is true and _distinct_row is also empty, 
it means it is in streaming mode, outputting what is input
-            // swap the column directly, to solve Check failed: 
d.column->use_count() == 1 (2 vs. 1)
+            // Share the evaluated keys as immutable columns. A repeated input 
position may
+            // already have been replaced, but its column is retained by the 
earlier output key.
             for (int i = 0; i < key_size; ++i) {
                 auto output_column = out_block->get_by_position(i).column;
-                auto input_column = IColumn::mutate(
-                        
std::move(in_block->get_by_position(result_idxs[i]).column));
-                out_block->replace_by_position(i, std::move(input_column));
+                out_block->replace_by_position(i, key_columns[i]->get_ptr());
                 in_block->replace_by_position(result_idxs[i], output_column);
             }
         } else {
@@ -243,11 +242,9 @@ Status 
DistinctStreamingAggLocalState::_distinct_pre_agg_with_serialized_key(
         ColumnsWithTypeAndName columns_with_schema;
         for (int i = 0; i < key_size; ++i) {
             if (_stop_emplace_flag) {
-                columns_with_schema.emplace_back(
-                        IColumn::mutate(
-                                
std::move(in_block->get_by_position(result_idxs[i]).column)),
-                        _probe_expr_ctxs[i]->root()->data_type(),
-                        _probe_expr_ctxs[i]->root()->expr_name());
+                columns_with_schema.emplace_back(key_columns[i]->get_ptr(),
+                                                 
_probe_expr_ctxs[i]->root()->data_type(),
+                                                 
_probe_expr_ctxs[i]->root()->expr_name());
             } else {
                 auto distinct_column = key_columns[i]->clone_empty();
                 key_columns[i]->append_data_by_selector(distinct_column, 
_distinct_row);
diff --git 
a/be/test/exec/operator/distinct_streaming_aggregation_operator_test.cpp 
b/be/test/exec/operator/distinct_streaming_aggregation_operator_test.cpp
index 17282356625..11296ade7ae 100644
--- a/be/test/exec/operator/distinct_streaming_aggregation_operator_test.cpp
+++ b/be/test/exec/operator/distinct_streaming_aggregation_operator_test.cpp
@@ -209,4 +209,72 @@ TEST_F(DistinctStreamingAggOperatorTest, test3) {
     { EXPECT_TRUE(op->close(state.get())); }
 }
 
+struct DistinctStreamingAggRepeatedKeyTest : public 
DistinctStreamingAggOperatorTest,
+                                             public 
::testing::WithParamInterface<bool> {
+    void SetUp() override {
+        DistinctStreamingAggOperatorTest::SetUp();
+        op->_is_streaming_preagg = true;
+        auto input_type = std::make_shared<DataTypeInt64>();
+        auto output_type = GetParam() ? make_nullable(input_type) : input_type;
+        create_op({input_type, input_type, input_type, input_type},
+                  {output_type, output_type, output_type, output_type});
+        local_state->_probe_expr_ctxs = {MockSlotRef::create_mock_context(0, 
input_type),
+                                         MockSlotRef::create_mock_context(1, 
input_type),
+                                         MockSlotRef::create_mock_context(0, 
input_type),
+                                         MockSlotRef::create_mock_context(0, 
input_type)};
+    }
+
+    Block expected_block(const std::vector<Int64>& first, const 
std::vector<Int64>& second) {
+        auto block = ColumnHelper::create_block<DataTypeInt64>(first, second);
+        block.insert(block.get_by_position(0));
+        block.insert(block.get_by_position(0));
+        if (GetParam()) {
+            for (size_t i = 0; i < block.columns(); ++i) {
+                auto& column = block.get_by_position(i);
+                column.column = make_nullable(column.column);
+                column.type = make_nullable(column.type);
+            }
+        }
+        return block;
+    }
+};
+
+TEST_P(DistinctStreamingAggRepeatedKeyTest, pass_through) {
+    local_state->_stop_emplace_flag = true;
+    auto input = ColumnHelper::create_block<DataTypeInt64>({1, 1, 2}, {10, 10, 
20});
+    ASSERT_TRUE(op->push(state.get(), &input, false));
+
+    auto output = expected_block({}, {});
+    bool eos = false;
+    ASSERT_TRUE(op->pull(state.get(), &output, &eos));
+    EXPECT_TRUE(ColumnHelper::block_equal(output, expected_block({1, 1, 2}, 
{10, 10, 20})));
+
+    // The output schema is now reused. Keep the previous output alive to 
check its ownership.
+    input = ColumnHelper::create_block<DataTypeInt64>({3, 4, 4}, {30, 40, 40});
+    ASSERT_TRUE(op->push(state.get(), &input, false));
+    auto next_output = expected_block({}, {});
+    ASSERT_TRUE(op->pull(state.get(), &next_output, &eos));
+    EXPECT_TRUE(ColumnHelper::block_equal(next_output, expected_block({3, 4, 
4}, {30, 40, 40})));
+    EXPECT_TRUE(ColumnHelper::block_equal(output, expected_block({1, 1, 2}, 
{10, 10, 20})));
+}
+
+TEST_P(DistinctStreamingAggRepeatedKeyTest, pass_through_after_deduplication) {
+    auto input = ColumnHelper::create_block<DataTypeInt64>({1, 1, 2}, {10, 10, 
20});
+    ASSERT_TRUE(op->push(state.get(), &input, false));
+    EXPECT_EQ(local_state->_aggregated_block->rows(), 2);
+    auto deduplicated_output = expected_block({}, {});
+    bool eos = false;
+    ASSERT_TRUE(op->pull(state.get(), &deduplicated_output, &eos));
+    EXPECT_TRUE(ColumnHelper::block_equal(deduplicated_output, 
expected_block({1, 2}, {10, 20})));
+
+    local_state->_stop_emplace_flag = true;
+    input = ColumnHelper::create_block<DataTypeInt64>({3, 3, 4}, {30, 30, 40});
+    ASSERT_TRUE(op->push(state.get(), &input, false));
+    auto output = expected_block({}, {});
+    ASSERT_TRUE(op->pull(state.get(), &output, &eos));
+    EXPECT_TRUE(ColumnHelper::block_equal(output, expected_block({3, 3, 4}, 
{30, 30, 40})));
+}
+
+INSTANTIATE_TEST_SUITE_P(NullableOutput, DistinctStreamingAggRepeatedKeyTest, 
::testing::Bool());
+
 } // namespace doris
diff --git 
a/regression-test/data/query_p0/set_operations/distinct_streaming_repeated_keys.out
 
b/regression-test/data/query_p0/set_operations/distinct_streaming_repeated_keys.out
new file mode 100644
index 00000000000..c61697af12f
--- /dev/null
+++ 
b/regression-test/data/query_p0/set_operations/distinct_streaming_repeated_keys.out
@@ -0,0 +1,13 @@
+-- This file is automatically generated. You should know what you did if you 
want to edit this
+-- !except --
+1638   2026-10-01      2026-10-01
+
+-- !except_nullable --
+2457   2457    2457    2026-10-01      2026-10-01
+
+-- !intersect --
+1639   2026-10-01      2026-10-01
+
+-- !union --
+4915   2026-10-01      2026-10-01
+
diff --git 
a/regression-test/suites/query_p0/set_operations/distinct_streaming_repeated_keys.groovy
 
b/regression-test/suites/query_p0/set_operations/distinct_streaming_repeated_keys.groovy
new file mode 100644
index 00000000000..d3b7a2b68dc
--- /dev/null
+++ 
b/regression-test/suites/query_p0/set_operations/distinct_streaming_repeated_keys.groovy
@@ -0,0 +1,146 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("distinct_streaming_repeated_keys", "query,p0") {
+    sql "DROP TABLE IF EXISTS distinct_streaming_repeated_keys"
+    sql """
+        CREATE TABLE distinct_streaming_repeated_keys (
+            id INT NOT NULL,
+            s STRING NULL,
+            d DATE NOT NULL
+        ) DISTRIBUTED BY HASH(id) BUCKETS 1
+        PROPERTIES ("replication_num" = "1")
+    """
+    sql """
+        INSERT INTO distinct_streaming_repeated_keys
+        SELECT number, IF(number % 5 = 0, NULL, CAST(number AS STRING)), DATE 
'2026-10-01'
+        FROM numbers("number" = "8192")
+    """
+
+    // Favor local distinct under the set operation, with a shuffle because id 
is not a key.
+    sql """
+        ALTER TABLE distinct_streaming_repeated_keys MODIFY COLUMN s
+        SET STATS ('row_count' = '8192', 'ndv' = '1', 'num_nulls' = '1639',
+                   'min_value' = '0', 'max_value' = '8191')
+    """
+    sql """
+        ALTER TABLE distinct_streaming_repeated_keys MODIFY COLUMN d
+        SET STATS ('row_count' = '8192', 'ndv' = '1', 'num_nulls' = '0',
+                   'min_value' = '2026-10-01', 'max_value' = '2026-10-01')
+    """
+    sql "SET agg_phase = 2"
+    sql "SET disable_streaming_preaggregations = false"
+    sql "SET enable_distinct_streaming_aggregation = true"
+    sql "SET parallel_pipeline_task_num = 1"
+    sql "SET batch_size = 256"
+    // Branch-4.2 uses the reduction ratio to switch distinct streaming to 
pass-through.
+    // Unit tests force pass-through to cover both output paths 
deterministically.
+
+    explain {
+        sql """
+            SELECT s, d, d FROM distinct_streaming_repeated_keys WHERE id < 
4096
+            EXCEPT DISTINCT
+            SELECT s, d, d FROM distinct_streaming_repeated_keys WHERE id % 2 
= 0
+        """
+        contains "STREAMING"
+    }
+
+    order_qt_except """
+        SELECT COUNT(*), MIN(d1), MAX(d2) FROM (
+            SELECT s, d AS d1, d AS d2
+            FROM distinct_streaming_repeated_keys WHERE id < 4096
+            EXCEPT DISTINCT
+            SELECT s, d, d FROM distinct_streaming_repeated_keys WHERE id % 2 
= 0
+        ) t
+    """
+
+    // The first date key becomes nullable, exercising the output path without 
memory reuse.
+    order_qt_except_nullable """
+        SELECT COUNT(*), COUNT(d1), COUNT(d2), MIN(d1), MAX(d2) FROM (
+            SELECT s, d AS d1, d AS d2
+            FROM distinct_streaming_repeated_keys WHERE id < 4096
+            EXCEPT DISTINCT
+            SELECT s, IF(id % 4 = 0, NULL, d), d
+            FROM distinct_streaming_repeated_keys WHERE id % 2 = 0
+        ) t
+    """
+
+    order_qt_intersect """
+        SELECT COUNT(*), MIN(d1), MAX(d2) FROM (
+            SELECT s, d AS d1, d AS d2
+            FROM distinct_streaming_repeated_keys WHERE id < 4096
+            INTERSECT DISTINCT
+            SELECT s, d, d FROM distinct_streaming_repeated_keys WHERE id % 2 
= 0
+        ) t
+    """
+
+    order_qt_union """
+        SELECT COUNT(*), MIN(d1), MAX(d2) FROM (
+            SELECT s, d AS d1, d AS d2
+            FROM distinct_streaming_repeated_keys WHERE id < 4096
+            UNION DISTINCT
+            SELECT s, d, d FROM distinct_streaming_repeated_keys WHERE id % 2 
= 0
+        ) t
+    """
+
+    // NormalizeAggregate removes repeated keys in a direct GROUP BY. Keep 
this as
+    // a control case, rather than assuming that repeated SQL text repeats 
input slots.
+    check_sqls_result_equal("""
+        SELECT s, d, d, d FROM distinct_streaming_repeated_keys
+        GROUP BY s, d, d, d
+        ORDER BY 1, 2, 3, 4
+    """, """
+        SELECT s, d, d, d FROM distinct_streaming_repeated_keys
+        GROUP BY s, d
+        ORDER BY 1, 2, 3, 4
+    """)
+
+    // PushDownLimitDistinctThroughUnion maps distinct union output slots back 
to
+    // the same branch slot after aggregate normalization. Three references to 
d
+    // exercise shared ownership even when the output Block is reused.
+    // Use ordinals to group all four union columns despite their repeated 
names.
+    // The limit covers every input row, so it cannot make the result 
nondeterministic.
+    def groupByUnionAllLimit = """
+        SELECT * FROM (
+            SELECT * FROM (
+                SELECT s, d, d, d FROM distinct_streaming_repeated_keys WHERE 
id < 4096
+                UNION ALL
+                SELECT s, d, d, d FROM distinct_streaming_repeated_keys WHERE 
id % 2 = 0
+            ) u
+            GROUP BY 1, 2, 3, 4
+            LIMIT 8192
+        ) grouped
+        ORDER BY 1, 2, 3, 4
+    """
+
+    explain {
+        sql groupByUnionAllLimit
+        verbose true
+        check { plan ->
+            // Require three references to one input slot in a streaming 
aggregate,
+            // not merely different slots projecting identical column values.
+            (plan =~ /STREAMING\s*\|\s*group by: s\[#\d+\], d\[#(\d+)\], 
d\[#\1\], d\[#\1\]/).find()
+        }
+    }
+
+    check_sqls_result_equal(groupByUnionAllLimit, """
+        SELECT s, d, d, d FROM distinct_streaming_repeated_keys
+        WHERE id < 4096 OR id % 2 = 0
+        GROUP BY s, d
+        ORDER BY 1, 2, 3, 4
+    """)
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to