This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 71d452d628c branch-4.2: [fix](be) Preserve shared columns in distinct
streaming pass-through (#68880) (#70533)
71d452d628c is described below
commit 71d452d628c4c4f87d0325da4a1edc435b45323b
Author: TengJianPing <[email protected]>
AuthorDate: Fri Nov 20 09:23:16 2026 +0800
branch-4.2: [fix](be) Preserve shared columns in distinct streaming
pass-through (#68880) (#70533)
### What problem does this PR solve?
Issue Number: N/A
Related PR: #68880
Backport of #68880 to `branch-4.2`, cherry-picked from
`0f987b6d24314a28a968449c718a008c8709a0fd` with source attribution
(`-x`). This branch is based directly on `apache/doris:branch-4.2` and
is independent of the other backports.
Repeated grouping keys can refer to the same input column. On 4.2,
moving a column out of the input block while constructing pass-through
output can leave later references without the original column. Share the
evaluated immutable key columns in both pass-through paths, preserving
repeated keys and retained output ownership.
### Branch-4.2 adaptations
Applied the shared-column fix to the 4.2 IColumn::mutate/move
implementation. Added only the repeated-key tests from this PR, without
importing unrelated master-only memory-limit tests. Removed the
regression setup for the master-only distinct memory-budget behavior:
4.2 uses the reduction ratio/low-memory mode, and the unit tests force
pass-through explicitly.
### Release note
Fix incorrect handling of repeated grouping keys in distinct streaming
aggregation pass-through.
### Validation
- Reviewed both fresh-output and reused-output paths against the 4.2
column ownership behavior.
- Verified that the added parameterized unit tests cover nullable
output, retained output ownership, repeated keys, and transition from
deduplication.
- Whitespace/conflict-marker checks passed (allowing the standard
trailing blank line in regression `.out` files).
- Exactly one backport commit on top of
`7523405f6aa8e9df3d110eb432fed1ff6e1c3418`; clean worktree.
- Full builds, unit tests and SQL regression suites were not run for
this backport on this macOS host. The local environment lacks the
required JDK 17/full Doris build dependencies; the offline Maven attempt
could not resolve the required BOMs. Submodules remain uninitialized.
BE repeated-key unit tests and EXCEPT/INTERSECT/UNION/GROUP BY
regression coverage from the original PR are included.
### Check List (For Author)
- Test: existing upstream unit/regression coverage included; local
checks and execution limits are listed above.
- Behavior changed: Yes; restores correct behavior as described above.
- Does this need documentation: No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---
.../distinct_streaming_aggregation_operator.cpp | 15 +--
...istinct_streaming_aggregation_operator_test.cpp | 68 ++++++++++
.../distinct_streaming_repeated_keys.out | 13 ++
.../distinct_streaming_repeated_keys.groovy | 146 +++++++++++++++++++++
4 files changed, 233 insertions(+), 9 deletions(-)
diff --git a/be/src/exec/operator/distinct_streaming_aggregation_operator.cpp
b/be/src/exec/operator/distinct_streaming_aggregation_operator.cpp
index b89f00fdd5a..da2d28deada 100644
--- a/be/src/exec/operator/distinct_streaming_aggregation_operator.cpp
+++ b/be/src/exec/operator/distinct_streaming_aggregation_operator.cpp
@@ -205,12 +205,11 @@ Status
DistinctStreamingAggLocalState::_distinct_pre_agg_with_serialized_key(
DCHECK_EQ(out_block->columns(), key_size);
if (_stop_emplace_flag && _distinct_row.empty()) {
// If _stop_emplace_flag is true and _distinct_row is also empty,
it means it is in streaming mode, outputting what is input
- // swap the column directly, to solve Check failed:
d.column->use_count() == 1 (2 vs. 1)
+ // Share the evaluated keys as immutable columns. A repeated input
position may
+ // already have been replaced, but its column is retained by the
earlier output key.
for (int i = 0; i < key_size; ++i) {
auto output_column = out_block->get_by_position(i).column;
- auto input_column = IColumn::mutate(
-
std::move(in_block->get_by_position(result_idxs[i]).column));
- out_block->replace_by_position(i, std::move(input_column));
+ out_block->replace_by_position(i, key_columns[i]->get_ptr());
in_block->replace_by_position(result_idxs[i], output_column);
}
} else {
@@ -243,11 +242,9 @@ Status
DistinctStreamingAggLocalState::_distinct_pre_agg_with_serialized_key(
ColumnsWithTypeAndName columns_with_schema;
for (int i = 0; i < key_size; ++i) {
if (_stop_emplace_flag) {
- columns_with_schema.emplace_back(
- IColumn::mutate(
-
std::move(in_block->get_by_position(result_idxs[i]).column)),
- _probe_expr_ctxs[i]->root()->data_type(),
- _probe_expr_ctxs[i]->root()->expr_name());
+ columns_with_schema.emplace_back(key_columns[i]->get_ptr(),
+
_probe_expr_ctxs[i]->root()->data_type(),
+
_probe_expr_ctxs[i]->root()->expr_name());
} else {
auto distinct_column = key_columns[i]->clone_empty();
key_columns[i]->append_data_by_selector(distinct_column,
_distinct_row);
diff --git
a/be/test/exec/operator/distinct_streaming_aggregation_operator_test.cpp
b/be/test/exec/operator/distinct_streaming_aggregation_operator_test.cpp
index 17282356625..11296ade7ae 100644
--- a/be/test/exec/operator/distinct_streaming_aggregation_operator_test.cpp
+++ b/be/test/exec/operator/distinct_streaming_aggregation_operator_test.cpp
@@ -209,4 +209,72 @@ TEST_F(DistinctStreamingAggOperatorTest, test3) {
{ EXPECT_TRUE(op->close(state.get())); }
}
+struct DistinctStreamingAggRepeatedKeyTest : public
DistinctStreamingAggOperatorTest,
+ public
::testing::WithParamInterface<bool> {
+ void SetUp() override {
+ DistinctStreamingAggOperatorTest::SetUp();
+ op->_is_streaming_preagg = true;
+ auto input_type = std::make_shared<DataTypeInt64>();
+ auto output_type = GetParam() ? make_nullable(input_type) : input_type;
+ create_op({input_type, input_type, input_type, input_type},
+ {output_type, output_type, output_type, output_type});
+ local_state->_probe_expr_ctxs = {MockSlotRef::create_mock_context(0,
input_type),
+ MockSlotRef::create_mock_context(1,
input_type),
+ MockSlotRef::create_mock_context(0,
input_type),
+ MockSlotRef::create_mock_context(0,
input_type)};
+ }
+
+ Block expected_block(const std::vector<Int64>& first, const
std::vector<Int64>& second) {
+ auto block = ColumnHelper::create_block<DataTypeInt64>(first, second);
+ block.insert(block.get_by_position(0));
+ block.insert(block.get_by_position(0));
+ if (GetParam()) {
+ for (size_t i = 0; i < block.columns(); ++i) {
+ auto& column = block.get_by_position(i);
+ column.column = make_nullable(column.column);
+ column.type = make_nullable(column.type);
+ }
+ }
+ return block;
+ }
+};
+
+TEST_P(DistinctStreamingAggRepeatedKeyTest, pass_through) {
+ local_state->_stop_emplace_flag = true;
+ auto input = ColumnHelper::create_block<DataTypeInt64>({1, 1, 2}, {10, 10,
20});
+ ASSERT_TRUE(op->push(state.get(), &input, false));
+
+ auto output = expected_block({}, {});
+ bool eos = false;
+ ASSERT_TRUE(op->pull(state.get(), &output, &eos));
+ EXPECT_TRUE(ColumnHelper::block_equal(output, expected_block({1, 1, 2},
{10, 10, 20})));
+
+ // The output schema is now reused. Keep the previous output alive to
check its ownership.
+ input = ColumnHelper::create_block<DataTypeInt64>({3, 4, 4}, {30, 40, 40});
+ ASSERT_TRUE(op->push(state.get(), &input, false));
+ auto next_output = expected_block({}, {});
+ ASSERT_TRUE(op->pull(state.get(), &next_output, &eos));
+ EXPECT_TRUE(ColumnHelper::block_equal(next_output, expected_block({3, 4,
4}, {30, 40, 40})));
+ EXPECT_TRUE(ColumnHelper::block_equal(output, expected_block({1, 1, 2},
{10, 10, 20})));
+}
+
+TEST_P(DistinctStreamingAggRepeatedKeyTest, pass_through_after_deduplication) {
+ auto input = ColumnHelper::create_block<DataTypeInt64>({1, 1, 2}, {10, 10,
20});
+ ASSERT_TRUE(op->push(state.get(), &input, false));
+ EXPECT_EQ(local_state->_aggregated_block->rows(), 2);
+ auto deduplicated_output = expected_block({}, {});
+ bool eos = false;
+ ASSERT_TRUE(op->pull(state.get(), &deduplicated_output, &eos));
+ EXPECT_TRUE(ColumnHelper::block_equal(deduplicated_output,
expected_block({1, 2}, {10, 20})));
+
+ local_state->_stop_emplace_flag = true;
+ input = ColumnHelper::create_block<DataTypeInt64>({3, 3, 4}, {30, 30, 40});
+ ASSERT_TRUE(op->push(state.get(), &input, false));
+ auto output = expected_block({}, {});
+ ASSERT_TRUE(op->pull(state.get(), &output, &eos));
+ EXPECT_TRUE(ColumnHelper::block_equal(output, expected_block({3, 3, 4},
{30, 30, 40})));
+}
+
+INSTANTIATE_TEST_SUITE_P(NullableOutput, DistinctStreamingAggRepeatedKeyTest,
::testing::Bool());
+
} // namespace doris
diff --git
a/regression-test/data/query_p0/set_operations/distinct_streaming_repeated_keys.out
b/regression-test/data/query_p0/set_operations/distinct_streaming_repeated_keys.out
new file mode 100644
index 00000000000..c61697af12f
--- /dev/null
+++
b/regression-test/data/query_p0/set_operations/distinct_streaming_repeated_keys.out
@@ -0,0 +1,13 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !except --
+1638 2026-10-01 2026-10-01
+
+-- !except_nullable --
+2457 2457 2457 2026-10-01 2026-10-01
+
+-- !intersect --
+1639 2026-10-01 2026-10-01
+
+-- !union --
+4915 2026-10-01 2026-10-01
+
diff --git
a/regression-test/suites/query_p0/set_operations/distinct_streaming_repeated_keys.groovy
b/regression-test/suites/query_p0/set_operations/distinct_streaming_repeated_keys.groovy
new file mode 100644
index 00000000000..d3b7a2b68dc
--- /dev/null
+++
b/regression-test/suites/query_p0/set_operations/distinct_streaming_repeated_keys.groovy
@@ -0,0 +1,146 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("distinct_streaming_repeated_keys", "query,p0") {
+ sql "DROP TABLE IF EXISTS distinct_streaming_repeated_keys"
+ sql """
+ CREATE TABLE distinct_streaming_repeated_keys (
+ id INT NOT NULL,
+ s STRING NULL,
+ d DATE NOT NULL
+ ) DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_num" = "1")
+ """
+ sql """
+ INSERT INTO distinct_streaming_repeated_keys
+ SELECT number, IF(number % 5 = 0, NULL, CAST(number AS STRING)), DATE
'2026-10-01'
+ FROM numbers("number" = "8192")
+ """
+
+ // Favor local distinct under the set operation, with a shuffle because id
is not a key.
+ sql """
+ ALTER TABLE distinct_streaming_repeated_keys MODIFY COLUMN s
+ SET STATS ('row_count' = '8192', 'ndv' = '1', 'num_nulls' = '1639',
+ 'min_value' = '0', 'max_value' = '8191')
+ """
+ sql """
+ ALTER TABLE distinct_streaming_repeated_keys MODIFY COLUMN d
+ SET STATS ('row_count' = '8192', 'ndv' = '1', 'num_nulls' = '0',
+ 'min_value' = '2026-10-01', 'max_value' = '2026-10-01')
+ """
+ sql "SET agg_phase = 2"
+ sql "SET disable_streaming_preaggregations = false"
+ sql "SET enable_distinct_streaming_aggregation = true"
+ sql "SET parallel_pipeline_task_num = 1"
+ sql "SET batch_size = 256"
+ // Branch-4.2 uses the reduction ratio to switch distinct streaming to
pass-through.
+ // Unit tests force pass-through to cover both output paths
deterministically.
+
+ explain {
+ sql """
+ SELECT s, d, d FROM distinct_streaming_repeated_keys WHERE id <
4096
+ EXCEPT DISTINCT
+ SELECT s, d, d FROM distinct_streaming_repeated_keys WHERE id % 2
= 0
+ """
+ contains "STREAMING"
+ }
+
+ order_qt_except """
+ SELECT COUNT(*), MIN(d1), MAX(d2) FROM (
+ SELECT s, d AS d1, d AS d2
+ FROM distinct_streaming_repeated_keys WHERE id < 4096
+ EXCEPT DISTINCT
+ SELECT s, d, d FROM distinct_streaming_repeated_keys WHERE id % 2
= 0
+ ) t
+ """
+
+ // The first date key becomes nullable, exercising the output path without
memory reuse.
+ order_qt_except_nullable """
+ SELECT COUNT(*), COUNT(d1), COUNT(d2), MIN(d1), MAX(d2) FROM (
+ SELECT s, d AS d1, d AS d2
+ FROM distinct_streaming_repeated_keys WHERE id < 4096
+ EXCEPT DISTINCT
+ SELECT s, IF(id % 4 = 0, NULL, d), d
+ FROM distinct_streaming_repeated_keys WHERE id % 2 = 0
+ ) t
+ """
+
+ order_qt_intersect """
+ SELECT COUNT(*), MIN(d1), MAX(d2) FROM (
+ SELECT s, d AS d1, d AS d2
+ FROM distinct_streaming_repeated_keys WHERE id < 4096
+ INTERSECT DISTINCT
+ SELECT s, d, d FROM distinct_streaming_repeated_keys WHERE id % 2
= 0
+ ) t
+ """
+
+ order_qt_union """
+ SELECT COUNT(*), MIN(d1), MAX(d2) FROM (
+ SELECT s, d AS d1, d AS d2
+ FROM distinct_streaming_repeated_keys WHERE id < 4096
+ UNION DISTINCT
+ SELECT s, d, d FROM distinct_streaming_repeated_keys WHERE id % 2
= 0
+ ) t
+ """
+
+ // NormalizeAggregate removes repeated keys in a direct GROUP BY. Keep
this as
+ // a control case, rather than assuming that repeated SQL text repeats
input slots.
+ check_sqls_result_equal("""
+ SELECT s, d, d, d FROM distinct_streaming_repeated_keys
+ GROUP BY s, d, d, d
+ ORDER BY 1, 2, 3, 4
+ """, """
+ SELECT s, d, d, d FROM distinct_streaming_repeated_keys
+ GROUP BY s, d
+ ORDER BY 1, 2, 3, 4
+ """)
+
+ // PushDownLimitDistinctThroughUnion maps distinct union output slots back
to
+ // the same branch slot after aggregate normalization. Three references to
d
+ // exercise shared ownership even when the output Block is reused.
+ // Use ordinals to group all four union columns despite their repeated
names.
+ // The limit covers every input row, so it cannot make the result
nondeterministic.
+ def groupByUnionAllLimit = """
+ SELECT * FROM (
+ SELECT * FROM (
+ SELECT s, d, d, d FROM distinct_streaming_repeated_keys WHERE
id < 4096
+ UNION ALL
+ SELECT s, d, d, d FROM distinct_streaming_repeated_keys WHERE
id % 2 = 0
+ ) u
+ GROUP BY 1, 2, 3, 4
+ LIMIT 8192
+ ) grouped
+ ORDER BY 1, 2, 3, 4
+ """
+
+ explain {
+ sql groupByUnionAllLimit
+ verbose true
+ check { plan ->
+ // Require three references to one input slot in a streaming
aggregate,
+ // not merely different slots projecting identical column values.
+ (plan =~ /STREAMING\s*\|\s*group by: s\[#\d+\], d\[#(\d+)\],
d\[#\1\], d\[#\1\]/).find()
+ }
+ }
+
+ check_sqls_result_equal(groupByUnionAllLimit, """
+ SELECT s, d, d, d FROM distinct_streaming_repeated_keys
+ WHERE id < 4096 OR id % 2 = 0
+ GROUP BY s, d
+ ORDER BY 1, 2, 3, 4
+ """)
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]