github-actions[bot] commented on code in PR #69304:
URL: https://github.com/apache/doris/pull/69304#discussion_r4579271307


##########
be/src/exprs/aggregate/aggregate_function_legacy_nullable.h:
##########
@@ -0,0 +1,141 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <cstdint>
+#include <memory>
+#include <utility>
+
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+template <typename T>
+struct FunctionStableTransfer;
+
+// Published flag-plus-state String columns retain a separate codec, while 
every
+// calculation delegates to V2. Keep the concrete function type so generic 
batch
+// readers can inline create/decode/merge/destroy instead of making virtual 
calls
+// for each state. Native readers retain their separate scratch capability.
+template <typename Function>
+class AggregateFunctionLegacyNullableState final
+        : public 
IAggregateFunctionHelper<AggregateFunctionLegacyNullableState<Function>> {
+    using Base = 
IAggregateFunctionHelper<AggregateFunctionLegacyNullableState<Function>>;
+
+public:
+    explicit AggregateFunctionLegacyNullableState(std::shared_ptr<Function> 
function)
+            : Base(function->get_argument_types()), 
_function(std::move(function)) {}
+
+    String get_name() const override { return _function->get_name(); }
+    IAggregateFunction* transmit_to_stable() override {
+        if (auto* stable = _function->transmit_to_stable()) {
+            using StableFunction = typename 
FunctionStableTransfer<Function>::FunctionStable;
+            return new AggregateFunctionLegacyNullableState<StableFunction>(
+                    
std::shared_ptr<StableFunction>(assert_cast<StableFunction*>(stable)));
+        }
+        return nullptr;
+    }
+    DataTypePtr get_return_type() const override { return 
_function->get_return_type(); }
+    void set_version(int version_) override {
+        Base::set_version(version_);
+        _function->set_version(version_);
+    }
+    void set_query_context(QueryContext* ctx) override { 
_function->set_query_context(ctx); }
+    bool is_blockable() const override { return _function->is_blockable(); }

Review Comment:
   [P2] Preserve window spill capabilities in the legacy adapter
   
   With the documented `be_exec_version=19` upgrade pin, new FE requests send 
the compatibility flag as false, so nullable SUM/AVG and numeric MIN/MAX 
windows are wrapped here. The adapter inherits `window_spill_strategy() = 
UNSUPPORTED` instead of forwarding the underlying `PARTITION_REDUCE` 
capability. `AnalyticSinkOperatorX::_prepare_spill` consequently disables spill 
for e.g. `SUM(nullable_v) OVER (PARTITION BY k)`, allowing a large partition to 
exceed its memory limit where the previous nullable wrapper could spill. Please 
forward the window spill capability methods and cover the legacy adapter in the 
spill tests.



##########
be/src/exprs/aggregate/aggregate_function_legacy_nullable.h:
##########
@@ -0,0 +1,141 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <cstdint>
+#include <memory>
+#include <utility>
+
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+template <typename T>
+struct FunctionStableTransfer;
+
+// Published flag-plus-state String columns retain a separate codec, while 
every
+// calculation delegates to V2. Keep the concrete function type so generic 
batch
+// readers can inline create/decode/merge/destroy instead of making virtual 
calls
+// for each state. Native readers retain their separate scratch capability.
+template <typename Function>
+class AggregateFunctionLegacyNullableState final
+        : public 
IAggregateFunctionHelper<AggregateFunctionLegacyNullableState<Function>> {
+    using Base = 
IAggregateFunctionHelper<AggregateFunctionLegacyNullableState<Function>>;
+
+public:
+    explicit AggregateFunctionLegacyNullableState(std::shared_ptr<Function> 
function)
+            : Base(function->get_argument_types()), 
_function(std::move(function)) {}
+
+    String get_name() const override { return _function->get_name(); }
+    IAggregateFunction* transmit_to_stable() override {
+        if (auto* stable = _function->transmit_to_stable()) {
+            using StableFunction = typename 
FunctionStableTransfer<Function>::FunctionStable;
+            return new AggregateFunctionLegacyNullableState<StableFunction>(
+                    
std::shared_ptr<StableFunction>(assert_cast<StableFunction*>(stable)));
+        }
+        return nullptr;
+    }
+    DataTypePtr get_return_type() const override { return 
_function->get_return_type(); }
+    void set_version(int version_) override {
+        Base::set_version(version_);
+        _function->set_version(version_);
+    }
+    void set_query_context(QueryContext* ctx) override { 
_function->set_query_context(ctx); }
+    bool is_blockable() const override { return _function->is_blockable(); }
+    bool supported_incremental_mode() const override {
+        return _function->supported_incremental_mode();
+    }
+    bool result_column_could_resize() const override {
+        return _function->result_column_could_resize();
+    }
+    void execute_function_with_incremental(int64_t partition_start, int64_t 
partition_end,
+                                           int64_t frame_start, int64_t 
frame_end,
+                                           AggregateDataPtr place, const 
IColumn** columns,
+                                           Arena& arena, bool previous_is_nul, 
bool end_is_nul,
+                                           bool has_null, UInt8* 
use_null_result,
+                                           UInt8* could_use_previous_result) 
const override {
+        _function->execute_function_with_incremental(
+                partition_start, partition_end, frame_start, frame_end, place, 
columns, arena,
+                previous_is_nul, end_is_nul, has_null, use_null_result, 
could_use_previous_result);
+    }
+    void insert_result_into_range(ConstAggregateDataPtr place, IColumn& 
column, size_t start,
+                                  size_t end) const override {
+        _function->insert_result_into_range(place, column, start, end);
+    }
+    // Do not forward the native reader's scratch capability: this adapter 
reads String
+    // columns through generic readers even when the nested function reads 
native fixed states.
+    void create(AggregateDataPtr place) const override { 
_function->create(place); }
+    void destroy(AggregateDataPtr place) const noexcept override { 
_function->destroy(place); }
+    void reset(AggregateDataPtr place) const override { 
_function->reset(place); }
+    bool is_trivial() const override { return _function->is_trivial(); }
+    size_t size_of_data() const override { return _function->size_of_data(); }
+    size_t align_of_data() const override { return _function->align_of_data(); 
}
+    void check_input_columns_type(const IColumn** columns) const override {
+        _function->check_input_columns_type(columns);
+    }
+    void check_result_column_type(const IColumn& column) const override {
+        _function->check_result_column_type(column);
+    }
+    void add(AggregateDataPtr place, const IColumn** columns, ssize_t row,
+             Arena& arena) const override {
+        _function->add(place, columns, row, arena);
+    }
+    void streaming_agg_serialize_to_column(const IColumn** columns, 
MutableColumnPtr& dst,
+                                           size_t rows, Arena& arena) const 
override {
+        _function->streaming_agg_serialize_to_column_legacy(columns, dst, 
rows, arena);
+    }
+    void add_batch(size_t rows, AggregateDataPtr* places, size_t offset, const 
IColumn** columns,
+                   Arena& arena, bool agg_many) const override {
+        _function->add_batch(rows, places, offset, columns, arena, agg_many);
+    }
+    void add_batch_selected(size_t rows, AggregateDataPtr* places, size_t 
offset,
+                            const IColumn** columns, Arena& arena) const 
override {
+        _function->add_batch_selected(rows, places, offset, columns, arena);
+    }
+    void add_batch_single_place(size_t rows, AggregateDataPtr place, const 
IColumn** columns,
+                                Arena& arena) const override {
+        _function->add_batch_single_place(rows, place, columns, arena);
+    }
+    void add_batch_range(size_t begin, size_t end, AggregateDataPtr place, 
const IColumn** columns,
+                         Arena& arena, bool has_null) override {
+        _function->add_batch_range(begin, end, place, columns, arena, 
has_null);
+    }
+    void merge(AggregateDataPtr place, ConstAggregateDataPtr rhs, Arena& 
arena) const override {
+        _function->merge(place, rhs, arena);
+    }
+    void serialize(ConstAggregateDataPtr place, BufferWritable& buf) const 
override {
+        _function->serialize(place, buf);
+    }
+    void deserialize(AggregateDataPtr place, BufferReadable& buf, Arena& 
arena) const override {
+        _function->deserialize(place, buf, arena);
+    }
+    void deserialize_and_merge(AggregateDataPtr place, AggregateDataPtr rhs, 
BufferReadable& buf,
+                               Arena& arena) const override {
+        _function->deserialize(rhs, buf, arena);

Review Comment:
   [P2] Preserve direct DISTINCT merging in the byte codec
   
   This eagerly deserializes a fresh RHS state into the destination's 
long-lived arena before merging it. For nullable 
`multi_distinct_count(VARIANT)`, the former V1 path called the nested 
`deserialize_and_merge` specialization, which decodes into the destination and 
checks membership before copying a key. Here every repeated partial first 
copies its keys into the empty RHS; destroying that scratch hash set does not 
release those arena bytes. Merging N identical partials containing one K-byte 
VARIANT value therefore retains roughly (N+1)*K key bytes despite cardinality 
remaining 1, instead of roughly K. This affects pre-20 wire merging and the 
persisted nullable AGG_STATE adapter. Preserve nullable/marker handling while 
delegating valid payloads to the nested direct merge, and cover repeated 
duplicate partials with an arena-growth test.



##########
be/src/exprs/aggregate/aggregate_function_null_v2.h:
##########
@@ -236,67 +281,121 @@ class AggregateFunctionNullBaseInlineV2 : public 
IAggregateFunctionHelper<Derive
             nested_function->serialize_to_column(places, offset + prefix_size, 
nested_col_ptr,
                                                  num_rows);
         } else {
-            nested_function->serialize_to_column(places, offset, dst, 
num_rows);
+            nested_function->serialize_to_column(places, offset + prefix_size, 
dst, num_rows);
         }
     }
 
     void streaming_agg_serialize_to_column(const IColumn** columns, 
MutableColumnPtr& dst,
                                            const size_t num_rows, Arena& 
arena) const override {
-        const auto* src_nullable_col = assert_cast<const 
ColumnNullable*>(columns[0]);
-        const auto* __restrict src_null_map_data = 
src_nullable_col->get_null_map_data().data();
+        assert_cast<const Derived*>(this)->prepare_streaming_columns(
+                columns, num_rows, [&](const IColumn** nested_columns, const 
UInt8* null_map) {
+                    streaming_serialize(nested_columns, null_map, dst, 
num_rows, arena);
+                });
+    }
 
-        size_t nested_size = nested_function->size_of_data();
-        std::vector<AggregateDataPtr> nested_places(num_rows);
-        std::vector<char> places_data(num_rows * nested_size);
-        for (size_t i = 0; i < num_rows; ++i) {
-            nested_places[i] = places_data.data() + i * nested_size;
-        }
+    template <typename Serialize>
+    void prepare_streaming_columns(const IColumn** columns, size_t num_rows,
+                                   Serialize&& serialize) const {
+        const auto& column = assert_cast<const ColumnNullable&>(*columns[0]);
+        const IColumn* nested_column = &column.get_nested_column();
+        serialize(&nested_column, column.get_null_map_data().data());
+    }
 
-        if (!nested_function->is_trivial()) {
-            for (int i = 0; i < num_rows; ++i) {
-                try {
-                    nested_function->create(nested_places[i]);
-                } catch (...) {
-                    for (int j = 0; j < i; ++j) {
-                        nested_function->destroy(nested_places[j]);
+    void streaming_agg_serialize_to_column_legacy(const IColumn** columns, 
MutableColumnPtr& dst,
+                                                  size_t num_rows, Arena&) 
const {
+        if (num_rows == 0) {
+            return;
+        }
+        // A hidden NULL array may shift later physical offsets, e.g. 
[NULL,[1]] vs [[2],[3]].
+        // Normalize visible rows once for both native and historical String 
output.
+        assert_cast<const Derived*>(this)->prepare_streaming_columns(
+                columns, num_rows, [&](const IColumn** nested_columns, const 
UInt8* null_map) {
+                    VectorBufferWriter 
writer(assert_cast<ColumnString&>(*dst));
+                    AlignedAggregateData storage(this->size_of_data(), 
this->align_of_data());
+                    auto* place = storage.data();
+                    Arena row_arena;
+                    for (size_t row = 0; row < num_rows; ++row) {
+                        row_arena.clear();
+                        create(place);
+                        DEFER({ destroy(place); });
+                        if (!null_map[row]) {
+                            set_flag(place);
+                            nested_function->add(nested_place(place), 
nested_columns, row,
+                                                 row_arena);
+                        }
+                        serialize(place, writer);
+                        writer.commit();
                     }
-                    throw;
-                }
+                });
+    }
+
+protected:
+    void streaming_serialize(const IColumn** columns, const UInt8* null_map, 
MutableColumnPtr& dst,
+                             size_t num_rows, Arena& arena) const {
+        if (num_rows == 0) {
+            return;
+        }
+        MutableColumnPtr nested_dst;
+        ColumnNullable* nullable_dst = nullptr;
+        if (native_state_is_nullable()) {
+            nullable_dst = &assert_cast<ColumnNullable&>(*dst);
+            nested_dst = nullable_dst->get_nested_column().assert_mutable();
+            auto& dst_null_map = nullable_dst->get_null_map_data();
+            dst_null_map.assign(null_map, null_map + num_rows);
+        } else {
+            nested_dst = std::move(dst);
+        }
+        DEFER({
+            if (!native_state_is_nullable()) {
+                dst = std::move(nested_dst);
             }
+        });
+        if constexpr (requires(const NestFunction& function) {
+                          
function.streaming_agg_serialize_to_column_with_null_map(
+                                  columns, nested_dst, num_rows, null_map);
+                      }) {
+            
nested_function->streaming_agg_serialize_to_column_with_null_map(columns, 
nested_dst,
+                                                                             
num_rows, null_map);
+            return;
         }
-        Defer destroy_places = {[&]() {
-            if (!nested_function->is_trivial()) {
-                for (int i = 0; i < num_rows; ++i) {
-                    nested_function->destroy(nested_places[i]);
+        const bool has_null =
+                std::any_of(null_map, null_map + num_rows, [](UInt8 value) { 
return value != 0; });
+        if (!has_null) {
+            nested_function->streaming_agg_serialize_to_column(columns, 
nested_dst, num_rows,
+                                                               arena);
+            return;
+        }
+        // A NULL row has no contribution (not even an empty bitmap 
intersection).
+        // Keep only one aligned state alive: wide/string/set states must not 
grow with batch size.
+        Arena state_arena;
+        Arena row_arena;
+        AggregateDataPtr place = nullptr;
+        for (size_t row = 0; row < num_rows; ++row) {
+            if (native_state_is_nullable()) {
+                if (null_map[row]) {
+                    nested_dst->insert_default();

Review Comment:
   [P2] Count skipped NULL rows in the streaming writers
   
   For a profiled `ai_agg(resource, nullable_text, task)` with high-cardinality 
GROUP BY, the local streaming operator can switch to passthrough and reach this 
loop even at execution version 20 with statistics correctly attached. NULL rows 
now insert a default and bypass the wrapper's `record_null_input` hook; only 
valid rows reach the nested `add`. Consequently both `AIFunctionInputRows` and 
`AIFunctionSkippedNullRows` omit every passed-through NULL row, and downstream 
merges deliberately do not recount input. The legacy streaming loop has the 
same omission. Account skipped rows once per prepared batch in both writers, 
preserving the previous variadic accounting, and add mixed/all-NULL streaming 
statistics coverage.



##########
be/src/exprs/aggregate/aggregate_function_legacy_nullable.h:
##########
@@ -0,0 +1,141 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <cstdint>
+#include <memory>
+#include <utility>
+
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+template <typename T>
+struct FunctionStableTransfer;
+
+// Published flag-plus-state String columns retain a separate codec, while 
every
+// calculation delegates to V2. Keep the concrete function type so generic 
batch
+// readers can inline create/decode/merge/destroy instead of making virtual 
calls
+// for each state. Native readers retain their separate scratch capability.
+template <typename Function>
+class AggregateFunctionLegacyNullableState final
+        : public 
IAggregateFunctionHelper<AggregateFunctionLegacyNullableState<Function>> {
+    using Base = 
IAggregateFunctionHelper<AggregateFunctionLegacyNullableState<Function>>;
+
+public:
+    explicit AggregateFunctionLegacyNullableState(std::shared_ptr<Function> 
function)
+            : Base(function->get_argument_types()), 
_function(std::move(function)) {}
+
+    String get_name() const override { return _function->get_name(); }
+    IAggregateFunction* transmit_to_stable() override {
+        if (auto* stable = _function->transmit_to_stable()) {
+            using StableFunction = typename 
FunctionStableTransfer<Function>::FunctionStable;
+            return new AggregateFunctionLegacyNullableState<StableFunction>(
+                    
std::shared_ptr<StableFunction>(assert_cast<StableFunction*>(stable)));
+        }
+        return nullptr;
+    }
+    DataTypePtr get_return_type() const override { return 
_function->get_return_type(); }
+    void set_version(int version_) override {
+        Base::set_version(version_);
+        _function->set_version(version_);
+    }
+    void set_query_context(QueryContext* ctx) override { 
_function->set_query_context(ctx); }

Review Comment:
   [P2] Forward AI statistics through the legacy codec
   
   For nullable `ai_agg` at execution versions below 20, 
`AggFnEvaluator::prepare` installs this adapter (the aggregate has multiple 
arguments), then calls `set_ai_statistics`. This class forwards the query 
context but inherits the base no-op statistics setter, leaving the nested 
`AggregateFunctionAIAgg` executor without its statistics owner. With profiling 
enabled, input/skipped rows, requests, parsing and token usage therefore 
disappear from the query profile throughout the documented version-19 upgrade 
pin. Forward `set_ai_statistics` to `_function` and exercise the version-19 
adapter in the AI statistics test; the changed test currently uses the raw 
factory function.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to