Doris-Breakwater commented on issue #65976: URL: https://github.com/apache/doris/issues/65976#issuecomment-5060511645
Breakwater-GitHub-Analysis-Slot: slot_c762c3070a5c ## Initial triage This is a valid BE aggregate-state serialization bug. It is high impact for affected table-stream/incremental queries, but the current evidence does not indicate data corruption or a BE crash. The issue is open with no labels, assignee, or milestone; adding the project’s bug plus BE/function and table-stream/binlog area labels would help routing. The reproduction, commit, schema, data, and error are sufficient to make the nested-array failure actionable. ## Code-level finding At the reported commit, the error text has a single relevant source: the complex-type specialization of `AggregateFunctionArrayAggData` stores values in `column_data`, but its binary [`write`, `read`, and `merge` methods are unconditional `NOT_IMPLEMENTED_ERROR` throws](https://github.com/apache/doris/blob/a8b1fd95e2a/be/src/exprs/aggregate/aggregate_function_array_agg.h#L229-L277). `array_agg_foreach`/`array_agg_foreachv2` keeps one nested aggregate state per array position. Its [`serialize()` loops over those positions and calls `nested_function->serialize()`](https://github.com/apache/doris/blob/a8b1fd95e2a/be/src/exprs/aggregate/aggregate_function_foreach.h#L182-L189). Therefore, for `b ARRAY<ARRAY<INT>>`, the nested function is `array_agg(ARRAY<INT>)`; it selects the complex-type state above and deterministically throws as soon as a partial aggregate state must be serialized. A one-stage base-table plan can avoid that method, which explains why an equivalent base-table query may appear healthy while a plan with an aggregate-state boundary fails. This is a latent `array_agg` complex-state contract gap exposed by the table-stream plan, not evidence of malformed ROW-binlog array decoding. There is one important discrepancy to resolve: for `a ARRAY<INT>`, the foreach nested function should be `array_agg(INT)`, and the primitive specialization does implement `write/read/merge`. The supplied combined log excerpt does not show why that query reportedly reaches the complex-type throw. Since the error literal comes from the complex specialization, the first query either has a different runtime aggregate parameter type/plan than expected or needs a separate per-query confirmation. ## Targeted missing evidence Please attach the following for each query separately: 1. `EXPLAIN VERBOSE` (or the most detailed available plan) for both the base-table and table-stream forms, showing local/global aggregate phases and exchanges. 2. The query ID and full BE stack trace for the first (`a`) query, or the serialized `TAggregateExpr.param_types` for its `array_agg_foreachv2` evaluator. 3. Separate success/error output for the two statements rather than one combined sanitized excerpt. No broader logs are needed initially. The nested `b` failure is already code-confirmed; these artifacts are only needed to pin down the scalar-array `a` path. ## Recommended next steps 1. Implement a complete complex-type state contract for `AggregateFunctionArrayAggData`: merge by appending `column_data`, and provide matching serialization/deserialization. The existing complex-type `AggregateFunctionCollectListData` SerDe-backed implementation is a useful reference, though maintainers should decide whether to reuse that format or introduce a column-native intermediate format with explicit compatibility considerations. 2. Add a BE unit test that exercises `serialize -> deserialize -> merge -> result` for `array_agg_foreach` with both `ARRAY<INT>` and `ARRAY<ARRAY<INT>>`, including NULLs and variable-length rows. Existing tests mostly exercise add/result and do not cover this failing contract. 3. Add a `table_stream_p0` regression using the supplied ROW-binlog/`APPEND_ONLY` setup, with base-table controls and both single- and multi-pipeline execution. 4. Before finalizing the fix, verify that the first query’s runtime nested argument is `INT`; if it is unexpectedly `ARRAY<INT>`, fix that planner/translator type propagation in addition to the complex-state serializer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
