matanper opened a new issue, #69931:
URL: https://github.com/apache/doris/issues/69931
### Search before asking
- [x] I had searched in the issues and found no similar issues.
### Version
4.1.x (the same code is on master)
### What's Wrong?
`collect_list` (and `array_agg`) over a struct with a STRING field fails in
two-phase aggregation when a value holds both `"` and `'`, for example JSON
text with an apostrophe:
```sql
SET agg_phase = 2;
SELECT SIZE(COLLECT_LIST(STRUCT(number, '{"a":"it''s","b":1}'))) FROM
numbers('number'='100000');
-- [INTERNAL_ERROR]Failed to deserialize data for Struct(BIGINT, String)
error:
-- [INVALID_ARGUMENT]Actual struct field number 1 is less than schema field
number 2.
```
With other values the error says `Actual struct field number is more than
schema field number 2`. `SET agg_phase = 1` works. A value with only one of the
two characters (`it''s`, or `x", "y`) works.
Cause: for a complex element type, `AggregateFunctionCollectListData`
(`be/src/exprs/aggregate/aggregate_function_collect.h`) sends its partial state
as one `serialize_one_cell_to_json` string per row, and the merge phase reads
it back with `deserialize_one_cell_from_json`. Inside a struct,
`DataTypeStringSerDe::serialize_one_cell_to_json` writes a STRING as
`"<value>"` without escaping, because `FormatOptions.escape_char` is 0 there.
`DataTypeStructSerDe::deserialize_one_cell_from_json` treats `"`, `'` and `\`
as quote and escape characters. The `"` inside the value ends the quoted field
early, a following `'` opens a single-quoted span, and the field delimiters are
miscounted. The code carries a TODO about replacing this text round trip.
### What You Expected?
`100000`.
### How to Reproduce?
Run the query above. It needs no table.
### Anything Else?
A possible fix: send the state in the data type's binary format
(`get_uncompressed_serialized_bytes` / `serialize` / `deserialize`), as the
MIN/MAX state for complex types already does. Because BEs of different versions
exchange this state during a rolling upgrade, the new form needs a version gate
or a reader that accepts both forms.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
