nooneuse commented on code in PR #67469:
URL: https://github.com/apache/doris/pull/67469#discussion_r4002108026
##########
be/src/exprs/aggregate/aggregate_function_datasketches_hll_union_agg.h:
##########
@@ -91,12 +84,32 @@ struct AggregateFunctionHllSketchData {
"Internal error happened when update HLL sketch:
unknown exception.");
}
}
- void reset() {
- if (hll_union_data.has_value()) {
- hll_union_data->reset();
+ void merge(const Sketch& sketch_data) {
+ if (sketch_data.is_empty()) {
+ return;
}
- hll_union_data.reset();
+ const auto lg_max_k = std::max<uint8_t>(sketch_data.get_lg_config_k(),
MIN_UNION_LOG_K);
Review Comment:
Thanks, this issue is valid.
I reproduced it with a non-empty LIST image whose serialized lgK byte was
changed to 22. DataSketches successfully deserializes it with lgK=22, and the
subsequent HllUnion construction throws std::invalid_argument("Invalid value of
k: 22").
The exception currently escapes the local CORRUPTION conversion boundary,
but it is caught by the pipeline TaskScheduler's std::exception boundary and
converted to an InternalError.
The root cause belongs to DataSketches: its deserializer returns an object
containing an lgK value that cannot be produced through its public
constructors. Adding a Doris-only validation would duplicate DataSketches'
invariant and leave its byte-array/stream deserializers and other consumers
inconsistent.
I will push an Datasketches fix to validate lgK for LIST and SET images in
both deserialization entry points, together with malformed-header tests, and
then update the dependency in Doris. I think we don't need to add a
Doris-specific workaround here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
