linrrzqqq opened a new pull request, #68498:
URL: https://github.com/apache/doris/pull/68498
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Materialized CTE consumers can copy QuantileState values into independent
aggregate accumulators while retaining the same mutable TDigest. Concurrent
merges or percentile queries can then modify shared centroid buffers, causing
backend crashes or cross-group state contamination.
Give each QuantileState an independent TDigest handle with copy-on-write
data.
Writes detach shared data and preserve vector capacity for continued
aggregation.
Percentile queries use an immutable processed snapshot. Only one reader
builds the snapshot, with allocation and compression outside the mutex. Other
readers reuse the result, and failed builds wake waiting readers to retry.
The serialized format remains unchanged. Concurrent mutation of the same
TDigest handle still requires external synchronization.
```sql
SET enable_cte_materialize = true;
SET inline_cte_referenced_threshold = 0;
WITH t AS (
SELECT number % 2 AS g,
quantile_union(to_quantile_state(100 * (number % 2), 2048)) AS q
FROM numbers("number" = "8192")
GROUP BY g
)
SELECT g, quantile_percent(q, 0.5) FROM t
UNION ALL
SELECT -1, quantile_percent(quantile_union(q), 0.5) FROM t;
```
before:
```text
std::vector<doris::Centroid>::operator[](size_type) const: Assertion '__n <
this->size()' failed.
*** Query id: e5437298fa6e4f14-9b6a546907f92425 ***
*** tablet id: 0 ***
*** Aborted at 1790250779 (unix time) try "date -d @1790250779" if you are
using GNU date ***
*** Current BE git commitID: 765e2dfefe ***
*** SIGABRT unknown detail explain (@0x3fe0007a9c2) received by PID 502210
(TID 547405 OR 0x113767afd640) from PID 502210; stack trace: ***
F20260924 19:52:59.574612 547409 tdigest.h:458] Check failed: index >=
_processed_weight - _weight(n - 1) / 2.0 (4095 vs. 8189)
*** Check failure stack trace: ***
@ 0x560d751ce0b6 google::LogMessageFatal::~LogMessageFatal()
@ 0x560d4c8a1600 doris::TDigest::quantile_processed()
@ 0x560d4c884644 doris::TDigest::quantile()
@ 0x560d4c876beb doris::QuantileState::get_value_by_percentile()
@ 0x560d555b14b5 doris::FunctionQuantileStatePercent::execute_impl()
@ 0x560d555b1902 doris::FunctionQuantileStatePercent::execute_impl()
@ 0x560d563a3d19
doris::PreparedFunctionImpl::_execute_skipped_constant_deal()
@ 0x560d55df44cb doris::PreparedFunctionImpl::default_execute()
@ 0x560d55d4aa7c doris::PreparedFunctionImpl::execute()
@ 0x560d4adbbe7f doris::IFunctionBase::execute()
@ 0x560d53f79b64 doris::VectorizedFnCall::_do_execute()
@ 0x560d53f147a3 doris::VectorizedFnCall::execute_column_impl()
@ 0x560d53f3d44c doris::VExpr::execute_column()
@ 0x560d53f9d63b doris::VExprContext::execute()
@ 0x560d535ee6cf doris::OperatorXBase::do_projections()
@ 0x560d535f091a doris::OperatorXBase::get_block_after_projects()
@ 0x560d4f121015 doris::PipelineTask::execute()
@ 0x560d51b28ca0 doris::TaskScheduler::_do_work()
@ 0x560d51d357c5 doris::TaskScheduler::start()::$_0::operator()()
@ 0x560d51d3570d std::__invoke_impl<>()
@ 0x560d51d3562d
_ZSt10__invoke_rIvRZN5doris13TaskScheduler5startEvE3$_0JEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EES5_E4typeEOS6_DpOS7_
@ 0x560d51d35305 std::_Function_handler<>::_M_invoke()
@ 0x560d4a838b3e std::function<>::operator()()
@ 0x560d710d3339 doris::FunctionRunnable::run()
@ 0x560d71013f64 doris::ThreadPool::dispatch_thread()
@ 0x560d710f38fd std::__invoke_impl<>()
@ 0x560d710f36b5 std::__invoke<>()
@ 0x560d710f35e1
_ZNSt5_BindIFMN5doris10ThreadPoolEFvvEPS1_EE6__callIvJEJLm0EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
@ 0x560d710f339c std::_Bind<>::operator()<>()
@ 0x560d710f328d std::__invoke_impl<>()
@ 0x560d710f318d
_ZSt10__invoke_rIvRSt5_BindIFMN5doris10ThreadPoolEFvvEPS2_EEJEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESA_E4typeEOSB_DpOSC_
@ 0x560d710f2a65 std::_Function_handler<>::_M_invoke()
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int,
siginfo_t*, void*) at ../src/common/signal_handler.h:418
1# 0x00001543D423FC60 in /lib64/libc.so.6
2# __pthread_kill_implementation in /lib64/libc.so.6
3# gsignal in /lib64/libc.so.6
4# abort in /lib64/libc.so.6
5# 0x0000560D7AD406A7 in
/mnt/disk9/linzhenqi/dv/doris/output/be/lib/doris_be
6# std::vector<doris::Centroid, std::allocator<doris::Centroid>
>::operator[](unsigned long) const at
/mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_vector.h:1282
7# doris::TDigest::_weight(long) const at ../src/util/tdigest.h:623
8# doris::TDigest::_update_cumulative() at ../src/util/tdigest.h:701
9# doris::TDigest::_process() at ../src/util/tdigest.h:749
10# doris::TDigest::quantile(float) in
/mnt/disk9/linzhenqi/dv/doris/output/be/lib/doris_be
11# doris::QuantileState::get_value_by_percentile(float) const at
./be/src/core/value/quantile_state.cpp:140
12#
doris::FunctionQuantileStatePercent::execute_impl(doris::FunctionContext*,
doris::Block&, std::vector<unsigned int, std::allocator<unsigned int> > const&,
unsigned int, unsigned long) const at
./be/src/exprs/function/function_quantile_state.cpp:202
13# non-virtual thunk to
doris::FunctionQuantileStatePercent::execute_impl(doris::FunctionContext*,
doris::Block&, std::vector<unsigned int, std::allocator<unsigned int> > const&,
unsigned int, unsigned long) const in
/mnt/disk9/linzhenqi/dv/doris/output/be/lib/doris_be
14#
doris::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*,
doris::Block&, std::vector<unsigned int, std::allocator<unsigned int> >
const&, unsigned int, unsigned long) const at
./be/src/exprs/function/function.cpp:135
15# doris::PreparedFunctionImpl::default_execute(doris::FunctionContext*,
doris::Block&, std::vector<unsigned int, std::allocator<unsigned int> > const&,
unsigned int, unsigned long) const at ./be/src/exprs/function/function.cpp:268
16# doris::PreparedFunctionImpl::execute(doris::FunctionContext*,
doris::Block&, std::vector<unsigned int, std::allocator<unsigned int> > const&,
unsigned int, unsigned long) const at ./be/src/exprs/function/function.cpp:274
17# doris::IFunctionBase::execute(doris::FunctionContext*, doris::Block&,
std::vector<unsigned int, std::allocator<unsigned int> > const&, unsigned int,
unsigned long) const at ../src/exprs/function/function.h:213
18# doris::VectorizedFnCall::_do_execute(doris::VExprContext*, doris::Block
const*, doris::PODArray<unsigned int, 4096ul, doris::Allocator<false, false,
false, doris::DefaultMemoryAllocator, true>, 16ul, 15ul> const*, unsigned long,
doris::COW<doris::IColumn>::immutable_ptr<doris::IColumn>&,
doris::COW<doris::IColumn>::immutable_ptr<doris::IColumn>*) const at
./be/src/exprs/vectorized_fn_call.cpp:441
19# doris::VectorizedFnCall::execute_column_impl(doris::VExprContext*,
doris::Block const*, doris::PODArray<unsigned int, 4096ul,
doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, true>,
16ul, 15ul> const*, unsigned long,
doris::COW<doris::IColumn>::immutable_ptr<doris::IColumn>&) const at
./be/src/exprs/vectorized_fn_call.cpp:477
20# doris::VExpr::execute_column(doris::VExprContext*, doris::Block const*,
doris::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false,
doris::DefaultMemoryAllocator, true>, 16ul, 15ul> const*, unsigned long,
doris::COW<doris::IColumn>::immutable_ptr<doris::IColumn>&) const at
./be/src/exprs/vexpr.cpp:1067
21# doris::VExprContext::execute(doris::Block const*,
doris::COW<doris::IColumn>::immutable_ptr<doris::IColumn>&) at
./be/src/exprs/vexpr_context.cpp:91
22#
doris::VExprContext::get_output_block_after_execute_exprs(std::vector<std::shared_ptr<doris::VExprContext>,
std::allocator<std::shared_ptr<doris::VExprContext> > > const&, doris::Block
const&, doris::Block*, bool) at ./be/src/exprs/vexpr_context.cpp:464
23#
doris::MultiCastDataStreamerSourceOperatorX::get_block_impl(doris::RuntimeState*,
doris::Block*, bool*) at
./be/src/exec/operator/multi_cast_data_stream_source.cpp:112
24# doris::OperatorXBase::get_block(doris::RuntimeState*, doris::Block*,
bool*) at ../src/exec/operator/operator.h:898
25# doris::OperatorXBase::get_block_after_projects(doris::RuntimeState*,
doris::Block*, bool*) at ./be/build_ASAN/../src/exec/operator/operator.cpp:435
26# doris::PipelineTask::execute(bool*) at
./be/src/exec/pipeline/pipeline_task.cpp:655
27# doris::TaskScheduler::_do_work(int) at
./be/src/exec/pipeline/task_scheduler.cpp:155
28# doris::TaskScheduler::start()::$_0::operator()() const at
./be/src/exec/pipeline/task_scheduler.cpp:64
29# void std::__invoke_impl<void,
doris::TaskScheduler::start()::$_0&>(std::__invoke_other,
doris::TaskScheduler::start()::$_0&) at
/mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63
30# std::enable_if<is_invocable_r_v<void,
doris::TaskScheduler::start()::$_0&>, void>::type std::__invoke_r<void,
doris::TaskScheduler::start()::$_0&>(doris::TaskScheduler::start()::$_0&) at
/mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:119
31# std::_Function_handler<void (),
doris::TaskScheduler::start()::$_0>::_M_invoke(std::_Any_data const&) at
/mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292
32# std::function<void ()>::operator()() const at
/mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593
33# doris::FunctionRunnable::run() at ./be/src/util/threadpool.cpp:60
34# doris::ThreadPool::dispatch_thread() at ./be/src/util/threadpool.cpp:621
35# void std::__invoke_impl<void, void (doris::ThreadPool::*&)(),
doris::ThreadPool*&>(std::__invoke_memfun_deref, void
(doris::ThreadPool::*&)(), doris::ThreadPool*&) at
/mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:76
36# std::__invoke_result<void (doris::ThreadPool::*&)(),
doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(),
doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at
/mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:98
37# void std::_Bind<void
(doris::ThreadPool::*(doris::ThreadPool*))()>::__call<void, ,
0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) at
/mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/functional:515
38# void std::_Bind<void
(doris::ThreadPool::*(doris::ThreadPool*))()>::operator()<, void>() at
/mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/functional:600
39# void std::__invoke_impl<void, std::_Bind<void
(doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::__invoke_other,
std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) at
/mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63
40# std::enable_if<is_invocable_r_v<void, std::_Bind<void
(doris::ThreadPool::*(doris::ThreadPool*))()>&>, void>::type
std::__invoke_r<void, std::_Bind<void
(doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::_Bind<void
(doris::ThreadPool::*(doris::ThreadPool*))()>&) at
/mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:119
41# std::_Function_handler<void (), std::_Bind<void
(doris::ThreadPool::*(doris::ThreadPool*))()> >::_M_invoke(std::_Any_data
const&) at
/mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292
42# std::function<void ()>::operator()() const at
/mnt/disk9/linzhenqi/dv/version-toolchain/ldb_toolchain_v28/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593
43# doris::Thread::supervise_thread(void*) at ./be/src/util/thread.cpp:460
44# asan_thread_start(void*) in
/mnt/disk9/linzhenqi/dv/doris/output/be/lib/doris_be
45# start_thread in /lib64/libc.so.6
46# clone3 in /lib64/libc.so.6
```
now:
```text
+------+--------------------------+
| g | quantile_percent(q, 0.5) |
+------+--------------------------+
| -1 | 0 |
| 0 | 0 |
| 1 | 100 |
+------+--------------------------+
```
### Release note
Fix backend crashes and incorrect quantile results when aggregate states
share inputs, including materialized CTE queries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
