Aleksandr Efimov created IMPALA-15381:
-----------------------------------------
Summary: Reservoir sampling RNG uses excessive per-group memory
Key: IMPALA-15381
URL: https://issues.apache.org/jira/browse/IMPALA-15381
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Aleksandr Efimov
ReservoirSampleState uses boost::mt19937_64. Its RNG state is 2504 bytes,
making the full state 2536 bytes. FunctionContext::Allocate() uses FreePool,
which rounds this allocation to 4 KiB. Together with the initial sample buffer
and allocation overhead, each sampling aggregate keeps 4624 bytes per group
before the sample grows.
This affects sample(), appx_median(), and histogram(). Queries with several
sampling aggregates and many groups can consume large amounts of untracked
memory.
Replace the RNG with a small-state generator already available in Impala and
update the planner estimate. The sampling sequence will change, so approximate
results may change.
This is a regression of IMPALA-4787 introduced by IMPALA-11005.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
