[
https://issues.apache.org/jira/browse/IMPALA-15381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksandr Efimov updated IMPALA-15381:
--------------------------------------
Description: ReservoirSampleState uses boost::mt19937_64. Its RNG state is
2504 bytes, making the full state 2536 bytes. FunctionContext::Allocate() uses
FreePool, which rounds this allocation to 4 KiB. Together with the initial
sample buffer and allocation overhead, each sampling aggregate keeps 4624 bytes
per group before the sample grows. This affects sample(), appx_median(), and
histogram(). Queries with several sampling aggregates and many groups can
consume large amounts of untracked memory. Replace the RNG with a small-state
generator already available in Impala and update the planner estimate. The
sampling sequence will change, so approximate results may change. This is a
regression of IMPALA-4787 introduced by IMPALA-11005. (was:
ReservoirSampleState uses boost::mt19937_64. Its RNG state is 2504 bytes,
making the full state 2536 bytes. FunctionContext::Allocate() uses FreePool,
which rounds this allocation to 4 KiB. Together with the initial sample buffer
and allocation overhead, each sampling aggregate keeps 4624 bytes per group
before the sample grows.
This affects sample(), appx_median(), and histogram(). Queries with several
sampling aggregates and many groups can consume large amounts of untracked
memory.
Replace the RNG with a small-state generator already available in Impala and
update the planner estimate. The sampling sequence will change, so approximate
results may change.
This is a regression of IMPALA-4787 introduced by IMPALA-11005.)
> Reservoir sampling RNG uses excessive per-group memory
> ------------------------------------------------------
>
> Key: IMPALA-15381
> URL: https://issues.apache.org/jira/browse/IMPALA-15381
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Aleksandr Efimov
> Priority: Major
>
> ReservoirSampleState uses boost::mt19937_64. Its RNG state is 2504 bytes,
> making the full state 2536 bytes. FunctionContext::Allocate() uses FreePool,
> which rounds this allocation to 4 KiB. Together with the initial sample
> buffer and allocation overhead, each sampling aggregate keeps 4624 bytes per
> group before the sample grows. This affects sample(), appx_median(), and
> histogram(). Queries with several sampling aggregates and many groups can
> consume large amounts of untracked memory. Replace the RNG with a small-state
> generator already available in Impala and update the planner estimate. The
> sampling sequence will change, so approximate results may change. This is a
> regression of IMPALA-4787 introduced by IMPALA-11005.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
