[
https://issues.apache.org/jira/browse/IMPALA-15381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksandr Efimov reassigned IMPALA-15381:
-----------------------------------------
Assignee: Aleksandr Efimov
> Reservoir sampling RNG uses excessive per-group memory
> ------------------------------------------------------
>
> Key: IMPALA-15381
> URL: https://issues.apache.org/jira/browse/IMPALA-15381
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Aleksandr Efimov
> Assignee: Aleksandr Efimov
> Priority: Major
>
> ReservoirSampleState uses boost::mt19937_64. Its RNG state is 2504 bytes,
> making the full state 2536 bytes. FunctionContext::Allocate() uses FreePool,
> which rounds this allocation to 4 KiB. Together with the initial sample
> buffer and allocation overhead, each sampling aggregate keeps 4624 bytes per
> group before the sample grows. This affects sample(), appx_median(), and
> histogram(). Queries with several sampling aggregates and many groups can
> consume large amounts of untracked memory. Replace the RNG with a small-state
> generator already available in Impala and update the planner estimate. The
> sampling sequence will change, so approximate results may change. This is a
> regression of IMPALA-4787 introduced by IMPALA-11005.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
