[
https://issues.apache.org/jira/browse/IMPALA-15381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118588#comment-18118588
]
ASF subversion and git services commented on IMPALA-15381:
----------------------------------------------------------
Commit a95c64c756f66156399f9e4de2a5067f34765e37 in impala's branch
refs/heads/master from Aleksandr Efimov
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=a95c64c75 ]
IMPALA-15381: Reduce reservoir RNG state
boost::mt19937_64 makes ReservoirSampleState 2536 bytes. FreePool
rounds the allocation to 4 KiB, so sampling aggregates use 4624
bytes per function and group with their initial sample array.
Use pcg32 to reduce the state to 48 bytes. The sampling sequence
changes, so approximate results can change.
Testing:
- Full release build
- HistogramTest.*
- Large appx_median(), sample(), and histogram() queries
- Updated the Init() allocation size in alloc-fail-init.test
Change-Id: I9e046e3f0c1b91f1761ce578601a330bd864eb84
Assisted-by: gpt-5.6-sol (OpenAI Codex)
Reviewed-on: http://gerrit.cloudera.org:8080/24896
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Reservoir sampling RNG uses excessive per-group memory
> ------------------------------------------------------
>
> Key: IMPALA-15381
> URL: https://issues.apache.org/jira/browse/IMPALA-15381
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Aleksandr Efimov
> Assignee: Aleksandr Efimov
> Priority: Major
>
> ReservoirSampleState uses boost::mt19937_64. Its RNG state is 2504 bytes,
> making the full state 2536 bytes. FunctionContext::Allocate() uses FreePool,
> which rounds this allocation to 4 KiB. Together with the initial sample
> buffer and allocation overhead, each sampling aggregate keeps 4624 bytes per
> group before the sample grows. This affects sample(), appx_median(), and
> histogram(). Queries with several sampling aggregates and many groups can
> consume large amounts of untracked memory. Replace the RNG with a small-state
> generator already available in Impala and update the planner estimate. The
> sampling sequence will change, so approximate results may change. This is a
> regression of IMPALA-4787 introduced by IMPALA-11005.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
