[
https://issues.apache.org/jira/browse/IMPALA-12161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Smith updated IMPALA-12161:
-----------------------------------
Component/s: Frontend
(was: fe)
> Add query hints for memory limits
> ---------------------------------
>
> Key: IMPALA-12161
> URL: https://issues.apache.org/jira/browse/IMPALA-12161
> Project: IMPALA
> Issue Type: New Feature
> Components: Frontend
> Affects Versions: Impala 4.1.2
> Reporter: Peter Ebert
> Priority: Major
>
> Support for a query hint to specify memory limits would make implementing
> memory limits in applications easier where multiple queries need different
> memory limits but are run through the same connection, e.g.: {{/*
> +mem_limit=4g */ select * from table}}
> ---
> Session settings for mem_limit are critical for getting large queries to
> avoid OOM when using admission control pools (for multi tenancy) and the
> memory estimation happens to severely under-estimate (raising the minimum
> memory for the pool would also reduce concurrency). However, for
> applications that run multiple queries through a single connection (e.g. data
> visualization tools) this pattern is cumbersome:
> # Set the memory limit
> # Run the large query
> # Reset the memory limit
> # Resume other queries with good estimates
> Due to this, some data visualization tools only support session settings at
> the connection level. If you need GBs for your worst query and MBs for your
> average query, this greatly limits the concurrency in that pool using a 'one
> size fits all' memory limit or it requires maintaining many connections of
> varying size memory limits to attempt to use memory efficiently.
> In one production use case this would limit concurrency by 10x or more.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
