Peter Ebert created IMPALA-12161:
------------------------------------
Summary: Add query hints for memory limits
Key: IMPALA-12161
URL: https://issues.apache.org/jira/browse/IMPALA-12161
Project: IMPALA
Issue Type: New Feature
Components: fe
Affects Versions: Impala 4.1.2
Reporter: Peter Ebert
Support for a query hint to specify memory limits would make implementing
memory limits in applications easier where multiple queries need different
memory limits but are run through the same connection, e.g.: {{/*
+mem_limit=4g */ select * from table}}
---
Session settings for mem_limit are critical for getting large queries to avoid
OOM when using admission control pools (for multi tenancy) and the memory
estimation happens to severely under-estimate (raising the minimum memory for
the pool would also reduce concurrency). However, for applications that run
multiple queries through a single connection (e.g. data visualization tools)
this pattern is cumbersome:
# Set the memory limit
# Run the large query
# Reset the memory limit
# Resume other queries with good estimates
Due to this, some data visualization tools only support session settings at the
connection level. If you need GBs for your worst query and MBs for your
average query, this greatly limits the concurrency in that pool using a 'one
size fits all' memory limit or it requires maintaining many connections of
varying size memory limits to attempt to use memory efficiently.
In one production use case this would limit concurrency by 10x or more.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
