Hello Fawze, Disabling this, per se, is not an option, but an equally simple workaround is using MEM_LIMIT.
The estimated stats are often very far from actual memory usage and shouldn't be relied on - a best practice is to set MEM_LIMIT as a query option (preferably have a default value set for each pool). Having that set will cause Impala to ignore the estimates and rely on this limit for admission control purposes. This works decently for well-understood workloads (ie. where the memory consumption is known to fit within certain limits). For ad-hoc workloads, if the query can't be executed within the default limit of the pool, you can override the limit on a per-query basis (just issue 'set MEM_LIMIT=...' before running the query). HTH On 16 June 2018 at 13:31, Fawze Abujaber <[email protected]> wrote: > Hi Community , > > In the last impala versions, impala is estimating the memory required for > the query, and in case the estimated memory required beyonds the configured > memory or the configured memory per pool, impala is not submitting this > query, taking the fact that many times and specially running query on > tables without stats there is a huge difference between the estimated and > the actual memory used, the estimated can be 31 GB per node and the actual > use is 1 or 2 GB, that’s mean to submit a query I need at least 1.5 T > memory configured which I see it too much. > > I’m curios to know if there is an option making this to configuration ( Un > submitting query if we he estimated memory required beyond the configured > memory) as an optional choice. > > Such issue can block using impala dynamic resource pools. > -- > Take Care > Fawze Abujaber >
