Hi Jeszy, Thanks for your response, Indeed this is what i was thinking about but, I have Default Query Memory Limit and Max memory set per pool which i think should be enough to cover this, shouldnot it? or i should pass the mem_limit in the default query options?
On Sun, Jun 17, 2018 at 8:36 PM, Jeszy <[email protected]> wrote: > Hello Fawze, > > Disabling this, per se, is not an option, but an equally simple workaround > is using MEM_LIMIT. > > The estimated stats are often very far from actual memory usage and > shouldn't be relied on - a best practice is to set MEM_LIMIT as a query > option (preferably have a default value set for each pool). Having that set > will cause Impala to ignore the estimates and rely on this limit for > admission control purposes. This works decently for well-understood > workloads (ie. where the memory consumption is known to fit within certain > limits). For ad-hoc workloads, if the query can't be executed within the > default limit of the pool, you can override the limit on a per-query basis > (just issue 'set MEM_LIMIT=...' before running the query). > > HTH > > On 16 June 2018 at 13:31, Fawze Abujaber <[email protected]> wrote: > >> Hi Community , >> >> In the last impala versions, impala is estimating the memory required for >> the query, and in case the estimated memory required beyonds the configured >> memory or the configured memory per pool, impala is not submitting this >> query, taking the fact that many times and specially running query on >> tables without stats there is a huge difference between the estimated and >> the actual memory used, the estimated can be 31 GB per node and the actual >> use is 1 or 2 GB, that’s mean to submit a query I need at least 1.5 T >> memory configured which I see it too much. >> >> I’m curios to know if there is an option making this to configuration ( >> Un submitting query if we he estimated memory required beyond the >> configured memory) as an optional choice. >> >> Such issue can block using impala dynamic resource pools. >> -- >> Take Care >> Fawze Abujaber >> > > -- Take Care Fawze Abujaber
