Hey Fawze, Default Query Memory Limit applies here, yes. If you submit a query to a pool with that setting, you should see something like this in the profile: Query Options (set by configuration): MEM_LIMIT=X
(YMMV based on version - what version are you running on?) If MEM_LIMIT is present in that line, Impala will (should) disregard estimates. Thanks! On 17 June 2018 at 21:04, Fawze Abujaber <[email protected]> wrote: > Hi Jeszy, > > Thanks for your response, Indeed this is what i was thinking about but, I > have Default Query Memory Limit and Max memory set per pool which i think > should be enough to cover this, shouldnot it? or i should pass the > mem_limit in the default query options? > > On Sun, Jun 17, 2018 at 8:36 PM, Jeszy <[email protected]> wrote: >> >> Hello Fawze, >> >> Disabling this, per se, is not an option, but an equally simple workaround >> is using MEM_LIMIT. >> >> The estimated stats are often very far from actual memory usage and >> shouldn't be relied on - a best practice is to set MEM_LIMIT as a query >> option (preferably have a default value set for each pool). Having that set >> will cause Impala to ignore the estimates and rely on this limit for >> admission control purposes. This works decently for well-understood >> workloads (ie. where the memory consumption is known to fit within certain >> limits). For ad-hoc workloads, if the query can't be executed within the >> default limit of the pool, you can override the limit on a per-query basis >> (just issue 'set MEM_LIMIT=...' before running the query). >> >> HTH >> >> On 16 June 2018 at 13:31, Fawze Abujaber <[email protected]> wrote: >>> >>> Hi Community , >>> >>> In the last impala versions, impala is estimating the memory required for >>> the query, and in case the estimated memory required beyonds the configured >>> memory or the configured memory per pool, impala is not submitting this >>> query, taking the fact that many times and specially running query on tables >>> without stats there is a huge difference between the estimated and the >>> actual memory used, the estimated can be 31 GB per node and the actual use >>> is 1 or 2 GB, that’s mean to submit a query I need at least 1.5 T memory >>> configured which I see it too much. >>> >>> I’m curios to know if there is an option making this to configuration ( >>> Un submitting query if we he estimated memory required beyond the configured >>> memory) as an optional choice. >>> >>> Such issue can block using impala dynamic resource pools. >>> -- >>> Take Care >>> Fawze Abujaber >> >> > > > > -- > Take Care > Fawze Abujaber
