Hi Jezy, Thanks for your quick response.
I’m using impala 2.10, I think since I’m using max memory memory per poo with the default memory limit, and here the estimate have to take place in order to estimate the max concurrent queries, and if the estimate beyond the max memory then the query will not submitted. In my case I set both values and getting errors of high memory needed for the query, it’s not occurring very often. On Mon, 18 Jun 2018 at 16:41 Jeszy <[email protected]> wrote: > Hey Fawze, > > Default Query Memory Limit applies here, yes. > If you submit a query to a pool with that setting, you should see > something like this in the profile: > Query Options (set by configuration): MEM_LIMIT=X > > (YMMV based on version - what version are you running on?) > If MEM_LIMIT is present in that line, Impala will (should) disregard > estimates. > > Thanks! > > On 17 June 2018 at 21:04, Fawze Abujaber <[email protected]> wrote: > > Hi Jeszy, > > > > Thanks for your response, Indeed this is what i was thinking about but, I > > have Default Query Memory Limit and Max memory set per pool which i > think > > should be enough to cover this, shouldnot it? or i should pass the > > mem_limit in the default query options? > > > > On Sun, Jun 17, 2018 at 8:36 PM, Jeszy <[email protected]> wrote: > >> > >> Hello Fawze, > >> > >> Disabling this, per se, is not an option, but an equally simple > workaround > >> is using MEM_LIMIT. > >> > >> The estimated stats are often very far from actual memory usage and > >> shouldn't be relied on - a best practice is to set MEM_LIMIT as a query > >> option (preferably have a default value set for each pool). Having that > set > >> will cause Impala to ignore the estimates and rely on this limit for > >> admission control purposes. This works decently for well-understood > >> workloads (ie. where the memory consumption is known to fit within > certain > >> limits). For ad-hoc workloads, if the query can't be executed within the > >> default limit of the pool, you can override the limit on a per-query > basis > >> (just issue 'set MEM_LIMIT=...' before running the query). > >> > >> HTH > >> > >> On 16 June 2018 at 13:31, Fawze Abujaber <[email protected]> wrote: > >>> > >>> Hi Community , > >>> > >>> In the last impala versions, impala is estimating the memory required > for > >>> the query, and in case the estimated memory required beyonds the > configured > >>> memory or the configured memory per pool, impala is not submitting > this > >>> query, taking the fact that many times and specially running query on > tables > >>> without stats there is a huge difference between the estimated and the > >>> actual memory used, the estimated can be 31 GB per node and the actual > use > >>> is 1 or 2 GB, that’s mean to submit a query I need at least 1.5 T > memory > >>> configured which I see it too much. > >>> > >>> I’m curios to know if there is an option making this to configuration ( > >>> Un submitting query if we he estimated memory required beyond the > configured > >>> memory) as an optional choice. > >>> > >>> Such issue can block using impala dynamic resource pools. > >>> -- > >>> Take Care > >>> Fawze Abujaber > >> > >> > > > > > > > > -- > > Take Care > > Fawze Abujaber > -- Take Care Fawze Abujaber
