That sounds weird (maybe a reporting issue?), please send a profile that shows the mem_limit set and the error message referring to the estimated memory instead - thanks!
On 19 June 2018 at 04:59, Fawze Abujaber <fawz...@gmail.com> wrote: > Hi Jezy, > > Thanks for your quick response. > > I’m using impala 2.10, I think since I’m using max memory memory per poo > with the default memory limit, and here the estimate have to take place in > order to estimate the max concurrent queries, and if the estimate beyond the > max memory then the query will not submitted. > > In my case I set both values and getting errors of high memory needed for > the query, it’s not occurring very often. > > On Mon, 18 Jun 2018 at 16:41 Jeszy <jes...@gmail.com> wrote: >> >> Hey Fawze, >> >> Default Query Memory Limit applies here, yes. >> If you submit a query to a pool with that setting, you should see >> something like this in the profile: >> Query Options (set by configuration): MEM_LIMIT=X >> >> (YMMV based on version - what version are you running on?) >> If MEM_LIMIT is present in that line, Impala will (should) disregard >> estimates. >> >> Thanks! >> >> On 17 June 2018 at 21:04, Fawze Abujaber <fawz...@gmail.com> wrote: >> > Hi Jeszy, >> > >> > Thanks for your response, Indeed this is what i was thinking about but, >> > I >> > have Default Query Memory Limit and Max memory set per pool which i >> > think >> > should be enough to cover this, shouldnot it? or i should pass the >> > mem_limit in the default query options? >> > >> > On Sun, Jun 17, 2018 at 8:36 PM, Jeszy <jes...@gmail.com> wrote: >> >> >> >> Hello Fawze, >> >> >> >> Disabling this, per se, is not an option, but an equally simple >> >> workaround >> >> is using MEM_LIMIT. >> >> >> >> The estimated stats are often very far from actual memory usage and >> >> shouldn't be relied on - a best practice is to set MEM_LIMIT as a query >> >> option (preferably have a default value set for each pool). Having that >> >> set >> >> will cause Impala to ignore the estimates and rely on this limit for >> >> admission control purposes. This works decently for well-understood >> >> workloads (ie. where the memory consumption is known to fit within >> >> certain >> >> limits). For ad-hoc workloads, if the query can't be executed within >> >> the >> >> default limit of the pool, you can override the limit on a per-query >> >> basis >> >> (just issue 'set MEM_LIMIT=...' before running the query). >> >> >> >> HTH >> >> >> >> On 16 June 2018 at 13:31, Fawze Abujaber <fawz...@gmail.com> wrote: >> >>> >> >>> Hi Community , >> >>> >> >>> In the last impala versions, impala is estimating the memory required >> >>> for >> >>> the query, and in case the estimated memory required beyonds the >> >>> configured >> >>> memory or the configured memory per pool, impala is not submitting >> >>> this >> >>> query, taking the fact that many times and specially running query on >> >>> tables >> >>> without stats there is a huge difference between the estimated and the >> >>> actual memory used, the estimated can be 31 GB per node and the actual >> >>> use >> >>> is 1 or 2 GB, that’s mean to submit a query I need at least 1.5 T >> >>> memory >> >>> configured which I see it too much. >> >>> >> >>> I’m curios to know if there is an option making this to configuration >> >>> ( >> >>> Un submitting query if we he estimated memory required beyond the >> >>> configured >> >>> memory) as an optional choice. >> >>> >> >>> Such issue can block using impala dynamic resource pools. >> >>> -- >> >>> Take Care >> >>> Fawze Abujaber >> >> >> >> >> > >> > >> > >> > -- >> > Take Care >> > Fawze Abujaber > > -- > Take Care > Fawze Abujaber
