Estimate peak memory VS used peak memory

2018-02-22 Thread Fawze Abujaber
Hi Guys, In Impala version 2.10 i see a lot that the Estimate per Mode peak memory can reach X30 times of the Per Node Peak Memory usage. For example one of the query the estimate was 9GB while the usage was 30GB. I have 3 questions: 1- Do you think the compute stats will help here? 2- How i

Re: Estimate peak memory VS used peak memory

2018-02-22 Thread Jim Apple
I think compute stats will help estimates be more accurate. On Thu, Feb 22, 2018 at 4:48 AM, Fawze Abujaber wrote: > Hi Guys, > > In Impala version 2.10 i see a lot that the Estimate per Mode peak memory > can reach X30 times of the Per Node Peak Memory usage. > > For example

Re: Estimate peak memory VS used peak memory

2018-02-22 Thread Mostafa Mokhtar
@Fawze, To best serve your question please share a profile for the problematic query. Thanks Mostafa On Thu, Feb 22, 2018 at 9:40 AM, Jim Apple wrote: > I think compute stats will help estimates be more accurate. > > On Thu, Feb 22, 2018 at 4:48 AM, Fawze Abujaber

Re: Estimate peak memory VS used peak memory

2018-02-22 Thread Mostafa Mokhtar
It is recommended to set a per query memory limit as part of admission and not rely on estimates as they are sometimes inaccurate. Can you please include the full query profile? On Thu, Feb 22, 2018 at 12:13 PM, Fawze Abujaber wrote: > Hi Mostafa, > > It's not a specific