Tim Armstrong created IMPALA-6847:

             Summary: Consider adding workaround for high memory estimates in 
admission control
                 Key: IMPALA-6847
                 URL: https://issues.apache.org/jira/browse/IMPALA-6847
             Project: IMPALA
          Issue Type: Improvement
          Components: Frontend
    Affects Versions: Product Backlog
            Reporter: Tim Armstrong
            Assignee: Tim Armstrong

A scenario we've seen play out a couple of times is this.

1. An Impala admin sets up memory-based admission control but sets the default 
query mem_limit to 0. This means that admission control will use memory 
estimates instead of mem_limit. Typically admins want some protection from 
large queries consuming excessive memory but can't or don't want to set a 
single mem_limit because the workload is unknown or unpredictable. This 
configuration has caveats and we recommend against it, but this happens and 
often works well enough as long as the workload is comprised of relatively 
simple queries. The caveats include:
* There is no enforcement that a query stays within the memory estimate. This 
means that a query can fail or force other queries to fail.
* Memory estimates are often inaccurate (this is unavoidable since they depend 
on cardinality estimates, which are commonly off by 10x even with 
state-of-the-art query planners). This means that runnable queries may be 
impossible to admit without setting a mem_limit.

The last problem is the problem this JIRA is intending to work around. The 
preferred solutions, which work most of the time, are:
1. Configure a default query memory limit.
2. Set a mem_limit for the problematic query

It would however, be useful to have a third fallback in cases where the first 
two options are difficult or impossible to apply. The basic requirement is to 
allow an administrator to configure a resource pool such that queries with very 
high estimates can run.

The long-term solution to this is IMPALA-6460. This is a short-term workaround 
that can tide users over until we have a comprehensive solution.

This message was sent by Atlassian JIRA

Reply via email to