Hi Tim,
Thanks for the follow-up. In the example I gave, I was actually assuming even if the 3 queries were exactly the same. The requirement I need to solve is to keep the runtime somewhat constant for the same query, regardless of the load. If that cannot be achieved, then the idea is that the admission control feature send the query to the queue instead of running them in parallel with others. Do you think that could be an issue with vcores allocation, instead of memory distribution? Just to confirm, for Impala admission control the specification related to vcores is going to be ignored, right? As I'm not using Cloudera Manager, I have set the fair-scheduler.xml and llama-site.xml files manually. It has been really hard to figure out which configuration from Yarn is accepted in the fair-scheduler.xml used by Impala. Is there any reference I can look at to find the subset of Yarn scheduler config considered by Impala? In particular, I would like to know if I can rely on weight and preemption configuration existing in Yarn, or even vcores and IO-buffers configuration. Thanks, Paulo. ________________________________ From: Tim Armstrong <[email protected]> Sent: Wednesday, February 21, 2018 4:44:40 PM To: [email protected] Subject: Re: Impala Admission Control ________________________________ NOTICE: This email was received from an EXTERNAL sender ________________________________ Hi Roberto, There's not really a good way to achieve exactly that at the moment that I can think of. One of the challenges is that Impala, right now, is quite opportunistic in the way it uses resources - if there are unused cores on the system, queries will often create new threads to take advantage of them. This makes it hard for admission control to anticipate how much CPU a query will use ahead of time. We have plans to make CPU consumption more deterministic (https://issues.apache.org/jira/browse/IMPALA-3902), which in turn would let admission control distinguish between the two kinds of queries you describe more automatically. I've recently been focusing a lot on some of these admission control and resource management problems, so I'm optimistic that there will be a lot of improvements in the pipeline to make it easier to solve problems like this. If the "resource-intensive" and "resource-light" queries come from different workloads, it is possible to set up different resource pools with different concurrency limits and route the queries to those. It doesn't really sound like that's the problem you're trying to solve though - Tim On Wed, Feb 21, 2018 at 11:16 AM, Roberto Cerioni, Paulo <[email protected]<mailto:[email protected]>> wrote: Hello, I have set admission control for Impala and I was able to create multiple queues and limit the resources of each queue. For example, if we have 3 queries that require 100mb to run each, submitted to a queue with 300mb maximum memory, then the 3 queries will be admitted immediately. However, if the runtime of each query normally is 1 second, all finish at the same time, but with 3 seconds runtime each. If we allow only 1 query to run in the queue at once using admission control, that query will finish in 1 second, but what I want is to be able to run as much queries as the queue resources allow in parallel without compromising the runtime. Thanks, Paulo;
