Hi Matt,
Thanks for suggestion. Even though we can solve "soft limit" problem
using single coordinator, the coordinator doesn't know how much memory will be
used during execution. The limit is actually based on the memory size estimated
by coordinator, right. So it is possible that the coordinator accept a query
which using estimating 5G memory which under limit and dispatch to workers, but
the query actually use 15G memory during execution so limit doesn’t make sense.
And as I know the estimation will be more accurate if the table has
stats, right ? But it does not promise to be accurate even if stats exists.
Does the wrongly estimation matters ? For example will it cause the out
of memory problem ? If yes, is there any best practice to avoid that ?
Thank you very much.
Songbo
-----邮件原件-----
发件人: Matthew Jacobs [mailto:[email protected]]
发送时间: 2016年7月19日 11:35
收件人: 廖松博
抄送: [email protected]
主题: Re: Impala user resource isolation best practice
Hi Songbo,
Yes, setting the MEM_LIMITs is a good idea. In Impala 2.5 and after, you can
set a per-pool default query mem limit. Cloudera Manager makes this easy in the
Dynamic Resource Pools page.
You are right about the possibility of overwhelming a single coordinator. You
can consider setting up one larger node to be a coordinator (e.g. a lot of RAM
& CPUs), and setting it so that it *does not* have a datanode on it to minimize
the number of query fragments scheduled on it. In order to do that you'll also
need to set the impalad cmd line argument "-abort_on_config_error=false"
because it normally expects a DN to be colocated with the Impalad. It is still
possible to overwhelm a single coordinator like this, but we've found it to
work reasonably well in our tests. Of course it is very dependent on your
hardware and workload.
Best,
Matt
On Mon, Jul 18, 2016 at 6:31 PM, 廖松博 <[email protected]> wrote:
> Hi Matt,
>
> If we use a single coordinator, if a lot of queries come in, the
> coordinator may be out of memory, right ? Because the coordinator take charge
> of merge results from workers and send back to client.
> Thanks.
>
> Songbo
>
> -----邮件原件-----
> 发件人: Matthew Jacobs [mailto:[email protected]]
> 发送时间: 2016年7月19日 7:50
> 收件人: 廖松博
> 抄送: [email protected]
> 主题: Re: Impala user resource isolation best practice
>
> The 'soft limits' refer to the behavior that, when multiple coordinators are
> used and queries are submitted concurrently, the admission control cannot
> guarantee the limits because the decisions are made based on some stale
> information that needs to be updated by the statestore. If you use a single
> coordinator you avoid that problem. That said, a separate problem is that
> there is no way to guarantee that other resources, e.g. network, IO, etc. are
> not abused by a single user.
>
> On Mon, Jul 18, 2016 at 4:29 PM, 廖松博 <[email protected]> wrote:
>> Hi Matthew，
>> Thanks for your reply. The point is , per admission control
>> documents, most of impala limits are "soft limit", are the 2 settings you
>> mentioned also "soft limit" ? The soft limit means the pool will exceed the
>> memory/concurrency limit at some moment when impala is not aware of. But it
>> is affecting other pool at that moment.
>> Thanks.
>>
>> Songbo
>>
>> -----邮件原件-----
>> 发件人: Matthew Jacobs [mailto:[email protected]]
>> 发送时间: 2016年7月19日 0:44
>> 收件人: [email protected]
>> 主题: Re: Impala user resource isolation best practice
>>
>> By the way, some of the controls I mentioned were added in Impala 2.5, so
>> you should consider upgrading if you're not already using a newer version of
>> Impala.
>>
>> Thanks,
>> Matt
>>
>> On Mon, Jul 18, 2016 at 9:20 AM, Matthew Jacobs <[email protected]> wrote:
>>> Hi Songbo,
>>>
>>> Right now the best you can do is with admission control with:
>>> (a) a single coordinator to avoid the possibility of over-admitting
>>> by different coordinators
>>> (b) setting default query mem limits so that individual queries are
>>> limited
>>>
>>> For your scenario, I'd recommend setting up 2 pools, one for user A
>>> and a second for user B. Set the max number of running queries for
>>> user A to something reasonable for the concurrency for that workload.
>>> Set the max memory for the user B pool to the portion of cluster
>>> memory you're willing to give to those queries. (Notice the pool
>>> with the small queries has the max number of running queries set and
>>> the pool with the fewer but larger big queries has the max memory
>>> set -- that is intentional, the former is faster for admission but
>>> doesn't limit based on memory.) How well this will work depends on
>>> how well you can pick good numbers for these settings, which can be
>>> difficult and requires studying your workload.
>>>
>>> This isn't perfect resource isolation because rogue queries can
>>> still consume too much CPU or other resources, but it's the best
>>> you'll be able to do right now. In the future we will have better
>>> tools to make this easier.
>>>
>>> Best,
>>> Matt
>>>
>>> On Mon, Jul 18, 2016 at 2:59 AM, 廖松博 <[email protected]> wrote:
>>>> Hello guys,
>>>>
>>>>
>>>>
>>>> My Company is using Cloudera Impala as our basic
>>>> infrastructure for online data analysis. The most difficult part we
>>>> met is resource isolation and instability.
>>>>
>>>> According to our experiences in Impala, some big query which
>>>> consume a vast amount of memory will crash impalad process(actually
>>>> as worker but not coordinator, right?).
>>>>
>>>> In our simplest scenario, user A is a very important customer and
>>>> his queries are relatively small, user B is a unimportant user who
>>>> may issue very large SQL to impala. It is unacceptable that the big
>>>> query from user B crash the impalad process and affect the user
>>>> experiences of user A. So resource isolation is the point.
>>>>
>>>> But per the Impala documents :
>>>> http://www.cloudera.com/documentation/enterprise/5-6-x/topics/impal
>>>> a _ admission.html , Impala resource isolation is soft limit,
>>>> cannot strictly prevent query from user B affecting user A.
>>>>
>>>> As I know llama(run impala with yarn) is not recommended and we
>>>> actually tried it but disappointed about the performance and accuracy.
>>>>
>>>> Is there any best practice for user resource isolation? So
>>>> different user will not affect each other.
>>>>
>>>> Thanks.
>>>>
>>>>
>>>>
>>>> Best Regards,
>>>>
>>>> Songbo
