[
https://issues.apache.org/jira/browse/AMBARI-14671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15098599#comment-15098599
]
Jayush Luniya commented on AMBARI-14671:
----------------------------------------
I recently noticed this behavior and after giving this a bit of a thought, and
came up with a way to dynamically scale up the pool size. Attached is the
prototype for DynamicScaleThreadPoolExecutor. Essentially the idea is to write
a custom blocking queue DynamicScaleBlockingQueue which takes scaleThreshold
and scaleUpPercentage as additional parameters to set thresholds on when to
scale up and by what amount and then implement the _offer() _method to use
these scaling parameters to decide when to scale up. Further a requeue
rejection policy can be use to requeue a rejected request assuming the queue is
unbounded.
> Reevaluate the use of threadpools in Ambari code base
> -----------------------------------------------------
>
> Key: AMBARI-14671
> URL: https://issues.apache.org/jira/browse/AMBARI-14671
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
> Affects Versions: 2.2.0
> Reporter: Jayush Luniya
> Assignee: Jayush Luniya
> Fix For: 2.4.0
>
> Attachments: DynamicScaling.tgz
>
>
> As part of investigation for BUG-43981, noticed that in many places in the
> Ambari code base, the way we use the threadpool is not quite correct. We
> will never scale up the number of threads on high load when we use unbounded
> queues. This could lead to performance bottlenecks especially if we are
> configure ThreadPoolExecutor with corePoolSize=0 and maxPoolSize=10, only one
> thread will ever be spawned. See observations below
> Observations:
> 1. When a ThreadPoolExecutor object is created, the pool size is 0 (i.e. no
> new threads are created then) unless prestartAllCoreThreads() is called. Also
> if we set allowCoreThreadTimeOut(true), idle core threads will also be
> reclaimed.
> 2. In our code base, I observed that we create a threadpool using unlimited
> queue. However the pool will never scale up from coreThreads -> maxThreads as
> the request will always get queued.
> {code:java}
> LinkedBlockingQueue<Runnable> queue = new LinkedBlockingQueue<Runnable>(); //
> unlimited Queue
> ThreadPoolExecutor threadPoolExecutor =
> new ThreadPoolExecutor(
> THREAD_POOL_CORE_SIZE,
> THREAD_POOL_MAX_SIZE,
> THREAD_POOL_TIMEOUT_MILLIS,
> TimeUnit.MILLISECONDS,
> queue);
> {code}
> http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ThreadPoolExecutor.html
> {quote}
> Unbounded queues. Using an unbounded queue (for example a LinkedBlockingQueue
> without a predefined capacity) will cause new tasks to wait in the queue when
> all corePoolSize threads are busy. Thus, no more than corePoolSize threads
> will ever be created. (And the value of the maximumPoolSize therefore doesn't
> have any effect.) This may be appropriate when each task is completely
> independent of others, so tasks cannot affect each others execution; for
> example, in a web page server. While this style of queuing can be useful in
> smoothing out transient bursts of requests, it admits the possibility of
> unbounded work queue growth when commands continue to arrive on average
> faster than they can be processed.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
