[
https://issues.apache.org/jira/browse/CLOUDSTACK-4855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13812699#comment-13812699
]
ASF subversion and git services commented on CLOUDSTACK-4855:
-------------------------------------------------------------
Commit 269a4ef11ee151fa408a7dd1f2e69cd1f7f05191 in branch refs/heads/master
from [~koushikd]
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=269a4ef ]
CLOUDSTACK-4855: Throttle based on the # of outstanding requests to the
directly managed HV host (direct agents)
Cloudstack sends requests to directly managed HV hosts (direct agents) using
the direct agent thread pool. The size of the pool is determined by global
config direct.agent.pool.size defaulted to 500.
Currently there is no restriction on the number of threads a direct agent can
use from this shared thread pool to send requests to the host. This is fine as
long as the host is responding to requests
in a reasonable amount of time. But if there is a considerable delay in getting
response, the thread remain blocked for that much time. As more commands are
send to the slow host threads keep getting
blocked. This can eventually lead to a situation where requests to healthy
hosts cannot be processed as there are not enough free threads.
The problem being addressed here is to localize the impact of few bad hosts, so
that entire management server is not affected.
One such way is to throttle based on the # of outstanding requests on per host
basis. The outstanding requests to a host will be a % of direct agent pool
size. This is configurable based on
direct.agent.thread.cap. The default value is 0.1 or 10%, a value of 1 would
mean the old behavior where there is no upper cap. This will ensure that the
impacted host will be bound by a upper cap on the number of threads it can use
to process requests and not the entire pool.
> Throttle based on the # of outstanding requests to the directly managed HV
> host (direct agents)
> -----------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-4855
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4855
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: pre-4.0.0, 4.1.0, 4.2.0
> Reporter: Koushik Das
> Assignee: Koushik Das
> Fix For: 4.3.0
>
>
> Currently requests to all direct managed HV hosts (direct agents) are handled
> by the direct agent thread pool. The size of the pool is determined by global
> config direct.agent.pool.size defaulted to 500.
> Currently there is no restriction on the number of requests that can be sent
> to a given HV host. The down side is if a lot commands are getting generated
> for some specific hosts (may be there is some issue with the host, the host
> is slow in responding and there is a pile up of outstanding requests), it may
> essentially starve the requests going to other hosts due to unavailability of
> direct agent threads as most of them will be serving a very few hosts.
> The problem being addressed is that a few bad hosts should not affect the
> entire management server. The solution is to localize the impact of the bad
> hosts.
> One such way is to throttle based on the # of outstanding requests on per
> host basis. The outstanding requests will be a % of the direct agent pool
> size.
> This will ensure that the impacted host will be bound by a upper cap on the
> number of threads it can use to process request and not the entire pool.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
