-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46714/
-----------------------------------------------------------
Review request for Ambari, Alejandro Fernandez, Jayush Luniya, Nate Cole, and
Sid Wagle.
Bugs: AMBARI-16131
https://issues.apache.org/jira/browse/AMBARI-16131
Repository: ambari
Description
-------
The underlying problem is that views are accessed off of the REST endpoint
({{/api/v1/views}}). This means that the Ambari REST API connector is going to
handle the request from its own threadpool. There is no way to configure Jetty
to use a different threadpool for the same connector. As a result, if a request
to load a view holds the Jetty thread hostage, eventually we will see thread
starvation and loss of service.
An example of this situation is a view which makes an innocent request to a
remote resource. If the view's request has a timeout of 60 seconds, then the
Jetty thread is going to be held for that amount of time. With concurrent users
and multiple instances of that view deployed, the Jetty threadpool can becomes
exhausted quickly.
Although there are more graceful ways of handling this situation, they mostly
involve substantial re-architecture and design:
- The use of a new connector and threadpool would require binding to another
port for view requests. This will cause problems with "local" views and their
assumption that if they run on the Ambari server they can share the same
session.
- The use of a
[Continuation|https://wiki.eclipse.org/Jetty/Feature/Continuations] in Jetty
which can suspend the incoming request. We would need the ability for views to
signal that they have completed their work in order to proceed with the
suspended request.
A quicker and far less invasive fix would be to create a filter which
intercepts requests for views. It will determine how many executing view
requests exist and decide if it will allow the new request through. For
example, if configured to allow a maximum of 10 concurrent view requests, then
the 11th request would be denied with an {{HTTP 503 - Service Unavailable}}.
Although the thread is temporarily used while the filter is processing, it's
quickly returned to the Jetty pool when it's determined there are too many
other running view requests.
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
5ff6a74
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
dc53172
ambari-server/src/main/java/org/apache/ambari/server/utils/VersionUtils.java
b07f7da
ambari-server/src/main/java/org/apache/ambari/server/view/ViewRegistry.java
d23fcad
ambari-server/src/main/java/org/apache/ambari/server/view/ViewThrottleFilter.java
PRE-CREATION
ambari-server/src/test/java/org/apache/ambari/server/view/ViewThrottleFilterTest.java
PRE-CREATION
Diff: https://reviews.apache.org/r/46714/diff/
Testing
-------
I modified an existing view to cause it to be very naughty. It held onto the
request for 10 seconds. Under normal Ambari operation, this caused two problems:
- The view partially rendered and needed to wait
- Ambari's API REST thread was held hostage
I turned down the number of available threads to only a handful and then had
several browsers open. This duplicated the loss of service until the view
returned control.
With the patch in place, Ambari was available and requests to the views beyond
the configured limit would return a 503.
Thanks,
Jonathan Hurley
