[
https://issues.apache.org/jira/browse/AMBARI-16131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hurley updated AMBARI-16131:
-------------------------------------
Fix Version/s: 2.2-next
> Prevent Views From Causing a Loss of Service For Ambari
> -------------------------------------------------------
>
> Key: AMBARI-16131
> URL: https://issues.apache.org/jira/browse/AMBARI-16131
> Project: Ambari
> Issue Type: Bug
> Components: ambari-views
> Affects Versions: 2.0.0
> Reporter: Jonathan Hurley
> Assignee: Jonathan Hurley
> Priority: Critical
> Fix For: 2.4.0, 2.2-next
>
> Attachments: AMBARI-16131.patch
>
>
> The underlying problem is that views are accessed off of the REST endpoint
> ({{/api/v1/views}}). This means that the Ambari REST API connector is going
> to handle the request from its own threadpool. There is no way to configure
> Jetty to use a different threadpool for the same connector. As a result, if a
> request to load a view holds the Jetty thread hostage, eventually we will see
> thread starvation and loss of service.
> An example of this situation is a view which makes an innocent request to a
> remote resource. If the view's request has a timeout of 60 seconds, then the
> Jetty thread is going to be held for that amount of time. With concurrent
> users and multiple instances of that view deployed, the Jetty threadpool can
> becomes exhausted quickly.
> Although there are more graceful ways of handling this situation, they mostly
> involve substantial re-architecture and design:
> - The use of a new connector and threadpool would require binding to another
> port for view requests. This will cause problems with "local" views and their
> assumption that if they run on the Ambari server they can share the same
> session.
> - The use of a
> [Continuation|https://wiki.eclipse.org/Jetty/Feature/Continuations] in Jetty
> which can suspend the incoming request. We would need the ability for views
> to signal that they have completed their work in order to proceed with the
> suspended request.
> A quicker and far less invasive fix would be to create a filter which
> intercepts requests for views. It will determine how many executing view
> requests exist and decide if it will allow the new request through. For
> example, if configured to allow a maximum of 10 concurrent view requests,
> then the 11th request would be denied with an {{HTTP 503 - Service
> Unavailable}}. Although the thread is temporarily used while the filter is
> processing, it's quickly returned to the Jetty pool when it's determined
> there are too many other running view requests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
