-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46714/#review130710
-----------------------------------------------------------


Ship it!




Ship It!

- Sid Wagle


On April 26, 2016, 10:03 p.m., Jonathan Hurley wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46714/
> -----------------------------------------------------------
> 
> (Updated April 26, 2016, 10:03 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jayush Luniya, Nate Cole, and 
> Sid Wagle.
> 
> 
> Bugs: AMBARI-16131
>     https://issues.apache.org/jira/browse/AMBARI-16131
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> The underlying problem is that views are accessed off of the REST endpoint 
> ({{/api/v1/views}}). This means that the Ambari REST API connector is going 
> to handle the request from its own threadpool. There is no way to configure 
> Jetty to use a different threadpool for the same connector. As a result, if a 
> request to load a view holds the Jetty thread hostage, eventually we will see 
> thread starvation and loss of service.
> 
> An example of this situation is a view which makes an innocent request to a 
> remote resource. If the view's request has a timeout of 60 seconds, then the 
> Jetty thread is going to be held for that amount of time. With concurrent 
> users and multiple instances of that view deployed, the Jetty threadpool can 
> becomes exhausted quickly.
> 
> Although there are more graceful ways of handling this situation, they mostly 
> involve substantial re-architecture and design:
> - The use of a new connector and threadpool would require binding to another 
> port for view requests. This will cause problems with "local" views and their 
> assumption that if they run on the Ambari server they can share the same 
> session.
> - The use of a 
> [Continuation|https://wiki.eclipse.org/Jetty/Feature/Continuations] in Jetty 
> which can suspend the incoming request. We would need the ability for views 
> to signal that they have completed their work in order to proceed with the 
> suspended request.
> 
> A quicker and far less invasive fix would be to create a filter which 
> intercepts requests for views. It will determine how many executing view 
> requests exist and decide if it will allow the new request through. For 
> example, if configured to allow a maximum of 10 concurrent view requests, 
> then the 11th request would be denied with an {{HTTP 503 - Service 
> Unavailable}}. Although the thread is temporarily used while the filter is 
> processing, it's quickly returned to the Jetty pool when it's determined 
> there are too many other running view requests.
> 
> 
> Diffs
> -----
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  5ff6a74 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
>  dc53172 
>   
> ambari-server/src/main/java/org/apache/ambari/server/utils/VersionUtils.java 
> b07f7da 
>   ambari-server/src/main/java/org/apache/ambari/server/view/ViewRegistry.java 
> d23fcad 
>   
> ambari-server/src/main/java/org/apache/ambari/server/view/ViewThrottleFilter.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/view/ViewThrottleFilterTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/46714/diff/
> 
> 
> Testing
> -------
> 
> I modified an existing view to cause it to be very naughty. It held onto the 
> request for 10 seconds. Under normal Ambari operation, this caused two 
> problems:
> - The view partially rendered and needed to wait
> - Ambari's API REST thread was held hostage
> 
> I turned down the number of available threads to only a handful and then had 
> several browsers open. This duplicated the loss of service until the view 
> returned control.
> 
> With the patch in place, Ambari was available and requests to the views 
> beyond the configured limit would return a 503.
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Reply via email to