> On April 26, 2016, 7:48 p.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java,
> >  lines 555-556
> > <https://reviews.apache.org/r/46714/diff/1/?file=1362444#file1362444line555>
> >
> >     I don't know what kind of views are out there, but it's conceivable 
> > that many of them will take more than 2 seconds to respond.  Will keeping 
> > this too "low" result in too-often 503?

In a happy world, there are enough threads and even if the view takes 10 
seconds to load, we have enough leftover threads to fullfil normal API 
requests. So the value doesn't matter.

In the case you're talking about, we've run out of threads and the view still 
takes 10 seconds to load. The hope is that while the new view request is 
waiting its 2 seconds, one of the other 10 or 15 view threads completes and 
releases the lock. If it doesn't you will get a 503. But the longer you hold 
onto that thread waiting, the less time there is for Ambari requests to be 
accepted.

You could argue it should only be 1 second. Initially, I had it failing fast 
with the 503, but figured a little bit of wait time is OK. Views are 2nd class 
citizens to Ambari; I'd rather them timeout then the entire webapp.


- Jonathan


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46714/#review130689
-----------------------------------------------------------


On April 26, 2016, 6:03 p.m., Jonathan Hurley wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46714/
> -----------------------------------------------------------
> 
> (Updated April 26, 2016, 6:03 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jayush Luniya, Nate Cole, and 
> Sid Wagle.
> 
> 
> Bugs: AMBARI-16131
>     https://issues.apache.org/jira/browse/AMBARI-16131
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> The underlying problem is that views are accessed off of the REST endpoint 
> ({{/api/v1/views}}). This means that the Ambari REST API connector is going 
> to handle the request from its own threadpool. There is no way to configure 
> Jetty to use a different threadpool for the same connector. As a result, if a 
> request to load a view holds the Jetty thread hostage, eventually we will see 
> thread starvation and loss of service.
> 
> An example of this situation is a view which makes an innocent request to a 
> remote resource. If the view's request has a timeout of 60 seconds, then the 
> Jetty thread is going to be held for that amount of time. With concurrent 
> users and multiple instances of that view deployed, the Jetty threadpool can 
> becomes exhausted quickly.
> 
> Although there are more graceful ways of handling this situation, they mostly 
> involve substantial re-architecture and design:
> - The use of a new connector and threadpool would require binding to another 
> port for view requests. This will cause problems with "local" views and their 
> assumption that if they run on the Ambari server they can share the same 
> session.
> - The use of a 
> [Continuation|https://wiki.eclipse.org/Jetty/Feature/Continuations] in Jetty 
> which can suspend the incoming request. We would need the ability for views 
> to signal that they have completed their work in order to proceed with the 
> suspended request.
> 
> A quicker and far less invasive fix would be to create a filter which 
> intercepts requests for views. It will determine how many executing view 
> requests exist and decide if it will allow the new request through. For 
> example, if configured to allow a maximum of 10 concurrent view requests, 
> then the 11th request would be denied with an {{HTTP 503 - Service 
> Unavailable}}. Although the thread is temporarily used while the filter is 
> processing, it's quickly returned to the Jetty pool when it's determined 
> there are too many other running view requests.
> 
> 
> Diffs
> -----
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  5ff6a74 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
>  dc53172 
>   
> ambari-server/src/main/java/org/apache/ambari/server/utils/VersionUtils.java 
> b07f7da 
>   ambari-server/src/main/java/org/apache/ambari/server/view/ViewRegistry.java 
> d23fcad 
>   
> ambari-server/src/main/java/org/apache/ambari/server/view/ViewThrottleFilter.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/view/ViewThrottleFilterTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/46714/diff/
> 
> 
> Testing
> -------
> 
> I modified an existing view to cause it to be very naughty. It held onto the 
> request for 10 seconds. Under normal Ambari operation, this caused two 
> problems:
> - The view partially rendered and needed to wait
> - Ambari's API REST thread was held hostage
> 
> I turned down the number of available threads to only a handful and then had 
> several browsers open. This duplicated the loss of service until the view 
> returned control.
> 
> With the patch in place, Ambari was available and requests to the views 
> beyond the configured limit would return a 503.
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Reply via email to