-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47961/
-----------------------------------------------------------

(Updated May 27, 2016, 5:20 p.m.)


Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, Robert 
Nettleton, and Srimanth Gunturi.


Changes
-------

Optimization recommendation from Srimanth


Bugs: AMBARI-16913
    https://issues.apache.org/jira/browse/AMBARI-16913


Repository: ambari


Description
-------

Incoming requests from the web client (or from any REST API) will eventually be 
routed to the property provider / subresource framework. It is here were any 
JMX data is queried for within the context of the REST request. In large 
clusters, these requests can backup quite easily (even with a massive 
threadpool), causing UX degradations in the web client:

```
Thread [qtp-ambari-client-38]
        
JMXPropertyProvider(ThreadPoolEnabledPropertyProvider).populateResources(Set<Resource>,
 Request, Predicate) line: 168   
        JMXPropertyProvider.populateResources(Set<Resource>, Request, 
Predicate) line: 156      
        StackDefinedPropertyProvider.populateResources(Set<Resource>, Request, 
Predicate) line: 200     
        ClusterControllerImpl.populateResources(Type, Set<Resource>, Request, 
Predicate) line: 155      
        QueryImpl.queryForResources() line: 407 
        QueryImpl.execute() line: 217   
        ReadHandler.handleRequest(Request) line: 69     
        GetRequest(BaseRequest).process() line: 145     
```

Consider one of the calls made by the web client:
```
GET api/v1/clusters/c1/components/?
ServiceComponentInfo/category=MASTER&
fields=
ServiceComponentInfo/service_name,
host_components/HostRoles/display_name,
host_components/HostRoles/host_name,
host_components/HostRoles/state,
host_components/HostRoles/maintenance_state,
host_components/HostRoles/stale_configs,
host_components/HostRoles/ha_state,
host_components/HostRoles/desired_admin_state,
host_components/metrics/jvm/memHeapUsedM,
host_components/metrics/jvm/HeapMemoryMax,
host_components/metrics/jvm/HeapMemoryUsed,
host_components/metrics/jvm/memHeapCommittedM,
host_components/metrics/mapred/jobtracker/trackers_decommissioned,
host_components/metrics/cpu/cpu_wio,
host_components/metrics/rpc/client/RpcQueueTime_avg_time,
host_components/metrics/dfs/FSNamesystem/*,
host_components/metrics/dfs/namenode/Version,
host_components/metrics/dfs/namenode/LiveNodes,
host_components/metrics/dfs/namenode/DeadNodes,
host_components/metrics/dfs/namenode/DecomNodes,
host_components/metrics/dfs/namenode/TotalFiles,
host_components/metrics/dfs/namenode/UpgradeFinalized,
host_components/metrics/dfs/namenode/Safemode,
host_components/metrics/runtime/StartTime
```

This query is essentially saying that for every {{MASTER}}, get metrics from 
them. The problem is that in a large cluster, there could be 100 masters, yet 
the metrics being asked for are only for NameNode. As a result, the JMX 
endpoints for all 100 masters are queried - *live* - as part of the request.

There are two inherent flaws with this approach:

- Even with millisecond JMX response times, multiplying this by 100's and then 
adding parsing overhead causes a noticeable delay in the web client as the 
federated requests are blocking the main UX request

- Although there is a threadpool which scales up to service these requests - 
that only really works for 1 user. With multiple users logged in, you'd need 
100's upon 100's of threads pulling in the same JMX data

This data should never be queried for directly as part of the incoming REST 
requests. Instead, an autonomous pool of threads should be constantly 
retrieving these point-in-time metrics and updating a cache. The cache is then 
used to service all live REST requests. 
- On the first request to a resource, a cache miss occurs and no data is 
returned. I think this is acceptable since metrics take a few moments to 
populate anyway right now. As the web client polls, the next request should 
pickup the newly cached metrics.
- Only URLs which are being asked for by incoming REST requests should be 
considered for retrieval. After sometime, if they haven't been requested, then 
the headless threadpool can stop trying to update their data
- All JMX data will be parsed and stored in-memory, in an expiring cache


Diffs (updated)
-----

  ambari-server/src/main/java/org/apache/ambari/server/AmbariService.java 
186e272 
  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 7cfaf61 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java
 d6b9d0e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 f4a615c 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
 99a6cab 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java
 617553b 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractProviderModule.java
 04a8f0a 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackDefinedPropertyProvider.java
 6c40d14 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/jmx/JMXPropertyProvider.java
 1ccc5df 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/metrics/MetricPropertyProviderFactory.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/metrics/RestMetricsPropertyProvider.java
 6f2a134 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/metrics/ThreadPoolEnabledPropertyProvider.java
 6f4a6ea 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/utilities/BufferedThreadPoolExecutorCompletionService.java
 4d6daa6 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/utilities/ScalingThreadPoolExecutor.java
 7a5e479 
  
ambari-server/src/main/java/org/apache/ambari/server/state/services/MetricsRetrievalService.java
 PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java
 5d65ea7 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackDefinedPropertyProviderTest.java
 32e84cb 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/metrics/JMXPropertyProviderTest.java
 1628f1f 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/metrics/RestMetricsPropertyProviderTest.java
 f78024f 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/test/BufferedThreadPoolExecutorCompletionServiceTest.java
 f47068c 
  
ambari-server/src/test/java/org/apache/ambari/server/utils/SynchronousThreadPoolExecutor.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/47961/diff/


Testing
-------

PENDING


Thanks,

Jonathan Hurley

Reply via email to