Re: Review Request 47757: Desired Configuration Cache Expiration Caused 10, 000's of Database Hits In Large Deployments

2016-05-24 Thread Nate Cole 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47757/#review134559
---


Ship it!




Ship It!

- Nate Cole


On May 23, 2016, 10:38 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47757/
> ---
> 
> (Updated May 23, 2016, 10:38 p.m.)
> 
> 
> Review request for Ambari and Nate Cole.
> 
> 
> Bugs: AMBARI-16830
> https://issues.apache.org/jira/browse/AMBARI-16830
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The stale configuration cache is set to expire every 1 minute. When it 
> expires, several areas of code invoke the same code path to rebuild the 
> cache, which essentially loops over all SCH's.
> 
> In a large deployment, that could be 10,000 SCHs every minute. At an average 
> of 10ms per database round trip, that's a lot more than 1 minute. Increasing 
> the cache invalidation doesn't really help since editing a configuration will 
> invalidate the cache anyway, causing the UI to hang.
> 
> Instead, we should be relying on the JPA EM relationships. At the same time, 
> there are several areas (like cluster health report) where the database/cache 
> hit can be done ahead of time once and passed in for every SCH.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/annotations/TransactionalLock.java
>  cd961ba 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatMonitor.java
>  a902a2c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  dd3d098 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java
>  8471825 
>   ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 
> d9df8b4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHelper.java 
> 854dd0b 
>   ambari-server/src/main/java/org/apache/ambari/server/state/Host.java 
> e9e5ea1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceComponentHost.java
>  4866148 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  6cf9933 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/host/HostImpl.java 
> e3665bf 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostImpl.java
>  feae6e5 
>   
> ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java
>  7e8c144 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java
>  1a7ff67 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/RefreshYarnCapacitySchedulerReleaseConfigTest.java
>  179f658 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/HostResourceProviderTest.java
>  ca332fd 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/ConfigHelperTest.java
>  15bcc53 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ClusterDeadlockTest.java
>  0a3286c 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostTest.java
>  57a7391 
> 
> Diff: https://reviews.apache.org/r/47757/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 47757: Desired Configuration Cache Expiration Caused 10, 000's of Database Hits In Large Deployments

2016-05-23 Thread Jonathan Hurley 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47757/#review134495
---




ambari-server/src/main/java/org/apache/ambari/annotations/TransactionalLock.java
 (lines 70 - 73)


The stale config cache can suffer the same transactional/cache-update 
problem as the HRC. We should be locking in some places to ensure that an 
invalidation doesn't re-populate a stale value before the transaction commits.



ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 (line 1168)


One of the many areas in which we can now fetch these once ahead of time 
and pass them around.



ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 


This was a bad idea; force a database hit every minute for every SCH ... 
The problem was that stale configs were not being reported correctly. It seems 
to be an issue with the cache invalidation while re-populating inside a 
transaction. Anway - bad idea - we cannot do this.



ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHelper.java 
(lines 190 - 193)


Method contract allows for null for those times we are lazy. Let's hope 
we're not lazy in a loop.



ambari-server/src/main/java/org/apache/ambari/server/state/ServiceComponentHost.java
 (line 181)


AMC loops over every SCH ... pass in the configs here as well so that they 
are only retrieved once.



ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
 (line 2382)


This should always be correct - there should be no reason to bypass it.


- Jonathan Hurley


On May 23, 2016, 10:38 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47757/
> ---
> 
> (Updated May 23, 2016, 10:38 p.m.)
> 
> 
> Review request for Ambari and Nate Cole.
> 
> 
> Bugs: AMBARI-16830
> https://issues.apache.org/jira/browse/AMBARI-16830
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The stale configuration cache is set to expire every 1 minute. When it 
> expires, several areas of code invoke the same code path to rebuild the 
> cache, which essentially loops over all SCH's.
> 
> In a large deployment, that could be 10,000 SCHs every minute. At an average 
> of 10ms per database round trip, that's a lot more than 1 minute. Increasing 
> the cache invalidation doesn't really help since editing a configuration will 
> invalidate the cache anyway, causing the UI to hang.
> 
> Instead, we should be relying on the JPA EM relationships. At the same time, 
> there are several areas (like cluster health report) where the database/cache 
> hit can be done ahead of time once and passed in for every SCH.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/annotations/TransactionalLock.java
>  cd961ba 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatMonitor.java
>  a902a2c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  dd3d098 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java
>  8471825 
>   ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 
> d9df8b4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHelper.java 
> 854dd0b 
>   ambari-server/src/main/java/org/apache/ambari/server/state/Host.java 
> e9e5ea1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceComponentHost.java
>  4866148 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  6cf9933 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/host/HostImpl.java 
> e3665bf 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostImpl.java
>  feae6e5 
>   
> ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java
>  7e8c144 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java
>  1a7ff67 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/RefreshYarnCapacitySchedulerReleaseConfigTest.java
>  179f658 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/HostResourceProviderTest.java
>  ca332fd 
>   
>

Review Request 47757: Desired Configuration Cache Expiration Caused 10, 000's of Database Hits In Large Deployments

2016-05-23 Thread Jonathan Hurley 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47757/
---

Review request for Ambari and Nate Cole.


Bugs: AMBARI-16830
https://issues.apache.org/jira/browse/AMBARI-16830


Repository: ambari


Description
---

The stale configuration cache is set to expire every 1 minute. When it expires, 
several areas of code invoke the same code path to rebuild the cache, which 
essentially loops over all SCH's.

In a large deployment, that could be 10,000 SCHs every minute. At an average of 
10ms per database round trip, that's a lot more than 1 minute. Increasing the 
cache invalidation doesn't really help since editing a configuration will 
invalidate the cache anyway, causing the UI to hang.

Instead, we should be relying on the JPA EM relationships. At the same time, 
there are several areas (like cluster health report) where the database/cache 
hit can be done ahead of time once and passed in for every SCH.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/annotations/TransactionalLock.java
 cd961ba 
  
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatMonitor.java
 a902a2c 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 dd3d098 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java
 8471825 
  ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 
d9df8b4 
  ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHelper.java 
854dd0b 
  ambari-server/src/main/java/org/apache/ambari/server/state/Host.java e9e5ea1 
  
ambari-server/src/main/java/org/apache/ambari/server/state/ServiceComponentHost.java
 4866148 
  
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
 6cf9933 
  ambari-server/src/main/java/org/apache/ambari/server/state/host/HostImpl.java 
e3665bf 
  
ambari-server/src/main/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostImpl.java
 feae6e5 
  
ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java
 7e8c144 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java
 1a7ff67 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/RefreshYarnCapacitySchedulerReleaseConfigTest.java
 179f658 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/HostResourceProviderTest.java
 ca332fd 
  
ambari-server/src/test/java/org/apache/ambari/server/state/ConfigHelperTest.java
 15bcc53 
  
ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ClusterDeadlockTest.java
 0a3286c 
  
ambari-server/src/test/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostTest.java
 57a7391 

Diff: https://reviews.apache.org/r/47757/diff/


Testing
---


Thanks,

Jonathan Hurley