----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35945/#review89614 -----------------------------------------------------------
Ship it! Ship It! - Alejandro Fernandez On June 27, 2015, 2:37 a.m., Jonathan Hurley wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/35945/ > ----------------------------------------------------------- > > (Updated June 27, 2015, 2:37 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Mahadev Konar, Nate Cole, > Sumit Mohanty, and Tom Beerbower. > > > Bugs: AMBARI-12178 > https://issues.apache.org/jira/browse/AMBARI-12178 > > > Repository: ambari > > > Description > ------- > > During an upgrade of a large cluster, the memory used by Ambari grows until > it is fully consumed. This, however, only happens when the Upgrade Dialog > page is open. If that popup is closed, the memory usage stays relatively > constant. Based on heap dumps, the largest offenders are StageEnity and, as a > result, byte[] > > Long story short here is that we have a cache in ActionDBAccessorImpl that > holds onto entities. Because of this, the underlying UnitOfWork map is never > released and holds onto all StageEntity instances. Eventually, items are > purged from this cache, but it's not fast enough to free up memory resources. > > Without ripping apart Ambari or making dangerous cache reference changes, the > easiest solution was to ensure that the fields causing the StageEntity to be > large were lazy loaded since most of the time these entities are just sitting > around in the EntityManager. > > Before change > ``` > Class Name | Objects | Shallow Heap | > Retained Heap > ------------------------------------------------------------------------------------------ > org.apache.ambari.server.orm.entities.StageEntity| 292,356 | 18,466,176 | > 3,575,693,136 > ------------------------------------------------------------------------------------------ > ``` > > First patch > ``` > Class Name | Objects | Shallow Heap | > Retained Heap > ------------------------------------------------------------------------------------------ > org.apache.ambari.server.orm.entities.StageEntity| 193,715 | 15,716,640 | > 255,318,392 > ------------------------------------------------------------------------------------------ > ``` > > Second patch > ``` > Class Name | Objects | Shallow Heap | > Retained Heap > ------------------------------------------------------------------------------------------ > org.apache.ambari.server.orm.entities.StageEntity| 4,410 | 423,360 | > 61,598,736 > ------------------------------------------------------------------------------------------ > ``` > > > Diffs > ----- > > > ambari-server/src/main/java/org/apache/ambari/server/actionmanager/HostRoleCommand.java > 20ec9ea > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java > 664fae3 > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeGroupResourceProvider.java > eb34d63 > ambari-server/src/main/java/org/apache/ambari/server/orm/dao/StageDAO.java > b354841 > > ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostEntity.java > 9f3f70c > > ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java > c2b97d6 > > ambari-server/src/main/java/org/apache/ambari/server/topology/HostRequest.java > f63ba3f > > ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java > 31363b4 > > Diff: https://reviews.apache.org/r/35945/diff/ > > > Testing > ------- > > Performed a cluster upgrade and monitored the memory usage. 200,000 > StageEntity used to occupy 3.5GB of heap; now they take up about 150MB. > > Tests run: 3099, Failures: 0, Errors: 0, Skipped: 28 > > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD SUCCESS > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 24:13 min > [INFO] Finished at: 2015-06-26T21:32:27-04:00 > [INFO] Final Memory: 46M/1414M > [INFO] > ------------------------------------------------------------------------ > > > Thanks, > > Jonathan Hurley > >
