> On Feb. 17, 2016, 12:41 a.m., Sid Wagle wrote: > > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java, > > line 358 > > <https://reviews.apache.org/r/43621/diff/1/?file=1251919#file1251919line358> > > > > Could you provide some guidance of size of the DTO object as a > > *comment* here to help gugage the size of the cache relative to memory > > available.
The values in the cache are HostRoleCommandStatusSummaryDTO objects grouped by stage id. The approx size of an entry in the cache is: 480 * nr_of_stages_in_request + 4 * nr_of_tasks. (I used this http://java-performance.info/memory-consumption-of-java-data-types-2/ as reference to estimate the footprint of HashMaps) Let's assume that there is 15000 node cluster that is being provisioned using Blueprint (this is one provision cluster request). That will create 15000 stages. Let's assume that we install all HDP services on each host (23 service * 5 avg components * 2 install_and_start tasks => roughly 200 tasks). I think we can consider the provision cluster request via Blueprints being the largest request that will yield the largest entry into the cache: 480 * 15000 + 4 * (200 * 15000) => 18 MB one entry in the cache. Most of the requests (like restart all components on a host, restart all components of a service etc) will result in a much smaller entry in the cache ~10 KB Hope this makes sense and my estimation is correct. > On Feb. 17, 2016, 12:41 a.m., Sid Wagle wrote: > > ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java, > > line 2433 > > <https://reviews.apache.org/r/43621/diff/1/?file=1251923#file1251923line2433> > > > > Why not store state in the DB and query active only? I don't know the reason why the active state state was never stored in the db. I'm just guessing that since this active flag is not used by the server it just needed by the client (eg. UI) of the server this was never stored in the db. The state is in the db already in a way (version and create time stamp) and the most recent records are considered the active ones. Having a separate active column is kind of redundant. > On Feb. 17, 2016, 12:41 a.m., Sid Wagle wrote: > > ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql, line 656 > > <https://reviews.apache.org/r/43621/diff/1/?file=1251925#file1251925line656> > > > > Status and role are not unique enough for an index, what sort of gain > > does this buy us ? The most often executed queries against host_role_command table are: SELECT DISTINCT task_id FROM host_role_command WHERE ((role = @P0) AND (status = @P1)) ORDER BY task_id SELECT COUNT(task_id) FROM host_role_command WHERE (status IN (@P0,@P1,@P2,@P3,@P4,@P5)) This index will cover these two queries. Also many of the other queries against this table filter by status and role so these two fields seems to be common accros the quries. - Sebastian ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/43621/#review119375 ----------------------------------------------------------- On Feb. 16, 2016, 10:39 p.m., Sebastian Toader wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/43621/ > ----------------------------------------------------------- > > (Updated Feb. 16, 2016, 10:39 p.m.) > > > Review request for Ambari, Myroslav Papirkovskyy, Sumit Mohanty, and Sid > Wagle. > > > Bugs: AMBARI-15011 > https://issues.apache.org/jira/browse/AMBARI-15011 > > > Repository: ambari > > > Description > ------- > > 1. Created new index on host_role_command table on (status, role) fields as > this table is being queries quite often by status and role > 2. Removed uneccessary querying of ambari server actions by hostname > 3. Cache HostRoleCommandStatusSummaryDTO computed objects using guava cache > 4. The method returning service configs made one call to the database to get > the active service config entities. Then retrieved from database all service > config entities and while building the response it was using the first > collection to determine which reponse item should be marked as > active/inactive. This has been modified to make only one roundtrip to the > database and determine the active flag using plain java code. > > > Diffs > ----- > > > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java > 91f2d30 > > ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java > d2fe4fc > > ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java > bd5fb5a > > ambari-server/src/main/java/org/apache/ambari/server/serveraction/ServerActionExecutor.java > 20cf5bb > > ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java > d3fdc65 > > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java > 88b3151 > ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql eded221 > ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 780e81a > ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql d4a1ddb > ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql > 64f2a4e > ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql c557cf6 > ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql adda30c > > ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java > bc4d397 > > ambari-server/src/test/java/org/apache/ambari/server/agent/AgentResourceTest.java > 510e1fb > > ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java > 1f90813 > > ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java > 862776f > > ambari-server/src/test/java/org/apache/ambari/server/state/ConfigHelperTest.java > 0cdf50a > > ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog222Test.java > aa4090e > > ambari-server/src/test/java/org/apache/ambari/server/utils/StageUtilsTest.java > 2d663d9 > > Diff: https://reviews.apache.org/r/43621/diff/ > > > Testing > ------- > > Manual testing: > > Created a cluster of 3 nodes waited 30 mins with UI running and collected > postgres sql statement stats, than left the cluster running for another 30 > mins wiht UI closed and getting new sql statement stats. Verified that the > number of execution of queries mentioned in the JIRA has decreased. > > Unit tests: > > Total run:884 > Total errors:0 > Total failures:0 > OK > > > Thanks, > > Sebastian Toader > >
