Re: Review Request 46785: Reduce NN start time by removing redundant haadmin calls.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46785/ --- (Updated April 28, 2016, 3:25 p.m.) Review request for Ambari and Vitalyi Brodetskyi. Bugs: AMBARI-16162 https://issues.apache.org/jira/browse/AMBARI-16162 Repository: ambari Description (updated) --- I found this while playing around with one of the clusters: Time 14:21:50 - 14:22:13 was spend just re-running haadmin calls, while we already figured This should give ~15s time reduce Diffs - ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py 357eb71 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/setup_ranger_hdfs.py 040c69c ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py f56a0d2 Diff: https://reviews.apache.org/r/46785/diff/ Testing --- mvn clean test Thanks, Andrew Onischuk
Re: Review Request 46785: Reduce NN start time by removing redundant haadmin calls.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46785/#review130922 --- Ship it! Ship It! - Vitalyi Brodetskyi On Квітень 28, 2016, 3:25 після полудня, Andrew Onischuk wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/46785/ > --- > > (Updated Квітень 28, 2016, 3:25 після полудня) > > > Review request for Ambari and Vitalyi Brodetskyi. > > > Bugs: AMBARI-16162 > https://issues.apache.org/jira/browse/AMBARI-16162 > > > Repository: ambari > > > Description > --- > > I found this while playing around with one of the clusters: > > > Time 14:21:50 - 14:22:13 was spend just re-running haadmin calls, while we > already figured > > This should give ~15s time reduce > > > Diffs > - > > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py > 357eb71 > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/setup_ranger_hdfs.py > 040c69c > ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py f56a0d2 > > Diff: https://reviews.apache.org/r/46785/diff/ > > > Testing > --- > > mvn clean test > > > Thanks, > > Andrew Onischuk > >
Re: Review Request 46785: Reduce NN start time by removing redundant haadmin calls.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46785/ --- (Updated April 28, 2016, 3:22 p.m.) Review request for Ambari and Vitalyi Brodetskyi. Bugs: AMBARI-16162 https://issues.apache.org/jira/browse/AMBARI-16162 Repository: ambari Description (updated) --- I found this while playing around with one of the clusters: 2016-04-27 14:21:50,062 - HdfsResource['/tmp'] {'security_enabled': False, 'only_if': 'ambari-sudo.sh su hdfs -l -s /bin/bash -c \'export PATH=\'"\'"\'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin\'"\'"\' ; /usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs --config /usr/hdp/current/hadoop-client/conf haadmin -getServiceState nn2 | grep active\'', 'keytab': [EMPTY], 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'default_fs': 'wasb://hwx-test-hdp24-had...@storagewuteam02.blob.core.windows.net', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': None, 'user': 'hdfs', 'owner': 'hdfs', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'mode' : 0777} 2016-04-27 14:21:58,266 - Skipping 'HdfsResource['/tmp']' because it is in ignore file /var/lib/ambari-agent/data/.hdfs_resource_ignore. 2016-04-27 14:21:58,267 - HdfsResource['/user/ambari-qa'] {'security_enabled': False, 'only_if': 'ambari-sudo.sh su hdfs -l -s /bin/bash -c \'export PATH=\'"\'"\'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin\'"\'"\' ; /usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs --config /usr/hdp/current/hadoop-client/conf haadmin -getServiceState nn2 | grep active\'', 'keytab': [EMPTY], 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'default_fs': 'wasb://hwx-test-hdp24-had...@storagewuteam02.blob.core.windows.net', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': None, 'user': 'hdfs', 'owner': 'ambari-qa', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_e xecute'], 'mode': 0770} 2016-04-27 14:22:05,522 - Skipping 'HdfsResource['/user/ambari-qa']' because it is in ignore file /var/lib/ambari-agent/data/.hdfs_resource_ignore. 2016-04-27 14:22:05,523 - HdfsResource[None] {'security_enabled': False, 'only_if': 'ambari-sudo.sh su hdfs -l -s /bin/bash -c \'export PATH=\'"\'"\'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin\'"\'"\' ; /usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs --config /usr/hdp/current/hadoop-client/conf haadmin -getServiceState nn2 | grep active\'', 'keytab': [EMPTY], 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'default_fs': 'wasb://hwx-test-hdp24-had...@storagewuteam02.blob.core.windows.net', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': None, 'user': 'hdfs', 'action': ['execute'], 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf'} 2016-04-27 14:22:13,846 - No resources to create. 'create_on_execute' or 'delete_on_execute' wasn't triggered before this 'execute' action. 2016-04-27 14:22:13,846 - Ranger admin not installed Time 14:21:50 - 14:22:13 was spend just re-running haadmin calls, while we already figured This should give ~15s time reduce Diffs - ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py 357eb71 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/setup_ranger_hdfs.py 040c69c ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py f56a0d2 Diff: https://reviews.apache.org/r/46785/diff/ Testing --- mvn clean test Thanks, Andrew Onischuk
Review Request 46785: Reduce NN start time by removing redundant haadmin calls.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46785/ --- Review request for Ambari and Vitalyi Brodetskyi. Bugs: AMBARI-16162 https://issues.apache.org/jira/browse/AMBARI-16162 Repository: ambari Description --- I found this while playing around with Humbold cluster: 2016-04-27 14:21:50,062 - HdfsResource['/tmp'] {'security_enabled': False, 'only_if': 'ambari-sudo.sh su hdfs -l -s /bin/bash -c \'export PATH=\'"\'"\'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin\'"\'"\' ; /usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs --config /usr/hdp/current/hadoop-client/conf haadmin -getServiceState nn2 | grep active\'', 'keytab': [EMPTY], 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'default_fs': 'wasb://hwx-test-hdp24-had...@storagewuteam02.blob.core.windows.net', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': None, 'user': 'hdfs', 'owner': 'hdfs', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'mode' : 0777} 2016-04-27 14:21:58,266 - Skipping 'HdfsResource['/tmp']' because it is in ignore file /var/lib/ambari-agent/data/.hdfs_resource_ignore. 2016-04-27 14:21:58,267 - HdfsResource['/user/ambari-qa'] {'security_enabled': False, 'only_if': 'ambari-sudo.sh su hdfs -l -s /bin/bash -c \'export PATH=\'"\'"\'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin\'"\'"\' ; /usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs --config /usr/hdp/current/hadoop-client/conf haadmin -getServiceState nn2 | grep active\'', 'keytab': [EMPTY], 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'default_fs': 'wasb://hwx-test-hdp24-had...@storagewuteam02.blob.core.windows.net', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': None, 'user': 'hdfs', 'owner': 'ambari-qa', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_e xecute'], 'mode': 0770} 2016-04-27 14:22:05,522 - Skipping 'HdfsResource['/user/ambari-qa']' because it is in ignore file /var/lib/ambari-agent/data/.hdfs_resource_ignore. 2016-04-27 14:22:05,523 - HdfsResource[None] {'security_enabled': False, 'only_if': 'ambari-sudo.sh su hdfs -l -s /bin/bash -c \'export PATH=\'"\'"\'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin\'"\'"\' ; /usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs --config /usr/hdp/current/hadoop-client/conf haadmin -getServiceState nn2 | grep active\'', 'keytab': [EMPTY], 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'default_fs': 'wasb://hwx-test-hdp24-had...@storagewuteam02.blob.core.windows.net', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': None, 'user': 'hdfs', 'action': ['execute'], 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf'} 2016-04-27 14:22:13,846 - No resources to create. 'create_on_execute' or 'delete_on_execute' wasn't triggered before this 'execute' action. 2016-04-27 14:22:13,846 - Ranger admin not installed Time 14:21:50 - 14:22:13 was spend just re-running haadmin calls, while we already figured This should give ~15s time reduce for Humboldt. Diffs - ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py 357eb71 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/setup_ranger_hdfs.py 040c69c ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py f56a0d2 Diff: https://reviews.apache.org/r/46785/diff/ Testing --- mvn clean test Thanks, Andrew Onischuk