Andrew Onischuk created AMBARI-10992:
----------------------------------------
Summary: Ambari is too slow to start/restart components
Key: AMBARI-10992
URL: https://issues.apache.org/jira/browse/AMBARI-10992
Project: Ambari
Issue Type: Epic
Reporter: Andrew Onischuk
Assignee: Andrew Onischuk
Priority: Critical
Fix For: Ambari-2.1
*PROBLEM REASONING*
This log of HiveServer2 restart (which took 5 minutes) brightly describes what
takes the most time usually for no big reason:
{noformat}
014-12-10 17:09:29,060 - ExecuteHadoop['fs -ls
hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True, 'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir':
'/etc/hadoop/conf'}
2014-12-10 17:09:29,073 - Execute['hadoop --config /etc/hadoop/conf fs -ls
hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True,
'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:09:46,301 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz':
No such file or directory
2014-12-10 17:09:46,301 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/hive']
{'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf',
'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs',
'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:09:46,303 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir
`rpm -q hadoop | grep -q "hadoop-1" || echo "-p"`
hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config /etc/hadoop/conf fs
-chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config
/etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/hive']
{'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER}
> /dev/null ; hadoop --config /etc/hadoop/conf fs -ls
hdfs:///hdp/apps/2.2.0.0-2041/hive'", 'user': 'hdfs', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:10:29,989 -
CopyFromLocal['/usr/hdp/current/hive-client/hive.tar.gz'] {'hadoop_bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs',
'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir':
'hdfs:///hdp/apps/2.2.0.0-2041/hive', 'hadoop_conf_dir': '/etc/hadoop/conf',
'mode': 0444}
2014-12-10 17:10:30,017 - ExecuteHadoop['fs -copyFromLocal
/usr/hdp/current/hive-client/hive.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/hive']
{'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER}
> /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls
hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'", 'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir':
'/etc/hadoop/conf'}
2014-12-10 17:10:48,275 - Execute['hadoop --config /etc/hadoop/conf fs
-copyFromLocal /usr/hdp/current/hive-client/hive.tar.gz
hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'logoutput': False, 'try_sleep': 0,
'environment': ..., 'tries': 1, 'user': 'hdfs', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:07,134 - ExecuteHadoop['fs -chown hdfs:hadoop
hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir':
'/etc/hadoop/conf'}
2014-12-10 17:11:07,135 - Execute['hadoop --config /etc/hadoop/conf fs -chown
hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput':
False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:16,533 - ExecuteHadoop['fs -chmod 444
hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir':
'/etc/hadoop/conf'}
2014-12-10 17:11:16,534 - Execute['hadoop --config /etc/hadoop/conf fs -chmod
444 hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': False,
'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:29,515 - ExecuteHadoop['fs -ls
hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir':
'/etc/hadoop/conf'}
2014-12-10 17:11:29,516 - Execute['hadoop --con014-12-10 17:09:29,060 -
ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz']
{'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user':
'hcat', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:09:29,073 - Execute['hadoop --config /etc/hadoop/conf fs -ls
hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True,
'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:09:46,301 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz':
No such file or directory
2014-12-10 17:09:46,301 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/hive']
{'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf',
'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs',
'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:09:46,303 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir
`rpm -q hadoop | grep -q "hadoop-1" || echo "-p"`
hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config /etc/hadoop/conf fs
-chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config
/etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/hive']
{'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER}
> /dev/null ; hadoop --config /etc/hadoop/conf fs -ls
hdfs:///hdp/apps/2.2.0.0-2041/hive'", 'user': 'hdfs', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:10:29,989 -
CopyFromLocal['/usr/hdp/current/hive-client/hive.tar.gz'] {'hadoop_bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs',
'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir':
'hdfs:///hdp/apps/2.2.0.0-2041/hive', 'hadoop_conf_dir': '/etc/hadoop/conf',
'mode': 0444}
2014-12-10 17:10:30,017 - ExecuteHadoop['fs -copyFromLocal
/usr/hdp/current/hive-client/hive.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/hive']
{'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER}
> /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls
hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'", 'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir':
'/etc/hadoop/conf'}
2014-12-10 17:10:48,275 - Execute['hadoop --config /etc/hadoop/conf fs
-copyFromLocal /usr/hdp/current/hive-client/hive.tar.gz
hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'logoutput': False, 'try_sleep': 0,
'environment': ..., 'tries': 1, 'user': 'hdfs', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:07,134 - ExecuteHadoop['fs -chown hdfs:hadoop
hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir':
'/etc/hadoop/conf'}
2014-12-10 17:11:07,135 - Execute['hadoop --config /etc/hadoop/conf fs -chown
hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput':
False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:16,533 - ExecuteHadoop['fs -chmod 444
hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir':
'/etc/hadoop/conf'}
2014-12-10 17:11:16,534 - Execute['hadoop --config /etc/hadoop/conf fs -chmod
444 hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': False,
'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:29,515 - ExecuteHadoop['fs -ls
hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir':
'/etc/hadoop/conf'}
2014-12-10 17:11:29,516 - Execute['hadoop --config /etc/hadoop/conf fs -ls
hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'try_sleep':
0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:45,791 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz':
No such file or directory
2014-12-10 17:11:45,791 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/pig']
{'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf',
'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs',
'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:11:45,794 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir
`rpm -q hadoop | grep -q "hadoop-1" || echo "-p"`
hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs -chmod
555 hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs
-chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'not_if': "/usr/bin/sudo su
hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config
/etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig'", 'user': 'hdfs',
'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:12:31,703 -
CopyFromLocal['/usr/hdp/current/pig-client/pig.tar.gz'] {'hadoop_bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs',
'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir':
'hdfs:///hdp/apps/2.2.0.0-2041/pig', 'hadoop_conf_dir': '/etc/hadoop/conf',
'mode': 0444}
2014-12-10 17:12:31,703 - ExecuteHadoop['fs -copyFromLocal
/usr/hdp/current/pig-client/pig.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/pig']
{'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER}
> /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls
hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'", 'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir':
'/etc/hadoop/conf'}
2014-12-10 17:12:49,508 - Execute['hadoop --config /etc/hadoop/conf fs
-copyFromLocal /usr/hdp/current/pig-client/pig.tar.gz
hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'logoutput': False, 'try_sleep': 0,
'environment': ..., 'tries': 1, 'user': 'hdfs', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:09,506 - ExecuteHadoop['fs -chown hdfs:hadoop
hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir':
'/etc/hadoop/conf'}
2014-12-10 17:13:09,507 - Execute['hadoop --config /etc/hadoop/conf fs -chown
hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False,
'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:18,968 - ExecuteHadoop['fs -chmod 444
hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir':
'/etc/hadoop/conf'}
2014-12-10 17:13:18,969 - Execute['hadoop --config /etc/hadoop/conf fs -chmod
444 hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False,
'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:32,936 - ExecuteHadoop['fs -ls
hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput':
True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat',
'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:13:32,937 - Execute['hadoop --config /etc/hadoop/conf fs -ls
hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput':
True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:52,891 - ls:
`hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar': No such file or
directory
2014-12-10 17:13:52,892 -
HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'security_enabled':
False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs',
'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:13:52,904 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir
`rpm -q hadoop | grep -q "hadoop-1" || echo "-p"`
hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs
-chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config
/etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce']
{'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER}
> /dev/null ; hadoop --config /etc/hadoop/conf fs -ls
hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'", 'user': 'hdfs', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:03,832 - Skipping Execute['hadoop --config /etc/hadoop/conf fs
-mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"`
hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs
-chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config
/etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] due
to not_if
2014-12-10 17:14:03,833 -
CopyFromLocal['/usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar']
{'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop',
'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir':
'hdfs:///hdp/apps/2.2.0.0-2041/mapreduce', 'hadoop_conf_dir':
'/etc/hadoop/conf', 'mode': 0444}
2014-12-10 17:14:03,836 - ExecuteHadoop['fs -copyFromLocal
/usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar
hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'not_if': "/usr/bin/sudo su hdfs -l
-s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ;
PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls
hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'", 'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir':
'/etc/hadoop/conf'}
2014-12-10 17:14:12,682 - Execute['hadoop --config /etc/hadoop/conf fs
-copyFromLocal /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar
hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'logoutput': False, 'try_sleep': 0,
'environment': ..., 'tries': 1, 'user': 'hdfs', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:22,350 - ExecuteHadoop['fs -chown hdfs:hadoop
hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir':
'/etc/hadoop/conf'}
2014-12-10 17:14:22,352 - Execute['hadoop --config /etc/hadoop/conf fs -chown
hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar']
{'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user':
'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:34,163 - ExecuteHadoop['fs -chmod 444
hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir':
'/etc/hadoop/conf'}
2014-12-10 17:14:34,164 - Execute['hadoop --config /etc/hadoop/conf fs -chmod
444 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar']
{'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user':
'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:50,851 - Could not find file:
/usr/hdp/current/sqoop-client/sqoop.tar.gz
2014-12-10 17:14:50,862 - XmlConfig['webhcat-site.xml'] {'owner': 'hcat',
'group': 'hadoop', 'conf_dir': '/etc/hive-webhcat/conf',
'configuration_attributes': ..., 'configurations': ...}
2014-12-10 17:14:50,979 - Generating config:
/etc/hive-webhcat/conf/webhcat-site.xml
2014-12-10 17:14:50,980 - File['/etc/hive-webhcat/conf/webhcat-site.xml']
{'owner': 'hcat', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode':
None, 'encoding': 'UTF-8'}
2014-12-10 17:14:50,983 - Writing
File['/etc/hive-webhcat/conf/webhcat-site.xml'] because it doesn't exist
2014-12-10 17:14:51,114 - Changing owner for
/etc/hive-webhcat/conf/webhcat-site.xml from 0 to hcat
2014-12-10 17:14:51,169 - Changing group for
/etc/hive-webhcat/conf/webhcat-site.xml from 0 to hadoop
2014-12-10 17:14:51,221 - File['/etc/hive-webhcat/conf/webhcat-env.sh']
{'content': InlineTemplate(...), 'owner': 'hcat', 'group': 'hadoop'}
2014-12-10 17:14:51,222 - Writing File['/etc/hive-webhcat/conf/webhcat-env.sh']
because it doesn't exist
2014-12-10 17:14:51,312 - Changing owner for
/etc/hive-webhcat/conf/webhcat-env.sh from 0 to hcat
2014-12-10 17:14:51,367 - Changing group for
/etc/hive-webhcat/conf/webhcat-env.sh from 0 to hadoop
2014-12-10 17:14:51,423 - Execute['env
HADOOP_HOME=/usr/hdp/current/hadoop-client
/usr/hdp/current/hive-webhcat/sbin/webhcat_server.sh start'] {'not_if': 'ls
/var/run/webhcat/webhcat.pid >/dev/null 2>&1 && ps -p `cat
/var/run/webhcat/webhcat.pid` >/dev/null 2>&1', 'user': 'hcat'}fig
/etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz']
{'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user':
'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:45,791 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz':
No such file or directory
2014-12-10 17:11:45,791 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/pig']
{'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf',
'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs',
'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:11:45,794 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir
`rpm -q hadoop | grep -q "hadoop-1" || echo "-p"`
hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs -chmod
555 hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs
-chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'not_if': "/usr/bin/sudo su
hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config
/etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig'", 'user': 'hdfs',
'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:12:31,703 -
CopyFromLocal['/usr/hdp/current/pig-client/pig.tar.gz'] {'hadoop_bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs',
'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir':
'hdfs:///hdp/apps/2.2.0.0-2041/pig', 'hadoop_conf_dir': '/etc/hadoop/conf',
'mode': 0444}
2014-12-10 17:12:31,703 - ExecuteHadoop['fs -copyFromLocal
/usr/hdp/current/pig-client/pig.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/pig']
{'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER}
> /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls
hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'", 'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir':
'/etc/hadoop/conf'}
2014-12-10 17:12:49,508 - Execute['hadoop --config /etc/hadoop/conf fs
-copyFromLocal /usr/hdp/current/pig-client/pig.tar.gz
hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'logoutput': False, 'try_sleep': 0,
'environment': ..., 'tries': 1, 'user': 'hdfs', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:09,506 - ExecuteHadoop['fs -chown hdfs:hadoop
hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir':
'/etc/hadoop/conf'}
2014-12-10 17:13:09,507 - Execute['hadoop --config /etc/hadoop/conf fs -chown
hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False,
'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:18,968 - ExecuteHadoop['fs -chmod 444
hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir':
'/etc/hadoop/conf'}
2014-12-10 17:13:18,969 - Execute['hadoop --config /etc/hadoop/conf fs -chmod
444 hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False,
'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:32,936 - ExecuteHadoop['fs -ls
hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput':
True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat',
'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:13:32,937 - Execute['hadoop --config /etc/hadoop/conf fs -ls
hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput':
True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:52,891 - ls:
`hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar': No such file or
directory
2014-12-10 17:13:52,892 -
HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'security_enabled':
False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs',
'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:13:52,904 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir
`rpm -q hadoop | grep -q "hadoop-1" || echo "-p"`
hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs
-chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config
/etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce']
{'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER}
> /dev/null ; hadoop --config /etc/hadoop/conf fs -ls
hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'", 'user': 'hdfs', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:03,832 - Skipping Execute['hadoop --config /etc/hadoop/conf fs
-mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"`
hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs
-chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config
/etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] due
to not_if
2014-12-10 17:14:03,833 -
CopyFromLocal['/usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar']
{'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop',
'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir':
'hdfs:///hdp/apps/2.2.0.0-2041/mapreduce', 'hadoop_conf_dir':
'/etc/hadoop/conf', 'mode': 0444}
2014-12-10 17:14:03,836 - ExecuteHadoop['fs -copyFromLocal
/usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar
hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'not_if': "/usr/bin/sudo su hdfs -l
-s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ;
PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls
hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'", 'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir':
'/etc/hadoop/conf'}
2014-12-10 17:14:12,682 - Execute['hadoop --config /etc/hadoop/conf fs
-copyFromLocal /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar
hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'logoutput': False, 'try_sleep': 0,
'environment': ..., 'tries': 1, 'user': 'hdfs', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:22,350 - ExecuteHadoop['fs -chown hdfs:hadoop
hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir':
'/etc/hadoop/conf'}
2014-12-10 17:14:22,352 - Execute['hadoop --config /etc/hadoop/conf fs -chown
hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar']
{'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user':
'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:34,163 - ExecuteHadoop['fs -chmod 444
hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir':
'/etc/hadoop/conf'}
2014-12-10 17:14:34,164 - Execute['hadoop --config /etc/hadoop/conf fs -chmod
444 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar']
{'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user':
'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:50,851 - Could not find file:
/usr/hdp/current/sqoop-client/sqoop.tar.gz
{noformat}
The problem is ‘hadoop fs’ calls. One call takes 3-30 seconds depending on node
load CPU/RAM etc. Mostly this is because every time call is done new jvm is
created and probably due to some DN/NN communication is redone. Since we have
more and more this calls triggered with every release, this became a problem
now.
*BASIC SOLUTION DESCRIPTION*
Instead of copying files to hdfs and creating directories one by one, by
calling countless number of ‘hadoop fs’ calls, we can do one single execution
of the jar file which will do all the creating it is asked.
For example:
{noformat}
hadoop -fs -mkdir /a
hadoop -fs -chmod 755 /a
hadoop -fs -chown ambari-qa /b
hadoop -fs -copyFromLocal /x /y
{noformat}
Can be replaced with
resources.json like this which is passed to our new jar (below):
{noformat}
{
{
“path”:”/a”
“type”: directory”
“mode”: “755”
},
{
“path”:”/b”
“type”: directory”
“owner”: “ambari-qa”
},
{
“path”:”/y”
“type”: directory”
“source”: “/x”
},
}
{noformat}
which will handle all that in one single call:
*hadoop jar create-hdfs-resources.jar resources.json*
For synchronization between nodes ZK for more details please read the doc below
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
