Starting this morning, we started seeing this on every single install. I think
someone at Hortonworks pushed out a broken RPM or something. Any ideas? This
is rather urgent as we are no longer able to provision HDP 2.2 clusters at all
because of it.
2015-03-18 15:58:05,982 - Group['hadoop'] {'ignore_failures': False}
2015-03-18 15:58:05,984 - Modifying group hadoop
2015-03-18 15:58:06,080 - Group['nobody'] {'ignore_failures': False}
2015-03-18 15:58:06,081 - Modifying group nobody
2015-03-18 15:58:06,219 - Group['users'] {'ignore_failures': False}
2015-03-18 15:58:06,220 - Modifying group users
2015-03-18 15:58:06,370 - Group['nagios'] {'ignore_failures': False}
2015-03-18 15:58:06,371 - Modifying group nagios
2015-03-18 15:58:06,474 - User['nobody'] {'gid': 'hadoop', 'ignore_failures':
False, 'groups': [u'nobody']}
2015-03-18 15:58:06,475 - Modifying user nobody
2015-03-18 15:58:06,558 - User['hive'] {'gid': 'hadoop', 'ignore_failures':
False, 'groups': [u'hadoop']}
2015-03-18 15:58:06,559 - Modifying user hive
2015-03-18 15:58:06,634 - User['mapred'] {'gid': 'hadoop', 'ignore_failures':
False, 'groups': [u'hadoop']}
2015-03-18 15:58:06,635 - Modifying user mapred
2015-03-18 15:58:06,722 - User['nagios'] {'gid': 'nagios', 'ignore_failures':
False, 'groups': [u'hadoop']}
2015-03-18 15:58:06,723 - Modifying user nagios
2015-03-18 15:58:06,841 - User['ambari-qa'] {'gid': 'hadoop',
'ignore_failures': False, 'groups': [u'users']}
2015-03-18 15:58:06,842 - Modifying user ambari-qa
2015-03-18 15:58:06,963 - User['zookeeper'] {'gid': 'hadoop',
'ignore_failures': False, 'groups': [u'hadoop']}
2015-03-18 15:58:06,964 - Modifying user zookeeper
2015-03-18 15:58:07,093 - User['tez'] {'gid': 'hadoop', 'ignore_failures':
False, 'groups': [u'users']}
2015-03-18 15:58:07,094 - Modifying user tez
2015-03-18 15:58:07,217 - User['hdfs'] {'gid': 'hadoop', 'ignore_failures':
False, 'groups': [u'hadoop']}
2015-03-18 15:58:07,218 - Modifying user hdfs
2015-03-18 15:58:07,354 - User['yarn'] {'gid': 'hadoop', 'ignore_failures':
False, 'groups': [u'hadoop']}
2015-03-18 15:58:07,355 - Modifying user yarn
2015-03-18 15:58:07,485 - User['hcat'] {'gid': 'hadoop', 'ignore_failures':
False, 'groups': [u'hadoop']}
2015-03-18 15:58:07,486 - Modifying user hcat
2015-03-18 15:58:07,629 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh']
{'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2015-03-18 15:58:07,631 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh
ambari-qa
/tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa
2>/dev/null'] {'not_if': 'test $(id -u ambari-qa) -gt 1000'}
2015-03-18 15:58:07,768 - Skipping
Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa
/tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa
2>/dev/null'] due to not_if
2015-03-18 15:58:07,769 - Directory['/etc/hadoop/conf.empty'] {'owner': 'root',
'group': 'root', 'recursive': True}
2015-03-18 15:58:07,770 - Link['/etc/hadoop/conf'] {'not_if': 'ls
/etc/hadoop/conf', 'to': '/etc/hadoop/conf.empty'}
2015-03-18 15:58:07,895 - Skipping Link['/etc/hadoop/conf'] due to not_if
2015-03-18 15:58:07,960 - File['/etc/hadoop/conf/hadoop-env.sh'] {'content':
InlineTemplate(...), 'owner': 'hdfs'}
2015-03-18 15:58:08,092 - Execute['/bin/echo 0 > /selinux/enforce'] {'only_if':
'test -f /selinux/enforce'}
2015-03-18 15:58:08,240 - Skipping Execute['/bin/echo 0 > /selinux/enforce']
due to only_if
2015-03-18 15:58:08,241 - Directory['/var/log/hadoop'] {'owner': 'root',
'group': 'hadoop', 'mode': 0775, 'recursive': True}
2015-03-18 15:58:08,244 - Directory['/var/run/hadoop'] {'owner': 'root',
'group': 'root', 'recursive': True}
2015-03-18 15:58:08,250 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs',
'recursive': True}
2015-03-18 15:58:08,278 - File['/etc/hadoop/conf/commons-logging.properties']
{'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2015-03-18 15:58:08,288 - File['/etc/hadoop/conf/health_check'] {'content':
Template('health_check-v2.j2'), 'owner': 'hdfs'}
2015-03-18 15:58:08,295 - File['/etc/hadoop/conf/log4j.properties'] {'content':
'...', 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2015-03-18 15:58:08,322 - File['/etc/hadoop/conf/hadoop-metrics2.properties']
{'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'}
2015-03-18 15:58:08,325 - File['/etc/hadoop/conf/task-log4j.properties']
{'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2015-03-18 15:58:08,330 - File['/etc/hadoop/conf/configuration.xsl'] {'owner':
'hdfs', 'group': 'hadoop'}
2015-03-18 15:58:09,219 - HdfsDirectory['/user/hcat'] {'security_enabled':
False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs',
'kinit_path_local': '', 'mode': 0755, 'owner': 'hcat', 'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'action': ['create_delayed']}
2015-03-18 15:58:09,220 - HdfsDirectory['None'] {'security_enabled': False,
'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs',
'kinit_path_local': '', 'action': ['create'], 'bin_dir':
'/usr/hdp/current/hadoop-client/bin'}
2015-03-18 15:58:09,228 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir
`rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` /user/hcat && hadoop --config
/etc/hadoop/conf fs -chmod 755 /user/hcat && hadoop --config /etc/hadoop/conf
fs -chown hcat /user/hcat'] {'not_if': "su - hdfs -c 'export
PATH=$PATH:/usr/hdp/current/hadoop-client/bin ; hadoop --config
/etc/hadoop/conf fs -ls /user/hcat'", 'user': 'hdfs', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2015-03-18 15:58:37,822 - Directory['/var/run/webhcat'] {'owner': 'hcat',
'group': 'hadoop', 'recursive': True, 'mode': 0755}
2015-03-18 15:58:37,823 - Changing group for /var/run/webhcat from 0 to hadoop
2015-03-18 15:58:37,823 - Directory['/var/log/webhcat'] {'owner': 'hcat',
'group': 'hadoop', 'recursive': True, 'mode': 0755}
2015-03-18 15:58:37,824 - Creating directory Directory['/var/log/webhcat']
2015-03-18 15:58:37,824 - Changing owner for /var/log/webhcat from 0 to hcat
2015-03-18 15:58:37,824 - Changing group for /var/log/webhcat from 0 to hadoop
2015-03-18 15:58:37,824 - Directory['/etc/hive-webhcat/conf'] {'owner': 'hcat',
'group': 'hadoop', 'recursive': True}
2015-03-18 15:58:37,825 - Changing owner for /etc/hive-webhcat/conf from 0 to
hcat
2015-03-18 15:58:37,825 - Changing group for /etc/hive-webhcat/conf from 0 to
hadoop
2015-03-18 15:58:37,893 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/hive/hive.tar.gz'] {'logoutput': True, 'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir':
'/etc/hadoop/conf'}
2015-03-18 15:58:37,896 - Execute['hadoop --config /etc/hadoop/conf fs -ls
hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/hive/hive.tar.gz'] {'logoutput': True, 'path':
['/usr/hdp/current/hadoop-client/bin'], 'tries': 1, 'user': 'hcat',
'try_sleep': 0}
2015-03-18 15:58:43,597 - -bash: line 1: 2.2.3.0-2611/hive/hive.tar.gz: No such
file or directory
2015-03-18 15:58:43,599 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/hive'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir':
'/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555,
'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action':
['create']}
2015-03-18 15:58:43,601 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir
`rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/hive && hadoop --config /etc/hadoop/conf fs -chmod 555
hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/hive && hadoop --config /etc/hadoop/conf fs -chown hdfs
hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/hive'] {'not_if': "su - hdfs -c 'export
PATH=$PATH:/usr/hdp/current/hadoop-client/bin ; hadoop --config
/etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041\n2.2.3.0-2611/hive'",
'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2015-03-18 15:58:54,904 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/pig/pig.tar.gz'] {'logoutput': True, 'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir':
'/etc/hadoop/conf'}
2015-03-18 15:58:54,906 - Execute['hadoop --config /etc/hadoop/conf fs -ls
hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/pig/pig.tar.gz'] {'logoutput': True, 'path':
['/usr/hdp/current/hadoop-client/bin'], 'tries': 1, 'user': 'hcat',
'try_sleep': 0}
2015-03-18 15:59:00,322 - -bash: line 1: 2.2.3.0-2611/pig/pig.tar.gz: No such
file or directory
2015-03-18 15:59:00,323 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/pig'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir':
'/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555,
'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action':
['create']}
2015-03-18 15:59:00,326 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir
`rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/hive hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/pig && hadoop --config /etc/hadoop/conf fs -chmod 555
hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/hive hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/pig && hadoop --config /etc/hadoop/conf fs -chown hdfs
hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/hive hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/pig'] {'not_if': "su - hdfs -c 'export
PATH=$PATH:/usr/hdp/current/hadoop-client/bin ; hadoop --config
/etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041\n2.2.3.0-2611/hive
hdfs:///hdp/apps/2.2.0.0-2041\n2.2.3.0-2611/pig'", 'user': 'hdfs', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2015-03-18 15:59:11,576 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'bin_dir':
'/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir':
'/etc/hadoop/conf'}
2015-03-18 15:59:11,578 - Execute['hadoop --config /etc/hadoop/conf fs -ls
hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'path':
['/usr/hdp/current/hadoop-client/bin'], 'tries': 1, 'user': 'hcat',
'try_sleep': 0}
2015-03-18 15:59:17,094 - -bash: line 1:
2.2.3.0-2611/mapreduce/hadoop-streaming.jar: No such file or directory
2015-03-18 15:59:17,097 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/mapreduce'] {'security_enabled': False, 'keytab': [EMPTY],
'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '',
'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin',
'action': ['create']}
2015-03-18 15:59:17,099 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir
`rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/hive hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/pig hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/mapreduce && hadoop --config /etc/hadoop/conf fs -chmod 555
hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/hive hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/pig hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/mapreduce && hadoop --config /etc/hadoop/conf fs -chown hdfs
hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/hive hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/pig hdfs:///hdp/apps/2.2.0.0-2041
2.2.3.0-2611/mapreduce'] {'not_if': '...', 'user': 'hdfs', 'path':
['/usr/hdp/current/hadoop-client/bin']}
2015-03-18 15:59:28,070 - Could not find file:
/usr/hdp/current/sqoop-client/sqoop.tar.gz
2015-03-18 15:59:28,071 - XmlConfig['webhcat-site.xml'] {'owner': 'hcat',
'group': 'hadoop', 'conf_dir': '/etc/hive-webhcat/conf',
'configuration_attributes': ..., 'configurations': ...}
2015-03-18 15:59:28,090 - Generating config:
/etc/hive-webhcat/conf/webhcat-site.xml
2015-03-18 15:59:28,091 - File['/etc/hive-webhcat/conf/webhcat-site.xml']
{'owner': 'hcat', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode':
None, 'encoding': 'UTF-8'}
2015-03-18 15:59:28,092 - Writing
File['/etc/hive-webhcat/conf/webhcat-site.xml'] because it doesn't exist
2015-03-18 15:59:28,093 - Changing owner for
/etc/hive-webhcat/conf/webhcat-site.xml from 0 to hcat
2015-03-18 15:59:28,093 - Changing group for
/etc/hive-webhcat/conf/webhcat-site.xml from 0 to hadoop
2015-03-18 15:59:28,095 - File['/etc/hive-webhcat/conf/webhcat-env.sh']
{'content': InlineTemplate(...), 'owner': 'hcat', 'group': 'hadoop'}
2015-03-18 15:59:28,096 - Writing File['/etc/hive-webhcat/conf/webhcat-env.sh']
because it doesn't exist
2015-03-18 15:59:28,096 - Changing owner for
/etc/hive-webhcat/conf/webhcat-env.sh from 0 to hcat
2015-03-18 15:59:28,096 - Changing group for
/etc/hive-webhcat/conf/webhcat-env.sh from 0 to hadoop
2015-03-18 15:59:28,097 - Execute['env
HADOOP_HOME=/usr/hdp/current/hadoop-client
/usr/hdp/current/hive-webhcat/sbin/webhcat_server.sh start'] {'not_if': 'ls
/var/run/webhcat/webhcat.pid >/dev/null 2>&1 && ps `cat
/var/run/webhcat/webhcat.pid` >/dev/null 2>&1', 'user': 'hcat'}
2015-03-18 15:59:28,179 - Error while executing command 'start':
Traceback (most recent call last):
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 123, in execute
method(env)
File
"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/webhcat_server.py",
line 39, in start
webhcat_service(action = 'start')
File
"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/webhcat_service.py",
line 33, in webhcat_service
not_if=no_op_test
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py",
line 148, in __init__
self.env.run()
File
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
line 149, in run
self.run_action(resource, action)
File
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
line 115, in run_action
provider_action()
File
"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
line 241, in action_run
raise ex
Fail: Execution of 'env HADOOP_HOME=/usr/hdp/current/hadoop-client
/usr/hdp/current/hive-webhcat/sbin/webhcat_server.sh start' returned 127. env:
/usr/hdp/current/hive-webhcat/sbin/webhcat_server.sh: No such file or directory
