-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62058/
-----------------------------------------------------------
(Updated Sept. 5, 2017, 2:07 p.m.)
Review request for Ambari, Di Li and Vitalyi Brodetskyi.
Bugs: AMBARI-21872
https://issues.apache.org/jira/browse/AMBARI-21872
Repository: ambari
Description
-------
**Problem:**
2017-08-30 01:31:33,674 - ExecuteHadoop['jar
/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples-3.*.jar
wordcount /user/ambari-qa/mapredsmokeinput /user/ambari-qa/mapredsmokeoutput']
{'bin_dir':
'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin:/usr/hdp/current/hadoop-yarn-client/bin',
'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'logoutput': True,
'try_sleep': 5, 'tries': 1, 'user': 'ambari-qa'}
2017-08-30 01:31:33,675 - Execute['hadoop --config
/usr/hdp/current/hadoop-client/conf jar
/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples-3.*.jar
wordcount /user/ambari-qa/mapredsmokeinput /user/ambari-qa/mapredsmokeoutput']
{'logoutput': True, 'try_sleep': 5, 'environment': {}, 'tries': 1, 'user':
'ambari-qa', 'path':
['/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin:/usr/hdp/current/hadoop-yarn-client/bin']}
######## Hortonworks #############
This is MOTD message, added for testing in qe infra
17/08/30 01:31:35 INFO client.RMProxy: Connecting to ResourceManager at
ctr-e134-1499953498516-113587-01-000003.hwx.site/172.27.31.9:8050
java.io.FileNotFoundException: File does not exist:
hdfs://ctr-e134-1499953498516-113587-01-000003.hwx.site:8020/hdp/apps/3.0.0.0-229/mapreduce/mapreduce.tar.gz
at org.apache.hadoop.fs.Hdfs.getFileStatus(Hdfs.java:137)
at
org.apache.hadoop.fs.AbstractFileSystem.resolvePath(AbstractFileSystem.java:480)
at org.apache.hadoop.fs.FileContext$25.next(FileContext.java:2191)
at org.apache.hadoop.fs.FileContext$25.next(FileContext.java:2187)
at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
at org.apache.hadoop.fs.FileContext.resolve(FileContext.java:2193)
at org.apache.hadoop.fs.FileContext.resolvePath(FileContext.java:598)
at
org.apache.hadoop.mapreduce.JobSubmitter.addMRFrameworkToDistributedCache(JobSubmitter.java:465)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:150)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
**Investigation:**
After starting History Server, (which includes adding mapreduce.tar.gz to
HDFS), it complains about HDFS_CLIENT not present.
2017-09-01 21:16:04,194 - Called copy_to_hdfs tarball: mapreduce
2017-09-01 21:16:04,194 - Stack Feature Version Info: Cluster Stack=3.0,
Command Stack=None, Command Version=3.0.0.0-229 -> 3.0.0.0-229
2017-09-01 21:16:04,194 - Tarball version was calcuated as 3.0.0.0-229. Use
Command Version: True
2017-09-01 21:16:04,194 - HDFS_CLIENT is not installed on the host. Skip
copying /usr/hdp/3.0.0.0-229/hadoop/mapreduce.tar.gz
In Ambari-2.5.2 history server host does not have MR client/Hadoop client and
has MR service check passed.
**Solution:**
Seems like Ambari checks if a certain component is installed on the cluster
(like HDFS_CLIENT). However not only HDFS_CLIENT but namenode, datanode,
nfsgateway, journalnode etc.
would include tar.gz. We have to check instead if cluster has a service, but
not that host has a component.
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
795160c
Diff: https://reviews.apache.org/r/62058/diff/2/
Changes: https://reviews.apache.org/r/62058/diff/1-2/
Testing
-------
mvn clean test
Thanks,
Andrew Onischuk
