----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62058/ -----------------------------------------------------------
Review request for Ambari and Vitalyi Brodetskyi. Bugs: AMBARI-21872 https://issues.apache.org/jira/browse/AMBARI-21872 Repository: ambari Description ------- **Problem:** 2017-08-30 01:31:33,674 - ExecuteHadoop['jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples-3.*.jar wordcount /user/ambari-qa/mapredsmokeinput /user/ambari-qa/mapredsmokeoutput'] {'bin_dir': '/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin:/usr/hdp/current/hadoop-yarn-client/bin', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'logoutput': True, 'try_sleep': 5, 'tries': 1, 'user': 'ambari-qa'} 2017-08-30 01:31:33,675 - Execute['hadoop --config /usr/hdp/current/hadoop-client/conf jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples-3.*.jar wordcount /user/ambari-qa/mapredsmokeinput /user/ambari-qa/mapredsmokeoutput'] {'logoutput': True, 'try_sleep': 5, 'environment': {}, 'tries': 1, 'user': 'ambari-qa', 'path': ['/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin:/usr/hdp/current/hadoop-yarn-client/bin']} ######## Hortonworks ############# This is MOTD message, added for testing in qe infra 17/08/30 01:31:35 INFO client.RMProxy: Connecting to ResourceManager at ctr-e134-1499953498516-113587-01-000003.hwx.site/172.27.31.9:8050 java.io.FileNotFoundException: File does not exist: hdfs://ctr-e134-1499953498516-113587-01-000003.hwx.site:8020/hdp/apps/3.0.0.0-229/mapreduce/mapreduce.tar.gz at org.apache.hadoop.fs.Hdfs.getFileStatus(Hdfs.java:137) at org.apache.hadoop.fs.AbstractFileSystem.resolvePath(AbstractFileSystem.java:480) at org.apache.hadoop.fs.FileContext$25.next(FileContext.java:2191) at org.apache.hadoop.fs.FileContext$25.next(FileContext.java:2187) at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) at org.apache.hadoop.fs.FileContext.resolve(FileContext.java:2193) at org.apache.hadoop.fs.FileContext.resolvePath(FileContext.java:598) at org.apache.hadoop.mapreduce.JobSubmitter.addMRFrameworkToDistributedCache(JobSubmitter.java:465) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:150) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344) **Investigation:** After starting History Server, (which includes adding mapreduce.tar.gz to HDFS), it complains about HDFS_CLIENT not present. 2017-09-01 21:16:04,194 - Called copy_to_hdfs tarball: mapreduce 2017-09-01 21:16:04,194 - Stack Feature Version Info: Cluster Stack=3.0, Command Stack=None, Command Version=3.0.0.0-229 -> 3.0.0.0-229 2017-09-01 21:16:04,194 - Tarball version was calcuated as 3.0.0.0-229. Use Command Version: True 2017-09-01 21:16:04,194 - HDFS_CLIENT is not installed on the host. Skip copying /usr/hdp/3.0.0.0-229/hadoop/mapreduce.tar.gz In Ambari-2.5.2 history server host does not have MR client/Hadoop client and has MR service check passed. **Solution:** Seems like Ambari checks if a certain component is installed on the cluster (like HDFS_CLIENT). However not only HDFS_CLIENT but namenode, datanode, nfsgateway, journalnode etc. would include tar.gz. We have to check instead if cluster has a service, but not that host has a component. Diffs ----- ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 795160c Diff: https://reviews.apache.org/r/62058/diff/1/ Testing ------- mvn clean test Thanks, Andrew Onischuk