----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62058/#review184541 -----------------------------------------------------------
Ship it! Ship It! - Vitalyi Brodetskyi On Вер. 4, 2017, 9:44 до полудня, Andrew Onischuk wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62058/ > ----------------------------------------------------------- > > (Updated Вер. 4, 2017, 9:44 до полудня) > > > Review request for Ambari, Di Li and Vitalyi Brodetskyi. > > > Bugs: AMBARI-21872 > https://issues.apache.org/jira/browse/AMBARI-21872 > > > Repository: ambari > > > Description > ------- > > **Problem:** > > > > 2017-08-30 01:31:33,674 - ExecuteHadoop['jar > /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples-3.*.jar > wordcount /user/ambari-qa/mapredsmokeinput > /user/ambari-qa/mapredsmokeoutput'] {'bin_dir': > '/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin:/usr/hdp/current/hadoop-yarn-client/bin', > 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'logoutput': True, > 'try_sleep': 5, 'tries': 1, 'user': 'ambari-qa'} > 2017-08-30 01:31:33,675 - Execute['hadoop --config > /usr/hdp/current/hadoop-client/conf jar > /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples-3.*.jar > wordcount /user/ambari-qa/mapredsmokeinput > /user/ambari-qa/mapredsmokeoutput'] {'logoutput': True, 'try_sleep': 5, > 'environment': {}, 'tries': 1, 'user': 'ambari-qa', 'path': > ['/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin:/usr/hdp/current/hadoop-yarn-client/bin']} > ######## Hortonworks ############# > This is MOTD message, added for testing in qe infra > 17/08/30 01:31:35 INFO client.RMProxy: Connecting to ResourceManager at > ctr-e134-1499953498516-113587-01-000003.hwx.site/172.27.31.9:8050 > java.io.FileNotFoundException: File does not exist: > hdfs://ctr-e134-1499953498516-113587-01-000003.hwx.site:8020/hdp/apps/3.0.0.0-229/mapreduce/mapreduce.tar.gz > at org.apache.hadoop.fs.Hdfs.getFileStatus(Hdfs.java:137) > at > org.apache.hadoop.fs.AbstractFileSystem.resolvePath(AbstractFileSystem.java:480) > at org.apache.hadoop.fs.FileContext$25.next(FileContext.java:2191) > at org.apache.hadoop.fs.FileContext$25.next(FileContext.java:2187) > at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) > at org.apache.hadoop.fs.FileContext.resolve(FileContext.java:2193) > at org.apache.hadoop.fs.FileContext.resolvePath(FileContext.java:598) > at > org.apache.hadoop.mapreduce.JobSubmitter.addMRFrameworkToDistributedCache(JobSubmitter.java:465) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:150) > at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344) > > > **Investigation:** > After starting History Server, (which includes adding mapreduce.tar.gz to > HDFS), it complains about HDFS_CLIENT not present. > > > > 2017-09-01 21:16:04,194 - Called copy_to_hdfs tarball: mapreduce > 2017-09-01 21:16:04,194 - Stack Feature Version Info: Cluster Stack=3.0, > Command Stack=None, Command Version=3.0.0.0-229 -> 3.0.0.0-229 > 2017-09-01 21:16:04,194 - Tarball version was calcuated as 3.0.0.0-229. > Use Command Version: True > 2017-09-01 21:16:04,194 - HDFS_CLIENT is not installed on the host. Skip > copying /usr/hdp/3.0.0.0-229/hadoop/mapreduce.tar.gz > > > In Ambari-2.5.2 history server host does not have MR client/Hadoop client and > has MR service check passed. > > **Solution:** > Seems like Ambari checks if a certain component is installed on the cluster > (like HDFS_CLIENT). However not only HDFS_CLIENT but namenode, datanode, > nfsgateway, journalnode etc. > would include tar.gz. We have to check instead if cluster has a service, but > not that host has a component. > > > Diffs > ----- > > > ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py > 795160c > > > Diff: https://reviews.apache.org/r/62058/diff/1/ > > > Testing > ------- > > mvn clean test > > > Thanks, > > Andrew Onischuk > >
