[
https://issues.apache.org/jira/browse/SAMZA-182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13933547#comment-13933547
]
Chris Riccomini commented on SAMZA-182:
---------------------------------------
I did some digging on this. You are not the first person to report this problem:
http://mail-archives.apache.org/mod_mbox/incubator-samza-dev/201403.mbox/%3CB84B01583BEBBC45AD442B3F9045B8AC0ED4673C%40048-CH1MPN3-331.048d.mgd.msft.net%3E
The problem appears to be that Samza's run-class.sh is adding YARN_HOME/conf to
the classpath. If YARN_HOME is not set, YARN can't find the yarn-site.xml, and
thus defaults to the 0.0.0.0 RM host.
Taking a look at:
http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
Says:
{noformat}
yarn.nodemanager.env-whitelist:
JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME
Environment variables that containers may override rather than use
NodeManager's default.
{noformat}
It looks like run-class.sh is still using the legacy YARN_HOME environment
variable. Found this in SVN:
{noformat}
Revision 1390218 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Tue Sep 25 23:37:32 2012 UTC (17 months, 2 weeks ago) by acmurthy
File length: 1353 byte(s)
Diff to previous 1370666 (colored)
YARN-9. Rename YARN_HOME to HADOOP_YARN_HOME. Contributed by Vinod K V.
{noformat}
Changing run-class.sh to have:
{code}
$ git diff
diff --git a/samza-shell/src/main/bash/run-class.sh
b/samza-shell/src/main/bash/run-class.sh
index 2fa2acf..bb4e0d2 100755
--- a/samza-shell/src/main/bash/run-class.sh
+++ b/samza-shell/src/main/bash/run-class.sh
@@ -33,8 +33,8 @@ if [ ! -d "$base_dir/lib" ]; then
exit 1
fi
-YARN_HOME="${YARN_HOME:-$HOME/.samza}"
-CLASSPATH=$YARN_HOME/conf
+HADOOP_YARN_HOME="${HADOOP_YARN_HOME:-$HOME/.samza}"
+CLASSPATH="${HADOOP_CONF_DIR:-$HADOOP_YARN_HOME/conf}"
for file in $base_dir/lib/*.[jw]ar;
do
{code}
Appears to fix the problem.
> RM Hostname Specification For SamzaAppMaster
> --------------------------------------------
>
> Key: SAMZA-182
> URL: https://issues.apache.org/jira/browse/SAMZA-182
> Project: Samza
> Issue Type: Improvement
> Components: yarn
> Affects Versions: 0.7.0
> Reporter: Ethan Setnik
> Attachments: yarn_remote_rm.patch
>
>
> When running Samza in a clustered environment, I found that SamzaAppMaster
> jobs running on remote Yarn NodeManagers do not know how to properly contact
> the ResourceManager.
> I have included a small patch that adds a directive to the YarnConfig to
> specify the hostname of the RM via "yarn.rm.hostname". I have tested this
> directive and it is working in my environment.
> Please take a look at the patch and provide advise on whether this is the
> correct approach. If there is no better suggestion, i'd be happy to writeup
> some test cases to support the patch.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
