RE: meaning of cryptic yarn nodemanager message:
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
I answered one of my own questions:
> Should I change the name be changed to "httpshuffle" or not ?
When I do change it, I get this error message and nodeserver fails to start:
2013-11-13 12:39:06,878 FATAL
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Failed
to initialize httpshuffle
java.lang.RuntimeException: No class defined for httpshuffle
So the correct name should be 'mapreduce_shuffle' ?
Why then the complaint in the log that it is not 'httpshuffle' ?
What issues might be caused by the name being 'mapreduce_shuffle' ?
Thanks & Regards,
Jason
-----Original Message-----
From: VasDias, Jason
Sent: 13 November 2013 12:36
To: [email protected]
Subject: meaning of cryptic yarn nodemanager message:
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
Good day - please could anyone explain the meaning of this message in
yarn-${USER}-nodemanager-${HOST}.log:
"
2013-11-13 12:17:29,704 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class
class org.apache.hadoop.mapred.ShuffleHandler which has a name of
'httpshuffle'. Because these are not the same tools trying to send ServiceData
and read Service Meta Data may have issues unless the refer to the name in the
config.
"
Love the creative spelling ! ('Auxilurary' should be 'Auxiliary' ? 'unless the
refer' should be 'unless they refer' ?)
I guess this is because of the setting in yarn-site.xml :
"
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<description>shuffle service that needs to be set for Map Reduce to run
</description> </property> "
that I installed according to the instructions at:
http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/SingleCluster.html
Why do these instructions specify this setting it may cause "issues" ? What
sort of issues may be caused ?
This sentence is nonsensical:
" Because these are not the same tools trying to send ServiceData and read
Service Meta Data may have issues unless the refer to the name in the config."
What are the "tools" being referred to here ?
What "may have issues" ?
What are the "issues" ?
What must refer to the name in the config ?
Should I change the name to "httpshuffle" or not ?
Any ideas / suggestions gratefully received.
Regards,
Jason
