Here is a comprehensive list of all the properties I had to add to
slider-client.xml to make Slider Client work against an RM HA enabled
cluster.
Note: I had a 3-node vagrant cluster c6401, c6402 and c6403. The thing that
I was not sure of is that I had to continue to specify the property
"yarn.resourcemanager.address". When my cluster was non-HA it's value was
set to "c6402.ambari.apache.org:8050" and I left it as is.
Removing that entry threw the following exception:
2014-09-18 00:19:25,760 [main] ERROR main.ServiceLauncher - No valid
Resource Manager address provided in the argument --manager or the
configuration property yarn.resourcemanager.address value :
0.0.0.0/0.0.0.0:8032
*Here are all the RM HA related properties -*
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>yarn-cluster</value>
</property>
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>c6402.ambari.apache.org</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>c6401.ambari.apache.org</value>
</property>
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property>
<property>
<name>yarn.resourcemanager.ha.automatic-failover.zk-base-path</name>
<value>/yarn-leader-election</value>
</property>
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>c6401.ambari.apache.org,c6402.ambari.apache.org,
c6403.ambari.apache.org</value>
</property>
-Gour
On Wed, Sep 17, 2014 at 7:53 AM, Jon Maron <[email protected]> wrote:
> How do we know we have a valid hadoop conf on the client? Should we
> augment client error messages/info messages to indicate where properties
> are being found?
>
> On Wed, Sep 17, 2014 at 10:28 AM, Billie Rinaldi <[email protected]
> >
> wrote:
>
> > That would be great if we could drop the hadoop conf properties from the
> > slider-client.xml when we have a good hadoop conf available.
> >
> > On Wed, Sep 17, 2014 at 2:19 AM, Steve Loughran <[email protected]>
> > wrote:
> >
> > > I think there's some JIRAs on that. What we could do is add the hadoop
> > conf
> > > dir env variables to the classpath (put them after slider-conf dir to
> > pick
> > > up slider's log4j first).
> > >
> > > I'd argue for only doing that in the .py script and then we change the
> > > build to copy bin/slider.py to bin/slider (somehow) so there is then
> only
> > > one script to keep up to date. And as the script gets more complex,
> > > switching to python gets more and more compelling
> > >
> > >
> > > On 17 September 2014 02:01, Sumit Mohanty <[email protected]>
> > wrote:
> > >
> > > > Ran some tests. Looks like the easiest option is to drop
> yarn-site.xml
> > in
> > > > Slider conf dir.
> > > >
> > > > However, if hadoop is installed on the host and /etc/hadoop/conf
> > exists,
> > > > should we just use the site.xml files at /etc/hadoop/conf?
> > > >
> > > > -Sumit
> > > >
> > > > On Tue, Sep 16, 2014 at 5:45 PM, Sumit Mohanty <
> > [email protected]
> > > >
> > > > wrote:
> > > >
> > > > > Looks like we need the following properties in the slider-client
> when
> > > RM
> > > > HA
> > > > > is enabled.
> > > > >
> > > > > * yarn.resourcemanager.ha.enabled
> > > > > * yarn.resourcemanager.ha.rm-ids
> > > > > * yarn.resourcemanager.hostname.[one entry per ids specified above]
> > > > > * yarn.resourcemanager.zk-address
> > > > > * yarn.resourcemanager.cluster-id
> > > > >
> > > > > Does the list look complete?
> > > > >
> > > > > *Steve*, related question:
> > > > > Should we go the route where we drop yarn-site.xml in the Slider
> > config
> > > > > directory and it gets read by the client? *Or, does it already
> work?*
> > > > >
> > > > > *-*Sumit
> > > > >
> > > > > --
> > > > > CONFIDENTIALITY NOTICE
> > > > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > > to
> > > > > which it is addressed and may contain information that is
> > confidential,
> > > > > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > > > of this message is not the intended recipient, you are hereby
> > notified
> > > > that
> > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > forwarding of this communication is strictly prohibited. If you
> have
> > > > > received this communication in error, please contact the sender
> > > > immediately
> > > > > and delete it from your system. Thank You.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > thanks
> > > > Sumit
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
