Re: finding Spark Master
This website explains it very clear, if you are using Yarn. https://www.cloudera.com/documentation/enterprise/5-6-x/topics/cdh_ig_running_spark_on_yarn.html Running Spark Applications on YARN - cloudera.com<https://www.cloudera.com/documentation/enterprise/5-6-x/topics/cdh_ig_running_spark_on_yarn.html> www.cloudera.com When Spark applications run on a YARN cluster manager, resource management, scheduling, and security are controlled by YARN. From: Adaryl Wakefield Sent: Tuesday, March 7, 2017 8:53 PM To: Koert Kuipers Cc: user@spark.apache.org Subject: RE: finding Spark Master Ah so I see setMaster(‘yarn-client’). Hmm. What I was ultimately trying to do was develop with Eclipse on my windows box and have the code point to my cluster so it executes there instead of my local windows machine. Perhaps I’m going about this wrong. Adaryl "Bob" Wakefield, MBA Principal Mass Street Analytics, LLC 913.938.6685 www.massstreet.net<http://www.massstreet.net> www.linkedin.com/in/bobwakefieldmba<http://www.linkedin.com/in/bobwakefieldmba> Twitter: @BobLovesData From: Koert Kuipers [mailto:ko...@tresata.com] Sent: Tuesday, March 7, 2017 7:47 PM To: Adaryl Wakefield Cc: user@spark.apache.org Subject: Re: finding Spark Master assuming this is running on yarn there is really spark-master. every job created its own "master" within a yarn application. On Tue, Mar 7, 2017 at 6:27 PM, Adaryl Wakefield mailto:adaryl.wakefi...@hotmail.com>> wrote: I’m running a three node cluster along with Spark along with Hadoop as part of a HDP stack. How do I find my Spark Master? I’m just seeing the clients. I’m trying to figure out what goes in setMaster() aside from local[*]. Adaryl "Bob" Wakefield, MBA Principal Mass Street Analytics, LLC 913.938.6685 www.massstreet.net<http://www.massstreet.net> www.linkedin.com/in/bobwakefieldmba<http://www.linkedin.com/in/bobwakefieldmba> Twitter: @BobLovesData
RE: finding Spark Master
Ah so I see setMaster(‘yarn-client’). Hmm. What I was ultimately trying to do was develop with Eclipse on my windows box and have the code point to my cluster so it executes there instead of my local windows machine. Perhaps I’m going about this wrong. Adaryl "Bob" Wakefield, MBA Principal Mass Street Analytics, LLC 913.938.6685 www.massstreet.net<http://www.massstreet.net> www.linkedin.com/in/bobwakefieldmba<http://www.linkedin.com/in/bobwakefieldmba> Twitter: @BobLovesData From: Koert Kuipers [mailto:ko...@tresata.com] Sent: Tuesday, March 7, 2017 7:47 PM To: Adaryl Wakefield Cc: user@spark.apache.org Subject: Re: finding Spark Master assuming this is running on yarn there is really spark-master. every job created its own "master" within a yarn application. On Tue, Mar 7, 2017 at 6:27 PM, Adaryl Wakefield mailto:adaryl.wakefi...@hotmail.com>> wrote: I’m running a three node cluster along with Spark along with Hadoop as part of a HDP stack. How do I find my Spark Master? I’m just seeing the clients. I’m trying to figure out what goes in setMaster() aside from local[*]. Adaryl "Bob" Wakefield, MBA Principal Mass Street Analytics, LLC 913.938.6685 www.massstreet.net<http://www.massstreet.net> www.linkedin.com/in/bobwakefieldmba<http://www.linkedin.com/in/bobwakefieldmba> Twitter: @BobLovesData
Re: finding Spark Master
assuming this is running on yarn there is really spark-master. every job created its own "master" within a yarn application. On Tue, Mar 7, 2017 at 6:27 PM, Adaryl Wakefield < adaryl.wakefi...@hotmail.com> wrote: > I’m running a three node cluster along with Spark along with Hadoop as > part of a HDP stack. How do I find my Spark Master? I’m just seeing the > clients. I’m trying to figure out what goes in setMaster() aside from > local[*]. > > > > Adaryl "Bob" Wakefield, MBA > Principal > Mass Street Analytics, LLC > 913.938.6685 <(913)%20938-6685> > > www.massstreet.net > > www.linkedin.com/in/bobwakefieldmba > Twitter: @BobLovesData > > >
RE: finding Spark Master
I’m sorry I don’t understand. Is that a question or the answer? Adaryl "Bob" Wakefield, MBA Principal Mass Street Analytics, LLC 913.938.6685 www.massstreet.net<http://www.massstreet.net> www.linkedin.com/in/bobwakefieldmba<http://www.linkedin.com/in/bobwakefieldmba> Twitter: @BobLovesData From: ayan guha [mailto:guha.a...@gmail.com] Sent: Tuesday, March 7, 2017 5:59 PM To: Adaryl Wakefield ; user@spark.apache.org Subject: Re: finding Spark Master yarn-client or yarn-cluster On Wed, 8 Mar 2017 at 10:28 am, Adaryl Wakefield mailto:adaryl.wakefi...@hotmail.com>> wrote: I’m running a three node cluster along with Spark along with Hadoop as part of a HDP stack. How do I find my Spark Master? I’m just seeing the clients. I’m trying to figure out what goes in setMaster() aside from local[*]. Adaryl "Bob" Wakefield, MBA Principal Mass Street Analytics, LLC 913.938.6685 www.massstreet.net<http://www.massstreet.net> www.linkedin.com/in/bobwakefieldmba<http://www.linkedin.com/in/bobwakefieldmba> Twitter: @BobLovesData -- Best Regards, Ayan Guha
Re: finding Spark Master
yarn-client or yarn-cluster On Wed, 8 Mar 2017 at 10:28 am, Adaryl Wakefield < adaryl.wakefi...@hotmail.com> wrote: > I’m running a three node cluster along with Spark along with Hadoop as > part of a HDP stack. How do I find my Spark Master? I’m just seeing the > clients. I’m trying to figure out what goes in setMaster() aside from > local[*]. > > > > Adaryl "Bob" Wakefield, MBA > Principal > Mass Street Analytics, LLC > 913.938.6685 > > www.massstreet.net > > www.linkedin.com/in/bobwakefieldmba > Twitter: @BobLovesData > > > -- Best Regards, Ayan Guha
