Re: Spark Mesos Dispatcher
I'm currently having the same issues. The documentation for Mesos dispatcher is sparse. I'll also add that I'm able to see the framework running in the mesos and spark driver UIs, but when viewing the spark job ui on a slave, no job is seen. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Mesos-Dispatcher-tp24238p24404.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Spark Mesos Dispatcher
Hi Bcjaes, Sorry I didn't see the previous thread so not sure what issues you are running into. In cluster mode the driver logs and results are all available through the Mesos UI, you need to look at terminated frameworks if it's a job that's already finished. I'll try to add more docs as we are still completing some other features around cluster mode on Mesos. Tim On Aug 23, 2015, at 7:22 AM, bcajes brian.ca...@gmail.com wrote: I'm currently having the same issues. The documentation for Mesos dispatcher is sparse. I'll also add that I'm able to see the framework running in the mesos and spark driver UIs, but when viewing the spark job ui on a slave, no job is seen. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Mesos-Dispatcher-tp24238p24404.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Spark Mesos Dispatcher
Yes. Sent from my iPhone On 19 Jul, 2015, at 10:52 pm, Jahagirdar, Madhu madhu.jahagir...@philips.com wrote: All, Can we run different version of Spark using the same Mesos Dispatcher. For example we can run drivers with Spark 1.3 and Spark 1.4 at the same time ? Regards, Madhu Jahagirdar The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.
Spark Mesos Dispatcher
All, Can we run different version of Spark using the same Mesos Dispatcher. For example we can run drivers with Spark 1.3 and Spark 1.4 at the same time ? Regards, Madhu Jahagirdar The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.
Re: Spark Mesos Dispatcher
I only used client mode both 1.3 and 1.4 versions on mesos. I skimmed through https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/mesos/MesosClusterDispatcher.scala. I would actually backport the Cluster Mode feature. Sorry, I don't have an answer for this. On Sun, Jul 19, 2015 at 11:16 PM, Jahagirdar, Madhu madhu.jahagir...@philips.com wrote: 1.3 does not have MesosDisptacher or does not have support for Mesos cluster mode , is it still possible to create a Dispatcher using 1.4 and run 1.3 using that dispatcher ? -- *From:* Jerry Lam [chiling...@gmail.com] *Sent:* Monday, July 20, 2015 8:27 AM *To:* Jahagirdar, Madhu *Cc:* user; d...@spark.apache.org *Subject:* Re: Spark Mesos Dispatcher Yes. Sent from my iPhone On 19 Jul, 2015, at 10:52 pm, Jahagirdar, Madhu madhu.jahagir...@philips.com wrote: All, Can we run different version of Spark using the same Mesos Dispatcher. For example we can run drivers with Spark 1.3 and Spark 1.4 at the same time ? Regards, Madhu Jahagirdar -- The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.
RE: Spark Mesos Dispatcher
1.3 does not have MesosDisptacher or does not have support for Mesos cluster mode , is it still possible to create a Dispatcher using 1.4 and run 1.3 using that dispatcher ? From: Jerry Lam [chiling...@gmail.com] Sent: Monday, July 20, 2015 8:27 AM To: Jahagirdar, Madhu Cc: user; d...@spark.apache.org Subject: Re: Spark Mesos Dispatcher Yes. Sent from my iPhone On 19 Jul, 2015, at 10:52 pm, Jahagirdar, Madhu madhu.jahagir...@philips.commailto:madhu.jahagir...@philips.com wrote: All, Can we run different version of Spark using the same Mesos Dispatcher. For example we can run drivers with Spark 1.3 and Spark 1.4 at the same time ? Regards, Madhu Jahagirdar The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.
Re: Spark Mesos Dispatcher
Depends on how you run 1.3/1.4 versions of Spark, if you're giving it different Docker images / tar balls of Spark, technically it should work since it's just launching a driver for you at the end of the day. However, I haven't really tried it so let me know if you run into problems with it. Tim On Sun, Jul 19, 2015 at 9:23 PM, Jerry Lam chiling...@gmail.com wrote: I only used client mode both 1.3 and 1.4 versions on mesos. I skimmed through https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/mesos/MesosClusterDispatcher.scala. I would actually backport the Cluster Mode feature. Sorry, I don't have an answer for this. On Sun, Jul 19, 2015 at 11:16 PM, Jahagirdar, Madhu madhu.jahagir...@philips.com wrote: 1.3 does not have MesosDisptacher or does not have support for Mesos cluster mode , is it still possible to create a Dispatcher using 1.4 and run 1.3 using that dispatcher ? -- *From:* Jerry Lam [chiling...@gmail.com] *Sent:* Monday, July 20, 2015 8:27 AM *To:* Jahagirdar, Madhu *Cc:* user; d...@spark.apache.org *Subject:* Re: Spark Mesos Dispatcher Yes. Sent from my iPhone On 19 Jul, 2015, at 10:52 pm, Jahagirdar, Madhu madhu.jahagir...@philips.com wrote: All, Can we run different version of Spark using the same Mesos Dispatcher. For example we can run drivers with Spark 1.3 and Spark 1.4 at the same time ? Regards, Madhu Jahagirdar -- The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.
