Hi Akila, Yes, there is a possibility to use a custom member discovery schema for Spark. In fact, that is what we are using at the moment. We are using Hazlecast for membership management (using WKA). We are extending the following extension points in Spark [1] to support this.
I agree with Imesh. We still have not extensively tested connecting DAS to an external Mesos Spark cluster. But it is in the roadmap and we are working on it. Best [1] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/master/RecoveryModeFactory.scala https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/master/PersistenceEngine.scala On Wed, Jul 6, 2016 at 11:36 AM, Imesh Gunaratne <im...@wso2.com> wrote: > Hi Akila, > > IMO it would better to use Spark Mesos Framework on Mesos DC/OS for > production deployments rather than trying to cluster the built-in Spark > module on Mesos. Then we might not need to worry much about this problem. > > None production environments such as Dev should be able to use a > standalone DAS server with a volume mount with auto healing. WDYT? > > Thanks > > On Wed, Jul 6, 2016 at 9:38 AM, Akila Ravihansa Perera <raviha...@wso2.com > > wrote: > >> [Looping Anjana and Isuru] >> >> @Anjana, Niranda: is it possible to plugin a custom member discovery >> scheme for Spark? In the current implementation how does a wso2das with >> embedded Spark instance discover the other members? >> I'm referring to Spark cluster mode deployment here. >> >> Thanks. >> >> On Wed, Jul 6, 2016 at 6:49 AM, Nirmal Fernando <nir...@wso2.com> wrote: >> >>> [Looping Niranda] >>> >>> On Tue, Jul 5, 2016 at 10:22 PM, Akila Ravihansa Perera < >>> raviha...@wso2.com> wrote: >>> >>>> Hi, >>>> >>>> Deploying wso2das in Mesos is bit tricky due to lack of overlay network >>>> support in Mesos OOTB. This is mainly because Spark cluster (when running >>>> in Spark cluster mode) is unable to communicate through Docker IP addresses >>>> when they are assigned to different host machines in a multi-node Mesos >>>> deployment. I checked the config options in [1] and tried setting >>>> SPARK_PUBLIC_DNS parameter to Mesos host IP without any success. >>>> >>>> The main problem is there is no way to instruct Spark members to bind >>>> to the local Docker IP and advertise a different IP (Mesos slave host IP) >>>> to other members. >>>> >>>> @Niranda, Nirmal: is this something we can fix from our side? AFAIU, we >>>> are using Hazelcast to discover Spark/wso2das members and adding them to >>>> Spark context, right? >>>> >>>> On a side note, there is a Wiki page explaining how Spark should be >>>> used with Mesos in [2]. This is available after Spark 1.2.0. In this >>>> approach, we treat each Mesos slave as a Spark member and Spark/Mesos >>>> driver can directly schedule tasks on Mesos slaves instead of running Spark >>>> itself as a container. We should consider this approach in our C5 based >>>> efforts. We can leverage Kubernetes in the same way. >>>> >>>> IMO, we should recommend users to use wso2das in Spark client mode on >>>> Mesos due to these complexities. >>>> >>> >>> +1, if we can't solve above from within DAS. >>> >>> >>>> There is a DCOS Mesos framework for Spark [3] which can be used to >>>> deploy Spark on Mesos very easily. We can even leverage DCOS Spark >>>> framework in our deploy scripts. >>>> >>>> [1] http://spark.apache.org/docs/latest/spark-standalone.html >>>> [2] http://spark.apache.org/docs/latest/running-on-mesos.html >>>> [3] https://docs.mesosphere.com/1.7/usage/service-guides/spark/ >>>> >>>> Thanks. >>>> >>>> -- >>>> Akila Ravihansa Perera >>>> WSO2 Inc.; http://wso2.com/ >>>> >>>> Blog: http://ravihansa3000.blogspot.com >>>> >>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Team Lead - WSO2 Machine Learner >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >> >> >> -- >> Akila Ravihansa Perera >> WSO2 Inc.; http://wso2.com/ >> >> Blog: http://ravihansa3000.blogspot.com >> > > > > -- > *Imesh Gunaratne* > Software Architect > WSO2 Inc: http://wso2.com > T: +94 11 214 5345 M: +94 77 374 2057 > W: https://medium.com/@imesh TW: @imesh > > -- *Niranda Perera* Software Engineer, WSO2 Inc. Mobile: +94-71-554-8430 Twitter: @n1r44 <https://twitter.com/N1R44> https://pythagoreanscript.wordpress.com/
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
