[Looping Niranda] On Tue, Jul 5, 2016 at 10:22 PM, Akila Ravihansa Perera <[email protected]> wrote:
> Hi, > > Deploying wso2das in Mesos is bit tricky due to lack of overlay network > support in Mesos OOTB. This is mainly because Spark cluster (when running > in Spark cluster mode) is unable to communicate through Docker IP addresses > when they are assigned to different host machines in a multi-node Mesos > deployment. I checked the config options in [1] and tried setting > SPARK_PUBLIC_DNS parameter to Mesos host IP without any success. > > The main problem is there is no way to instruct Spark members to bind to > the local Docker IP and advertise a different IP (Mesos slave host IP) to > other members. > > @Niranda, Nirmal: is this something we can fix from our side? AFAIU, we > are using Hazelcast to discover Spark/wso2das members and adding them to > Spark context, right? > > On a side note, there is a Wiki page explaining how Spark should be used > with Mesos in [2]. This is available after Spark 1.2.0. In this approach, > we treat each Mesos slave as a Spark member and Spark/Mesos driver can > directly schedule tasks on Mesos slaves instead of running Spark itself as > a container. We should consider this approach in our C5 based efforts. We > can leverage Kubernetes in the same way. > > IMO, we should recommend users to use wso2das in Spark client mode on > Mesos due to these complexities. > +1, if we can't solve above from within DAS. > There is a DCOS Mesos framework for Spark [3] which can be used to deploy > Spark on Mesos very easily. We can even leverage DCOS Spark framework in > our deploy scripts. > > [1] http://spark.apache.org/docs/latest/spark-standalone.html > [2] http://spark.apache.org/docs/latest/running-on-mesos.html > [3] https://docs.mesosphere.com/1.7/usage/service-guides/spark/ > > Thanks. > > -- > Akila Ravihansa Perera > WSO2 Inc.; http://wso2.com/ > > Blog: http://ravihansa3000.blogspot.com > -- Thanks & regards, Nirmal Team Lead - WSO2 Machine Learner Associate Technical Lead - Data Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
