Thnx Rad, sounds pretty cool :). Elizabeth one note for the jira ticket i do not run the cluster with zookeeper, i use mesos master in stand alone mode... i guess it makes no difference right?
On Thu, Oct 22, 2015 at 10:38 PM, Rad Gruchalski <[email protected]> wrote: > There are 2 things: > > - Akka remote in 2.3.x does not support advertising hostname / port > different to what it binds to > - All other services: file server, broadcast server, repl class server do > not support advertising hostnames / ports different than what they bind to > > Just to expand on the previous one, we are in the process of contributing > the following bits: > > - akka-remote bind-hostname and bind-port backport to akka 2.3.x (not the > typesafe closed support implementation, our own implementation) > - spark patches for spark 1.4.0+ which enable running Spark on Mesos in > Docker Bridge networking > > Unfortunately, due to the nature of how my employer operates I can’t share > the code yet. We are working with our legal team to make these available > asap. > > We do run this stuff in production. > > Kind regards, > Radek Gruchalski > [email protected] <[email protected]> > de.linkedin.com/in/radgruchalski/ > > > *Confidentiality:*This communication is intended for the above-named > person and may be confidential and/or legally privileged. > If it has come to you in error you must take no action based on it, nor > must you copy or show it to anyone; please delete/destroy and inform the > sender immediately. > > On Thursday, 22 October 2015 at 22:28, Iulian Dragoș wrote: > > > > On Thu, Oct 22, 2015 at 9:10 PM, Rad Gruchalski <[email protected]> > wrote: > > Stavros, > > Spark does not support this. I am currently in the process of submitting > patches for it however it first has to pass through the legal team at the > company I work for. > > > What exactly is missing in Spark? > > > Kind regards, > Radek Gruchalski > [email protected] <[email protected]> > de.linkedin.com/in/radgruchalski/ > > > *Confidentiality:*This communication is intended for the above-named > person and may be confidential and/or legally privileged. > If it has come to you in error you must take no action based on it, nor > must you copy or show it to anyone; please delete/destroy and inform the > sender immediately. > > On Thursday, 22 October 2015 at 21:08, Stavros Kontopoulos wrote: > > Bridge... with the latest mesos library vesion 0.25... > > On Thu, Oct 22, 2015 at 9:07 PM, Elizabeth Lingg <[email protected]> > wrote: > > Are you using Bridge or Host Networking? > > -Elizabeth > > > > On Thu, Oct 22, 2015 at 12:02 PM, Stavros Kontopoulos < > [email protected]> wrote: > > Hi, > > Im using spark on mesos on docker. I have linked my slaves to the master > and a > spark repl works fine inside the master container. > > If i try to crate the same spark repl form the host i get stuck at the > point when the framework tries to register to the mesos master (here the > framework is the spark repl itself). > I can ping the container from my host and vice versa. So networking its > not the problem. > What i noticed form the logs is that mesos does not resolve the correct ip: > > Framework failover timeout, removing framework > b3605c33-f573-4d40-806f-b9b0abee2e32-0012 (Spark shell) at > [email protected]:40186 > > docker0 interface is on 172.17.x.x and my host is one such ip so i didnt > expect there to see > 127.0.1.1. I have tried several things like spark.driver.host, > SPARK_LOCAL_IP to be set correctly but with no result... > I suspect this is a mesos problem on docker... > > Thnx, > > S. > > > > > > > > -- > > -- > Iulian Dragos > > ------ > Reactive Apps on the JVM > www.typesafe.com > > >
