Stavros, As mentioned a couple of weeks ago: https://issues.apache.org/jira/browse/SPARK-11638 Happy to answer any questions.
Kind regards, Radek Gruchalski [email protected] (mailto:[email protected]) (mailto:[email protected]) de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/) Confidentiality: This communication is intended for the above-named person and may be confidential and/or legally privileged. If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately. On Thursday, 22 October 2015 at 22:45, Stavros Kontopoulos wrote: > Thnx Rad, sounds pretty cool :). Elizabeth one note for the jira ticket i do > not run the cluster with zookeeper, i use mesos master in stand alone mode... > i guess it makes no difference right? > > On Thu, Oct 22, 2015 at 10:38 PM, Rad Gruchalski <[email protected] > (mailto:[email protected])> wrote: > > There are 2 things: > > > > - Akka remote in 2.3.x does not support advertising hostname / port > > different to what it binds to > > - All other services: file server, broadcast server, repl class server do > > not support advertising hostnames / ports different than what they bind to > > > > Just to expand on the previous one, we are in the process of contributing > > the following bits: > > > > - akka-remote bind-hostname and bind-port backport to akka 2.3.x (not the > > typesafe closed support implementation, our own implementation) > > - spark patches for spark 1.4.0+ which enable running Spark on Mesos in > > Docker Bridge networking > > > > Unfortunately, due to the nature of how my employer operates I can’t share > > the code yet. We are working with our legal team to make these available > > asap. > > > > We do run this stuff in production. > > > > > > > > > > > > > > > > > > > > > > Kind regards, > > Radek Gruchalski > > [email protected] (mailto:[email protected]) > > (mailto:[email protected]) > > de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/) > > > > Confidentiality: > > This communication is intended for the above-named person and may be > > confidential and/or legally privileged. > > If it has come to you in error you must take no action based on it, nor > > must you copy or show it to anyone; please delete/destroy and inform the > > sender immediately. > > > > > > > > On Thursday, 22 October 2015 at 22:28, Iulian Dragoș wrote: > > > > > > > > > > > On Thu, Oct 22, 2015 at 9:10 PM, Rad Gruchalski <[email protected] > > > (mailto:[email protected])> wrote: > > > > Stavros, > > > > > > > > Spark does not support this. I am currently in the process of > > > > submitting patches for it however it first has to pass through the > > > > legal team at the company I work for. > > > > > > What exactly is missing in Spark? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Kind regards, > > > > Radek Gruchalski > > > > [email protected] (mailto:[email protected]) > > > > (mailto:[email protected]) > > > > de.linkedin.com/in/radgruchalski/ > > > > (http://de.linkedin.com/in/radgruchalski/) > > > > > > > > Confidentiality: > > > > This communication is intended for the above-named person and may be > > > > confidential and/or legally privileged. > > > > If it has come to you in error you must take no action based on it, nor > > > > must you copy or show it to anyone; please delete/destroy and inform > > > > the sender immediately. > > > > > > > > > > > > > > > > On Thursday, 22 October 2015 at 21:08, Stavros Kontopoulos wrote: > > > > > > > > > Bridge... with the latest mesos library vesion 0.25... > > > > > > > > > > On Thu, Oct 22, 2015 at 9:07 PM, Elizabeth Lingg > > > > > <[email protected] (mailto:[email protected])> wrote: > > > > > > Are you using Bridge or Host Networking? > > > > > > > > > > > > -Elizabeth > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Oct 22, 2015 at 12:02 PM, Stavros Kontopoulos > > > > > > <[email protected] (mailto:[email protected])> wrote: > > > > > > > Hi, > > > > > > > > > > > > > > Im using spark on mesos on docker. I have linked my slaves to the > > > > > > > master and a > > > > > > > spark repl works fine inside the master container. > > > > > > > > > > > > > > If i try to crate the same spark repl form the host i get stuck > > > > > > > at the point when the framework tries to register to the mesos > > > > > > > master (here the framework is the spark repl itself). > > > > > > > I can ping the container from my host and vice versa. So > > > > > > > networking its not the problem. > > > > > > > What i noticed form the logs is that mesos does not resolve the > > > > > > > correct ip: > > > > > > > > > > > > > > Framework failover timeout, removing framework > > > > > > > b3605c33-f573-4d40-806f-b9b0abee2e32-0012 (Spark shell) at > > > > > > > [email protected]:40186 > > > > > > > (http://[email protected]:40186) > > > > > > > > > > > > > > docker0 interface is on 172.17.x.x and my host is one such ip so > > > > > > > i didnt expect there to see > > > > > > > 127.0.1.1. I have tried several things like spark.driver.host, > > > > > > > SPARK_LOCAL_IP to be set correctly but with no result... > > > > > > > I suspect this is a mesos problem on docker... > > > > > > > > > > > > > > Thnx, > > > > > > > > > > > > > > S. > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > -- > > > Iulian Dragos > > > > > > ------ > > > Reactive Apps on the JVM > > > www.typesafe.com (http://www.typesafe.com) > > > > > >
