Re: Launch docker container from Marathon UI

[Rad Gruchalski]

Marathon does not know it has to run it. You need to tell Marathon to start 
mysql container and then tell your wordpress container where is your mysql.










Best regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Tuesday, 26 April 2016 at 17:43, Stefano Bianchi wrote:

> My problem is this, where can i find the mysql container? I have just said to 
> marathon to rim a wordpress docker container, without specify mysql one.
> Il 26/apr/2016 17:39, "Aaron Carey" <aca...@ilm.com (mailto:aca...@ilm.com)> 
> ha scritto:
> > If you run the wordpress container on a different host to the mysql 
> > container and use --link on the command line, does that work?
> >  
> > --
> >  
> > Aaron Carey
> > Production Engineer - Cloud Pipeline Industrial Light & Magic London
> > 020 3751 9150  
> > From: Stefano Bianchi [jazzist...@gmail.com (mailto:jazzist...@gmail.com)]
> > Sent: 26 April 2016 16:23
> > To: user@mesos.apache.org (mailto:user@mesos.apache.org)
> > Subject: RE: Launch docker container from Marathon UI
> >  
> > Hi Aaron
> > Actually both mesos-dns and calico are running even though these containers 
> > im running are not, yet, using the calico ipaddr specific in json 
> > description.
> > So i guess it is a problem of bridging, i guess simply specify HOST option 
> > on container field in Marathon UI  
> > Il 26/apr/2016 16:59, "Aaron Carey" <aca...@ilm.com 
> > (mailto:aca...@ilm.com)> ha scritto:
> > > --link in docker should really be avoided when using marathon/mesos as it 
> > > implies the containers are on the same host, but this will not always be 
> > > the case when mesos schedules your containers (also I think it's being 
> > > deprecated in docker anyway.. not sure though?).
> > >  
> > > This problem looks like one of service discovery within the mesos 
> > > cluster: how does one service contact the other when it doesn't know 
> > > which host the other service may have landed on?
> > >  
> > > There are several different solutions for service discovery, you can look 
> > > into some like Project Calico to offer a network layer to docker or try 
> > > dns based solutions like Mesos-dns or Consul (along with mesos-consul). I 
> > > think marathon also has some concept of service discovery built in too if 
> > > you use something like haproxy.
> > >  
> > > I hope this helps!
> > >  
> > > Aaron
> > >  
> > > --
> > >  
> > > Aaron Carey
> > > Production Engineer - Cloud Pipeline Industrial Light & Magic London
> > > 020 3751 9150  
> > > From: June Taylor [j...@umn.edu (mailto:j...@umn.edu)]
> > > Sent: 26 April 2016 15:22
> > > To: user@mesos.apache.org (mailto:user@mesos.apache.org)
> > > Subject: Re: Launch docker container from Marathon UI
> > >  
> > > Stefano,  
> > >  
> > > The docker run flag --link is intended to connect the container to 
> > > another running container. I do not know how this would operate in 
> > > marathon. Perhaps it would be an application group which starts up the 
> > > mysql docker image first, then the Wordpress docker image after it.  
> > >  
> > >  
> > > Thanks,  
> > > June Taylor
> > >  
> > > System Administrator, Minnesota Population Center
> > > University of Minnesota
> > >  
> > >  
> > >  
> > >  
> > >  
> > >  
> > >  
> > >  
> > > On Tue, Apr 26, 2016 at 9:20 AM, Stefano Bianchi <jazzist...@gmail.com 
> > > (mailto:jazzist...@gmail.com)> wrote:
> > > > Thanks Rad Gruchalski, actually i'm trying to make a json file that 
> > > > "translate" this command in marathon:  
> > > >  
> > > > docker run --name some-wordpress --link some-mysql:mysql -d wordpress  
> > > >  
> > > > i guess the error is relate to the fact that i'm not specifying --link 
> > > > some-mysql:mysql  
> > > > My problem is that i don't know how to do that, i tried to fill 
> > > > e

Re: Launch docker container from Marathon UI

[Rad Gruchalski]

It says exactly what the problem is.  

Start a marathon task with correct environment variables in env and you will be 
fine.  











Best regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Tuesday, 26 April 2016 at 15:56, Stefano Bianchi wrote:

> jupyter is working fine.
> i tried to run wordpress and i get this error in stderr of mesos:
>  
> error: missing WORDPRESS_DB_HOST and MYSQL_PORT_3306_TCP environment 
> variables  
>   Did you forget to --link some_mysql_container:mysql or set an external db
>   with -e WORDPRESS_DB_HOST=hostname:port?
>  
>  
> Some one of you know this issue?
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
> 2016-04-26 15:51 GMT+02:00 Stefano Bianchi  (mailto:jazzist...@gmail.com)>:
> > However thank you so much to all!
> >  
> > 2016-04-26 15:22 GMT+02:00 Stefano Bianchi  > (mailto:jazzist...@gmail.com)>:
> > > i found the answer by my self sorry if i disturbed you.
> > >  
> > > 2016-04-26 15:19 GMT+02:00 Stefano Bianchi  > > (mailto:jazzist...@gmail.com)>:
> > > > Now that it is running, how can i check the User Interface?
> > > >  
> > > >  
> > > > 2016-04-26 15:18 GMT+02:00 Stefano Bianchi  > > > (mailto:jazzist...@gmail.com)>:
> > > > > Yes, now it's running!!! June you are awesome!!!
> > > > >  
> > > > > 2016-04-26 15:16 GMT+02:00 Stefano Bianchi  > > > > (mailto:jazzist...@gmail.com)>:
> > > > > > i have done these 2 commands, now jupyter is in deploying in 
> > > > > > marathon, staging in mesos.
> > > > > > Is there some additional configuration needed?
> > > > > >  
> > > > > >  
> > > > > > 2016-04-26 15:13 GMT+02:00 Stefano Bianchi  > > > > > (mailto:jazzist...@gmail.com)>:
> > > > > > > June i tried to run your json, but the task is waiting, and if 
> > > > > > > starts it failed immediately.
> > > > > > >  
> > > > > > > I guess because i did not type this commands:
> > > > > > >  
> > > > > > > echo 'docker,mesos' > /etc/mesos-slave/containerizers
> > > > > > > $ echo '5mins' > /etc/mesos-slave/executor_registration_timeout
> > > > > > >  
> > > > > > >  
> > > > > > >  
> > > > > > >  
> > > > > > > Could it be the problem?
> > > > > > >  
> > > > > > >  
> > > > > > > 2016-04-26 15:02 GMT+02:00 Stefano Bianchi  > > > > > > (mailto:jazzist...@gmail.com)>:
> > > > > > > > Thank you June taylor, it is axactly what i was intending.
> > > > > > > > if it is not disturbing you, i try jupyter notebook as well, 
> > > > > > > > just to make some tests on how to launch marathon.
> > > > > > > > stay tuned :)
> > > > > > > >  
> > > > > > > >  
> > > > > > > > 2016-04-26 14:58 GMT+02:00 Stefano Bianchi 
> > > > > > > > :
> > > > > > > > > thanks haosdent.
> > > > > > > > > actually i have run this kind of apps through rest api.
> > > > > > > > > Now i want learn how to laung docker files. I found this 
> > > > > > > > > exaple:
> > > > > > > > >  
> > > > > > > > > { "container": { "type": "DOCKER", "docker": { "network": 
> > > > > > > > > "HOST", "image": "group/image" }, "volumes": [ { 
> > > > > > > > > "containerPath": "/etc/a", "hostPath": "/var/data/a", "mode": 
> > > > > > > > > "RO" }, { "containerPath": "/etc/b", "hostPath": 
> > > > > > > > > "/var/data/b", "mode": "RW" } ] } }
> > > > > > > > >  
> > > > > > > > > I am able to link the correspondent field in the marathon UI 
> > > > > > > > > with what is written in this json.
> > > > > > > > > What i need to understand is, what i should write in:
> > > > > > > > > "image": "group/image"
> > > > > > > > > i mean, with what i have to replace group and image?
> > > > > > > > > For instance, if i would run wordpress container on docker 
> > > > > > > > > hub what i should put replacing "group/image" ?
> > > > > > > > >  
> > > > > > > > >  
> > > > > > > > >  
> > > > > > > > >  
> > > > > > > > >  
> > > > > > > > >  
> > > > > > > > >  
> > > > > > > > > 2016-04-26 14:30 GMT+02:00 haosdent  > > > > > > > > (mailto:haosd...@gmail.com)>:
> > > > > > > > > > This is marathon REST API document: 
> > > > > > > > > > https://mesosphere.github.io/marathon/docs/rest-api.html#post-v2-apps
> > > > > > > > > >  . Usually we define a task json file and call the rest api.
> > > > > > > > > >  
> > > > > > > > > > For example, create tasks.json which content is  
> > > > > > > > > >  
> > > > > > > > > > ```
> > > > > > > > > > {
> > > > > > > 

Re: Launch docker container from Marathon UI

[Rad Gruchalski]

Here’s the up to date URL:  
https://mesosphere.github.io/marathon/docs/generated/api.html#v2_apps_post










Best regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Tuesday, 26 April 2016 at 14:30, haosdent wrote:

> This is marathon REST API document: 
> https://mesosphere.github.io/marathon/docs/rest-api.html#post-v2-apps . 
> Usually we define a task json file and call the rest api.
>  
> For example, create tasks.json which content is  
>  
> ```
> {
> "id": "/test-sleep",
> "cmd": "sleep 200"
> }
> ```
>  
> And  
>  
> ```
> curl -X POST -H "Content-type: application/json" 
> http://${YOUR_MARATHON_ADDR}:8080/v2/apps -d @tasks.json
> ```
>  
> You could find more field definitions in their document.
>  
> On Tue, Apr 26, 2016 at 8:24 PM, Stefano Bianchi  (mailto:jazzist...@gmail.com)> wrote:
> > Hi guys
> >  
> > i'm trying to follow the instructions on mesosphere documentation, but it 
> > is a little bit unlear for me that i am new to this world.
> > i' would like to launch a wordpress container, present on docker hub here 
> > https://hub.docker.com/_/wordpress/ , from marathon UI but i don't 
> > understand the mechanism.  
> > I understood that i don't have to specify a command in "Command" field, but 
> > i don't know how to fill the dockers fields in marathon UI to launch 
> > wordpress container.
> > Could you please help me or tell me where i can find a clearer 
> > documentation?
> >  
> > Thanks.  
>  
>  
> --  
> Best Regards,
> Haosdent Huang  

Re: Custom IPTables rules

[Rad Gruchalski]

Alfredo,  

I have no examples of locking that one on hand but I can imagine that if should 
be feasible to lock that down.










Best regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Wednesday, 13 April 2016 at 22:14, Alfredo Carneiro wrote:

> Unfortunately, I am facing some problemseven with my INPUT rules allowing 
> just some subnetworks, Docker is accepting connections from everywhere.
>  
> On Wed, Apr 13, 2016 at 5:06 PM, Rad Gruchalski <ra...@gruchalski.com 
> (mailto:ra...@gruchalski.com)> wrote:
> > I actually found the complete thing you need. Here we go:  
> >  
> > *nat
> > …
> >  
> > :DOCKER - [0:0]
> > -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
> > -A OUTPUT ! -d 127.0.0.0/8 (http://127.0.0.0/8) -m addrtype --dst-type 
> > LOCAL -j DOCKER
> > -A POSTROUTING -s 172.17.0.0/16 (http://172.17.0.0/16) ! -o docker0 -j 
> > MASQUERADE
> > # This is where the docker NAT rules go
> >  
> >  
> > # NAT chains
> >  
> > COMMIT
> >  
> > *filter
> > …
> > :DOCKER - [0:0]
> >  
> > …
> >  
> > -A FORWARD -o docker0 -j DOCKER
> > -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
> > -A FORWARD -i docker0 ! -o docker0 -j ACCEPT
> > -A FORWARD -i docker0 -o docker0 -j ACCEPT
> >  
> >  
> > This gives you everything you need. Thanks to Avinash for pointing this 
> > out.  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> > Best regards,

> > Radek Gruchalski
> > 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
> > (mailto:ra...@gruchalski.com)
> > de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
> >  
> > Confidentiality:
> > This communication is intended for the above-named person and may be 
> > confidential and/or legally privileged.
> > If it has come to you in error you must take no action based on it, nor 
> > must you copy or show it to anyone; please delete/destroy and inform the 
> > sender immediately.
> >  
> >  
> >  
> > On Wednesday, 13 April 2016 at 21:59, Alfredo Carneiro wrote:
> >  
> > > Oh man! Really thanks! It worked!
> > >  
> > > On Wed, Apr 13, 2016 at 4:57 PM, Rad Gruchalski <ra...@gruchalski.com 
> > > (mailto:ra...@gruchalski.com)> wrote:
> > > > Have you tried restarting docker daemon afterwards?
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > > Best regards,

> > > > Radek Gruchalski
> > > > 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
> > > > (mailto:ra...@gruchalski.com)
> > > > de.linkedin.com/in/radgruchalski/ 
> > > > (http://de.linkedin.com/in/radgruchalski/)
> > > >  
> > > > Confidentiality:
> > > > This communication is intended for the above-named person and may be 
> > > > confidential and/or legally privileged.
> > > > If it has come to you in error you must take no action based on it, nor 
> > > > must you copy or show it to anyone; please delete/destroy and inform 
> > > > the sender immediately.
> > > >  
> > > >  
> > > >  
> > > > On Wednesday, 13 April 2016 at 21:53, Alfredo Carneiro wrote:
> > > >  
> > > > > Hey Rad,
> > > > >  
> > > > > Thanks for your answer! I have added theses lines and now looks very 
> > > > > similar before.
> > > > >  
> > > > > iptables -N DOCKER
> > > > > iptables -A FORWARD -o docker0 -j DOCKER
> > > > > iptables -A FORWARD -o docker0 -m conntrack --ctstate 
> > > > > RELATED,ESTABLISHED -j ACCEPT
> > > > > iptables -A FORWARD -i docker0 ! -o docker0 -j ACCEPT
> > > > > iptables -A FORWARD -i docker0 -o docker0 -j ACCEPT
> > > > >  
> > > > >  
> > > > > However, I am

Re: Custom IPTables rules

[Rad Gruchalski]

I actually found the complete thing you need. Here we go:  

*nat
…

:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
# This is where the docker NAT rules go


# NAT chains

COMMIT

*filter
…
:DOCKER - [0:0]

…

-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT


This gives you everything you need. Thanks to Avinash for pointing this out.  











Best regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Wednesday, 13 April 2016 at 21:59, Alfredo Carneiro wrote:

> Oh man! Really thanks! It worked!
>  
> On Wed, Apr 13, 2016 at 4:57 PM, Rad Gruchalski <ra...@gruchalski.com 
> (mailto:ra...@gruchalski.com)> wrote:
> > Have you tried restarting docker daemon afterwards?
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> > Best regards,

> > Radek Gruchalski
> > 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
> > (mailto:ra...@gruchalski.com)
> > de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
> >  
> > Confidentiality:
> > This communication is intended for the above-named person and may be 
> > confidential and/or legally privileged.
> > If it has come to you in error you must take no action based on it, nor 
> > must you copy or show it to anyone; please delete/destroy and inform the 
> > sender immediately.
> >  
> >  
> >  
> > On Wednesday, 13 April 2016 at 21:53, Alfredo Carneiro wrote:
> >  
> > > Hey Rad,
> > >  
> > > Thanks for your answer! I have added theses lines and now looks very 
> > > similar before.
> > >  
> > > iptables -N DOCKER
> > > iptables -A FORWARD -o docker0 -j DOCKER
> > > iptables -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED 
> > > -j ACCEPT
> > > iptables -A FORWARD -i docker0 ! -o docker0 -j ACCEPT
> > > iptables -A FORWARD -i docker0 -o docker0 -j ACCEPT
> > >  
> > >  
> > > However, I am still getting errors.
> > >  
> > > docker: Error response from daemon: failed to create endpoint 
> > > cranky_kilby on network bridge: iptables failed: iptables --wait -t nat 
> > > -A DOCKER -p tcp -d 0/0 --dport 8080 -j DNAT --to-destination 
> > > 172.17.0.2:8080 (http://172.17.0.2:8080) ! -i docker0: iptables: No 
> > > chain/target/match by that name.
> > >  (exit status 1).
> > >  
> > >  
> > > This is my iptables -L output:
> > >  
> > > Chain FORWARD (policy DROP)
> > > target prot opt source   destination  
> > > DOCKER all  --  anywhere anywhere 
> > > ACCEPT all  --  anywhere anywhere ctstate 
> > > RELATED,ESTABLISHED
> > > ACCEPT all  --  anywhere anywhere 
> > > ACCEPT all  --  anywhere anywhere 
> > >  
> > > Chain OUTPUT (policy ACCEPT)
> > > target prot opt source   destination  
> > > ACCEPT all  --  anywhere anywhere 
> > >  
> > > Chain DOCKER (1 references)
> > > target prot opt source   destination
> > >  
> > >  
> > > I hid the INPUT chain because is very big!
> > >  
> > > Best Regards,
> > >  
> > > On Wed, Apr 13, 2016 at 4:29 PM, Rad Gruchalski <ra...@gruchalski.com 
> > > (mailto:ra...@gruchalski.com)> wrote:
> > > > Hi Alfredo,  
> > > >  
> > > > The only thing you need is:
> > > >  
> > > > -A FORWARD -o docker0 -j DOCKER
> > > > -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j 
> > > > ACCEPT
> > > > -A FORWARD -i docker0 ! -o docker0 -j ACCEPT
> > > > -A FORWARD -i docker0 -o docker0 -j ACCEPT
> > > >  
> > > >  
> >

Re: Custom IPTables rules

[Rad Gruchalski]

Have you tried restarting docker daemon afterwards?










Best regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Wednesday, 13 April 2016 at 21:53, Alfredo Carneiro wrote:

> Hey Rad,
>  
> Thanks for your answer! I have added theses lines and now looks very similar 
> before.
>  
> iptables -N DOCKER
> iptables -A FORWARD -o docker0 -j DOCKER
> iptables -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j 
> ACCEPT
> iptables -A FORWARD -i docker0 ! -o docker0 -j ACCEPT
> iptables -A FORWARD -i docker0 -o docker0 -j ACCEPT
>  
>  
> However, I am still getting errors.
>  
> docker: Error response from daemon: failed to create endpoint cranky_kilby on 
> network bridge: iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 
> 0/0 --dport 8080 -j DNAT --to-destination 172.17.0.2:8080 
> (http://172.17.0.2:8080) ! -i docker0: iptables: No chain/target/match by 
> that name.
>  (exit status 1).
>  
>  
> This is my iptables -L output:
>  
> Chain FORWARD (policy DROP)
> target prot opt source   destination  
> DOCKER all  --  anywhere anywhere 
> ACCEPT all  --  anywhere anywhere ctstate 
> RELATED,ESTABLISHED
> ACCEPT all  --  anywhere anywhere 
> ACCEPT all  --  anywhere anywhere 
>  
> Chain OUTPUT (policy ACCEPT)
> target prot opt source   destination  
> ACCEPT all  --  anywhere anywhere 
>  
> Chain DOCKER (1 references)
> target prot opt source   destination
>  
>  
> I hid the INPUT chain because is very big!
>  
> Best Regards,
>  
> On Wed, Apr 13, 2016 at 4:29 PM, Rad Gruchalski <ra...@gruchalski.com 
> (mailto:ra...@gruchalski.com)> wrote:
> > Hi Alfredo,  
> >  
> > The only thing you need is:
> >  
> > -A FORWARD -o docker0 -j DOCKER
> > -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
> > -A FORWARD -i docker0 ! -o docker0 -j ACCEPT
> > -A FORWARD -i docker0 -o docker0 -j ACCEPT
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> > Best regards,

> > Radek Gruchalski
> > 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
> > (mailto:ra...@gruchalski.com)
> > de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
> >  
> > Confidentiality:
> > This communication is intended for the above-named person and may be 
> > confidential and/or legally privileged.
> > If it has come to you in error you must take no action based on it, nor 
> > must you copy or show it to anyone; please delete/destroy and inform the 
> > sender immediately.
> >  
> >  
> >  
> > On Wednesday, 13 April 2016 at 21:27, Alfredo Carneiro wrote:
> >  
> > > Hello guys,
> > >  
> > > I don't know if that is the right place to ask. So, since we use public 
> > > cloud, we are trying to hardening our servers allowing traffic just from 
> > > our subnetworks. However, when I tried to implement some iptables rules I 
> > > got problems with Docker, which couldn't find its chain anymore.
> > >  
> > > Then, I am wondering if anyone has ever implemented any iptables rule in 
> > > this scenario.
> > >  
> > > I've seen this[1] "tip", however, I think that it is not apply to this 
> > > case, because it is very "static".
> > >  
> > > [1] - https://fralef.me/docker-and-iptables.html
> > >  
> > > Best Regards,
> > >  
> > > --  
> > > Alfredo Miranda  
> >  
>  
>  
>  
> --  
> Alfredo Miranda  

Re: Custom IPTables rules

[Rad Gruchalski]

Hi Alfredo,  

The only thing you need is:

-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT











Best regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Wednesday, 13 April 2016 at 21:27, Alfredo Carneiro wrote:

> Hello guys,
>  
> I don't know if that is the right place to ask. So, since we use public 
> cloud, we are trying to hardening our servers allowing traffic just from our 
> subnetworks. However, when I tried to implement some iptables rules I got 
> problems with Docker, which couldn't find its chain anymore.
>  
> Then, I am wondering if anyone has ever implemented any iptables rule in this 
> scenario.
>  
> I've seen this[1] "tip", however, I think that it is not apply to this case, 
> because it is very "static".
>  
> [1] - https://fralef.me/docker-and-iptables.html
>  
> Best Regards,
>  
> --  
> Alfredo Miranda  

Re: marathon issue in running a docker container.

[Rad Gruchalski]

Do you have anything like haproxy for port mappings installed on your Mesos 
cluster?  
When using BRIDGE network, your process inside of the container, say SSH, is 
running on a standard port 22. Marathon allocates a random port in the default 
range of 31000 to 32000. However, it is your task to map the :31xxx to 
:22.

The simplest is to use this:
https://github.com/mesosphere/marathon/blob/master/examples/haproxy-marathon-bridge

The haproxy-marathon-bridge needs to run as a cron job on every agent. Because 
it runs as a cron job every minute, your ports become accessible after up to 
one minute from going into RUNNING state.

There are obviously moe advanced ways of getting this done - 
haproxy-marathon-bridge is the simplest one.










Best regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Tuesday, 12 April 2016 at 10:19, nirmalendu swain wrote:

> From the stderr log, nothing can be figure out. From stout log, it says 
> server running at host:port. But If I do telnet or curl, it does not show me 
> any reponse.
> output of stderr log :
>  
> I0412 08:16:12.842341  9909 exec.cpp:134] Version: 0.27.1
> I0412 08:16:12.844701  9934 exec.cpp:208] Executor registered on slave 
> 87849fd2-fda9-4d6a-870f-de101a5bdc59-S3
> js-bson: Failed to load c++ bson extension, using pure JS version
>  
>  
> On Tuesday, 12 April 2016 1:39 PM, Abhishek Amralkar 
>  
> wrote:
>  
>  
> anything is sandbox logs, why the tasks are getting killed? `stderr` and 
> `stdout`?
> > On 12-Apr-2016, at 1:35 PM, haosdent  > (mailto:haosd...@gmail.com)> wrote:  
> > >Its frequently changing the deployment status to Staged  
> >  
> > Do you find any related log in mesos when marathon lauch the task?  
> >  
> > On Tue, Apr 12, 2016 at 3:55 PM, nirmalendu swain 
> >  wrote:
> > > Changing the network type to HOST does not work. Its frequently changing 
> > > the deployment status to Staged and then to no task.  
> > >  
> > >  
> > > On Tuesday, 12 April 2016 11:49 AM, haosdent  > > (mailto:haosd...@gmail.com)> wrote:
> > >  
> > >  
> > > How about change the network type from BRIDGE to HOST?  
> > >  
> > > On Tue, Apr 12, 2016 at 2:13 PM, nirmalendu swain 
> > >  wrote:
> > > > Hi Mesos user,  
> > > > I am running mesos marathon using dcos for spinning up AWS instances.  
> > > > I have successfully built the mongodb as a docker container, but when I 
> > > > try to deploy my dockerized app, it does n't deploy. My App is 
> > > > dependent upon mongo which is passed as environment variable in the 
> > > > json file to be run by dcos command. If a i do a telnet/curl, it does 
> > > > n't receive at the desired host:port. from the mesos logs, it does not 
> > > > seem to throw any error/exception. Doing a copy-past of my 
> > > > backend-app.json file which falis to deploy.
> > > >  
> > > >  
> > > > {  
> > > > "id": "/todo-with-backend",
> > > > "instances": 2,
> > > > "container": {
> > > > "type": "DOCKER",
> > > > "docker": {
> > > > "image": "tldr/todo-backend",
> > > > "network": "BRIDGE",
> > > > "portMappings": [
> > > > {
> > > > "containerPort": 8080,
> > > > "hostPort": 0,
> > > > "protocol": "tcp"
> > > > }
> > > > ]
> > > > }
> > > > },
> > > > "env":{
> > > >"MONGO_URL":"10.0.2.252:5530 (http://10.0.2.252:5530/)"
> > > >  },
> > > > "healthChecks": [{
> > > > "protocol": "HTTP",
> > > > "portIndex": 0
> > > > }],
> > > > "labels":{
> > > > "HAPROXY_GROUP":"external",
> > > > ""
> > > > },
> > > >  
> > > > "cpus": 0.25,
> > > > "mem": 256.0
> > > > }
> > > >  
> > > > I have gone inside the host and checked that env value is reflecting 
> > > > correctly.  
> > > > Please help me out in analyzing the issue.
> > > >  
> > > > Regards,  
> > > > Nirmal
> > > >  
> > > >  
> > >  
> > >  
> > >  
> > > --  
> > > Best Regards,
> > > Haosdent Huang  
> > >  
> >  
> >  
> >  
> > --  
> > Best Regards,
> > Haosdent Huang  
>  
>  

Re: orphaned_tasks cleanup and prevention method

[Rad Gruchalski]

Greg,

All you need to do is tell Spark that the master is mesos://…, as in the 
example from June.
It’s all nicely documented here:

http://spark.apache.org/docs/latest/running-on-mesos.html

I’d suggest running in coarse mode as fine grained is a bit choppy.










Best regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Saturday, 9 April 2016 at 00:48, Greg Mann wrote:

> Unfortunately I'm not able to glean much from that command, but perhaps 
> someone out there with more Spark experience can? I do know that there are a 
> couple ways to launch Spark jobs on a cluster: you can run them in client 
> mode, where the Spark driver runs locally on your machine and exits when it's 
> finished, or they can be run in cluster mode where the Spark driver runs 
> persistently on the cluster as a Mesos framework. How exactly are you 
> launching these tasks on the Mesos cluster?
>  
> On Fri, Apr 8, 2016 at 5:41 AM, June Taylor  (mailto:j...@umn.edu)> wrote:
> > Greg,
> >  
> > I'm on the ops side and fairly new to spark/mesos, so I'm not quite sure I 
> > understand your question, here's how the task shows up in a process listing:
> >  
> > /usr/lib/jvm/java-8-oracle/bin/java -cp 
> > /path/to/spark/spark-installations/spark-1.6.0-bin-hadoop2.6/conf/:/path/to/spark/spark-installations/spark-1.6.0-bin-hadoop2.6/lib/spark-assembly-1.6.0-hadoop2.6.0.jar:/path/to/spark/spark-installations/spark-1.6.0-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar:/path/to/spark/spark-installations/spark-1.6.0-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar:/path/to/spark/spark-installations/spark-1.6.0-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar
> >  -Xms10G -Xmx10G org.apache.spark.deploy.SparkSubmit --master 
> > mesos://master.ourdomain.com (http://master.ourdomain.com):5050 --conf 
> > spark.driver.memory=10G --executor-memory 100G --total-executor-cores 90 
> > pyspark-shell
> >  
> >  
> > Thanks,
> > June Taylor
> >  
> > System Administrator, Minnesota Population Center
> > University of Minnesota
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> > On Thu, Apr 7, 2016 at 3:37 PM, Greg Mann  > (mailto:g...@mesosphere.io)> wrote:
> > > Hi June,
> > > Are these Spark tasks being run in cluster mode or client mode? If it's 
> > > client mode, then perhaps your local Spark scheduler is tearing itself 
> > > down before the executors exit, thus leaving them orphaned.
> > >  
> > > I'd love to see master/agent logs during the time that the tasks are 
> > > becoming orphaned if you have them available.
> > >  
> > > Cheers,
> > > Greg
> > >  
> > >  
> > > On Thu, Apr 7, 2016 at 1:08 PM, June Taylor  > > (mailto:j...@umn.edu)> wrote:
> > > > Just a quick update... I was only able to get the orphans cleared by 
> > > > stopping mesos-slave, deleting the contents of the scratch directory, 
> > > > and then restarting mesos-slave.
> > > >  
> > > >  
> > > > Thanks,
> > > > June Taylor
> > > >  
> > > > System Administrator, Minnesota Population Center
> > > > University of Minnesota
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > > On Thu, Apr 7, 2016 at 12:01 PM, Vinod Kone  > > > (mailto:vinodk...@apache.org)> wrote:
> > > > > A task/executor is called "orphaned" if the corresponding scheduler 
> > > > > doesn't register with Mesos. Is your framework scheduler running or 
> > > > > gone for good? The resources should be cleaned up if the agent (and 
> > > > > consequently the master) have realized that the executor exited.
> > > > >  
> > > > > Can you paste the master and agent logs for one of orphaned 
> > > > > tasks/executors (grep the log with the task/executor id)?
> > > > >  
> > > > > On Thu, Apr 7, 2016 at 9:00 AM, haosdent  > > > > (mailto:haosd...@gmail.com)> wrote:
> > > > > > Hmm, sorry for didn't express my idea clear. I mean kill those 
> > > > > > orphan tasks here.
> > > > > >  
> > > > > > On Thu, Apr 7, 2016 at 11:57 PM, June Taylor  > > > > > (mailto:j...@umn.edu)> wrote:
> > > > > > > Forgive my ignorance, are you literally saying I should just 
> > > > > > > sigkill these instances? How will that clean up the mesos orphans?
> > > > > > >  
> > > > > > >  
> > > > > > > Thanks,
> > > > > > > June Taylor
> > > > > > >  
> > > > > > > System Administrator, Minnesota Population Center
> > > > > > > University of Minnesota
> > > > > > >  
> > > > > > >  
> > > > > > >  
> > > > > > >  
> > > > > > >  
> > > > > > >  
> > > > > > >  

Re: Set LIBPROCESS_IP for frameworks launched with marathon

[Rad Gruchalski]

Hi Gmail,  

AFAIK not. The only way to do so is setting up the env variable as you do now.










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Sunday, 3 April 2016 at 16:09, Gmail wrote:

> I'm pretty new to mesos and marathon, and I'm running a couple of frameworks 
> with marathon (Kafka and elastic search). However, in order for the 
> frameworks to receive resource offers I need to set the LIBPROCESS_IP 
> environment variable to the hosts IP address for the docker container running 
> the frameworks. Currently I am working around me this by using a constraint 
> to hard wire the slave that the framework gets launched on, so then I can put 
> the slaves ip in the marathon json file.
>  
> Obviously this is not ideal. Is there a better way to define the host ip 
> Inside the docker container?
>  
> Sent from my iPad  

Re: Apache Spark Over Mesos

[Rad Gruchalski]

As Tim suggested: spark.mesos.executor.docker.image is your friend.










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Tuesday, 15 March 2016 at 17:23, Pradeep Chhetri wrote:

> Hello Radoslaw,
>  
> Thank you for the quick reply. Few questions:
>  
> 1) Do you mean mounting spark artifacts as a volume on each mesos agent node? 
>  This means number of volumes = number of mesos agents.
>  
> 2) Since I am not using HDFS at all, that is definitely not an option for me.
>  
> Isn't there a way to just launch the spark tasks also as docker containers 
> which are self contained with spark artifacts ?
>  
> Thanks.
>  
> On Tue, Mar 15, 2016 at 3:49 PM, Radoslaw Gruchalski  (mailto:ra...@gruchalski.com)> wrote:
> > Pradeep,
> >  
> > You can mount a spark directory as a volume. This means you have to have 
> > spark deployed on every agent.
> >  
> > Another thing you can do, place spark in hdfs, assuming that you have hdfs 
> > available but that too will download a copy to the sandbox.
> >  
> > I'd prefer the former.
> >  
> > Sent from Outlook Mobile (https://aka.ms/qtex0l)
> > _
> > From: Pradeep Chhetri  > (mailto:pradeep.chhetr...@gmail.com)>
> > Sent: Tuesday, March 15, 2016 4:41 pm
> > Subject: Apache Spark Over Mesos
> > To: 
> >  
> >  
> >  
> > Hello,  
> >  
> > I am able to run Apache Spark over Mesos. Its quite simple to run Spark 
> > Dispatcher over marathon and ask it to run Spark Executor (I guess also can 
> > be called as Spark Driver) as docker container.  
> >  
> > I have a query regarding this:  
> >  
> > All spark tasks are spawned directly by first downloading the spark 
> > artifacts. I was thinking if there is some way I can start them too as 
> > docker containers. This will save the time for downloading the spark 
> > artifacts. I am running spark in fine-grained mode.  
> >  
> > I have attached a screenshot of a sample job  
> >  
> >  
> > ​  
> > Thanks,  
> >  
> > --  
> > Pradeep Chhetri  
> >  
>  
>  
>  
> --  
> Pradeep Chhetri  

Re: Apache Spark Over Mesos

[Rad Gruchalski]

That’s a great alternative!










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Tuesday, 15 March 2016 at 17:19, Timothy Chen wrote:

> You can launch the driver and executor in docker containers as well by 
> setting spark.mesos.executor.docker.image to the image you want to use to 
> launch them.
>  
> Tim
>  
> On Mar 15, 2016, at 8:49 AM, Radoslaw Gruchalski  (mailto:ra...@gruchalski.com)> wrote:
>  
> > Pradeep,
> >  
> > You can mount a spark directory as a volume. This means you have to have 
> > spark deployed on every agent.
> >  
> > Another thing you can do, place spark in hdfs, assuming that you have hdfs 
> > available but that too will download a copy to the sandbox.
> >  
> > I'd prefer the former.
> >  
> > Sent from Outlook Mobile (https://aka.ms/qtex0l)
> > _
> > From: Pradeep Chhetri  > (mailto:pradeep.chhetr...@gmail.com)>
> > Sent: Tuesday, March 15, 2016 4:41 pm
> > Subject: Apache Spark Over Mesos
> > To: 
> >  
> >  
> > Hello,  
> >  
> > I am able to run Apache Spark over Mesos. Its quite simple to run Spark 
> > Dispatcher over marathon and ask it to run Spark Executor (I guess also can 
> > be called as Spark Driver) as docker container.  
> >  
> > I have a query regarding this:  
> >  
> > All spark tasks are spawned directly by first downloading the spark 
> > artifacts. I was thinking if there is some way I can start them too as 
> > docker containers. This will save the time for downloading the spark 
> > artifacts. I am running spark in fine-grained mode.  
> >  
> > I have attached a screenshot of a sample job  
> >  
> >   
> > ​  
> > Thanks,  
> >  
> > --  
> > Pradeep Chhetri  
> >  

Re: mess spark cluster mode error

[Rad Gruchalski]

I believe you are missing either

—spark.executor.uri pointing to spark tar.gz
or —spark.mesos.executor.home

http://spark.apache.org/docs/latest/running-on-mesos.html#configuration










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Monday, 14 March 2016 at 14:56, sjk wrote:

> hi,all, when i run task on mesos, task error below.  for help, thanks a lot.
>  
>  
> cluster mode, command:
>  
> $SPARK_HOME/spark-submit --class com.xxx.ETL --master 
> mesos://192.168.191.116:7077 --deploy-mode cluster --supervise 
> --driver-memory 2G --executor-memory 10G —
> total-executor-cores 4 http://jar.xxx.info/streaming-etl-assembly-1.0.jar  
>  
>  
> task stderr:
>  
>  
> I0314 21:13:17.520845 29008 fetcher.cpp:424] Fetcher Info: 
> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7\/appweb","items":[{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/data\/program\/spark-1.6.0-bin-hadoop2.6.tgz"}}],"sandbox_directory":"\/data\/mesos\/slaves\/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7\/frameworks\/dd8e95f7-3626-4e46-b48c-b3b58b573c4d-0044\/executors\/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7\/runs\/92509aa4-7804-459b-857d-cfc08c31a993","user":"appweb"}
> I0314 21:13:17.522541 29008 fetcher.cpp:379] Fetching URI 
> '/data/program/spark-1.6.0-bin-hadoop2.6.tgz'
> I0314 21:13:17.522562 29008 fetcher.cpp:250] Fetching directly into the 
> sandbox directory
> I0314 21:13:17.522586 29008 fetcher.cpp:187] Fetching URI 
> '/data/program/spark-1.6.0-bin-hadoop2.6.tgz'
> I0314 21:13:17.522603 29008 fetcher.cpp:167] Copying resource with command:cp 
> '/data/program/spark-1.6.0-bin-hadoop2.6.tgz' 
> '/data/mesos/slaves/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/frameworks/dd8e95f7-3626-4e46-b48c-b3b58b573c4d-0044/executors/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/runs/92509aa4-7804-459b-857d-cfc08c31a993/spark-1.6.0-bin-hadoop2.6.tgz'
> I0314 21:13:17.880008 29008 fetcher.cpp:84] Extracting with command: tar -C 
> '/data/mesos/slaves/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/frameworks/dd8e95f7-3626-4e46-b48c-b3b58b573c4d-0044/executors/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/runs/92509aa4-7804-459b-857d-cfc08c31a993'
>  -xf 
> '/data/mesos/slaves/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/frameworks/dd8e95f7-3626-4e46-b48c-b3b58b573c4d-0044/executors/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/runs/92509aa4-7804-459b-857d-cfc08c31a993/spark-1.6.0-bin-hadoop2.6.tgz'
> I0314 21:13:20.911213 29008 fetcher.cpp:92] Extracted 
> '/data/mesos/slaves/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/frameworks/dd8e95f7-3626-4e46-b48c-b3b58b573c4d-0044/executors/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/runs/92509aa4-7804-459b-857d-cfc08c31a993/spark-1.6.0-bin-hadoop2.6.tgz'
>  into 
> '/data/mesos/slaves/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/frameworks/dd8e95f7-3626-4e46-b48c-b3b58b573c4d-0044/executors/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/runs/92509aa4-7804-459b-857d-cfc08c31a993'
> I0314 21:13:20.911278 29008 fetcher.cpp:456] Fetched 
> '/data/program/spark-1.6.0-bin-hadoop2.6.tgz' to 
> '/data/mesos/slaves/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/frameworks/dd8e95f7-3626-4e46-b48c-b3b58b573c4d-0044/executors/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/runs/92509aa4-7804-459b-857d-cfc08c31a993/spark-1.6.0-bin-hadoop2.6.tgz'
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/spark/launcher/Main
> Caused by: java.lang.ClassNotFoundException: org.apache.spark.launcher.Main
> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
> Could not find the main class: org.apache.spark.launcher.Main. Program will 
> exit.  

Re: Mesos 0.27 and docker

[Rad Gruchalski]

I like my life easy so I use Marathon.










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Friday, 11 March 2016 at 12:24, Walter Heestermans (TME) wrote:

> You are specifying two ways, what’s the preferred way?
>   
> Walter
>   
>   
> From: Rad Gruchalski [mailto:ra...@gruchalski.com]  
> Sent: 11 March 2016 12:20
> To: user@mesos.apache.org (mailto:user@mesos.apache.org)
> Subject: Re: Mesos 0.27 and docker  
>   
> Walter,  
>  
>   
>  
> All you need to know to start is documented here: 
> https://mesosphere.github.io/marathon/docs/native-docker.html.
>  
> That’s with Marathon, if you are planning on using it directly with Mesos, 
> http://mesos.apache.org/documentation/latest/docker-containerizer/
>  
> No problem using latest Docker, I have a 0.27.2 cluster with Docker 1.10.2 
> (docker-engine). All working perfectly fine.
>  
>  
> Kind regards,

> Radek Gruchalski
> 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
> (mailto:ra...@gruchalski.com)
> de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
>  
> Confidentiality:
> This communication is intended for the above-named person and may be 
> confidential and/or legally privileged.
> If it has come to you in error you must take no action based on it, nor must 
> you copy or show it to anyone; please delete/destroy and inform the sender 
> immediately.
>  
>  
> On Friday, 11 March 2016 at 11:20, Walter Heestermans (TME) wrote:
> >  
> > Hi,
> >  
> >  
> >   
> >  
> >  
> > I’m new using mesos, and I like to make study of the docker 
> > containerization inside mesos.
> >  
> >  
> >   
> >  
> >  
> > Can somebody provide me some interesting links and some links to samples on 
> > how to use, configure,…
> >  
> >  
> >   
> >  
> >  
> > Walter
> >  
> >  
> >   
> >  
> >  
> >   
> >  
> >  
> > This e-mail may contain confidential information. If you are not an 
> > addressee or otherwise authorised to receive this message, you should not 
> > use, copy, disclose or take any action based on this e-mail. If you have 
> > received this e-mail in error, please inform the sender promptly and delete 
> > this message and any attachments immediately.
> >  
> >  
>  
>   
>  
>  
>  
> This e-mail may contain confidential information. If you are not an addressee 
> or otherwise authorised to receive this message, you should not use, copy, 
> disclose or take any action based on this e-mail. If you have received this 
> e-mail in error, please inform the sender promptly and delete this message 
> and any attachments immediately.
>  
>  
>  

mesos.interface 0.27.2

[Rad Gruchalski]

Hi everyone,  

I have successfully upgraded my cluster to 0.27.2 but I have a couple of 
questions regarding mesos.interface python package.  
First, I do not see the 0.27.2 version in pypi: 
https://pypi.python.org/pypi/mesos.interface. Is that intentional?

Second, I thought it would be better to use the modules built with Mesos. I am 
building Mesos myself and package it using mesos-deb-packaging. After 
installation with apt, I see that the modules are installed in:

/usr/lib/python2.7/site-packages

I tried the following:

export PYTHONPATH=$PYTHONPATH:/usr/lib/python2.7/site-packages
/usr/local/lib/python2.7/dist-packages/mesos/native$ mesos-ps

But the result is:

Traceback (most recent call last):
  File "/usr/bin/mesos-ps", line 10, in 
from mesos import http
ImportError: cannot import name http


Am I missing some important step here?
Thanks for any pointers.










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.

Re: 0.27.2 RPMs

[Rad Gruchalski]

Oh, cool, I was expecting some info here.
Going to build one now :)










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Wednesday, 9 March 2016 at 16:44, craig w wrote:

> I thought so based on the website: 
> http://mesos.apache.org/blog/mesos-0-27-2-released/

Re: 0.27.2 RPMs

[Rad Gruchalski]

Was it actually released?  
I can see the version tagged in git but there was no official release note?










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Wednesday, 9 March 2016 at 14:38, craig w wrote:

> When will the 0.27.2 RPMs be available?

Re: How to deploy a Database cluster

[Rad Gruchalski]

Alfredo,  

Not sure how to do this with Calico and Co but you need to investigate 
LIBPROCESS_ADVERTISE_IP and LIBPROCESS_ADVERTICE_PORT.
https://github.com/apache/mesos/blob/master/docs/configuration.md#libprocess-options

Basically, what you need to do is, in your container you need to:

export LIBPROCESS_ADVERTISE_IP=
export LIBPROCESS_ADVERTISE_PORT=$PORT0

assuming that port 0 is the one MariaDB would use.










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Wednesday, 24 February 2016 at 23:20, Alfredo Carneiro wrote:

> Hello guys,
>  
> I have been trying to deploy a Galera MariaDB cluster on my Mesos Cluster 
> following this tutorial [1], but I am facing some problems.
>  
> After I set Mesos-DNS up, I noticed that nodes use their internal cointainer 
> IP addresses to communicate with other nodes, so the other nodes will be not 
> able to send the messages back to them.
>  
> So, I have tried to create a Virtual Network using Calico and Docker 
> libnertwork, but I got stucked in other communication problems.
>  
> Then, I wonder if anyone has some use case on deploying database on Mesos 
> cluster and could help me with that.
>  
> Best Regards,
>  
> --  
> Alfredo Miranda  

Learning Agent ID without reading full state?

[Rad Gruchalski]

Hi there,  

I’m trying to figure out what would the best way to learn the agent id without 
having to read the complete state of mesos. Currently, I do the following:

mesos-state 2>/dev/null | jq '.slaves[] | select(.hostname == 
"’$(AGENT_HOSTNAME)'") | .id' -r

However, depending on how much stuff is happening, the amount of data thrown 
away is less massive or even more massive.
I am currently looking at 2 options, I only want to learn the agent id on a 
host running that agent:

cat $MESOS_WORKPLACE/meta/slaves/latest/slave.info

but I’m not sure what format the data in this file is in? It looks encoded but 
not sure what the exact encoding is.
The other option I’m looking at is:

basename $(readlink -f $(find $MESOS_WORKPLACE/meta/slaves -maxdepth 1 -type l))

But I’m not sure if there are any caveats I should be aware of? Any pointers?  










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.

Re: Learning Agent ID without reading full state?

[Rad Gruchalski]

TBH, after having a second look at the marathon event bus, I don’t need it, it 
gives me the host of where the event happened. That should be sufficient.










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Tuesday, 8 December 2015 at 15:42, tommy xiao wrote:

> why are you need the Agent ID? could you please input more information?
>  
> 2015-12-08 19:27 GMT+08:00 Rad Gruchalski <ra...@gruchalski.com 
> (mailto:ra...@gruchalski.com)>:
> > Hi there,  
> >  
> > I’m trying to figure out what would the best way to learn the agent id 
> > without having to read the complete state of mesos. Currently, I do the 
> > following:
> >  
> > mesos-state 2>/dev/null | jq '.slaves[] | select(.hostname == 
> > "’$(AGENT_HOSTNAME)'") | .id' -r
> >  
> > However, depending on how much stuff is happening, the amount of data 
> > thrown away is less massive or even more massive.
> > I am currently looking at 2 options, I only want to learn the agent id on a 
> > host running that agent:
> >  
> > cat $MESOS_WORKPLACE/meta/slaves/latest/slave.info (http://slave.info)
> >  
> > but I’m not sure what format the data in this file is in? It looks encoded 
> > but not sure what the exact encoding is.
> > The other option I’m looking at is:
> >  
> > basename $(readlink -f $(find $MESOS_WORKPLACE/meta/slaves -maxdepth 1 
> > -type l))
> >  
> > But I’m not sure if there are any caveats I should be aware of? Any 
> > pointers?  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> > Kind regards,

> > Radek Gruchalski
> > 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
> > (mailto:ra...@gruchalski.com)
> > de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
> >  
> > Confidentiality:
> > This communication is intended for the above-named person and may be 
> > confidential and/or legally privileged.
> > If it has come to you in error you must take no action based on it, nor 
> > must you copy or show it to anyone; please delete/destroy and inform the 
> > sender immediately.
> >  
> >  
> >  
>  
>  
>  
>  
>  
> --  
> Deshi Xiao
> Twitter: xds2000
> E-mail: xiaods(AT)gmail.com (http://gmail.com)  

Re: Spark task sometimes won't start

[Rad Gruchalski]

Mitsutoshi,

Is this in a fine-grained mode?










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Tuesday, 24 November 2015 at 07:18, 木内満歳 wrote:

> Hi, Tim
>  
> I've reproduced and taken debug logs(attached).
> I cannot understand what is going on, but it seems that the slave is 
> repeatedly sending ACCEPT message to master.
>  
> Please have your comment.
>  
> Best Regards,
> Mitsutoshi Kiuchi
>  
>  
> 2015-11-24 5:28 GMT+09:00 Tim Chen  (mailto:t...@mesosphere.io)>:
> > Hi Mitsutoshi,
> >  
> > Can you enable TRACING log on Spark (modify your log4j.properties file)?
> >  
> > It should have more information on why offers are being rejected, but most 
> > of the time it's due to not enough resources in your cluster to satifsy 
> > launching your Spark job. You can either increase your slave(s) resources 
> > or lower your cpu/memory requirement for your job through configuration.
> >  
> > Tim
> >  
> > On Mon, Nov 23, 2015 at 6:30 AM, 木内満歳  > (mailto:m-kiu...@creationline.com)> wrote:
> > > Hi,
> > >  
> > > I'm experiencing that some spark task on Mesos 0.25 occasionally won't 
> > > start.
> > > Please tell some advice how to see more detail against it.
> > >  
> > > Here is the slave log about bad task
> > >  
> > > Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.677291 18516 
> > > slave.cpp:2379] Got registration for executor 
> > > '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework 
> > > 235498ca-6603-4cfe-bfc7-94005bb235fb-1442 from 
> > > executor(1)@10.130.91.16:60295 (http://10.130.91.16:60295)
> > > Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.679875 18516 
> > > slave.cpp:1760] Sending queued task '0' to executor 
> > > '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework 
> > > 235498ca-6603-4cfe-bfc7-94005bb235fb-1442
> > > (no more log about this task)
> > >  
> > > When task succeed to run, slave log shows like that.
> > >  
> > > Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.637285  
> > > 8658 slave.cpp:2379] Got registration for executor 
> > > '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework 
> > > 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from 
> > > executor(1)@10.130.98.65:52273 (http://10.130.98.65:52273)
> > > Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.639233  
> > > 8658 slave.cpp:1760] Sending queued task '6' to executor 
> > > '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework 
> > > 235498ca-6603-4cfe-bfc7-94005bb235fb-1437
> > > Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.608182  
> > > 8658 slave.cpp:2717] Handling status update TASK_RUNNING (UUID: 
> > > ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework 
> > > 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from 
> > > executor(1)@10.130.98.65:52273 (http://10.130.98.65:52273)
> > > Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.612318  
> > > 8658 status_update_manager.cpp:322] Received status update TASK_RUNNING 
> > > (UUID: ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework 
> > > 235498ca-6603-4cfe-bfc7-94005bb235fb-1437
> > >  
> > > Any advice is welcome.
> > >  
> > > Best Regards,
> > > Mitsutoshi Kiuchi
> > >  
> >  
>  
>  
> Attachments:  
> - log.driverStdErr.gz
>  
> - log.mesosMaster.gz
>  
> - log.mesosSlave.gz
>  

Re: Spark task sometimes won't start

[Rad Gruchalski]

Ah, I see, I am experiencing a similar thing with fine-grained where one of the 
tasks would stay in staging and fail the whole job but never in coarse mode.










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Tuesday, 24 November 2015 at 10:07, 木内満歳 wrote:

> Hi Rad,
>  
> I've tried both. I've experienced same symptom on both case.
>  
> Thanks,
> Mitsutoshi Kiuchi
>  
>  
> 2015-11-24 17:57 GMT+09:00 Rad Gruchalski <ra...@gruchalski.com 
> (mailto:ra...@gruchalski.com)>:
> > Mitsutoshi,
> >  
> > Is this in a fine-grained mode?
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> > Kind regards,

> > Radek Gruchalski
> > 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
> > (mailto:ra...@gruchalski.com)
> > de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
> >  
> > Confidentiality:
> > This communication is intended for the above-named person and may be 
> > confidential and/or legally privileged.
> > If it has come to you in error you must take no action based on it, nor 
> > must you copy or show it to anyone; please delete/destroy and inform the 
> > sender immediately.
> >  
> >  
> >  
> > On Tuesday, 24 November 2015 at 07:18, 木内満歳 wrote:
> >  
> > > Hi, Tim
> > >  
> > > I've reproduced and taken debug logs(attached).
> > > I cannot understand what is going on, but it seems that the slave is 
> > > repeatedly sending ACCEPT message to master.
> > >  
> > > Please have your comment.
> > >  
> > > Best Regards,
> > > Mitsutoshi Kiuchi
> > >  
> > >  
> > > 2015-11-24 5:28 GMT+09:00 Tim Chen <t...@mesosphere.io 
> > > (mailto:t...@mesosphere.io)>:
> > > > Hi Mitsutoshi,
> > > >  
> > > > Can you enable TRACING log on Spark (modify your log4j.properties file)?
> > > >  
> > > > It should have more information on why offers are being rejected, but 
> > > > most of the time it's due to not enough resources in your cluster to 
> > > > satifsy launching your Spark job. You can either increase your slave(s) 
> > > > resources or lower your cpu/memory requirement for your job through 
> > > > configuration.
> > > >  
> > > > Tim
> > > >  
> > > > On Mon, Nov 23, 2015 at 6:30 AM, 木内満歳 <m-kiu...@creationline.com 
> > > > (mailto:m-kiu...@creationline.com)> wrote:
> > > > > Hi,
> > > > >  
> > > > > I'm experiencing that some spark task on Mesos 0.25 occasionally 
> > > > > won't start.
> > > > > Please tell some advice how to see more detail against it.
> > > > >  
> > > > > Here is the slave log about bad task
> > > > >  
> > > > > Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.677291 
> > > > > 18516 slave.cpp:2379] Got registration for executor 
> > > > > '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework 
> > > > > 235498ca-6603-4cfe-bfc7-94005bb235fb-1442 from 
> > > > > executor(1)@10.130.91.16:60295 (http://10.130.91.16:60295)
> > > > > Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.679875 
> > > > > 18516 slave.cpp:1760] Sending queued task '0' to executor 
> > > > > '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework 
> > > > > 235498ca-6603-4cfe-bfc7-94005bb235fb-1442
> > > > > (no more log about this task)
> > > > >  
> > > > > When task succeed to run, slave log shows like that.
> > > > >  
> > > > > Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.637285  
> > > > > 8658 slave.cpp:2379] Got registration for executor 
> > > > > '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework 
> > > > > 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from 
> > > > > executor(1)@10.130.98.65:52273 (http://10.130.98.65:52273)
> > > > > Nov 23 08:44:39 al-mesos-

Re: Spark task sometimes won't start

[Rad Gruchalski]

Indeed, also, keep in mind, there’s a talk here that fine-grained mode might be 
removed in Spark 2.










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Tuesday, 24 November 2015 at 11:32, 木内満歳 wrote:

> I agree. "coarse grain mode" is "take me to safe side" mode, I think.
> Unfortunately, it looks more hard to resolve...
>  
> Anyway, I appreciate your advice. Thanks much !
>  
> Mitsutoshi
>  
>  
> 2015-11-24 19:23 GMT+09:00 Rad Gruchalski <ra...@gruchalski.com 
> (mailto:ra...@gruchalski.com)>:
> > Ah, I see, I am experiencing a similar thing with fine-grained where one of 
> > the tasks would stay in staging and fail the whole job but never in coarse 
> > mode.
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> > Kind regards,

> > Radek Gruchalski
> > 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
> > (mailto:ra...@gruchalski.com)
> > de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
> >  
> > Confidentiality:
> > This communication is intended for the above-named person and may be 
> > confidential and/or legally privileged.
> > If it has come to you in error you must take no action based on it, nor 
> > must you copy or show it to anyone; please delete/destroy and inform the 
> > sender immediately.
> >  
> >  
> >  
> > On Tuesday, 24 November 2015 at 10:07, 木内満歳 wrote:
> >  
> > > Hi Rad,
> > >  
> > > I've tried both. I've experienced same symptom on both case.
> > >  
> > > Thanks,
> > > Mitsutoshi Kiuchi
> > >  
> > >  
> > > 2015-11-24 17:57 GMT+09:00 Rad Gruchalski <ra...@gruchalski.com 
> > > (mailto:ra...@gruchalski.com)>:
> > > > Mitsutoshi,
> > > >  
> > > > Is this in a fine-grained mode?
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > > Kind regards,

> > > > Radek Gruchalski
> > > > 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
> > > > (mailto:ra...@gruchalski.com)
> > > > de.linkedin.com/in/radgruchalski/ 
> > > > (http://de.linkedin.com/in/radgruchalski/)
> > > >  
> > > > Confidentiality:
> > > > This communication is intended for the above-named person and may be 
> > > > confidential and/or legally privileged.
> > > > If it has come to you in error you must take no action based on it, nor 
> > > > must you copy or show it to anyone; please delete/destroy and inform 
> > > > the sender immediately.
> > > >  
> > > >  
> > > >  
> > > > On Tuesday, 24 November 2015 at 07:18, 木内満歳 wrote:
> > > >  
> > > > > Hi, Tim
> > > > >  
> > > > > I've reproduced and taken debug logs(attached).
> > > > > I cannot understand what is going on, but it seems that the slave is 
> > > > > repeatedly sending ACCEPT message to master.
> > > > >  
> > > > > Please have your comment.
> > > > >  
> > > > > Best Regards,
> > > > > Mitsutoshi Kiuchi
> > > > >  
> > > > >  
> > > > > 2015-11-24 5:28 GMT+09:00 Tim Chen <t...@mesosphere.io 
> > > > > (mailto:t...@mesosphere.io)>:
> > > > > > Hi Mitsutoshi,
> > > > > >  
> > > > > > Can you enable TRACING log on Spark (modify your log4j.properties 
> > > > > > file)?
> > > > > >  
> > > > > > It should have more information on why offers are being rejected, 
> > > > > > but most of the time it's due to not enough resources in your 
> > > > > > cluster to satifsy launching your Spark job. You can either 
> > > > > > increase your slave(s) resources or lower your cpu/memory 
> > > > > > requirement for your job through confi

Re: resolving hosts with mesos-dns not working with "/" in the appid

[Rad Gruchalski]

According to the docs, it is possible:  

Mesos-DNS follows RFC 952 for name formatting. All fields used to construct 
hostnames for A records and service names for SRV records must be up to 24 
characters and drawn from the alphabet (A-Z), digits (0-9) and minus sign (-). 
No distinction is made between upper and lower case. If the task name does not 
comply with these constraints, Mesos-DNS will trim it, remove all invalid 
characters, and replace period (.) with sign (-) for task names.

However, it is a bit dodgy, specially if your app id is longer than 24 
characters. It sometimes work, sometimes does not.  

I think your record will go along the lines of:

mesosspark-shuffle-service.marathon.mesos and so on.

Be careful about the 24 character limit. Depending on where the invalid 
characters happen in the string, it may render the whole record unavailable 
(mesos-dns will not be able to return it).  










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Tuesday, 17 November 2015 at 20:40, Rodrick Brown wrote:

> Is it possible to resolve app-ids with / in them when using mesos-dns?  
>  
>  
> I have apps defined like the following:  
>  
> /kafkadirectconsumer/es-services  
> /mesos/spark-shuffle-service
>  
>  
> however trying to resolve any appID with a “/“ in the name returns NXDOMAIN  
> In the above case I thought the following should work  
>  
> $ dig mess_spark-shuffle-service.marathon.mesos  
>  
> I don’t get the IP of those service.  
>  
>  
> --  
>  
>  
>  
>  
> Rodrick Brown / DevOPs Engineer  
> +1 917 445 6839 / rodr...@orchardplatform.com 
> (mailto:char...@orchardplatform.com)
>  
>  
> Orchard Platform  
> 101 5th Avenue, 4th Floor, New York, NY 10003  
> http://www.orchardplatform.com (http://www.orchardplatform.com/)
>  
>  
> Orchard Blog (http://www.orchardplatform.com/blog/) | Marketplace Lending 
> Meetup (http://www.meetup.com/Peer-to-Peer-Lending-P2P/)
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
> NOTICE TO RECIPIENTS: This communication is confidential and intended for the 
> use of the addressee only. If you are not an intended recipient of this 
> communication, please delete it immediately and notify the sender by return 
> email. Unauthorized reading, dissemination, distribution or copying of this 
> communication is prohibited. This communication does not constitute an offer 
> to sell or a solicitation of an indication of interest to purchase any loan, 
> security or any other financial product or instrument, nor is it an offer to 
> sell or a solicitation of an indication of interest to purchase any products 
> or services to any persons who are prohibited from receiving such information 
> under applicable law. The contents of this communication may not be accurate 
> or complete and are subject to change without notice. As such, Orchard App, 
> Inc. (including its subsidiaries and affiliates, "Orchard") makes no 
> representation regarding the accuracy or completeness of the information 
> contained herein. The intended recipient is advised to consult its own 
> professional advisors, including those specializing in legal, tax and 
> accounting matters. Orchard does not provide legal, tax or accounting advice. 
>  

Re: spark on mesos with docker issue

[Rad Gruchalski]

Stavros,  

As mentioned a couple of weeks ago: 
https://issues.apache.org/jira/browse/SPARK-11638
Happy to answer any questions.










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Thursday, 22 October 2015 at 22:45, Stavros Kontopoulos wrote:

> Thnx Rad, sounds pretty cool :). Elizabeth one note for the jira ticket i do 
> not run the cluster with zookeeper, i use mesos master in stand alone mode... 
> i guess it makes no difference right?
>  
> On Thu, Oct 22, 2015 at 10:38 PM, Rad Gruchalski <ra...@gruchalski.com 
> (mailto:ra...@gruchalski.com)> wrote:
> > There are 2 things:  
> >  
> >  - Akka remote in 2.3.x does not support advertising hostname / port 
> > different to what it binds to
> >  - All other services: file server, broadcast server, repl class server do 
> > not support advertising hostnames / ports different than what they bind to
> >  
> > Just to expand on the previous one, we are in the process of contributing 
> > the following bits:  
> >  
> >  - akka-remote bind-hostname and bind-port backport to akka 2.3.x (not the 
> > typesafe closed support implementation, our own implementation)
> >  - spark patches for spark 1.4.0+ which enable running Spark on Mesos in 
> > Docker Bridge networking
> >  
> > Unfortunately, due to the nature of how my employer operates I can’t share 
> > the code yet. We are working with our legal team to make these available 
> > asap.
> >  
> > We do run this stuff in production.  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> > Kind regards,

> > Radek Gruchalski
> > 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
> > (mailto:ra...@gruchalski.com)
> > de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
> >  
> > Confidentiality:
> > This communication is intended for the above-named person and may be 
> > confidential and/or legally privileged.
> > If it has come to you in error you must take no action based on it, nor 
> > must you copy or show it to anyone; please delete/destroy and inform the 
> > sender immediately.
> >  
> >  
> >  
> > On Thursday, 22 October 2015 at 22:28, Iulian Dragoș wrote:
> >  
> > >  
> > >  
> > > On Thu, Oct 22, 2015 at 9:10 PM, Rad Gruchalski <ra...@gruchalski.com 
> > > (mailto:ra...@gruchalski.com)> wrote:
> > > > Stavros,  
> > > >  
> > > > Spark does not support this. I am currently in the process of 
> > > > submitting patches for it however it first has to pass through the 
> > > > legal team at the company I work for.
> > >  
> > > What exactly is missing in Spark?
> > >   
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > > Kind regards,

> > > > Radek Gruchalski
> > > > 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
> > > > (mailto:ra...@gruchalski.com)
> > > > de.linkedin.com/in/radgruchalski/ 
> > > > (http://de.linkedin.com/in/radgruchalski/)
> > > >  
> > > > Confidentiality:
> > > > This communication is intended for the above-named person and may be 
> > > > confidential and/or legally privileged.
> > > > If it has come to you in error you must take no action based on it, nor 
> > > > must you copy or show it to anyone; please delete/destroy and inform 
> > > > the sender immediately.
> > > >  
> > > >  
> > > >  
> > > > On Thursday, 22 October 2015 at 21:08, Stavros Kontopoulos wrote:
> > > >  
> > > > > Bridge... with the latest mesos library vesion 0.25...
> > > > >  
> > > > > On Thu, Oct 22, 2015 at 9:07 PM, Elizabeth Lingg 
> > > > > <elizab...@mesosphere.io (mailto:elizab...@mesosphere.io)> wrote:
> > > > > > Are you using Bridge or Host Networking?
> > > > > >  
> >

Re: How is Mesos doing certificate verification for resources in URIs?

[Rad Gruchalski]

Kamil,  

It’s perfect, thank you.










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Wednesday, 4 November 2015 at 12:31, Rad Gruchalski wrote:

> Kamil,
>  
> Will give it a shot. Thanks for the pointer.
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
> Kind regards,

> Radek Gruchalski
> 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
> (mailto:ra...@gruchalski.com)
> de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
>  
> Confidentiality:
> This communication is intended for the above-named person and may be 
> confidential and/or legally privileged.
> If it has come to you in error you must take no action based on it, nor must 
> you copy or show it to anyone; please delete/destroy and inform the sender 
> immediately.
>  
>  
>  
> On Wednesday, 4 November 2015 at 12:28, Kamil Chmielewski wrote:
>  
> > We had similiar issues with custom built Mesos linked with libcurl4-nss 
> > https://github.com/apache/mesos/pull/48.
> > Everythng works like expected when we use libcurl4-openssl.
> >  
> > Cheers,
> > Kamil
> >  
> > 2015-11-04 12:19 GMT+01:00 Rad Gruchalski <ra...@gruchalski.com 
> > (mailto:ra...@gruchalski.com)>:
> > > Yes, this is from the agent:  
> > >  
> > > ~$ curl -i https://raw.githubusercontent.com/apache/spark/master/pom.xml
> > > HTTP/1.1 200 OK
> > > Content-Security-Policy: default-src 'none'
> > > X-XSS-Protection: 1; mode=block
> > > X-Frame-Options: deny
> > > X-Content-Type-Options: nosniff
> > > Strict-Transport-Security: max-age=31536000
> > > ETag: "762bfc728233533ab49336ff68dc02203407ea43"
> > > Content-Type: text/plain; charset=utf-8
> > > Cache-Control: max-age=300
> > > X-GitHub-Request-Id: B91F1318:509A:EEE5F90:5639E92E
> > > Content-Length: 87329
> > > Accept-Ranges: bytes
> > > Date: Wed, 04 Nov 2015 11:17:02 GMT
> > > Via: 1.1 varnish
> > > Connection: keep-alive
> > > X-Served-By: cache-lhr6327-LHR
> > > X-Cache: MISS
> > > X-Cache-Hits: 0
> > > Vary: Authorization,Accept-Encoding
> > > Access-Control-Allow-Origin: *
> > > X-Fastly-Request-ID: f3120a4d90968291aa84609c786626599809456d
> > > Expires: Wed, 04 Nov 2015 11:22:02 GMT
> > > Source-Age: 0
> > >  
> > > 
> > >  > > > Best Regards,
> > > > Haosdent Huang  
> > >  
> >  
>  

How is Mesos doing certificate verification for resources in URIs?

[Rad Gruchalski]

Hi everyone,  

I’ve added the following URI to the URIs for the task: 
https://raw.githubusercontent.com/apache/spark/master/pom.xml. However, my task 
has failed because of:

Failed to fetch 
'https://raw.githubusercontent.com/apache/spark/master/pom.xml': Error 
downloading resource: Peer certificate cannot be authenticated with given CA 
certificates

This surely is a problem in mesos. Everybody else in the world claims that the 
certificate is valid. Or is there a setting for making this work?










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.

Re: How is Mesos doing certificate verification for resources in URIs?

[Rad Gruchalski]

Yes, this is from the agent:  

~$ curl -i https://raw.githubusercontent.com/apache/spark/master/pom.xml
HTTP/1.1 200 OK
Content-Security-Policy: default-src 'none'
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000
ETag: "762bfc728233533ab49336ff68dc02203407ea43"
Content-Type: text/plain; charset=utf-8
Cache-Control: max-age=300
X-GitHub-Request-Id: B91F1318:509A:EEE5F90:5639E92E
Content-Length: 87329
Accept-Ranges: bytes
Date: Wed, 04 Nov 2015 11:17:02 GMT
Via: 1.1 varnish
Connection: keep-alive
X-Served-By: cache-lhr6327-LHR
X-Cache: MISS
X-Cache-Hits: 0
Vary: Authorization,Accept-Encoding
Access-Control-Allow-Origin: *
X-Fastly-Request-ID: f3120a4d90968291aa84609c786626599809456d
Expires: Wed, 04 Nov 2015 11:22:02 GMT
Source-Age: 0


 Best Regards,
> Haosdent Huang  

Re: Tasks that run docker images consistently fail while downloading

[Rad Gruchalski]

Indeed, I have images which take more than a minute to download and I do not 
see such behaviour. No immediate idea why this would happen.
Can you share your application request?










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Wednesday, 28 October 2015 at 11:38, James Vanns wrote:

> Yes I have - I mention that in my Email ;) I set it to the same as the 
> 'executor_registration_timeout'. Both effectively set to 5 minutes - but my 
> tasks are killed off after 1 minute without being allowed to fully download 
> the image.
>  
> Jim
>  
> On 28 October 2015 at 10:26, Rad Gruchalski <ra...@gruchalski.com 
> (mailto:ra...@gruchalski.com)> wrote:
> > Jim,  
> >  
> > Have you tried —task_launch_timeout? From: 
> > https://mesosphere.github.io/marathon/docs/native-docker.html
> >  
> > Configure Marathon
> > Increase the Marathon command line option 
> > (https://mesosphere.github.io/marathon/docs/command-line-flags.html) 
> > --task_launch_timeout to at least the executor timeout, in milliseconds, 
> > you set on your slaves in the previous step.
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> > Kind regards,

> > Radek Gruchalski
> > 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
> > (mailto:ra...@gruchalski.com)
> > de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
> >  
> > Confidentiality:
> > This communication is intended for the above-named person and may be 
> > confidential and/or legally privileged.
> > If it has come to you in error you must take no action based on it, nor 
> > must you copy or show it to anyone; please delete/destroy and inform the 
> > sender immediately.
> >  
> >  
> >  
> > On Wednesday, 28 October 2015 at 11:21, James Vanns wrote:
> >  
> > > Hi all.
> > >  
> > > Mesos version = 0.23.0-1.0.ubuntu1404 (mesosphere APT repo)
> > > Marathon version = 0.10.1 (mesosphere APT repo)
> > >  
> > > Hopefully this is a simple one for someone to answer, though I couldn't 
> > > find anything immediately  
> > > obvious in the documentation. We're trialling Mesos in a cloud (EC2/GCE) 
> > > environment and the one  
> > > thing that continues to bite us in the ass is this; continued task 
> > > failures until the docker image is  
> > > fully downloaded! Why is this!? Some of our images a small (say 200MB), 
> > > some much larger (2GB)  
> > > due to the nature of the software packages we're containerising. 
> > > Regardless of this size, they fail the  
> > > first dozen (or more) times until one of the slaves has pulled the image. 
> > > Why is there an apparent  
> > > hard time-out and how can I avoid it? I don't want the task to register 
> > > as a fail - it hasn't even had a  
> > > chance to run yet! Up until now we've just been tolerating the bouncing 
> > > around of these tasks but it's  
> > > now reached a point where it's darn annoying ;)
> > >  
> > > I've tried setting executor_registration_timeout to '5mins' but this made 
> > > no apparent difference (every  
> > > minute the task is killed still). I should note that these tasks are 
> > > launched using the Marathon  
> > > framework and I've tried setting 'task_launch_timeout' to '3000' and 
> > > again, it makes no difference.
> > >  
> > > Based on a brief glance of a mesos slave log file it seems the master 
> > > instructs the slave to kill the task off after 1 minute.
> > >  
> > > Please advise.
> > >  
> > > Cheers,
> > >  
> > > Jim
> > > --
> > > Senior Code Pig
> > > Industrial Light & Magic
> > >  
> > >  
> > >  
> > >  
> > >  
> > >  
> >  
> >  
>  
>  
>  
> --  
> --
> Senior Code Pig
> Industrial Light & Magic
>  
>  
>  
>  
>  
>  

Re: Tasks that run docker images consistently fail while downloading

[Rad Gruchalski]

However, it should be 30, not 3. It’s milliseconds - 5 mins in 
milliseconds is 30.  

Optional. Default: 30 (5 minutes)  











Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Wednesday, 28 October 2015 at 13:08, Rad Gruchalski wrote:

> Nobody getting those today ;) Good catch. Worth keeping in mind!
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
> Kind regards,

> Radek Gruchalski
> 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
> (mailto:ra...@gruchalski.com)
> de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
>  
> Confidentiality:
> This communication is intended for the above-named person and may be 
> confidential and/or legally privileged.
> If it has come to you in error you must take no action based on it, nor must 
> you copy or show it to anyone; please delete/destroy and inform the sender 
> immediately.
>  
>  
>  
> On Wednesday, 28 October 2015 at 13:06, James Vanns wrote:
>  
> > I shall fix my own problem it's embarrassing. Top marks to those of you 
> > that notice I supplied 3000 instead of 3 (which I understand is 
> > actually the default anyway) to task_launch_timeout!
> >  
> > Jim
> >  
> >  
> > On 28 October 2015 at 10:21, James Vanns <jvanns@gmail.com 
> > (mailto:jvanns@gmail.com)> wrote:
> > > Hi all.
> > >  
> > > Mesos version = 0.23.0-1.0.ubuntu1404 (mesosphere APT repo)
> > > Marathon version = 0.10.1 (mesosphere APT repo)
> > >  
> > > Hopefully this is a simple one for someone to answer, though I couldn't 
> > > find anything immediately  
> > > obvious in the documentation. We're trialling Mesos in a cloud (EC2/GCE) 
> > > environment and the one  
> > > thing that continues to bite us in the ass is this; continued task 
> > > failures until the docker image is  
> > > fully downloaded! Why is this!? Some of our images a small (say 200MB), 
> > > some much larger (2GB)  
> > > due to the nature of the software packages we're containerising. 
> > > Regardless of this size, they fail the  
> > > first dozen (or more) times until one of the slaves has pulled the image. 
> > > Why is there an apparent  
> > > hard time-out and how can I avoid it? I don't want the task to register 
> > > as a fail - it hasn't even had a  
> > > chance to run yet! Up until now we've just been tolerating the bouncing 
> > > around of these tasks but it's  
> > > now reached a point where it's darn annoying ;)
> > >  
> > > I've tried setting executor_registration_timeout to '5mins' but this made 
> > > no apparent difference (every  
> > > minute the task is killed still). I should note that these tasks are 
> > > launched using the Marathon  
> > > framework and I've tried setting 'task_launch_timeout' to '3000' and 
> > > again, it makes no difference.
> > >  
> > > Based on a brief glance of a mesos slave log file it seems the master 
> > > instructs the slave to kill the task off after 1 minute.
> > >  
> > > Please advise.
> > >  
> > > Cheers,
> > >  
> > > Jim
> > > --
> > > Senior Code Pig
> > > Industrial Light & Magic
> > >  
> > >  
> > >  
> > >  
> >  
> >  
> >  
> >  
> >  
> > --  
> > --
> > Senior Code Pig
> > Industrial Light & Magic
> >  
> >  
> >  
> >  
> >  
> >  
>  
>  

Re: spark on mesos with docker issue

[Rad Gruchalski]

There are 2 things:  

 - Akka remote in 2.3.x does not support advertising hostname / port different 
to what it binds to
 - All other services: file server, broadcast server, repl class server do not 
support advertising hostnames / ports different than what they bind to

Just to expand on the previous one, we are in the process of contributing the 
following bits:  

 - akka-remote bind-hostname and bind-port backport to akka 2.3.x (not the 
typesafe closed support implementation, our own implementation)
 - spark patches for spark 1.4.0+ which enable running Spark on Mesos in Docker 
Bridge networking

Unfortunately, due to the nature of how my employer operates I can’t share the 
code yet. We are working with our legal team to make these available asap.

We do run this stuff in production.  










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Thursday, 22 October 2015 at 22:28, Iulian Dragoș wrote:

>  
>  
> On Thu, Oct 22, 2015 at 9:10 PM, Rad Gruchalski <ra...@gruchalski.com 
> (mailto:ra...@gruchalski.com)> wrote:
> > Stavros,  
> >  
> > Spark does not support this. I am currently in the process of submitting 
> > patches for it however it first has to pass through the legal team at the 
> > company I work for.
>  
> What exactly is missing in Spark?
>   
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> > Kind regards,

> > Radek Gruchalski
> > 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
> > (mailto:ra...@gruchalski.com)
> > de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
> >  
> > Confidentiality:
> > This communication is intended for the above-named person and may be 
> > confidential and/or legally privileged.
> > If it has come to you in error you must take no action based on it, nor 
> > must you copy or show it to anyone; please delete/destroy and inform the 
> > sender immediately.
> >  
> >  
> >  
> > On Thursday, 22 October 2015 at 21:08, Stavros Kontopoulos wrote:
> >  
> > > Bridge... with the latest mesos library vesion 0.25...
> > >  
> > > On Thu, Oct 22, 2015 at 9:07 PM, Elizabeth Lingg <elizab...@mesosphere.io 
> > > (mailto:elizab...@mesosphere.io)> wrote:
> > > > Are you using Bridge or Host Networking?
> > > >  
> > > > -Elizabeth
> > > >  
> > > >  
> > > >  
> > > > On Thu, Oct 22, 2015 at 12:02 PM, Stavros Kontopoulos 
> > > > <st.kontopou...@gmail.com (mailto:st.kontopou...@gmail.com)> wrote:
> > > > > Hi,
> > > > >  
> > > > > Im using spark on mesos on docker. I have linked my slaves to the 
> > > > > master and a
> > > > > spark repl works fine inside the master container.
> > > > >  
> > > > > If i try to crate the same spark repl form the host i get stuck at 
> > > > > the point when the framework tries to register to the mesos master 
> > > > > (here the framework is the spark repl itself).
> > > > > I can ping the container from my host and vice versa. So networking 
> > > > > its not the problem.
> > > > > What i noticed form the logs is that mesos does not resolve the 
> > > > > correct ip:
> > > > >  
> > > > > Framework failover timeout, removing framework 
> > > > > b3605c33-f573-4d40-806f-b9b0abee2e32-0012 (Spark shell) at 
> > > > > scheduler-b6dbe4f8-4760-4cbe-bb5e-a991e35aba1b@127.0.1.1:40186 
> > > > > (http://scheduler-b6dbe4f8-4760-4cbe-bb5e-a991e35aba1b@127.0.1.1:40186)
> > > > >  
> > > > > docker0 interface is on 172.17.x.x and my host is one such ip so i 
> > > > > didnt expect there to see  
> > > > > 127.0.1.1. I have tried several things like spark.driver.host, 
> > > > > SPARK_LOCAL_IP to be set correctly but with no result...
> > > > > I suspect this is a mesos problem on docker...
> > > > >  
> > > > > Thnx,
> > > > >  
> > > > > S.
> > > >  
> > >  
> >  
>  
>  
>  
> --  
>  
> --
> Iulian Dragos
>  
> --
> Reactive Apps on the JVM
> www.typesafe.com (http://www.typesafe.com)
>  

Re:

[Rad Gruchalski]

Rafael,  

According to the cassandra documentation, you should not be affected at all:
http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_tune_jvm_c.html

However, your performance with these settings will be rather poor.










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Monday, 12 October 2015 at 02:42, Rafael Capucho wrote:

> Hello!,
>  
> I'm using the follow marathon script [1] to launch cassandra non-seeds nodes, 
> it is working properly.
>  
> [1] - http://hastebin.com/visujikela.lua
>  
> As you can see in the script, I'm limiting CPU and Memory.
>  
> But some nodes of my cluster is not that big, even because the cluster isn't 
> big yet. As we know, cassandra generally uses a lot of memory as cache etc..
>  
> 1) I would like to know if Mesos will kill (and keep killing) Cassandra 
> process if it reach the memory limit? if Yes, how can I block Mesos from kill 
> it?
>  
> 2) If I have one server with 4gb memory where I deployed Mesos Slave, and I 
> create a container (by using marathon) with mem=1024 (for example) the 
> processing within that container, when they ask about the memory available, 
> they will receive 4gb or 1gb?
>  
> Thank you!
>  
>  
> --
> Rafael Capucho
>  
> Bachelor of Computer Science
> Federal University of São Paulo
> Institute of Science and Technology - ICT
>  
> PGP-Public Key: 2048R/7389A96F pgp.mit.edu (http://pgp.mit.edu/)
> FP: EDB5 CDEE 8442 99CC C92D 9173 6B32 A5C9 7389 A96F  

Re: Old docker version deployed

[Rad Gruchalski]

Paul,

Are you using the same tag every time?










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Tuesday, 6 October 2015 at 11:37, haosdent wrote:

> You could see the stdout/stderr of your container from mesos webui.
>  
> On Tue, Oct 6, 2015 at 5:30 PM, Paul Wolfe  (mailto:paul.wo...@imc.nl)> wrote:
> > Hello all,  
> >   
> > I'm new to this list, so please let me know if there is a better/more 
> > appropriate forum for this question.
> >   
> > We are currently experimenting with marathon and mesos for deploying a 
> > simple webapp.  We ship the app as a docker container.
> >   
> > Sporadically (ie 1 out of 100) we find an old version of the app is 
> > deployed.  It is obvious from the logs and the appearance of the GUI that 
> > the version is old.  If I download and run the docker container locally, I 
> > see it is indeed the latest version of the code.  That leads me to believe 
> > that somewhere in the marathon deploy or the mesos running of the image, 
> > versions are getting confused.
> >   
> > I guess my first question is what additional information can I get from 
> > marathon or mesos logs to help diagnose? I've checked the mesos-SLAVE.* but 
> > haven't been able to garner anything interesting there.
> >   
> > Thanks for any help!
> > Paul Wolfe
> >   
> >  
> >  
> >  
> > The information in this e-mail is intended only for the person or entity to 
> > which it is addressed.
> >  
> > It may contain confidential and /or privileged material. If someone other 
> > than the intended recipient should receive this e-mail, he / she shall not 
> > be entitled to read, disseminate, disclose or duplicate it.
> >  
> > If you receive this e-mail unintentionally, please inform us immediately by 
> > "reply" and then delete it from your system. Although this information has 
> > been compiled with great care, neither IMC Financial Markets & Asset 
> > Management nor any of its related entities shall accept any responsibility 
> > for any errors, omissions or other inaccuracies in this information or for 
> > the consequences thereof, nor shall it be bound in any way by the contents 
> > of this e-mail or its attachments. In the event of incomplete or incorrect 
> > transmission, please return the e-mail to the sender and permanently delete 
> > this message and any attachments.
> >  
> > Messages and attachments are scanned for all known viruses. Always scan 
> > attachments before opening them.
>  
>  
>  
> --  
> Best Regards,
> Haosdent Huang  

Increasing a number of ports for a Docker container

[Rad Gruchalski]

Hi everyone,  

I remember reading somewhere in the documentation that there’s a limit of 
number of port mappings for a single task / container. I believe it is 10, 
however, I can’t find the specific documentation link describing this.
Is it possible to increase this number? Say, I need 15 not 10.










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.

Re: mesos-tail in 0.24.1

[Rad Gruchalski]

Haosdent,  

This only happens with the zookeeper resolver.










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Wednesday, 30 September 2015 at 10:13, haosdent wrote:

> For mesos-tail, I build from mesos-0.24.1, seems work for me.
>  
> ```
> $ mesos tail --master=127.0.0.1:5050 (http://127.0.0.1:5050) 
> --framework=721b7682-afa1-49a7-8d3a-20292ba666a4- --task=test 
> --file=stdout
> Registered executor on 127.0.0.1
> Starting task test
> sh -c 'ls /'
> Forked command at 18827
> afs
> bin
> boot
> dev
> etc
> home
> initrd.img
> lib
> lib64
> lost+found
> media
> ```
>  
> mesos-resolve also works for me.
>  
>  
>  
> On Wed, Sep 30, 2015 at 12:26 AM, Marco Massenzio <ma...@mesosphere.io 
> (mailto:ma...@mesosphere.io)> wrote:
> > Provided that I'm not familiar at all with mesos-tail and/or mesos-resolve, 
> > you are correct in that this is due to the recent changes (in 0.24) to the 
> > way we write MasterInfo data to ZooKeeper.
> >  
> > This is a genuine bug, thanks for reporting: would you mind terribly to 
> > file a Jira and assign to me, please?
> > (marco-mesos)
> >  
> > Thanks!
> >  
> > Marco Massenzio
> > Distributed Systems Engineer
> > http://codetrips.com
> >  
> > On Tue, Sep 29, 2015 at 6:28 AM, Rad Gruchalski <ra...@gruchalski.com 
> > (mailto:ra...@gruchalski.com)> wrote:
> > > Thank you, that’s some progress:  
> > >  
> > > I changed the code at this line:
> > >  
> > > https://github.com/mesosphere/mesos-cli/blob/master/mesos/cli/master.py#L107
> > >  
> > > to:  
> > >  
> > > try:
> > > parsed =  json.loads(val)
> > > return parsed["address"]["ip"] + ":" + 
> > > str(parsed["address"]["port"])
> > > except Exception:
> > > return val.split("@")[-1]
> > >  
> > >  
> > > And now it gives me the correct master. However, executing mesos-tail or 
> > > mesos-ps does not do anything, just hangs there without any output. 
> > > Something obviously does not work as advertised.
> > > Or I should possibly switch to https://github.com/mesosphere/dcos-cli 
> > > (https://pypi.python.org/pypi/dcoscli), but will this work with just a 
> > > regular mesos 0.24.1 installation?
> > >  
> > >  
> > >  
> > >  
> > >  
> > >  
> > >  
> > >  
> > >  
> > >  
> > > Kind regards,

> > > Radek Gruchalski
> > > 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
> > > (mailto:ra...@gruchalski.com)
> > > de.linkedin.com/in/radgruchalski/ 
> > > (http://de.linkedin.com/in/radgruchalski/)
> > >  
> > > Confidentiality:
> > > This communication is intended for the above-named person and may be 
> > > confidential and/or legally privileged.
> > > If it has come to you in error you must take no action based on it, nor 
> > > must you copy or show it to anyone; please delete/destroy and inform the 
> > > sender immediately.
> > >  
> > >  
> > >  
> > > On Tuesday, 29 September 2015 at 15:20, haosdent wrote:
> > >  
> > > > I think the problem here is you use zk as schema in your config 
> > > > file(.mesos.json) or MESOS_CLI_CONFIG 
> > > > (https://github.com/mesosphere/mesos-cli/blob/master/mesos/cli/cfg.py#L42
> > > >  and 
> > > > https://github.com/mesosphere/mesos-cli/blob/master/mesos/cli/master.py#L119).
> > > >  Not because 0.24.1, you use 0.24.0 should have same issue.
> > > >  
> > > > On Tue, Sep 29, 2015 at 9:14 PM, haosdent <haosd...@gmail.com 
> > > > (mailto:haosd...@gmail.com)> wrote:
> > > > > I think you install mesos-cli from 
> > > > > https://github.com/mesosphere/mesos-cli
> > > > >  
> > > > > On Tue, Sep 29, 2015 at 8:51 PM, Rad Gruchalski <ra...@gruchalski.com 
> > > &

Re: mesos-tail in 0.24.1

[Rad Gruchalski]

Marco,  

Done: https://issues.apache.org/jira/browse/MESOS-3556










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Tuesday, 29 September 2015 at 18:26, Marco Massenzio wrote:

> Provided that I'm not familiar at all with mesos-tail and/or mesos-resolve, 
> you are correct in that this is due to the recent changes (in 0.24) to the 
> way we write MasterInfo data to ZooKeeper.
>  
> This is a genuine bug, thanks for reporting: would you mind terribly to file 
> a Jira and assign to me, please?
> (marco-mesos)
>  
> Thanks!
>  
> Marco Massenzio
> Distributed Systems Engineer
> http://codetrips.com  
> On Tue, Sep 29, 2015 at 6:28 AM, Rad Gruchalski <ra...@gruchalski.com 
> (mailto:ra...@gruchalski.com)> wrote:
> > Thank you, that’s some progress:  
> >  
> > I changed the code at this line:
> >  
> > https://github.com/mesosphere/mesos-cli/blob/master/mesos/cli/master.py#L107
> >  
> > to:  
> >  
> > try:
> > parsed =  json.loads(val)
> > return parsed["address"]["ip"] + ":" + 
> > str(parsed["address"]["port"])
> > except Exception:
> > return val.split("@")[-1]
> >  
> >  
> > And now it gives me the correct master. However, executing mesos-tail or 
> > mesos-ps does not do anything, just hangs there without any output. 
> > Something obviously does not work as advertised.
> > Or I should possibly switch to https://github.com/mesosphere/dcos-cli 
> > (https://pypi.python.org/pypi/dcoscli), but will this work with just a 
> > regular mesos 0.24.1 installation?
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> > Kind regards,

> > Radek Gruchalski
> > 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
> > (mailto:ra...@gruchalski.com)
> > de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
> >  
> > Confidentiality:
> > This communication is intended for the above-named person and may be 
> > confidential and/or legally privileged.
> > If it has come to you in error you must take no action based on it, nor 
> > must you copy or show it to anyone; please delete/destroy and inform the 
> > sender immediately.
> >  
> >  
> >  
> > On Tuesday, 29 September 2015 at 15:20, haosdent wrote:
> >  
> > > I think the problem here is you use zk as schema in your config 
> > > file(.mesos.json) or MESOS_CLI_CONFIG 
> > > (https://github.com/mesosphere/mesos-cli/blob/master/mesos/cli/cfg.py#L42 
> > > and 
> > > https://github.com/mesosphere/mesos-cli/blob/master/mesos/cli/master.py#L119).
> > >  Not because 0.24.1, you use 0.24.0 should have same issue.
> > >  
> > > On Tue, Sep 29, 2015 at 9:14 PM, haosdent <haosd...@gmail.com 
> > > (mailto:haosd...@gmail.com)> wrote:
> > > > I think you install mesos-cli from 
> > > > https://github.com/mesosphere/mesos-cli
> > > >  
> > > > On Tue, Sep 29, 2015 at 8:51 PM, Rad Gruchalski <ra...@gruchalski.com 
> > > > (mailto:ra...@gruchalski.com)> wrote:
> > > > > It seems that I found the reason for this behaviour.  
> > > > > When I execute mesos-resolve, I get an output like this:
> > > > > > 10.100.1.100:5050 
> > > > > > (http://10.100.1.100:5050)","port":5050,"version":"0.24.1"}
> > > > >  
> > > > >  
> > > > > I managed to get to the python sources on the machine, especially 
> > > > > master.py. I verified that in my case the zookeeper_resolver is used.
> > > > > However, what gets returned from zookeeper resolver is:
> > > > >  
> > > > > return val.split("@")[-1]
> > > > >  
> > > > > Where the val is a JSON string:
> > > > >  
> > > > >
> > > > > {"address":{"hostname”:”mesos-master","ip":"10.100.1.100","port":5

Re: mesos-tail in 0.24.1

[Rad Gruchalski]

Thank you, that’s some progress:  

I changed the code at this line:

https://github.com/mesosphere/mesos-cli/blob/master/mesos/cli/master.py#L107

to:  

try:
parsed =  json.loads(val)
return parsed["address"]["ip"] + ":" + 
str(parsed["address"]["port"])
except Exception:
return val.split("@")[-1]


And now it gives me the correct master. However, executing mesos-tail or 
mesos-ps does not do anything, just hangs there without any output. Something 
obviously does not work as advertised.
Or I should possibly switch to https://github.com/mesosphere/dcos-cli 
(https://pypi.python.org/pypi/dcoscli), but will this work with just a regular 
mesos 0.24.1 installation?










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Tuesday, 29 September 2015 at 15:20, haosdent wrote:

> I think the problem here is you use zk as schema in your config 
> file(.mesos.json) or MESOS_CLI_CONFIG 
> (https://github.com/mesosphere/mesos-cli/blob/master/mesos/cli/cfg.py#L42 and 
> https://github.com/mesosphere/mesos-cli/blob/master/mesos/cli/master.py#L119).
>  Not because 0.24.1, you use 0.24.0 should have same issue.
>  
> On Tue, Sep 29, 2015 at 9:14 PM, haosdent <haosd...@gmail.com 
> (mailto:haosd...@gmail.com)> wrote:
> > I think you install mesos-cli from https://github.com/mesosphere/mesos-cli
> >  
> > On Tue, Sep 29, 2015 at 8:51 PM, Rad Gruchalski <ra...@gruchalski.com 
> > (mailto:ra...@gruchalski.com)> wrote:
> > > It seems that I found the reason for this behaviour.  
> > > When I execute mesos-resolve, I get an output like this:
> > > > 10.100.1.100:5050 
> > > > (http://10.100.1.100:5050)","port":5050,"version":"0.24.1"}
> > >  
> > >  
> > > I managed to get to the python sources on the machine, especially 
> > > master.py. I verified that in my case the zookeeper_resolver is used.
> > > However, what gets returned from zookeeper resolver is:
> > >  
> > > return val.split("@")[-1]
> > >  
> > > Where the val is a JSON string:
> > >  
> > >
> > > {"address":{"hostname”:”mesos-master","ip":"10.100.1.100","port":5050},"hostname”:”mesos-master","id":"20150929-113531-244404234-5050-18065","ip”:...,"pid":"master@10.100.1.100:5050
> > >  (http://master@10.100.1.100:5050)","port":5050,"version":"0.24.1”}
> > >  
> > > Looking at these two, it is obvious why it does not work. I’m trying to 
> > > find the code for master.py but it does not exist in 
> > > https://github.com/apache/mesos/tree/master/src/python/interface/src/mesos/interface.
> > > Where does it come from? Is it somehow generated or is it a separate repo?
> > >  
> > >  
> > >  
> > >  
> > >  
> > >  
> > >  
> > >  
> > >  
> > >  
> > > Kind regards,

> > > Radek Gruchalski
> > > 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
> > > (mailto:ra...@gruchalski.com)
> > > de.linkedin.com/in/radgruchalski/ 
> > > (http://de.linkedin.com/in/radgruchalski/)
> > >  
> > > Confidentiality:
> > > This communication is intended for the above-named person and may be 
> > > confidential and/or legally privileged.
> > > If it has come to you in error you must take no action based on it, nor 
> > > must you copy or show it to anyone; please delete/destroy and inform the 
> > > sender immediately.
> > >  
> > >  
> > >  
> > > On Tuesday, 29 September 2015 at 13:02, Rad Gruchalski wrote:
> > >  
> > > > Hi everyone,  
> > > >  
> > > > I have upgraded my development mesos environment to 0.24.1 this 
> > > > morning. It’s a clean installation with new zookeeper and everything.
> > > > Since the upgrade I get an error while executing mesos-tail:
> > > >  
> > > > mesos-master ~$ mesos tail -f -n 50 service
>

Re: mesos-tail in 0.24.1

[Rad Gruchalski]

It seems that I found the reason for this behaviour.  
When I execute mesos-resolve, I get an output like this:
> 10.100.1.100:5050","port":5050,"version":"0.24.1"}

I managed to get to the python sources on the machine, especially master.py. I 
verified that in my case the zookeeper_resolver is used.
However, what gets returned from zookeeper resolver is:

return val.split("@")[-1]

Where the val is a JSON string:

   
{"address":{"hostname”:”mesos-master","ip":"10.100.1.100","port":5050},"hostname”:”mesos-master","id":"20150929-113531-244404234-5050-18065","ip”:...,"pid":"master@10.100.1.100:5050","port":5050,"version":"0.24.1”}

Looking at these two, it is obvious why it does not work. I’m trying to find 
the code for master.py but it does not exist in 
https://github.com/apache/mesos/tree/master/src/python/interface/src/mesos/interface.
Where does it come from? Is it somehow generated or is it a separate repo?










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Tuesday, 29 September 2015 at 13:02, Rad Gruchalski wrote:

> Hi everyone,  
>  
> I have upgraded my development mesos environment to 0.24.1 this morning. It’s 
> a clean installation with new zookeeper and everything.
> Since the upgrade I get an error while executing mesos-tail:
>  
> mesos-master ~$ mesos tail -f -n 50 service
> Traceback (most recent call last):
>   File "/usr/local/bin/mesos-tail", line 11, in 
> sys.exit(main())
>   File "/usr/local/lib/python2.7/dist-packages/mesos/cli/cli.py", line 61, in 
> wrapper
> return fn(*args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/mesos/cli/cmds/tail.py", line 
> 55, in main
> args.task, args.file, fail=(not args.follow)):
>   File "/usr/local/lib/python2.7/dist-packages/mesos/cli/cluster.py", line 
> 27, in files
> tlist = MASTER.tasks(fltr)
>   File "/usr/local/lib/python2.7/dist-packages/mesos/cli/master.py", line 
> 174, in tasks
> self._task_list(active_only
>   File "/usr/local/lib/python2.7/dist-packages/mesos/cli/master.py", line 
> 153, in _task_list
> *[util.merge(x, *keys) for x in self.frameworks(active_only)])
>   File "/usr/local/lib/python2.7/dist-packages/mesos/cli/master.py", line 
> 185, in frameworks
> return util.merge(self.state, *keys)
>   File "/usr/local/lib/python2.7/dist-packages/mesos/cli/util.py", line 58, 
> in __get__
> value = self.fget(inst)
>   File "/usr/local/lib/python2.7/dist-packages/mesos/cli/master.py", line 
> 123, in state
> return self.fetch("/master/state.json").json()
>   File "/usr/local/lib/python2.7/dist-packages/mesos/cli/master.py", line 64, 
> in fetch
> return requests.get(urlparse.urljoin(self.host, url), **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 69, in 
> get
> return request('get', url, params=params, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 50, in 
> request
> response = session.request(method=method, url=url, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 
> 451, in request
> prep = self.prepare_request(req)
>   File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 
> 382, in prepare_request
> hooks=merge_hooks(request.hooks, self.hooks),
>   File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 304, 
> in prepare
> self.prepare_url(url, params)
>   File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 357, 
> in prepare_url
> raise InvalidURL(*e.args)
> requests.exceptions.InvalidURL: Failed to parse: 
> 10.100.1.100:5050","port":5050,"version":"0.24.1"}
>  
>  
> I used mesos-deb-packaging for packaging this release, exactly the same way 
> as 0.24.0, which works without the problem. Is this a known issue?  
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
> Kind regards,

> Radek Gruchalski
> 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
> (mailto:ra...@gruchalski.com)
> de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
>  
> Confidentiality:
> This communication is intended for the above-named person and may be 
> confidential and/or legally privileged.
> If it has come to you in error you must take no action based on it, nor must 
> you copy or show it to anyone; please delete/destroy and inform the sender 
> immediately.
>  
>  
>  
>  

mesos-tail in 0.24.1

[Rad Gruchalski]

Hi everyone,  

I have upgraded my development mesos environment to 0.24.1 this morning. It’s a 
clean installation with new zookeeper and everything.
Since the upgrade I get an error while executing mesos-tail:

mesos-master ~$ mesos tail -f -n 50 service
Traceback (most recent call last):
  File "/usr/local/bin/mesos-tail", line 11, in 
sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/mesos/cli/cli.py", line 61, in 
wrapper
return fn(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/mesos/cli/cmds/tail.py", line 
55, in main
args.task, args.file, fail=(not args.follow)):
  File "/usr/local/lib/python2.7/dist-packages/mesos/cli/cluster.py", line 27, 
in files
tlist = MASTER.tasks(fltr)
  File "/usr/local/lib/python2.7/dist-packages/mesos/cli/master.py", line 174, 
in tasks
self._task_list(active_only
  File "/usr/local/lib/python2.7/dist-packages/mesos/cli/master.py", line 153, 
in _task_list
*[util.merge(x, *keys) for x in self.frameworks(active_only)])
  File "/usr/local/lib/python2.7/dist-packages/mesos/cli/master.py", line 185, 
in frameworks
return util.merge(self.state, *keys)
  File "/usr/local/lib/python2.7/dist-packages/mesos/cli/util.py", line 58, in 
__get__
value = self.fget(inst)
  File "/usr/local/lib/python2.7/dist-packages/mesos/cli/master.py", line 123, 
in state
return self.fetch("/master/state.json").json()
  File "/usr/local/lib/python2.7/dist-packages/mesos/cli/master.py", line 64, 
in fetch
return requests.get(urlparse.urljoin(self.host, url), **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 69, in get
return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 50, in 
request
response = session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 451, 
in request
prep = self.prepare_request(req)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 382, 
in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 304, 
in prepare
self.prepare_url(url, params)
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 357, 
in prepare_url
raise InvalidURL(*e.args)
requests.exceptions.InvalidURL: Failed to parse: 
10.100.1.100:5050","port":5050,"version":"0.24.1"}


I used mesos-deb-packaging for packaging this release, exactly the same way as 
0.24.0, which works without the problem. Is this a known issue?  










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.

Re: Official RPMs

[Rad Gruchalski]

Should be rather easy to package it with this little tool from Mesosphere: 
https://github.com/mesosphere/mesos-deb-packaging. I’ve done it myself for 
ubuntu 12.04 and 14.04.
The only thing that needs to be changed are the dependencies, for ubuntu this 
was:

diff --git a/build_mesos b/build_mesos
index 81561bc..f756ef0 100755
--- a/build_mesos
+++ b/build_mesos
@@ -313,9 +313,10 @@ function deb_ {
--deb-recommends zookeeperd
--deb-recommends zookeeper-bin
-d 'java-runtime-headless'
-   -d libcurl3
-   -d libsvn1
-   -d libsasl2-modules
+   -d libcurl4-nss-dev
+   -d libsasl2-dev
+   -d libapr1-dev
+   -d libsvn-dev


It does look like the tool can build RPMs.










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Saturday, 19 September 2015 at 04:09, craig w wrote:

> Mesosphere provides packages, you can find more information here: 
> https://mesosphere.com/downloads/
>  
> As of right now, they don't seem to have a 0.24.0 package.
>  
> On Fri, Sep 18, 2015 at 8:51 PM, Brian Hicks  (mailto:br...@brianthicks.com)> wrote:
> > We've got some experimental packages at bintray.com/asteris/mantl-rpm 
> > (http://bintray.com/asteris/mantl-rpm), source is at 
> > github.com/asteris-llc/mesos-packaging 
> > (http://github.com/asteris-llc/mesos-packaging). They can really use some 
> > testing if you wanted to give them a try. Configuration is a bit different 
> > than the Mesosphere packages, see the repo for details.
> > On Sep 18, 2015 7:01 PM, "Zameer Manji"  > (mailto:zma...@apache.org)> wrote:
> > > Hey,
> > >  
> > > Does the Apache Mesos project provide OS packages for installation? I 
> > > haven't been able to find any for the 0.24 release and I think having 
> > > them would make installing Mesos a lot easier.
> > >  
> > > --  
> > > Zameer Manji
>  
>  
>  
> --  
> https://github.com/mindscratch
> https://www.google.com/+CraigWickesser
> https://twitter.com/mind_scratch
> https://twitter.com/craig_links

Re: Mesos 0.24.0 with spark in docker - error

[Rad Gruchalski]

Marco,  

The same setup with mesos 0.23.0 attempts registering the framework but 
obviously fails as the framework request has an incorrect address (docker in 
bridge mode).










Kind regards,

Radek Gruchalski

ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 
(mailto:ra...@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.



On Wednesday, 16 September 2015 at 22:58, Marco Massenzio wrote:

> Radek:
>  
> I'm afraid I have some difficulty in following all your steps, so won't be 
> able to comment on all that.
> However, last I tried to run a Spark shell (both Python and Scala) against a 
> deployed Spark cluster (I was using DCOS CE) that did not work.
>  
> AFAIK running a Spark shell is not supported (but others may have succeeded 
> in doing so, please chime in) - did you try to run a spark-submit job and see 
> if you could get that one running?
>  
> As per your error:
>  
> >  java.lang.UnsatisfiedLinkError: /usr/lib/libmesos-0.24.0.so 
> > (http://libmesos-0.24.0.so/): /usr/lib/x86_64-linux-gnu/libcurl-nss.so.4: 
> > version `CURL_3' not found (required by /usr/lib/libmesos-0.24.0.so 
> > (http://libmesos-0.24.0.so/))
>  
> You are right: it's clearly a missing dependency in libcurl (FWIW, I don't 
> think Mesos supports 12.04).
>  
> Marco Massenzio
> Distributed Systems Engineer
> http://codetrips.com  
> On Wed, Sep 16, 2015 at 1:20 PM, Rad Gruchalski <ra...@gruchalski.com 
> (mailto:ra...@gruchalski.com)> wrote:
> > Dear list,  
> >  
> > I’m here for the first time so I do apologise if I’m misbehaving. I am 
> > currently trying running Scala Spark Notebook on Mesos with Docker in 
> > bridge networking. I would like to launch spark executors using the 
> > cluster, the master is running on. This requires registering a framework 
> > from inside of the container and being able to accept the offers back 
> > inside. I found out that the necessary settings, LIBPROCESS_ADVERTISE_IP 
> > and LIBPROCESS_ADVERTISE_PORT, were added in mesos 0.24.0, which seems to 
> > be released (at least tagged in git). I have the cluster running mesos 
> > 0.24.0 and this seems to be behaving really good so far.
> > However, I’m having a problem with the Spark Notebook docker image.
> >  
> > The way I build mesos itself, I use mesosphere/mesos-deb-packaging:
> >  
> > git clone https://github.com/mesosphere/mesos-deb-packaging.git .
> > git checkout d7e5b7b5a8a04b11eaee6a1f9a0962ef3e77864a
> >  
> >  
> > And I’m applying this patch (mesos 0.24.0):
> >  
> > diff --git a/build_mesos b/build_mesos
> > index 81561bc..f756ef0 100755
> > --- a/build_mesos
> > +++ b/build_mesos
> >  
> > @@ -313,9 +313,10 @@ function deb_ {
> > --deb-recommends zookeeperd
> > --deb-recommends zookeeper-bin
> > -d 'java-runtime-headless'
> > -   -d libcurl3
> > -   -d libsvn1
> > -   -d libsasl2-modules
> > +   -d libcurl4-nss-dev
> > +   -d libsasl2-dev
> > +   -d libapr1-dev
> > +   -d libsvn-dev
> > --after-install "$this/$asset_dir/mesos.postinst"
> > --after-remove "$this/$asset_dir/mesos.postrm" )
> >rm -f "$this”/pkg.deb
> >  
> >  
> > My mesos setup
> >  
> > git clone https://github.com/apache/mesos.git .
> > git fetch origin
> > git checkout 0.24.0
> >  
> >  
> > Having gcc-4.9.2, oracle-java7-installer and the following installed 
> > (ubuntu 12.04):
> >  
> > apt-get install -y autoconf libtool
> > apt-get install -y python-dev python-boto libcurl4-nss-dev libsasl2-dev 
> > maven libapr1-dev libsvn-dev git-core software-properties-common \
> >python-software-properties ruby1.9.1 
> > ruby1.9.1-dev \
> >build-essential libxslt-dev libxml2-dev \
> >wget zlibc zlib1g zlib1g-dev
> >  
> > gem install fpm --no-ri --no-rdoc
> >  
> > Building mesos with command:
> >  
> > mkdir -p versions/0.24.0
> > cd versions/0.24

Mesos 0.24.0 with spark in docker - error

[Rad Gruchalski]

Dear list,  

I’m here for the first time so I do apologise if I’m misbehaving. I am 
currently trying running Scala Spark Notebook on Mesos with Docker in bridge 
networking. I would like to launch spark executors using the cluster, the 
master is running on. This requires registering a framework from inside of the 
container and being able to accept the offers back inside. I found out that the 
necessary settings, LIBPROCESS_ADVERTISE_IP and LIBPROCESS_ADVERTISE_PORT, were 
added in mesos 0.24.0, which seems to be released (at least tagged in git). I 
have the cluster running mesos 0.24.0 and this seems to be behaving really good 
so far.
However, I’m having a problem with the Spark Notebook docker image.

The way I build mesos itself, I use mesosphere/mesos-deb-packaging:

git clone https://github.com/mesosphere/mesos-deb-packaging.git .
git checkout d7e5b7b5a8a04b11eaee6a1f9a0962ef3e77864a


And I’m applying this patch (mesos 0.24.0):

diff --git a/build_mesos b/build_mesos
index 81561bc..f756ef0 100755
--- a/build_mesos
+++ b/build_mesos

@@ -313,9 +313,10 @@ function deb_ {
--deb-recommends zookeeperd
--deb-recommends zookeeper-bin
-d 'java-runtime-headless'
-   -d libcurl3
-   -d libsvn1
-   -d libsasl2-modules
+   -d libcurl4-nss-dev
+   -d libsasl2-dev
+   -d libapr1-dev
+   -d libsvn-dev
--after-install "$this/$asset_dir/mesos.postinst"
--after-remove "$this/$asset_dir/mesos.postrm" )
   rm -f "$this”/pkg.deb


My mesos setup

git clone https://github.com/apache/mesos.git .
git fetch origin
git checkout 0.24.0


Having gcc-4.9.2, oracle-java7-installer and the following installed (ubuntu 
12.04):

apt-get install -y autoconf libtool
apt-get install -y python-dev python-boto libcurl4-nss-dev libsasl2-dev 
maven libapr1-dev libsvn-dev git-core software-properties-common \
   python-software-properties ruby1.9.1 ruby1.9.1-dev \
   build-essential libxslt-dev libxml2-dev \
   wget zlibc zlib1g zlib1g-dev

gem install fpm --no-ri --no-rdoc

Building mesos with command:
 
mkdir -p versions/0.24.0
cd versions/0.24.0
git clone https://github.com/apache/mesos.git .
git fetch origin
git checkout 0.24.0
cd ../..
./build_mesos --src-dir versions/0.24.0

Gives me a DEB package which can be successfully installed on a clean 
ubuntu:14.04 docker image by installing only these:

apt-get install -y openjdk-7-jre build-essential python-dev python-boto 
libcurl4-nss-dev libsasl2-dev maven libapr1-dev libsvn-dev

I build on ubuntu:12.04. The solution needs to run on 12.04.

This is how I build the notebook, there are 2 images, the first one tagged as 
myorg/openjdk-7-jre-trusty, the second generated by sbt, however the most 
important commands are listed.

# myorg/openjdk-7-jre-trusty
FROM ubuntu:trusty
MAINTAINER ...
RUN \
  apt-get update && \
  apt-get install -y openjdk-7-jre && \
  rm -rf /var/lib/apt/lists/*
ENV JAVA_HOME /usr/lib/jvm/java-7-openjdk-amd64


The notebook itself:

FROM myorg/openjdk-7-jre-trusty
MAINTAINER …
RUN apt-get update -y && apt-get install -y wget build-essential python-dev 
python-boto libcurl4-nss-dev libsasl2-dev maven libapr1-dev libsvn-dev
RUN cd /tmp && wget 
https://to.my/location/where/I/store/built/mesos-0.24.0.deb && dpkg -i 
mesos-0.24.0.deb
ENV MESOS_JAVA_NATIVE_LIBRARY /usr/lib/libmesos.so
ENV MESOS_LOG_DIR /var/log/mesos

I attempt running this code in Spark Notebook:

# all the hosts used are resolvable, files can be accessed on HDFS:
reset("spark-notebook-tests", lastChanges = (c:SparkConf) => {
  c.set("spark.mesos.executor.uri", 
"hdfs:///resources/spark-1.4.1-cdh4.tar.gz")
   .set("spark.master", 
"mesos://zk://zookeeper-0:2181,zookeeper-1:2181,zookeeper-2:2181/mesos")
   .set("spark.local.dir", System.getenv("MESOS_SANDBOX"))
})


Spark notebook launched with this command:

export LIBPROCESS_PORT=1; export 
LIBPROCESS_ADVERTISE_IP=10.XXX.XXX.XXX; export 
LIBPROCESS_ADVERTISE_PORT=$PORT1; /opt/docker/bin/spark-notebook

The result should be: at least a trace in logs that the offers can’t be sent 
and framework registration attempt. Instead, I receive an error, listed below:

java.lang.UnsatisfiedLinkError: /usr/lib/libmesos-0.24.0.so: 
/usr/lib/x86_64-linux-gnu/libcurl-nss.so.4: version `CURL_3' not found 
(required by /usr/lib/libmesos-0.24.0.so)
  at java.lang.ClassLoader$NativeLibrary.load(Native Method)
  at java.lang.ClassLoader.loadLibrary1(ClassLoader.java:1965)
  at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1890)
  at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1880)