Yes, the executor will try to connect to agent via 127.0.0.1:5051 in its own network namespace, but your agent is actually listening on 127.0.0.1:5051 in the host network namespace, so executor can never connect to agent. You need to have your agent listen on a host interface.
Thanks, Qian Zhang On Wed, Aug 24, 2016 at 10:04 PM, Avinash Sridharan <[email protected]> wrote: > Oliver, you can't have the agent running on 127.0.0.1. The agent needs to > be running in a routeabl IP address (choose an IP from one of the > interfaces). > > Reason being that if agent is on local host the executor running in its own > network namespace will try to make a connection in its own network > namespace and fail. > On Wed, Aug 24, 2016 at 5:15 AM Olivier Sallou <[email protected]> > wrote: > > > I have the same behavior with Calico. Task get IP from CNI plugin, but > > task remains in STAGING and same logs. > > > > mesos-execute --containerizer=mesos \ > > > --name=cni \ > > > --master=127.0.0.1:5050 \ > > > --networks=calico-net-1 \ > > > --command="ifconfig" > > I0824 14:12:03.202328 24912 scheduler.cpp:172] Version: 1.0.0 > > I0824 14:12:03.203009 24911 scheduler.cpp:461] New master detected at > > [email protected]:5050 > > Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0017' > > Submitted task 'cni' to agent 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0' > > > > REMAINS STAGING! > > > > > > I0824 14:12:03.857158 24806 cni.cpp:1109] Got assigned IPv4 address > > '192.168.0.0/32' from CNI network 'calico-net-1' for container > > bdbb275a-ec5f-4a50-aca0-5e694ae57324 > > I0824 14:12:03.857348 24805 cni.cpp:838] Unable to find DNS nameservers > > for container bdbb275a-ec5f-4a50-aca0-5e694ae57324. Using host > > '/etc/resolv.conf' > > > > No more logs > > > > > > Olivier > > > > On 08/24/2016 08:23 AM, Olivier Sallou wrote: > > > > > > On 08/23/2016 06:13 PM, Jie Yu wrote: > > >> The DNS related logging means that the weave plugin does not return > DNS > > >> information, the agent uses the host resolv.conf for the container. > So I > > >> think is irrelevant to your problem. > > >> > > >> Mesos requires that executor can talk to agent. Can you see if there > is > > a > > >> route from 10.32.0.1 to the agent IP? > > > How can I check this as task does not start ? I have exposed weave > > > network on host: > > > > > > osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo ./weave > expose > > > 10.32.0.2 > > > osallou@tifenn~/Development/NOSAVE/go-docker/weave $ ping 10.32.0.2 > > > PING 10.32.0.2 (10.32.0.2) 56(84) bytes of data. > > > 64 bytes from 10.32.0.2: icmp_seq=1 ttl=64 time=0.032 ms > > > 64 bytes from 10.32.0.2: icmp_seq=2 ttl=64 time=0.029 ms > > > 64 bytes from 10.32.0.2: icmp_seq=3 ttl=64 time=0.029 ms > > > 64 bytes from 10.32.0.2: icmp_seq=4 ttl=64 time=0.031 ms > > > > > > And why is it blocking? > > > > > > I am on a single host environement, so agent is on 127.0.0.1. > > > > > > Olivier > > >> On Tue, Aug 23, 2016 at 9:05 AM, Olivier Sallou < > > [email protected]> > > >> wrote: > > >> > > >>> HI, > > >>> > > >>> I have setup Mesos 1.0.0-2 to use CNI with Weave (1.6.1) > > >>> > > >>> Weave works nicely with the Docker containerizer. > > >>> > > >>> When I try to launch a task via my framework with unified > > containerizer, > > >>> the job remains waiting forever (no RUNNING message). I can see > however > > >>> that weave cni allocated an IP address to Mesos. > > >>> > > >>> I tried with a simple mesos-execute test. > > >>> > > >>> Example with a mesos-execute with no CNI, everything is OK > > >>> > > >>> > > >>> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo > > mesos-execute > > >>> --command="sleep 2" -docker_image=centos:latest --master= > > 127.0.0.1:5050 > > >>> --name=test0 I0823 17:56:50.067520 28815 scheduler.cpp:172] Version: > > 1.0.0 > > >>> I0823 17:56:50.068260 28822 scheduler.cpp:461] New master detected at > > >>> [email protected]:5050 > > >>> Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0005' > > >>> Submitted task 'test0' to agent > > 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0' > > >>> Received status update TASK_RUNNING for task 'test0' > > >>> source: SOURCE_EXECUTOR > > >>> Received status update TASK_FINISHED for task 'test0' > > >>> message: 'Command exited with status 0' > > >>> > > >>> > > >>> Sample example specifying the weave network > > >>> > > >>> > > >>> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo > > mesos-execute > > >>> --command="sleep 2" -docker_image=centos:latest --master= > > 127.0.0.1:5050 > > >>> --name=test0 --networks=weave > > >>> I0823 17:57:15.845304 28856 scheduler.cpp:172] Version: 1.0.0 > > >>> I0823 17:57:15.846248 28857 scheduler.cpp:461] New master detected at > > >>> [email protected]:5050 > > >>> Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0006' > > >>> Submitted task 'test0' to agent > > 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0' > > >>> ==> REMAINS WAITING HERE, job is in STAGING in Mesos UI > > >>> > > >>> mesos-slave logs: > > >>> > > >>> I0823 17:57:15.873872 26522 cni.cpp:716] Bind mounted > > >>> '/proc/28869/ns/net' to > > >>> > > '/run/mesos/isolators/network/cni/4f91a5df-2e9a-4cfc-93f5- > aa197646db09/ns' > > >>> for container 4f91a5df-2e9a-4cfc-93f5-aa197646db09 > > >>> I0823 17:57:16.257063 26519 cni.cpp:1109] Got assigned IPv4 address > > >>> '10.32.0.1/12' from CNI network 'weave' for container > > >>> 4f91a5df-2e9a-4cfc-93f5-aa197646db09 > > >>> I0823 17:57:16.257258 26525 cni.cpp:838] Unable to find DNS > nameservers > > >>> for container 4f91a5df-2e9a-4cfc-93f5-aa197646db09. Using host > > >>> '/etc/resolv.conf' > > >>> > > >>> There are no other logs until I kill the job. > > >>> We can see that Mesos container got an IP but it seems to block on > DNS, > > >>> > > >>> Thanks for hints.... > > >>> > > >>> -- > > >>> > > >>> gpg key id: 4096R/326D8438 (keyring.debian.org) > > >>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 > > >>> > > >>> > > >>> > > > > -- > > Olivier Sallou > > IRISA / University of Rennes 1 > > Campus de Beaulieu, 35000 RENNES - FRANCE > > Tel: 02.99.84.71.95 > > > > gpg key id: 4096R/326D8438 (keyring.debian.org) > > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 > > > > >
