Hi Andy, Good catch, I also missed that as I was just looking at the Docker configurations.
You'll set the Executor when you have an custom executor. Let us know if you have any other problems. Tim On Tue, Sep 30, 2014 at 11:02 AM, Andy Grove <[email protected]> wrote: > OK. So I figured out the issue with this and it was my misunderstanding of > executors and tasks. > > My task info had: > > .setExecutor(Protos.ExecutorInfo.newBuilder(executor)) > > I should have had this: > > .setContainer(containerInfoBuilder) > .setCommand(Protos.CommandInfo.newBuilder().setShell(false)) > > I didn't have a mesos executor deployed inside my container which explains > the timeout issue. > > Thanks again for the support. > > > Thanks, > > Andy. > > -- > Andy Grove > VP Engineering > CodeFutures Corporation > > > > On Tue, Sep 30, 2014 at 10:20 AM, Andy Grove <[email protected]> > wrote: > >> Hi Tim, >> >> Thanks for helping with this. I am running mesos-master and mesos-slave >> natively on the same host (my desktop). The only container in use is the >> one being launched by the mesos-slave. >> >> I will try your suggestion of running a simple command next. >> >> Here is the output from the slave from this issue though: >> >> I0930 10:13:52.053177 30722 main.cpp:126] Build: 2014-09-29 15:35:37 by >> andy >> I0930 10:13:52.053228 30722 main.cpp:128] Version: 0.20.1 >> I0930 10:13:53.055480 30722 containerizer.cpp:89] Using isolation: >> posix/cpu,posix/mem >> I0930 10:13:53.058353 30722 main.cpp:149] Starting Mesos slave >> I0930 10:13:53.059651 30722 slave.cpp:167] Slave started on 1)@ >> 127.0.1.1:5051 >> I0930 10:13:53.060072 30722 slave.cpp:278] Slave resources: cpus(*):8; >> mem(*):14963; disk(*):1.85648e+06; ports(*):[31000-32000] >> I0930 10:13:53.060226 30722 slave.cpp:306] Slave hostname: davros >> I0930 10:13:53.060253 30722 slave.cpp:307] Slave checkpoint: true >> I0930 10:13:53.064975 30729 state.cpp:33] Recovering state from >> '/tmp/mesos/meta' >> I0930 10:13:53.065352 30725 status_update_manager.cpp:193] Recovering >> status update manager >> I0930 10:13:53.065626 30729 docker.cpp:577] Recovering Docker containers >> I0930 10:13:53.065690 30724 containerizer.cpp:252] Recovering >> containerizer >> I0930 10:13:54.055233 30723 slave.cpp:3198] Finished recovery >> I0930 10:13:54.055448 30723 slave.cpp:589] New master detected at >> [email protected]:5050 >> I0930 10:13:54.055532 30723 slave.cpp:625] No credentials provided. >> Attempting to register without authentication >> I0930 10:13:54.055537 30730 status_update_manager.cpp:167] New master >> detected at [email protected]:5050 >> I0930 10:13:54.055552 30723 slave.cpp:636] Detecting new master >> I0930 10:13:54.928225 30724 slave.cpp:754] Registered with master >> [email protected]:5050; given slave ID >> 20140930-101303-16777343-5050-30690-0 >> I0930 10:13:54.928598 30724 slave.cpp:767] Checkpointing SlaveInfo to >> '/tmp/mesos/meta/slaves/20140930-101303-16777343-5050-30690-0/slave.info' >> I0930 10:14:17.330390 30725 slave.cpp:1002] Got assigned task 0 for >> framework 20140930-101303-16777343-5050-30690-0000 >> I0930 10:14:17.330557 30725 slave.cpp:1112] Launching task 0 for >> framework 20140930-101303-16777343-5050-30690-0000 >> I0930 10:14:17.331296 30725 slave.cpp:1222] Queuing task '0' for executor >> default of framework '20140930-101303-16777343-5050-30690-0000 >> *I0930 10:14:17.333109 30730 docker.cpp:984] Starting container >> 'ebb1dca6-cc9d-427f-8faa-f3f723f6ab81' for executor 'default' and framework >> '20140930-101303-16777343-5050-30690-0000'* >> I0930 10:14:20.062705 30730 slave.cpp:2538] Monitoring executor 'default' >> of framework '20140930-101303-16777343-5050-30690-0000' in container >> 'ebb1dca6-cc9d-427f-8faa-f3f723f6ab81' >> >> The container is running quite happily at this point. >> >> I0930 10:14:53.061337 30724 slave.cpp:3053] Current usage 0.76%. Max >> allowed age: 6.247043850997720days >> *I0930 10:15:17.331712 30730 slave.cpp:3010] Terminating executor default >> of framework 20140930-101303-16777343-5050-30690-0000 because it did not >> register within 1mins* >> I0930 10:15:17.332221 30728 docker.cpp:1473] Destroying container >> 'ebb1dca6-cc9d-427f-8faa-f3f723f6ab81' >> I0930 10:15:17.332308 30728 docker.cpp:1568] Running docker kill on >> container 'ebb1dca6-cc9d-427f-8faa-f3f723f6ab81' >> I0930 10:15:18.109361 30730 docker.cpp:1646] Executor for container >> 'ebb1dca6-cc9d-427f-8faa-f3f723f6ab81' has exited >> >> >> Thanks, >> >> Andy. >> >> -- >> Andy Grove >> VP Engineering >> CodeFutures Corporation >> >> >> >> On Mon, Sep 29, 2014 at 6:25 PM, Tim Chen <[email protected]> wrote: >> >>> Hi Andy, >>> >>> You don't need to specifiy -d as the docker containerizer will set it >>> for you since we run all docker images detached. >>> >>> It seems like the executor just simply can't register with the slave. >>> Can you try just running a simple command without Docker that takes longer >>> than the executor registration timeout to see if you see the same error? >>> >>> Also do you run the mesos slave in a docker container as well? >>> >>> Will be great if you can share the slave log as Vinod suggested too. >>> >>> Tim >>> >>> >>> >>> >>> >>> >>> On Mon, Sep 29, 2014 at 5:15 PM, Vinod Kone <[email protected]> wrote: >>> >>>> I'll let Tim Chen help you out here since he has more context. Some >>>> slave logs around the failed container launch would be helpful. >>>> >>>> >>>> On Mon, Sep 29, 2014 at 5:03 PM, Andy Grove <[email protected] >>>> > wrote: >>>> >>>>> Ignore my comment about docker run not returning. That is incorrect. >>>>> >>>>> Thanks, >>>>> >>>>> Andy. >>>>> >>>>> -- >>>>> Andy Grove >>>>> VP Engineering >>>>> CodeFutures Corporation >>>>> >>>>> >>>>> >>>>> On Mon, Sep 29, 2014 at 5:59 PM, Andy Grove < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Vinod, >>>>>> >>>>>> Thanks for the quick response but the image is already on the slave >>>>>> and I see the container being launched almost immediately when my >>>>>> framework >>>>>> starts (within 1-2 seconds). If I keep running docker ps, this is the >>>>>> last >>>>>> output I see before the container is killed: >>>>>> >>>>>> $ docker ps >>>>>> CONTAINER ID IMAGE COMMAND >>>>>> CREATED STATUS PORTS >>>>>> NAMES >>>>>> 45f992c2781f codefutures/dbshards_zookeeper:latest "/bin/sh >>>>>> -c '/opt/zo 59 seconds ago Up 58 seconds >>>>>> >>>>>> I am using mesos 0.20.1 and docker 1.2.0 on Ubuntu 14.04. >>>>>> >>>>>> So the container is running fine. It is a long running service i.e. >>>>>> the docker run command will never return. Should I be providing some >>>>>> option >>>>>> so that the docker executor passed the -d flag to the docker run >>>>>> command? I >>>>>> guess I should start looking through the mesos source so I can see how >>>>>> this >>>>>> works. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Andy. >>>>>> >>>>>> -- >>>>>> Andy Grove >>>>>> VP Engineering >>>>>> CodeFutures Corporation >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Sep 29, 2014 at 5:49 PM, Vinod Kone <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Trying increasing the executor registration timeout on the slave >>>>>>> (--executor_registration_timeout) to give docker more time to do a pull >>>>>>> of >>>>>>> the image. >>>>>>> >>>>>>> On Mon, Sep 29, 2014 at 4:41 PM, Andy Grove < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I've working on a prototype Mesos framework to launch docker >>>>>>>> containers. I'm getting as far as seeing my container start up but >>>>>>>> after >>>>>>>> one minute if gets killed due to: >>>>>>>> >>>>>>>> Terminating executor default of framework >>>>>>>> 20140929-155916-16777343-5050-2708-0004 because it did not register >>>>>>>> within >>>>>>>> 1mins >>>>>>>> >>>>>>>> Here is the code I am using in my scheduler, which was based on one >>>>>>>> of the examples: >>>>>>>> >>>>>>>> @Override >>>>>>>> public void resourceOffers(SchedulerDriver schedulerDriver, >>>>>>>> List<Protos.Offer> offers) { >>>>>>>> logger.info("resourceOffers() with {} offers", offers.size()); >>>>>>>> >>>>>>>> for (Protos.Offer offer : offers) { >>>>>>>> >>>>>>>> List<Protos.TaskInfo> tasks = new >>>>>>>> ArrayList<Protos.TaskInfo>(); >>>>>>>> if (launchedTasks < totalTasks) { >>>>>>>> Protos.TaskID taskId = Protos.TaskID.newBuilder() >>>>>>>> .setValue(Integer.toString(launchedTasks++)).build(); >>>>>>>> >>>>>>>> logger.info("Launching task " + taskId.getValue()); >>>>>>>> >>>>>>>> // docker image info >>>>>>>> Protos.ContainerInfo.DockerInfo.Builder dockerInfoBuilder = >>>>>>>> Protos.ContainerInfo.DockerInfo.newBuilder(); >>>>>>>> >>>>>>>> dockerInfoBuilder.setImage("codefutures/dbshards_zookeeper"); >>>>>>>> >>>>>>>> // container info >>>>>>>> Protos.ContainerInfo.Builder containerInfoBuilder = >>>>>>>> Protos.ContainerInfo.newBuilder(); >>>>>>>> >>>>>>>> containerInfoBuilder.setType(Protos.ContainerInfo.Type.DOCKER); >>>>>>>> containerInfoBuilder.setDocker(dockerInfoBuilder.build()); >>>>>>>> >>>>>>>> // create executor for the container >>>>>>>> Protos.ExecutorInfo executor = >>>>>>>> Protos.ExecutorInfo.newBuilder() >>>>>>>> >>>>>>>> .setExecutorId(Protos.ExecutorID.newBuilder().setValue("default")) >>>>>>>> >>>>>>>> .setCommand(Protos.CommandInfo.newBuilder().setShell(false)) >>>>>>>> .setContainer(containerInfoBuilder) >>>>>>>> .setName("Test Executor (Docker)") >>>>>>>> .setSource("docker_test") >>>>>>>> .build(); >>>>>>>> >>>>>>>> // create task to run >>>>>>>> Protos.TaskInfo task = Protos.TaskInfo.newBuilder() >>>>>>>> .setName("task " + taskId.getValue()) >>>>>>>> .setTaskId(taskId) >>>>>>>> .setSlaveId(offer.getSlaveId()) >>>>>>>> .addResources(Protos.Resource.newBuilder() >>>>>>>> .setName("cpus") >>>>>>>> .setType(Protos.Value.Type.SCALAR) >>>>>>>> >>>>>>>> .setScalar(Protos.Value.Scalar.newBuilder().setValue(1))) >>>>>>>> .addResources(Protos.Resource.newBuilder() >>>>>>>> .setName("mem") >>>>>>>> .setType(Protos.Value.Type.SCALAR) >>>>>>>> >>>>>>>> .setScalar(Protos.Value.Scalar.newBuilder().setValue(128))) >>>>>>>> .setExecutor(Protos.ExecutorInfo.newBuilder(executor)) >>>>>>>> .build(); >>>>>>>> >>>>>>>> tasks.add(task); >>>>>>>> } >>>>>>>> Protos.Filters filters = >>>>>>>> Protos.Filters.newBuilder().setRefuseSeconds(1).build(); >>>>>>>> >>>>>>>> schedulerDriver.launchTasks(offer.getId(), tasks, filters); >>>>>>>> } >>>>>>>> >>>>>>>> } >>>>>>>> >>>>>>>> Am I missing some steps with this approach? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Andy. >>>>>>>> >>>>>>>> -- >>>>>>>> Andy Grove >>>>>>>> VP Engineering >>>>>>>> CodeFutures Corporation >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
