OK. So I figured out the issue with this and it was my misunderstanding of
executors and tasks.
My task info had:
.setExecutor(Protos.ExecutorInfo.newBuilder(executor))
I should have had this:
.setContainer(containerInfoBuilder)
.setCommand(Protos.CommandInfo.newBuilder().setShell(false))
I didn't have a mesos executor deployed inside my container which explains
the timeout issue.
Thanks again for the support.
Thanks,
Andy.
--
Andy Grove
VP Engineering
CodeFutures Corporation
On Tue, Sep 30, 2014 at 10:20 AM, Andy Grove <[email protected]>
wrote:
> Hi Tim,
>
> Thanks for helping with this. I am running mesos-master and mesos-slave
> natively on the same host (my desktop). The only container in use is the
> one being launched by the mesos-slave.
>
> I will try your suggestion of running a simple command next.
>
> Here is the output from the slave from this issue though:
>
> I0930 10:13:52.053177 30722 main.cpp:126] Build: 2014-09-29 15:35:37 by
> andy
> I0930 10:13:52.053228 30722 main.cpp:128] Version: 0.20.1
> I0930 10:13:53.055480 30722 containerizer.cpp:89] Using isolation:
> posix/cpu,posix/mem
> I0930 10:13:53.058353 30722 main.cpp:149] Starting Mesos slave
> I0930 10:13:53.059651 30722 slave.cpp:167] Slave started on 1)@
> 127.0.1.1:5051
> I0930 10:13:53.060072 30722 slave.cpp:278] Slave resources: cpus(*):8;
> mem(*):14963; disk(*):1.85648e+06; ports(*):[31000-32000]
> I0930 10:13:53.060226 30722 slave.cpp:306] Slave hostname: davros
> I0930 10:13:53.060253 30722 slave.cpp:307] Slave checkpoint: true
> I0930 10:13:53.064975 30729 state.cpp:33] Recovering state from
> '/tmp/mesos/meta'
> I0930 10:13:53.065352 30725 status_update_manager.cpp:193] Recovering
> status update manager
> I0930 10:13:53.065626 30729 docker.cpp:577] Recovering Docker containers
> I0930 10:13:53.065690 30724 containerizer.cpp:252] Recovering containerizer
> I0930 10:13:54.055233 30723 slave.cpp:3198] Finished recovery
> I0930 10:13:54.055448 30723 slave.cpp:589] New master detected at
> [email protected]:5050
> I0930 10:13:54.055532 30723 slave.cpp:625] No credentials provided.
> Attempting to register without authentication
> I0930 10:13:54.055537 30730 status_update_manager.cpp:167] New master
> detected at [email protected]:5050
> I0930 10:13:54.055552 30723 slave.cpp:636] Detecting new master
> I0930 10:13:54.928225 30724 slave.cpp:754] Registered with master
> [email protected]:5050; given slave ID
> 20140930-101303-16777343-5050-30690-0
> I0930 10:13:54.928598 30724 slave.cpp:767] Checkpointing SlaveInfo to
> '/tmp/mesos/meta/slaves/20140930-101303-16777343-5050-30690-0/slave.info'
> I0930 10:14:17.330390 30725 slave.cpp:1002] Got assigned task 0 for
> framework 20140930-101303-16777343-5050-30690-0000
> I0930 10:14:17.330557 30725 slave.cpp:1112] Launching task 0 for framework
> 20140930-101303-16777343-5050-30690-0000
> I0930 10:14:17.331296 30725 slave.cpp:1222] Queuing task '0' for executor
> default of framework '20140930-101303-16777343-5050-30690-0000
> *I0930 10:14:17.333109 30730 docker.cpp:984] Starting container
> 'ebb1dca6-cc9d-427f-8faa-f3f723f6ab81' for executor 'default' and framework
> '20140930-101303-16777343-5050-30690-0000'*
> I0930 10:14:20.062705 30730 slave.cpp:2538] Monitoring executor 'default'
> of framework '20140930-101303-16777343-5050-30690-0000' in container
> 'ebb1dca6-cc9d-427f-8faa-f3f723f6ab81'
>
> The container is running quite happily at this point.
>
> I0930 10:14:53.061337 30724 slave.cpp:3053] Current usage 0.76%. Max
> allowed age: 6.247043850997720days
> *I0930 10:15:17.331712 30730 slave.cpp:3010] Terminating executor default
> of framework 20140930-101303-16777343-5050-30690-0000 because it did not
> register within 1mins*
> I0930 10:15:17.332221 30728 docker.cpp:1473] Destroying container
> 'ebb1dca6-cc9d-427f-8faa-f3f723f6ab81'
> I0930 10:15:17.332308 30728 docker.cpp:1568] Running docker kill on
> container 'ebb1dca6-cc9d-427f-8faa-f3f723f6ab81'
> I0930 10:15:18.109361 30730 docker.cpp:1646] Executor for container
> 'ebb1dca6-cc9d-427f-8faa-f3f723f6ab81' has exited
>
>
> Thanks,
>
> Andy.
>
> --
> Andy Grove
> VP Engineering
> CodeFutures Corporation
>
>
>
> On Mon, Sep 29, 2014 at 6:25 PM, Tim Chen <[email protected]> wrote:
>
>> Hi Andy,
>>
>> You don't need to specifiy -d as the docker containerizer will set it for
>> you since we run all docker images detached.
>>
>> It seems like the executor just simply can't register with the slave. Can
>> you try just running a simple command without Docker that takes longer than
>> the executor registration timeout to see if you see the same error?
>>
>> Also do you run the mesos slave in a docker container as well?
>>
>> Will be great if you can share the slave log as Vinod suggested too.
>>
>> Tim
>>
>>
>>
>>
>>
>>
>> On Mon, Sep 29, 2014 at 5:15 PM, Vinod Kone <[email protected]> wrote:
>>
>>> I'll let Tim Chen help you out here since he has more context. Some
>>> slave logs around the failed container launch would be helpful.
>>>
>>>
>>> On Mon, Sep 29, 2014 at 5:03 PM, Andy Grove <[email protected]>
>>> wrote:
>>>
>>>> Ignore my comment about docker run not returning. That is incorrect.
>>>>
>>>> Thanks,
>>>>
>>>> Andy.
>>>>
>>>> --
>>>> Andy Grove
>>>> VP Engineering
>>>> CodeFutures Corporation
>>>>
>>>>
>>>>
>>>> On Mon, Sep 29, 2014 at 5:59 PM, Andy Grove <[email protected]
>>>> > wrote:
>>>>
>>>>> Hi Vinod,
>>>>>
>>>>> Thanks for the quick response but the image is already on the slave
>>>>> and I see the container being launched almost immediately when my
>>>>> framework
>>>>> starts (within 1-2 seconds). If I keep running docker ps, this is the last
>>>>> output I see before the container is killed:
>>>>>
>>>>> $ docker ps
>>>>> CONTAINER ID IMAGE COMMAND
>>>>> CREATED STATUS PORTS
>>>>> NAMES
>>>>> 45f992c2781f codefutures/dbshards_zookeeper:latest "/bin/sh
>>>>> -c '/opt/zo 59 seconds ago Up 58 seconds
>>>>>
>>>>> I am using mesos 0.20.1 and docker 1.2.0 on Ubuntu 14.04.
>>>>>
>>>>> So the container is running fine. It is a long running service i.e.
>>>>> the docker run command will never return. Should I be providing some
>>>>> option
>>>>> so that the docker executor passed the -d flag to the docker run command?
>>>>> I
>>>>> guess I should start looking through the mesos source so I can see how
>>>>> this
>>>>> works.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Andy.
>>>>>
>>>>> --
>>>>> Andy Grove
>>>>> VP Engineering
>>>>> CodeFutures Corporation
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Sep 29, 2014 at 5:49 PM, Vinod Kone <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Trying increasing the executor registration timeout on the slave
>>>>>> (--executor_registration_timeout) to give docker more time to do a pull
>>>>>> of
>>>>>> the image.
>>>>>>
>>>>>> On Mon, Sep 29, 2014 at 4:41 PM, Andy Grove <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I've working on a prototype Mesos framework to launch docker
>>>>>>> containers. I'm getting as far as seeing my container start up but after
>>>>>>> one minute if gets killed due to:
>>>>>>>
>>>>>>> Terminating executor default of framework
>>>>>>> 20140929-155916-16777343-5050-2708-0004 because it did not register
>>>>>>> within
>>>>>>> 1mins
>>>>>>>
>>>>>>> Here is the code I am using in my scheduler, which was based on one
>>>>>>> of the examples:
>>>>>>>
>>>>>>> @Override
>>>>>>> public void resourceOffers(SchedulerDriver schedulerDriver,
>>>>>>> List<Protos.Offer> offers) {
>>>>>>> logger.info("resourceOffers() with {} offers", offers.size());
>>>>>>>
>>>>>>> for (Protos.Offer offer : offers) {
>>>>>>>
>>>>>>> List<Protos.TaskInfo> tasks = new ArrayList<Protos.TaskInfo>();
>>>>>>> if (launchedTasks < totalTasks) {
>>>>>>> Protos.TaskID taskId = Protos.TaskID.newBuilder()
>>>>>>> .setValue(Integer.toString(launchedTasks++)).build();
>>>>>>>
>>>>>>> logger.info("Launching task " + taskId.getValue());
>>>>>>>
>>>>>>> // docker image info
>>>>>>> Protos.ContainerInfo.DockerInfo.Builder dockerInfoBuilder =
>>>>>>> Protos.ContainerInfo.DockerInfo.newBuilder();
>>>>>>> dockerInfoBuilder.setImage("codefutures/dbshards_zookeeper");
>>>>>>>
>>>>>>> // container info
>>>>>>> Protos.ContainerInfo.Builder containerInfoBuilder =
>>>>>>> Protos.ContainerInfo.newBuilder();
>>>>>>>
>>>>>>> containerInfoBuilder.setType(Protos.ContainerInfo.Type.DOCKER);
>>>>>>> containerInfoBuilder.setDocker(dockerInfoBuilder.build());
>>>>>>>
>>>>>>> // create executor for the container
>>>>>>> Protos.ExecutorInfo executor =
>>>>>>> Protos.ExecutorInfo.newBuilder()
>>>>>>>
>>>>>>> .setExecutorId(Protos.ExecutorID.newBuilder().setValue("default"))
>>>>>>>
>>>>>>> .setCommand(Protos.CommandInfo.newBuilder().setShell(false))
>>>>>>> .setContainer(containerInfoBuilder)
>>>>>>> .setName("Test Executor (Docker)")
>>>>>>> .setSource("docker_test")
>>>>>>> .build();
>>>>>>>
>>>>>>> // create task to run
>>>>>>> Protos.TaskInfo task = Protos.TaskInfo.newBuilder()
>>>>>>> .setName("task " + taskId.getValue())
>>>>>>> .setTaskId(taskId)
>>>>>>> .setSlaveId(offer.getSlaveId())
>>>>>>> .addResources(Protos.Resource.newBuilder()
>>>>>>> .setName("cpus")
>>>>>>> .setType(Protos.Value.Type.SCALAR)
>>>>>>>
>>>>>>> .setScalar(Protos.Value.Scalar.newBuilder().setValue(1)))
>>>>>>> .addResources(Protos.Resource.newBuilder()
>>>>>>> .setName("mem")
>>>>>>> .setType(Protos.Value.Type.SCALAR)
>>>>>>>
>>>>>>> .setScalar(Protos.Value.Scalar.newBuilder().setValue(128)))
>>>>>>> .setExecutor(Protos.ExecutorInfo.newBuilder(executor))
>>>>>>> .build();
>>>>>>>
>>>>>>> tasks.add(task);
>>>>>>> }
>>>>>>> Protos.Filters filters =
>>>>>>> Protos.Filters.newBuilder().setRefuseSeconds(1).build();
>>>>>>>
>>>>>>> schedulerDriver.launchTasks(offer.getId(), tasks, filters);
>>>>>>> }
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> Am I missing some steps with this approach?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Andy.
>>>>>>>
>>>>>>> --
>>>>>>> Andy Grove
>>>>>>> VP Engineering
>>>>>>> CodeFutures Corporation
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
