On 05/20/2016 03:26 PM, Guangya Liu wrote:
> Since you are using docker image which means that your container will have
> rootfs, so it is not required to have the absolute path exist, the linux
> file system isolator will help create the path automatically
> https://github.com/apache/mesos/blob/0.28.x/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp#L390-L402
>
> Can you please share your framework? How did you set the volume part in
> your framework?
@Guangya
I use Python API.
Here is related code:
....
# Define container volumes
for v in job['container']['volumes']:
volume = container.volumes.add()
volume.container_path = v['mount']
volume.host_path = v['path']
if v['acl'] == 'rw':
volume.mode = 1 # mesos_pb2.Volume.Mode.RW
else:
volume.mode = 2 # mesos_pb2.Volume.Mode.RO
=> In my test case, I add 2 volumes from a host shared directory,
mounted in container as /mnt/go-docker and /mnt/god-data.
...
# Define docker image and network
docker = mesos_pb2.ContainerInfo.MesosInfo()
docker.image.type = 2 # Docker
docker.image.docker.name ='centos:latest'
# Request an IP from a network module
network_info = container.network_infos.add()
network_info_name = 'sampletest'
# Get an IP V4 address
ip_address = network_info.ip_addresses.add()
ip_address.protocol = 1
# The network group to join
group = network_info.groups.append(network_info_name)
port_list = [22]
if port_list:
for port in port_list:
job['container']['port_mapping'].append({'host':
port, 'container': port})
container.mesos.MergeFrom(docker)
It results in error message:
+ mount -n --rbind
/tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
/tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
+ mount -n --rbind
/home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
/tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
mount: mount point
/tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
does not exist
Failed to execute a preparation shell command
We can see the .rootfs, but mnt/god-data under .rootfs fails. Local
directory exists, it does not pre-exists in container. What is strange
is , if I look in mesos task dir, .rootfs/mnt/go-data, directory is present.
Or, is the error message (.rootfs/mnt/god-data does not exist) simply a
warning, and it creates it, and final error (Failed to execute a
preparation shell command) not related (and unclear...)
Olivier
>
> Thanks,
>
> Guangya
>
> On Fri, May 20, 2016 at 4:54 AM, Olivier Sallou <[email protected]>
> wrote:
>
>>
>> ----- Mail original -----
>>> De: "Gilbert Song" <[email protected]>
>>> À: "dev" <[email protected]>
>>> Envoyé: Jeudi 19 Mai 2016 01:57:16
>>> Objet: Re: volume / mount point error with Unified Containerizer
>>>
>>> @Olivier,
>>> In mesos 0.28.1, you are supposed to be able bind mount a volume from
>>> the host into the mesos container. Did you specify a docker image (we
>>> determine
>>> the mount point differently depending whether the container has a
>> rootfs)?
>>
>> Yes I specified an image, a Docker image URI.
>>
>>> How
>>> do you specify your 'container_path' (the mount point in the container)?
>> If
>>> it is an
>>> absolute path, we require that dir to be pre-existed. If it is a relative
>>> path, we will
>>> mkdir for it.
>> It is an absolute path, but it does not exists in image (this is the
>> issue). Images are custom Docker images (images containing tools for batch
>> computing), and I want, for example, to mount some shared resources (user
>> home dir, common data, etc.) in the image. Of course those directories do
>> not pre-exists in container images as they are specific to the environment.
>> Requiring existence of the directory in the image is not issue as it
>> prevents using any existing image from a repo.
>>
>> When using Docker containerizer it works fine, I can mount any external
>> storage in the container.
>>
>> Olivie
>>
>>
>>> @Joshua,
>>> Thank for posting your workaround on mesos. As I mentioned above, in
>> 0.28.1
>>> or
>>> older, we only mkdir for container_path which is relative path (not
>>> starting with "/").
>>> Because if no rootfs specified for a mesos container, the container
>> shares
>>> the host
>>> root filesystem. Obviously we don't want any random files to be created
>>> implicitly
>>> on your host fs.
>>> From mesos 0.29 (release by the end of this month), we will mkdir the
>> mount
>>> point in the container except for the command task case that specify an
>>> absolute
>>> container_path without a rootfs. Because we simplify the mounting logic,
>> and
>>> sandbox bind mount will only be done in container mount namespace
>> instead of
>>> host mount namespace (what we did before). Please keep tuned.
>>>
>>> Cheers,
>>> Gilbert
>>>
>>> On Wed, May 18, 2016 at 8:14 AM, Joshua Cohen <[email protected]> wrote:
>>>
>>>> Hi Olivier,
>>>>
>>>> I touched on this issue as part of
>>>> https://issues.apache.org/jira/browse/MESOS-5229. It would be nice if
>>>> Mesos
>>>> automatically created container mount points if they don't already
>> exist.
>>>> In the meantime, as a workaround for this, I've updated my filesystem
>>>> images to include the path (e.g. in Dockerfile, add `RUN mkdir -p
>>>> /some/mount/point`). Not the best solution, but the only thing I've
>> seen
>>>> that works at the moment.
>>>>
>>>> Cheers,
>>>>
>>>> Joshua
>>>>
>>>> On Wed, May 18, 2016 at 7:36 AM, Guangya Liu <[email protected]>
>> wrote:
>>>>> It's pretty simple for you from scratch with source code
>>>>>
>>>>>
>> https://github.com/apache/mesos/blob/master/docs/getting-started.md#building-mesos
>>>>> ;-)
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Guangya
>>>>>
>>>>> On Wed, May 18, 2016 at 8:30 PM, Olivier Sallou <
>> [email protected]
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> On 05/18/2016 02:31 PM, Guangya Liu wrote:
>>>>>>> Just saw that you are working with 0.28.1, the "docker volume
>> driver"
>>>>>> code
>>>>>>> was not in 0.28.1, can you please have a try with mesos master
>> branch
>>>>> if
>>>>>>> you are only doing some test?
>>>>>> this is indeed test only for the moment. But I will have to
>>>>>> recompile/install mesos :-( (I used packages for install).
>>>>>>
>>>>>> I will try when possible, but thanks for the hint.
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Guangya
>>>>>>>
>>>>>>> On Wed, May 18, 2016 at 8:28 PM, Guangya Liu <[email protected]
>>>>> wrote:
>>>>>>>> Hi Olivier,
>>>>>>>>
>>>>>>>> I think that you need to enable "docker volume isolator" if you
>> want
>>>>> use
>>>>>>>> external storage with unified container I was writing a document
>>>> here
>>>>>>>> https://reviews.apache.org/r/47511/, perhaps you can have a try
>>>>>> according
>>>>>>>> to the document and post some comments there if you find any
>> issues.
>>>>>>>> Also you can patch mesos-execute here
>>>>>> https://reviews.apache.org/r/46762/ to
>>>>>>>> have a try with mesos-execute.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Guangya
>>>>>>>>
>>>>>>>> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou <
>>>>>> [email protected]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Answering (partially) to myself.
>>>>>>>>>
>>>>>>>>> I seems issue is container_path does not exists inside
>> container.
>>>> On
>>>>>>>>> Docker, path is created and mounted. With pure mesos,
>>>> container_path
>>>>>>>>> must exists.
>>>>>>>>>
>>>>>>>>> mesos.proto says: "If the path is an absolute path, that path
>> must
>>>>>>>>> already exist."
>>>>>>>>>
>>>>>>>>> This is an issue however, using Docker images, the path I want
>> to
>>>>> mount
>>>>>>>>> does not exists, and it cannot be modified "on the fly".
>>>>>>>>>
>>>>>>>>> Is there a workaround for this ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 05/18/2016 12:24 PM, Olivier Sallou wrote:
>>>>>>>>>> Hi,
>>>>>>>>>> I am trying unified containerizer on a single server
>>>> (master/slave)
>>>>> on
>>>>>>>>>> mesos 0.28.1, to switch from docker containerizer to
>> mesos+docker
>>>>>> image
>>>>>>>>>> container.
>>>>>>>>>>
>>>>>>>>>> I have setup slave config as suggested in documentation:
>>>>>>>>>>
>>>>>>>>>> containerizers=docker,mesos
>>>>>>>>>> image_providers=docker \
>>>>>>>>>> isolation=filesystem/linux,docker/runtime
>>>>>>>>>>
>>>>>>>>>> However, when I execute my task with a volume I have an error:
>>>>>>>>>>
>>>>>>>>>> ....
>>>>>>>>>> + mount -n --rbind
>>>>>>>>>>
>> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
>>>>>>>>>> + mount -n --rbind
>>>>>>>>>>
>> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
>>>>>>>>>> mount: mount point
>>>>>>>>>>
>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
>>>>>>>>>> does not exist
>>>>>>>>>> Failed to execute a preparation shell command
>>>>>>>>>>
>>>>>>>>>> Then, my task switches to FAILED.
>>>>>>>>>>
>>>>>>>>>> I define a local volume to bind mount in my "container"
>>>>>>>>>>
>> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
>>>>>>>>>> => /mnt/god-data
>>>>>>>>>> My directory exists on local server.
>>>>>>>>>> In mesos UI, I can see the .rootfs directory along stdout and
>>>> stderr
>>>>>>>>>> files, and inside .rootfs, I can see /mnt/god-data (empty).
>>>>>>>>>>
>>>>>>>>>> Running the same using Docker containerizer instead of mesos
>>>>>>>>>> containerizer (with a Docker image) works fine.
>>>>>>>>>>
>>>>>>>>>> It seems it fails to mount my local directory in the
>> container.
>>>> Any
>>>>>> idea
>>>>>>>>>> of what is going wrong or how to debug this?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Olivier Sallou
>>>>>>>>> IRISA / University of Rennes 1
>>>>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
>>>>>>>>> Tel: 02.99.84.71.95
>>>>>>>>>
>>>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>>>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D
>>>> 8438
>>>>>>>>>
>>>>>> --
>>>>>> Olivier Sallou
>>>>>> IRISA / University of Rennes 1
>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
>>>>>> Tel: 02.99.84.71.95
>>>>>>
>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D
>> 8438
>>>>>>
--
Olivier Sallou
IRISA / University of Rennes 1
Campus de Beaulieu, 35000 RENNES - FRANCE
Tel: 02.99.84.71.95
gpg key id: 4096R/326D8438 (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
