On 05/23/2016 09:33 AM, Olivier Sallou wrote: > > On 05/20/2016 03:26 PM, Guangya Liu wrote: >> Since you are using docker image which means that your container will have >> rootfs, so it is not required to have the absolute path exist, the linux >> file system isolator will help create the path automatically >> https://github.com/apache/mesos/blob/0.28.x/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp#L390-L402 >> >> Can you please share your framework? How did you set the volume part in >> your framework? > @Guangya > > I use Python API. > > Here is related code: > > .... > # Define container volumes > for v in job['container']['volumes']: > volume = container.volumes.add() > volume.container_path = v['mount'] > volume.host_path = v['path'] > if v['acl'] == 'rw': > volume.mode = 1 # mesos_pb2.Volume.Mode.RW > else: > volume.mode = 2 # mesos_pb2.Volume.Mode.RO > > => In my test case, I add 2 volumes from a host shared directory, > mounted in container as /mnt/go-docker and /mnt/god-data. > > ... > # Define docker image and network > docker = mesos_pb2.ContainerInfo.MesosInfo() > docker.image.type = 2 # Docker > docker.image.docker.name ='centos:latest' > # Request an IP from a network module > network_info = container.network_infos.add() > network_info_name = 'sampletest' > # Get an IP V4 address > ip_address = network_info.ip_addresses.add() > ip_address.protocol = 1 > # The network group to join > group = network_info.groups.append(network_info_name) > port_list = [22] > if port_list: > for port in port_list: > job['container']['port_mapping'].append({'host': > port, 'container': port}) > container.mesos.MergeFrom(docker) > > It results in error message: > > + mount -n --rbind > /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f > /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs > + mount -n --rbind > /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task > /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data > mount: mount point > /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data > does not exist > Failed to execute a preparation shell command > > > We can see the .rootfs, but mnt/god-data under .rootfs fails. Local > directory exists, it does not pre-exists in container. What is strange > is , if I look in mesos task dir, .rootfs/mnt/go-data, directory is present. > > Or, is the error message (.rootfs/mnt/god-data does not exist) simply a > warning, and it creates it, and final error (Failed to execute a > preparation shell command) not related (and unclear...) Additional info: command to execute in container is located in one of mounted volume. > > Olivier >> Thanks, >> >> Guangya >> >> On Fri, May 20, 2016 at 4:54 AM, Olivier Sallou <[email protected]> >> wrote: >> >>> ----- Mail original ----- >>>> De: "Gilbert Song" <[email protected]> >>>> À: "dev" <[email protected]> >>>> Envoyé: Jeudi 19 Mai 2016 01:57:16 >>>> Objet: Re: volume / mount point error with Unified Containerizer >>>> >>>> @Olivier, >>>> In mesos 0.28.1, you are supposed to be able bind mount a volume from >>>> the host into the mesos container. Did you specify a docker image (we >>>> determine >>>> the mount point differently depending whether the container has a >>> rootfs)? >>> >>> Yes I specified an image, a Docker image URI. >>> >>>> How >>>> do you specify your 'container_path' (the mount point in the container)? >>> If >>>> it is an >>>> absolute path, we require that dir to be pre-existed. If it is a relative >>>> path, we will >>>> mkdir for it. >>> It is an absolute path, but it does not exists in image (this is the >>> issue). Images are custom Docker images (images containing tools for batch >>> computing), and I want, for example, to mount some shared resources (user >>> home dir, common data, etc.) in the image. Of course those directories do >>> not pre-exists in container images as they are specific to the environment. >>> Requiring existence of the directory in the image is not issue as it >>> prevents using any existing image from a repo. >>> >>> When using Docker containerizer it works fine, I can mount any external >>> storage in the container. >>> >>> Olivie >>> >>> >>>> @Joshua, >>>> Thank for posting your workaround on mesos. As I mentioned above, in >>> 0.28.1 >>>> or >>>> older, we only mkdir for container_path which is relative path (not >>>> starting with "/"). >>>> Because if no rootfs specified for a mesos container, the container >>> shares >>>> the host >>>> root filesystem. Obviously we don't want any random files to be created >>>> implicitly >>>> on your host fs. >>>> From mesos 0.29 (release by the end of this month), we will mkdir the >>> mount >>>> point in the container except for the command task case that specify an >>>> absolute >>>> container_path without a rootfs. Because we simplify the mounting logic, >>> and >>>> sandbox bind mount will only be done in container mount namespace >>> instead of >>>> host mount namespace (what we did before). Please keep tuned. >>>> >>>> Cheers, >>>> Gilbert >>>> >>>> On Wed, May 18, 2016 at 8:14 AM, Joshua Cohen <[email protected]> wrote: >>>> >>>>> Hi Olivier, >>>>> >>>>> I touched on this issue as part of >>>>> https://issues.apache.org/jira/browse/MESOS-5229. It would be nice if >>>>> Mesos >>>>> automatically created container mount points if they don't already >>> exist. >>>>> In the meantime, as a workaround for this, I've updated my filesystem >>>>> images to include the path (e.g. in Dockerfile, add `RUN mkdir -p >>>>> /some/mount/point`). Not the best solution, but the only thing I've >>> seen >>>>> that works at the moment. >>>>> >>>>> Cheers, >>>>> >>>>> Joshua >>>>> >>>>> On Wed, May 18, 2016 at 7:36 AM, Guangya Liu <[email protected]> >>> wrote: >>>>>> It's pretty simple for you from scratch with source code >>>>>> >>>>>> >>> https://github.com/apache/mesos/blob/master/docs/getting-started.md#building-mesos >>>>>> ;-) >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Guangya >>>>>> >>>>>> On Wed, May 18, 2016 at 8:30 PM, Olivier Sallou < >>> [email protected] >>>>>> wrote: >>>>>> >>>>>>> On 05/18/2016 02:31 PM, Guangya Liu wrote: >>>>>>>> Just saw that you are working with 0.28.1, the "docker volume >>> driver" >>>>>>> code >>>>>>>> was not in 0.28.1, can you please have a try with mesos master >>> branch >>>>>> if >>>>>>>> you are only doing some test? >>>>>>> this is indeed test only for the moment. But I will have to >>>>>>> recompile/install mesos :-( (I used packages for install). >>>>>>> >>>>>>> I will try when possible, but thanks for the hint. >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Guangya >>>>>>>> >>>>>>>> On Wed, May 18, 2016 at 8:28 PM, Guangya Liu <[email protected] >>>>>> wrote: >>>>>>>>> Hi Olivier, >>>>>>>>> >>>>>>>>> I think that you need to enable "docker volume isolator" if you >>> want >>>>>> use >>>>>>>>> external storage with unified container I was writing a document >>>>> here >>>>>>>>> https://reviews.apache.org/r/47511/, perhaps you can have a try >>>>>>> according >>>>>>>>> to the document and post some comments there if you find any >>> issues. >>>>>>>>> Also you can patch mesos-execute here >>>>>>> https://reviews.apache.org/r/46762/ to >>>>>>>>> have a try with mesos-execute. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Guangya >>>>>>>>> >>>>>>>>> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou < >>>>>>> [email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Answering (partially) to myself. >>>>>>>>>> >>>>>>>>>> I seems issue is container_path does not exists inside >>> container. >>>>> On >>>>>>>>>> Docker, path is created and mounted. With pure mesos, >>>>> container_path >>>>>>>>>> must exists. >>>>>>>>>> >>>>>>>>>> mesos.proto says: "If the path is an absolute path, that path >>> must >>>>>>>>>> already exist." >>>>>>>>>> >>>>>>>>>> This is an issue however, using Docker images, the path I want >>> to >>>>>> mount >>>>>>>>>> does not exists, and it cannot be modified "on the fly". >>>>>>>>>> >>>>>>>>>> Is there a workaround for this ? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 05/18/2016 12:24 PM, Olivier Sallou wrote: >>>>>>>>>>> Hi, >>>>>>>>>>> I am trying unified containerizer on a single server >>>>> (master/slave) >>>>>> on >>>>>>>>>>> mesos 0.28.1, to switch from docker containerizer to >>> mesos+docker >>>>>>> image >>>>>>>>>>> container. >>>>>>>>>>> >>>>>>>>>>> I have setup slave config as suggested in documentation: >>>>>>>>>>> >>>>>>>>>>> containerizers=docker,mesos >>>>>>>>>>> image_providers=docker \ >>>>>>>>>>> isolation=filesystem/linux,docker/runtime >>>>>>>>>>> >>>>>>>>>>> However, when I execute my task with a volume I have an error: >>>>>>>>>>> >>>>>>>>>>> .... >>>>>>>>>>> + mount -n --rbind >>>>>>>>>>> >>> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f >>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs >>>>>>>>>>> + mount -n --rbind >>>>>>>>>>> >>> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task >>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data >>>>>>>>>>> mount: mount point >>>>>>>>>>> >>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data >>>>>>>>>>> does not exist >>>>>>>>>>> Failed to execute a preparation shell command >>>>>>>>>>> >>>>>>>>>>> Then, my task switches to FAILED. >>>>>>>>>>> >>>>>>>>>>> I define a local volume to bind mount in my "container" >>>>>>>>>>> >>> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task >>>>>>>>>>> => /mnt/god-data >>>>>>>>>>> My directory exists on local server. >>>>>>>>>>> In mesos UI, I can see the .rootfs directory along stdout and >>>>> stderr >>>>>>>>>>> files, and inside .rootfs, I can see /mnt/god-data (empty). >>>>>>>>>>> >>>>>>>>>>> Running the same using Docker containerizer instead of mesos >>>>>>>>>>> containerizer (with a Docker image) works fine. >>>>>>>>>>> >>>>>>>>>>> It seems it fails to mount my local directory in the >>> container. >>>>> Any >>>>>>> idea >>>>>>>>>>> of what is going wrong or how to debug this? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Olivier Sallou >>>>>>>>>> IRISA / University of Rennes 1 >>>>>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE >>>>>>>>>> Tel: 02.99.84.71.95 >>>>>>>>>> >>>>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) >>>>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D >>>>> 8438 >>>>>>> -- >>>>>>> Olivier Sallou >>>>>>> IRISA / University of Rennes 1 >>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE >>>>>>> Tel: 02.99.84.71.95 >>>>>>> >>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) >>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D >>> 8438
-- Olivier Sallou IRISA / University of Rennes 1 Campus de Beaulieu, 35000 RENNES - FRANCE Tel: 02.99.84.71.95 gpg key id: 4096R/326D8438 (keyring.debian.org) Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
