It is a bit strange to me, I also did some test and review code for relative path, and found that relative path works well.
In 0.28.1, if deploy a docker container with MesosContaineirizer, then if using absolute path as continer_path, the mesos agent will update the container_path to a relative path by adding a prefix ./rootfs to the container_path, e.g. /file/path = > ./rootfs/file/path. If deploy a docker container with MesosContaineirizer with relative path as container_path, then the mesos agent will not update the container_path. So the final mount point for the container should be either 1) /tmp/mesos/slaves/agent_id/frameworks/framework_id/ executors/51/runs/container_id/.rootfs/file/path 2) /tmp/mesos/slaves/agent_id/frameworks/framework_id/executors/51/runs/ container_id/file/path The only difference is adding ./rootfs as a prefix or not, the test result is that 1) does not work and 2) works well. And even the mount for 1) failed, but I can see the mount point path does exist. @Yu Jie and @Gilbert, any comments for this? @Oilivier, In order not to block your test, can you please use mesos after 0.28.1? You can use either 0.28.2 or above version. Thanks, Guangya On Mon, May 23, 2016 at 10:30 PM, Guangya Liu <[email protected]> wrote: > Thanks Olivier, I can reproduce this issue now and still checking what is > wrong. > > What I did is as following: > 1) Check out code with tag of 0.28.1 > 2) update mesos-execute to add a host path volume > diff --git a/src/cli/execute.cpp b/src/cli/execute.cpp > index 81a0388..0ff913c 100644 > --- a/src/cli/execute.cpp > +++ b/src/cli/execute.cpp > @@ -72,6 +72,8 @@ using mesos::v1::TaskID; > using mesos::v1::TaskInfo; > using mesos::v1::TaskState; > using mesos::v1::TaskStatus; > +using mesos::v1::Volume; > +using mesos::v1::Parameters; > > using mesos::v1::scheduler::Call; > using mesos::v1::scheduler::Event; > @@ -572,6 +574,12 @@ private: > } > } > > + Volume* volume1 = containerInfo.add_volumes(); > + volume1->set_container_path("/tmp/abcd"); > + volume1->set_mode(Volume::RW); > + volume1->set_host_path("/root/convoy"); > + cout << "Add Voume 1" << endl; > + > return containerInfo; > } else if (containerizer == "docker") { > // 'docker' containerizer only supports 'docker' images. > 3) launch a task with docker image, task failed. > > 4) Check sandbox: > + /root/src/mesos/m1/mesos/build/src/mesos-containerizer mount > --help=false --operation=make-rslave --path=/ > + grep -E /tmp/mesos/.+ /proc/self/mountinfo > + grep -v 3239aafc-78d8-4f70-81e5-f32fb3733339 > + cut+ -d -f5 > xargs --no-run-if-empty umount -l > + mount -n --rbind > /tmp/mesos/provisioner/containers/3239aafc-78d8-4f70-81e5-f32fb3733339/backends/copy/rootfses/5e8bf3fa-53b1-4bd5-bb3d-525ddc7900b6 > /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs > + mount -n --rbind /root/convoy > /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs/tmp/abcd > mount: mount point > /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs/tmp/abcd > does not exist > Failed to execute a preparation shell command > > Will check more for this. > > Thanks, > > Guangya > > On Mon, May 23, 2016 at 3:35 PM, Olivier Sallou <[email protected]> > wrote: > >> >> >> On 05/23/2016 09:33 AM, Olivier Sallou wrote: >> > >> > On 05/20/2016 03:26 PM, Guangya Liu wrote: >> >> Since you are using docker image which means that your container will >> have >> >> rootfs, so it is not required to have the absolute path exist, the >> linux >> >> file system isolator will help create the path automatically >> >> >> https://github.com/apache/mesos/blob/0.28.x/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp#L390-L402 >> >> >> >> Can you please share your framework? How did you set the volume part in >> >> your framework? >> > @Guangya >> > >> > I use Python API. >> > >> > Here is related code: >> > >> > .... >> > # Define container volumes >> > for v in job['container']['volumes']: >> > volume = container.volumes.add() >> > volume.container_path = v['mount'] >> > volume.host_path = v['path'] >> > if v['acl'] == 'rw': >> > volume.mode = 1 # mesos_pb2.Volume.Mode.RW >> > else: >> > volume.mode = 2 # mesos_pb2.Volume.Mode.RO >> > >> > => In my test case, I add 2 volumes from a host shared directory, >> > mounted in container as /mnt/go-docker and /mnt/god-data. >> > >> > ... >> > # Define docker image and network >> > docker = mesos_pb2.ContainerInfo.MesosInfo() >> > docker.image.type = 2 # Docker >> > docker.image.docker.name ='centos:latest' >> > # Request an IP from a network module >> > network_info = container.network_infos.add() >> > network_info_name = 'sampletest' >> > # Get an IP V4 address >> > ip_address = network_info.ip_addresses.add() >> > ip_address.protocol = 1 >> > # The network group to join >> > group = network_info.groups.append(network_info_name) >> > port_list = [22] >> > if port_list: >> > for port in port_list: >> > job['container']['port_mapping'].append({'host': >> > port, 'container': port}) >> > container.mesos.MergeFrom(docker) >> > >> > It results in error message: >> > >> > + mount -n --rbind >> > >> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f >> > >> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs >> > + mount -n --rbind >> > >> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task >> > >> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data >> > mount: mount point >> > >> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data >> > does not exist >> > Failed to execute a preparation shell command >> > >> > >> > We can see the .rootfs, but mnt/god-data under .rootfs fails. Local >> > directory exists, it does not pre-exists in container. What is strange >> > is , if I look in mesos task dir, .rootfs/mnt/go-data, directory is >> present. >> > >> > Or, is the error message (.rootfs/mnt/god-data does not exist) simply a >> > warning, and it creates it, and final error (Failed to execute a >> > preparation shell command) not related (and unclear...) >> Additional info: command to execute in container is located in one of >> mounted volume. >> > >> > Olivier >> >> Thanks, >> >> >> >> Guangya >> >> >> >> On Fri, May 20, 2016 at 4:54 AM, Olivier Sallou < >> [email protected]> >> >> wrote: >> >> >> >>> ----- Mail original ----- >> >>>> De: "Gilbert Song" <[email protected]> >> >>>> À: "dev" <[email protected]> >> >>>> Envoyé: Jeudi 19 Mai 2016 01:57:16 >> >>>> Objet: Re: volume / mount point error with Unified Containerizer >> >>>> >> >>>> @Olivier, >> >>>> In mesos 0.28.1, you are supposed to be able bind mount a volume from >> >>>> the host into the mesos container. Did you specify a docker image (we >> >>>> determine >> >>>> the mount point differently depending whether the container has a >> >>> rootfs)? >> >>> >> >>> Yes I specified an image, a Docker image URI. >> >>> >> >>>> How >> >>>> do you specify your 'container_path' (the mount point in the >> container)? >> >>> If >> >>>> it is an >> >>>> absolute path, we require that dir to be pre-existed. If it is a >> relative >> >>>> path, we will >> >>>> mkdir for it. >> >>> It is an absolute path, but it does not exists in image (this is the >> >>> issue). Images are custom Docker images (images containing tools for >> batch >> >>> computing), and I want, for example, to mount some shared resources >> (user >> >>> home dir, common data, etc.) in the image. Of course those >> directories do >> >>> not pre-exists in container images as they are specific to the >> environment. >> >>> Requiring existence of the directory in the image is not issue as it >> >>> prevents using any existing image from a repo. >> >>> >> >>> When using Docker containerizer it works fine, I can mount any >> external >> >>> storage in the container. >> >>> >> >>> Olivie >> >>> >> >>> >> >>>> @Joshua, >> >>>> Thank for posting your workaround on mesos. As I mentioned above, in >> >>> 0.28.1 >> >>>> or >> >>>> older, we only mkdir for container_path which is relative path (not >> >>>> starting with "/"). >> >>>> Because if no rootfs specified for a mesos container, the container >> >>> shares >> >>>> the host >> >>>> root filesystem. Obviously we don't want any random files to be >> created >> >>>> implicitly >> >>>> on your host fs. >> >>>> From mesos 0.29 (release by the end of this month), we will mkdir the >> >>> mount >> >>>> point in the container except for the command task case that specify >> an >> >>>> absolute >> >>>> container_path without a rootfs. Because we simplify the mounting >> logic, >> >>> and >> >>>> sandbox bind mount will only be done in container mount namespace >> >>> instead of >> >>>> host mount namespace (what we did before). Please keep tuned. >> >>>> >> >>>> Cheers, >> >>>> Gilbert >> >>>> >> >>>> On Wed, May 18, 2016 at 8:14 AM, Joshua Cohen <[email protected]> >> wrote: >> >>>> >> >>>>> Hi Olivier, >> >>>>> >> >>>>> I touched on this issue as part of >> >>>>> https://issues.apache.org/jira/browse/MESOS-5229. It would be nice >> if >> >>>>> Mesos >> >>>>> automatically created container mount points if they don't already >> >>> exist. >> >>>>> In the meantime, as a workaround for this, I've updated my >> filesystem >> >>>>> images to include the path (e.g. in Dockerfile, add `RUN mkdir -p >> >>>>> /some/mount/point`). Not the best solution, but the only thing I've >> >>> seen >> >>>>> that works at the moment. >> >>>>> >> >>>>> Cheers, >> >>>>> >> >>>>> Joshua >> >>>>> >> >>>>> On Wed, May 18, 2016 at 7:36 AM, Guangya Liu <[email protected]> >> >>> wrote: >> >>>>>> It's pretty simple for you from scratch with source code >> >>>>>> >> >>>>>> >> >>> >> https://github.com/apache/mesos/blob/master/docs/getting-started.md#building-mesos >> >>>>>> ;-) >> >>>>>> >> >>>>>> Thanks, >> >>>>>> >> >>>>>> Guangya >> >>>>>> >> >>>>>> On Wed, May 18, 2016 at 8:30 PM, Olivier Sallou < >> >>> [email protected] >> >>>>>> wrote: >> >>>>>> >> >>>>>>> On 05/18/2016 02:31 PM, Guangya Liu wrote: >> >>>>>>>> Just saw that you are working with 0.28.1, the "docker volume >> >>> driver" >> >>>>>>> code >> >>>>>>>> was not in 0.28.1, can you please have a try with mesos master >> >>> branch >> >>>>>> if >> >>>>>>>> you are only doing some test? >> >>>>>>> this is indeed test only for the moment. But I will have to >> >>>>>>> recompile/install mesos :-( (I used packages for install). >> >>>>>>> >> >>>>>>> I will try when possible, but thanks for the hint. >> >>>>>>>> Thanks, >> >>>>>>>> >> >>>>>>>> Guangya >> >>>>>>>> >> >>>>>>>> On Wed, May 18, 2016 at 8:28 PM, Guangya Liu <[email protected] >> >>>>>> wrote: >> >>>>>>>>> Hi Olivier, >> >>>>>>>>> >> >>>>>>>>> I think that you need to enable "docker volume isolator" if you >> >>> want >> >>>>>> use >> >>>>>>>>> external storage with unified container I was writing a document >> >>>>> here >> >>>>>>>>> https://reviews.apache.org/r/47511/, perhaps you can have a try >> >>>>>>> according >> >>>>>>>>> to the document and post some comments there if you find any >> >>> issues. >> >>>>>>>>> Also you can patch mesos-execute here >> >>>>>>> https://reviews.apache.org/r/46762/ to >> >>>>>>>>> have a try with mesos-execute. >> >>>>>>>>> >> >>>>>>>>> Thanks, >> >>>>>>>>> >> >>>>>>>>> Guangya >> >>>>>>>>> >> >>>>>>>>> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou < >> >>>>>>> [email protected]> >> >>>>>>>>> wrote: >> >>>>>>>>> >> >>>>>>>>>> Answering (partially) to myself. >> >>>>>>>>>> >> >>>>>>>>>> I seems issue is container_path does not exists inside >> >>> container. >> >>>>> On >> >>>>>>>>>> Docker, path is created and mounted. With pure mesos, >> >>>>> container_path >> >>>>>>>>>> must exists. >> >>>>>>>>>> >> >>>>>>>>>> mesos.proto says: "If the path is an absolute path, that path >> >>> must >> >>>>>>>>>> already exist." >> >>>>>>>>>> >> >>>>>>>>>> This is an issue however, using Docker images, the path I want >> >>> to >> >>>>>> mount >> >>>>>>>>>> does not exists, and it cannot be modified "on the fly". >> >>>>>>>>>> >> >>>>>>>>>> Is there a workaround for this ? >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> On 05/18/2016 12:24 PM, Olivier Sallou wrote: >> >>>>>>>>>>> Hi, >> >>>>>>>>>>> I am trying unified containerizer on a single server >> >>>>> (master/slave) >> >>>>>> on >> >>>>>>>>>>> mesos 0.28.1, to switch from docker containerizer to >> >>> mesos+docker >> >>>>>>> image >> >>>>>>>>>>> container. >> >>>>>>>>>>> >> >>>>>>>>>>> I have setup slave config as suggested in documentation: >> >>>>>>>>>>> >> >>>>>>>>>>> containerizers=docker,mesos >> >>>>>>>>>>> image_providers=docker \ >> >>>>>>>>>>> isolation=filesystem/linux,docker/runtime >> >>>>>>>>>>> >> >>>>>>>>>>> However, when I execute my task with a volume I have an error: >> >>>>>>>>>>> >> >>>>>>>>>>> .... >> >>>>>>>>>>> + mount -n --rbind >> >>>>>>>>>>> >> >>> >> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f >> >>> >> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs >> >>>>>>>>>>> + mount -n --rbind >> >>>>>>>>>>> >> >>> >> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task >> >>> >> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data >> >>>>>>>>>>> mount: mount point >> >>>>>>>>>>> >> >>> >> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data >> >>>>>>>>>>> does not exist >> >>>>>>>>>>> Failed to execute a preparation shell command >> >>>>>>>>>>> >> >>>>>>>>>>> Then, my task switches to FAILED. >> >>>>>>>>>>> >> >>>>>>>>>>> I define a local volume to bind mount in my "container" >> >>>>>>>>>>> >> >>> >> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task >> >>>>>>>>>>> => /mnt/god-data >> >>>>>>>>>>> My directory exists on local server. >> >>>>>>>>>>> In mesos UI, I can see the .rootfs directory along stdout and >> >>>>> stderr >> >>>>>>>>>>> files, and inside .rootfs, I can see /mnt/god-data (empty). >> >>>>>>>>>>> >> >>>>>>>>>>> Running the same using Docker containerizer instead of mesos >> >>>>>>>>>>> containerizer (with a Docker image) works fine. >> >>>>>>>>>>> >> >>>>>>>>>>> It seems it fails to mount my local directory in the >> >>> container. >> >>>>> Any >> >>>>>>> idea >> >>>>>>>>>>> of what is going wrong or how to debug this? >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> Thanks >> >>>>>>>>>>> >> >>>>>>>>>> -- >> >>>>>>>>>> Olivier Sallou >> >>>>>>>>>> IRISA / University of Rennes 1 >> >>>>>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE >> >>>>>>>>>> Tel: 02.99.84.71.95 >> >>>>>>>>>> >> >>>>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) >> >>>>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D >> >>>>> 8438 >> >>>>>>> -- >> >>>>>>> Olivier Sallou >> >>>>>>> IRISA / University of Rennes 1 >> >>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE >> >>>>>>> Tel: 02.99.84.71.95 >> >>>>>>> >> >>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) >> >>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D >> >>> 8438 >> >> -- >> Olivier Sallou >> IRISA / University of Rennes 1 >> Campus de Beaulieu, 35000 RENNES - FRANCE >> Tel: 02.99.84.71.95 >> >> gpg key id: 4096R/326D8438 (keyring.debian.org) >> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >> >> >
