----- Mail original ----- > De: "Guangya Liu" <[email protected]> > À: "dev" <[email protected]>, "Jie Yu" <[email protected]>, "Gilbert > Song" <[email protected]> > Envoyé: Lundi 23 Mai 2016 17:34:41 > Objet: Re: volume / mount point error with Unified Containerizer > > It is a bit strange to me, I also did some test and review code for > relative path, and found that relative path works well. > > In 0.28.1, if deploy a docker container with MesosContaineirizer, then if > using absolute path as continer_path, the mesos agent will update the > container_path to a relative path by adding a prefix ./rootfs to the > container_path, e.g. /file/path = > ./rootfs/file/path. > > If deploy a docker container with MesosContaineirizer with relative path as > container_path, then the mesos agent will not update the container_path. > > So the final mount point for the container should be either > > 1) /tmp/mesos/slaves/agent_id/frameworks/framework_id/ > executors/51/runs/container_id/.rootfs/file/path > 2) /tmp/mesos/slaves/agent_id/frameworks/framework_id/executors/51/runs/ > container_id/file/path > > The only difference is adding ./rootfs as a prefix or not, the test result > is that 1) does not work and 2) works well. And even the mount for 1) > failed, but I can see the mount point path does exist. >
@Guangya I confirm that using relative path works fine, I get volumes in mesos path (but it does not help for my implementation). If I use the Docker containerizer, absolute paths are fine, this is what I use for the moment in my code, and am investigating to switch to unified container. > @Yu Jie and @Gilbert, any comments for this? > > @Oilivier, > > In order not to block your test, can you please use mesos after 0.28.1? You > can use either 0.28.2 or above version. Well, as this is not an urgent matter, I am waiting 0.29 to test against this release (with other features I am waiting for). > > Thanks, > > Guangya > > > On Mon, May 23, 2016 at 10:30 PM, Guangya Liu <[email protected]> wrote: > > > Thanks Olivier, I can reproduce this issue now and still checking what is > > wrong. > > > > What I did is as following: > > 1) Check out code with tag of 0.28.1 > > 2) update mesos-execute to add a host path volume > > diff --git a/src/cli/execute.cpp b/src/cli/execute.cpp > > index 81a0388..0ff913c 100644 > > --- a/src/cli/execute.cpp > > +++ b/src/cli/execute.cpp > > @@ -72,6 +72,8 @@ using mesos::v1::TaskID; > > using mesos::v1::TaskInfo; > > using mesos::v1::TaskState; > > using mesos::v1::TaskStatus; > > +using mesos::v1::Volume; > > +using mesos::v1::Parameters; > > > > using mesos::v1::scheduler::Call; > > using mesos::v1::scheduler::Event; > > @@ -572,6 +574,12 @@ private: > > } > > } > > > > + Volume* volume1 = containerInfo.add_volumes(); > > + volume1->set_container_path("/tmp/abcd"); > > + volume1->set_mode(Volume::RW); > > + volume1->set_host_path("/root/convoy"); > > + cout << "Add Voume 1" << endl; > > + > > return containerInfo; > > } else if (containerizer == "docker") { > > // 'docker' containerizer only supports 'docker' images. > > 3) launch a task with docker image, task failed. > > > > 4) Check sandbox: > > + /root/src/mesos/m1/mesos/build/src/mesos-containerizer mount > > --help=false --operation=make-rslave --path=/ > > + grep -E /tmp/mesos/.+ /proc/self/mountinfo > > + grep -v 3239aafc-78d8-4f70-81e5-f32fb3733339 > > + cut+ -d -f5 > > xargs --no-run-if-empty umount -l > > + mount -n --rbind > > /tmp/mesos/provisioner/containers/3239aafc-78d8-4f70-81e5-f32fb3733339/backends/copy/rootfses/5e8bf3fa-53b1-4bd5-bb3d-525ddc7900b6 > > /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs > > + mount -n --rbind /root/convoy > > /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs/tmp/abcd > > mount: mount point > > /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs/tmp/abcd > > does not exist > > Failed to execute a preparation shell command > > > > Will check more for this. > > > > Thanks, > > > > Guangya > > > > On Mon, May 23, 2016 at 3:35 PM, Olivier Sallou <[email protected]> > > wrote: > > > >> > >> > >> On 05/23/2016 09:33 AM, Olivier Sallou wrote: > >> > > >> > On 05/20/2016 03:26 PM, Guangya Liu wrote: > >> >> Since you are using docker image which means that your container will > >> have > >> >> rootfs, so it is not required to have the absolute path exist, the > >> linux > >> >> file system isolator will help create the path automatically > >> >> > >> https://github.com/apache/mesos/blob/0.28.x/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp#L390-L402 > >> >> > >> >> Can you please share your framework? How did you set the volume part in > >> >> your framework? > >> > @Guangya > >> > > >> > I use Python API. > >> > > >> > Here is related code: > >> > > >> > .... > >> > # Define container volumes > >> > for v in job['container']['volumes']: > >> > volume = container.volumes.add() > >> > volume.container_path = v['mount'] > >> > volume.host_path = v['path'] > >> > if v['acl'] == 'rw': > >> > volume.mode = 1 # mesos_pb2.Volume.Mode.RW > >> > else: > >> > volume.mode = 2 # mesos_pb2.Volume.Mode.RO > >> > > >> > => In my test case, I add 2 volumes from a host shared directory, > >> > mounted in container as /mnt/go-docker and /mnt/god-data. > >> > > >> > ... > >> > # Define docker image and network > >> > docker = mesos_pb2.ContainerInfo.MesosInfo() > >> > docker.image.type = 2 # Docker > >> > docker.image.docker.name ='centos:latest' > >> > # Request an IP from a network module > >> > network_info = container.network_infos.add() > >> > network_info_name = 'sampletest' > >> > # Get an IP V4 address > >> > ip_address = network_info.ip_addresses.add() > >> > ip_address.protocol = 1 > >> > # The network group to join > >> > group = network_info.groups.append(network_info_name) > >> > port_list = [22] > >> > if port_list: > >> > for port in port_list: > >> > job['container']['port_mapping'].append({'host': > >> > port, 'container': port}) > >> > container.mesos.MergeFrom(docker) > >> > > >> > It results in error message: > >> > > >> > + mount -n --rbind > >> > > >> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f > >> > > >> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs > >> > + mount -n --rbind > >> > > >> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task > >> > > >> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data > >> > mount: mount point > >> > > >> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data > >> > does not exist > >> > Failed to execute a preparation shell command > >> > > >> > > >> > We can see the .rootfs, but mnt/god-data under .rootfs fails. Local > >> > directory exists, it does not pre-exists in container. What is strange > >> > is , if I look in mesos task dir, .rootfs/mnt/go-data, directory is > >> present. > >> > > >> > Or, is the error message (.rootfs/mnt/god-data does not exist) simply a > >> > warning, and it creates it, and final error (Failed to execute a > >> > preparation shell command) not related (and unclear...) > >> Additional info: command to execute in container is located in one of > >> mounted volume. > >> > > >> > Olivier > >> >> Thanks, > >> >> > >> >> Guangya > >> >> > >> >> On Fri, May 20, 2016 at 4:54 AM, Olivier Sallou < > >> [email protected]> > >> >> wrote: > >> >> > >> >>> ----- Mail original ----- > >> >>>> De: "Gilbert Song" <[email protected]> > >> >>>> À: "dev" <[email protected]> > >> >>>> Envoyé: Jeudi 19 Mai 2016 01:57:16 > >> >>>> Objet: Re: volume / mount point error with Unified Containerizer > >> >>>> > >> >>>> @Olivier, > >> >>>> In mesos 0.28.1, you are supposed to be able bind mount a volume from > >> >>>> the host into the mesos container. Did you specify a docker image (we > >> >>>> determine > >> >>>> the mount point differently depending whether the container has a > >> >>> rootfs)? > >> >>> > >> >>> Yes I specified an image, a Docker image URI. > >> >>> > >> >>>> How > >> >>>> do you specify your 'container_path' (the mount point in the > >> container)? > >> >>> If > >> >>>> it is an > >> >>>> absolute path, we require that dir to be pre-existed. If it is a > >> relative > >> >>>> path, we will > >> >>>> mkdir for it. > >> >>> It is an absolute path, but it does not exists in image (this is the > >> >>> issue). Images are custom Docker images (images containing tools for > >> batch > >> >>> computing), and I want, for example, to mount some shared resources > >> (user > >> >>> home dir, common data, etc.) in the image. Of course those > >> directories do > >> >>> not pre-exists in container images as they are specific to the > >> environment. > >> >>> Requiring existence of the directory in the image is not issue as it > >> >>> prevents using any existing image from a repo. > >> >>> > >> >>> When using Docker containerizer it works fine, I can mount any > >> external > >> >>> storage in the container. > >> >>> > >> >>> Olivie > >> >>> > >> >>> > >> >>>> @Joshua, > >> >>>> Thank for posting your workaround on mesos. As I mentioned above, in > >> >>> 0.28.1 > >> >>>> or > >> >>>> older, we only mkdir for container_path which is relative path (not > >> >>>> starting with "/"). > >> >>>> Because if no rootfs specified for a mesos container, the container > >> >>> shares > >> >>>> the host > >> >>>> root filesystem. Obviously we don't want any random files to be > >> created > >> >>>> implicitly > >> >>>> on your host fs. > >> >>>> From mesos 0.29 (release by the end of this month), we will mkdir the > >> >>> mount > >> >>>> point in the container except for the command task case that specify > >> an > >> >>>> absolute > >> >>>> container_path without a rootfs. Because we simplify the mounting > >> logic, > >> >>> and > >> >>>> sandbox bind mount will only be done in container mount namespace > >> >>> instead of > >> >>>> host mount namespace (what we did before). Please keep tuned. > >> >>>> > >> >>>> Cheers, > >> >>>> Gilbert > >> >>>> > >> >>>> On Wed, May 18, 2016 at 8:14 AM, Joshua Cohen <[email protected]> > >> wrote: > >> >>>> > >> >>>>> Hi Olivier, > >> >>>>> > >> >>>>> I touched on this issue as part of > >> >>>>> https://issues.apache.org/jira/browse/MESOS-5229. It would be nice > >> if > >> >>>>> Mesos > >> >>>>> automatically created container mount points if they don't already > >> >>> exist. > >> >>>>> In the meantime, as a workaround for this, I've updated my > >> filesystem > >> >>>>> images to include the path (e.g. in Dockerfile, add `RUN mkdir -p > >> >>>>> /some/mount/point`). Not the best solution, but the only thing I've > >> >>> seen > >> >>>>> that works at the moment. > >> >>>>> > >> >>>>> Cheers, > >> >>>>> > >> >>>>> Joshua > >> >>>>> > >> >>>>> On Wed, May 18, 2016 at 7:36 AM, Guangya Liu <[email protected]> > >> >>> wrote: > >> >>>>>> It's pretty simple for you from scratch with source code > >> >>>>>> > >> >>>>>> > >> >>> > >> https://github.com/apache/mesos/blob/master/docs/getting-started.md#building-mesos > >> >>>>>> ;-) > >> >>>>>> > >> >>>>>> Thanks, > >> >>>>>> > >> >>>>>> Guangya > >> >>>>>> > >> >>>>>> On Wed, May 18, 2016 at 8:30 PM, Olivier Sallou < > >> >>> [email protected] > >> >>>>>> wrote: > >> >>>>>> > >> >>>>>>> On 05/18/2016 02:31 PM, Guangya Liu wrote: > >> >>>>>>>> Just saw that you are working with 0.28.1, the "docker volume > >> >>> driver" > >> >>>>>>> code > >> >>>>>>>> was not in 0.28.1, can you please have a try with mesos master > >> >>> branch > >> >>>>>> if > >> >>>>>>>> you are only doing some test? > >> >>>>>>> this is indeed test only for the moment. But I will have to > >> >>>>>>> recompile/install mesos :-( (I used packages for install). > >> >>>>>>> > >> >>>>>>> I will try when possible, but thanks for the hint. > >> >>>>>>>> Thanks, > >> >>>>>>>> > >> >>>>>>>> Guangya > >> >>>>>>>> > >> >>>>>>>> On Wed, May 18, 2016 at 8:28 PM, Guangya Liu <[email protected] > >> >>>>>> wrote: > >> >>>>>>>>> Hi Olivier, > >> >>>>>>>>> > >> >>>>>>>>> I think that you need to enable "docker volume isolator" if you > >> >>> want > >> >>>>>> use > >> >>>>>>>>> external storage with unified container I was writing a document > >> >>>>> here > >> >>>>>>>>> https://reviews.apache.org/r/47511/, perhaps you can have a try > >> >>>>>>> according > >> >>>>>>>>> to the document and post some comments there if you find any > >> >>> issues. > >> >>>>>>>>> Also you can patch mesos-execute here > >> >>>>>>> https://reviews.apache.org/r/46762/ to > >> >>>>>>>>> have a try with mesos-execute. > >> >>>>>>>>> > >> >>>>>>>>> Thanks, > >> >>>>>>>>> > >> >>>>>>>>> Guangya > >> >>>>>>>>> > >> >>>>>>>>> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou < > >> >>>>>>> [email protected]> > >> >>>>>>>>> wrote: > >> >>>>>>>>> > >> >>>>>>>>>> Answering (partially) to myself. > >> >>>>>>>>>> > >> >>>>>>>>>> I seems issue is container_path does not exists inside > >> >>> container. > >> >>>>> On > >> >>>>>>>>>> Docker, path is created and mounted. With pure mesos, > >> >>>>> container_path > >> >>>>>>>>>> must exists. > >> >>>>>>>>>> > >> >>>>>>>>>> mesos.proto says: "If the path is an absolute path, that path > >> >>> must > >> >>>>>>>>>> already exist." > >> >>>>>>>>>> > >> >>>>>>>>>> This is an issue however, using Docker images, the path I want > >> >>> to > >> >>>>>> mount > >> >>>>>>>>>> does not exists, and it cannot be modified "on the fly". > >> >>>>>>>>>> > >> >>>>>>>>>> Is there a workaround for this ? > >> >>>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>>> On 05/18/2016 12:24 PM, Olivier Sallou wrote: > >> >>>>>>>>>>> Hi, > >> >>>>>>>>>>> I am trying unified containerizer on a single server > >> >>>>> (master/slave) > >> >>>>>> on > >> >>>>>>>>>>> mesos 0.28.1, to switch from docker containerizer to > >> >>> mesos+docker > >> >>>>>>> image > >> >>>>>>>>>>> container. > >> >>>>>>>>>>> > >> >>>>>>>>>>> I have setup slave config as suggested in documentation: > >> >>>>>>>>>>> > >> >>>>>>>>>>> containerizers=docker,mesos > >> >>>>>>>>>>> image_providers=docker \ > >> >>>>>>>>>>> isolation=filesystem/linux,docker/runtime > >> >>>>>>>>>>> > >> >>>>>>>>>>> However, when I execute my task with a volume I have an error: > >> >>>>>>>>>>> > >> >>>>>>>>>>> .... > >> >>>>>>>>>>> + mount -n --rbind > >> >>>>>>>>>>> > >> >>> > >> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f > >> >>> > >> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs > >> >>>>>>>>>>> + mount -n --rbind > >> >>>>>>>>>>> > >> >>> > >> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task > >> >>> > >> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data > >> >>>>>>>>>>> mount: mount point > >> >>>>>>>>>>> > >> >>> > >> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data > >> >>>>>>>>>>> does not exist > >> >>>>>>>>>>> Failed to execute a preparation shell command > >> >>>>>>>>>>> > >> >>>>>>>>>>> Then, my task switches to FAILED. > >> >>>>>>>>>>> > >> >>>>>>>>>>> I define a local volume to bind mount in my "container" > >> >>>>>>>>>>> > >> >>> > >> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task > >> >>>>>>>>>>> => /mnt/god-data > >> >>>>>>>>>>> My directory exists on local server. > >> >>>>>>>>>>> In mesos UI, I can see the .rootfs directory along stdout and > >> >>>>> stderr > >> >>>>>>>>>>> files, and inside .rootfs, I can see /mnt/god-data (empty). > >> >>>>>>>>>>> > >> >>>>>>>>>>> Running the same using Docker containerizer instead of mesos > >> >>>>>>>>>>> containerizer (with a Docker image) works fine. > >> >>>>>>>>>>> > >> >>>>>>>>>>> It seems it fails to mount my local directory in the > >> >>> container. > >> >>>>> Any > >> >>>>>>> idea > >> >>>>>>>>>>> of what is going wrong or how to debug this? > >> >>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>>> Thanks > >> >>>>>>>>>>> > >> >>>>>>>>>> -- > >> >>>>>>>>>> Olivier Sallou > >> >>>>>>>>>> IRISA / University of Rennes 1 > >> >>>>>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE > >> >>>>>>>>>> Tel: 02.99.84.71.95 > >> >>>>>>>>>> > >> >>>>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) > >> >>>>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D > >> >>>>> 8438 > >> >>>>>>> -- > >> >>>>>>> Olivier Sallou > >> >>>>>>> IRISA / University of Rennes 1 > >> >>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE > >> >>>>>>> Tel: 02.99.84.71.95 > >> >>>>>>> > >> >>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) > >> >>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D > >> >>> 8438 > >> > >> -- > >> Olivier Sallou > >> IRISA / University of Rennes 1 > >> Campus de Beaulieu, 35000 RENNES - FRANCE > >> Tel: 02.99.84.71.95 > >> > >> gpg key id: 4096R/326D8438 (keyring.debian.org) > >> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 > >> > >> > > >
