Thanks Olivier, I can reproduce this issue now and still checking what is
wrong.
What I did is as following:
1) Check out code with tag of 0.28.1
2) update mesos-execute to add a host path volume
diff --git a/src/cli/execute.cpp b/src/cli/execute.cpp
index 81a0388..0ff913c 100644
--- a/src/cli/execute.cpp
+++ b/src/cli/execute.cpp
@@ -72,6 +72,8 @@ using mesos::v1::TaskID;
using mesos::v1::TaskInfo;
using mesos::v1::TaskState;
using mesos::v1::TaskStatus;
+using mesos::v1::Volume;
+using mesos::v1::Parameters;
using mesos::v1::scheduler::Call;
using mesos::v1::scheduler::Event;
@@ -572,6 +574,12 @@ private:
}
}
+ Volume* volume1 = containerInfo.add_volumes();
+ volume1->set_container_path("/tmp/abcd");
+ volume1->set_mode(Volume::RW);
+ volume1->set_host_path("/root/convoy");
+ cout << "Add Voume 1" << endl;
+
return containerInfo;
} else if (containerizer == "docker") {
// 'docker' containerizer only supports 'docker' images.
3) launch a task with docker image, task failed.
4) Check sandbox:
+ /root/src/mesos/m1/mesos/build/src/mesos-containerizer mount --help=false
--operation=make-rslave --path=/
+ grep -E /tmp/mesos/.+ /proc/self/mountinfo
+ grep -v 3239aafc-78d8-4f70-81e5-f32fb3733339
+ cut+ -d -f5
xargs --no-run-if-empty umount -l
+ mount -n --rbind
/tmp/mesos/provisioner/containers/3239aafc-78d8-4f70-81e5-f32fb3733339/backends/copy/rootfses/5e8bf3fa-53b1-4bd5-bb3d-525ddc7900b6
/tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs
+ mount -n --rbind /root/convoy
/tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs/tmp/abcd
mount: mount point
/tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs/tmp/abcd
does not exist
Failed to execute a preparation shell command
Will check more for this.
Thanks,
Guangya
On Mon, May 23, 2016 at 3:35 PM, Olivier Sallou <[email protected]>
wrote:
>
>
> On 05/23/2016 09:33 AM, Olivier Sallou wrote:
> >
> > On 05/20/2016 03:26 PM, Guangya Liu wrote:
> >> Since you are using docker image which means that your container will
> have
> >> rootfs, so it is not required to have the absolute path exist, the linux
> >> file system isolator will help create the path automatically
> >>
> https://github.com/apache/mesos/blob/0.28.x/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp#L390-L402
> >>
> >> Can you please share your framework? How did you set the volume part in
> >> your framework?
> > @Guangya
> >
> > I use Python API.
> >
> > Here is related code:
> >
> > ....
> > # Define container volumes
> > for v in job['container']['volumes']:
> > volume = container.volumes.add()
> > volume.container_path = v['mount']
> > volume.host_path = v['path']
> > if v['acl'] == 'rw':
> > volume.mode = 1 # mesos_pb2.Volume.Mode.RW
> > else:
> > volume.mode = 2 # mesos_pb2.Volume.Mode.RO
> >
> > => In my test case, I add 2 volumes from a host shared directory,
> > mounted in container as /mnt/go-docker and /mnt/god-data.
> >
> > ...
> > # Define docker image and network
> > docker = mesos_pb2.ContainerInfo.MesosInfo()
> > docker.image.type = 2 # Docker
> > docker.image.docker.name ='centos:latest'
> > # Request an IP from a network module
> > network_info = container.network_infos.add()
> > network_info_name = 'sampletest'
> > # Get an IP V4 address
> > ip_address = network_info.ip_addresses.add()
> > ip_address.protocol = 1
> > # The network group to join
> > group = network_info.groups.append(network_info_name)
> > port_list = [22]
> > if port_list:
> > for port in port_list:
> > job['container']['port_mapping'].append({'host':
> > port, 'container': port})
> > container.mesos.MergeFrom(docker)
> >
> > It results in error message:
> >
> > + mount -n --rbind
> >
> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> > + mount -n --rbind
> >
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > mount: mount point
> >
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > does not exist
> > Failed to execute a preparation shell command
> >
> >
> > We can see the .rootfs, but mnt/god-data under .rootfs fails. Local
> > directory exists, it does not pre-exists in container. What is strange
> > is , if I look in mesos task dir, .rootfs/mnt/go-data, directory is
> present.
> >
> > Or, is the error message (.rootfs/mnt/god-data does not exist) simply a
> > warning, and it creates it, and final error (Failed to execute a
> > preparation shell command) not related (and unclear...)
> Additional info: command to execute in container is located in one of
> mounted volume.
> >
> > Olivier
> >> Thanks,
> >>
> >> Guangya
> >>
> >> On Fri, May 20, 2016 at 4:54 AM, Olivier Sallou <
> [email protected]>
> >> wrote:
> >>
> >>> ----- Mail original -----
> >>>> De: "Gilbert Song" <[email protected]>
> >>>> À: "dev" <[email protected]>
> >>>> Envoyé: Jeudi 19 Mai 2016 01:57:16
> >>>> Objet: Re: volume / mount point error with Unified Containerizer
> >>>>
> >>>> @Olivier,
> >>>> In mesos 0.28.1, you are supposed to be able bind mount a volume from
> >>>> the host into the mesos container. Did you specify a docker image (we
> >>>> determine
> >>>> the mount point differently depending whether the container has a
> >>> rootfs)?
> >>>
> >>> Yes I specified an image, a Docker image URI.
> >>>
> >>>> How
> >>>> do you specify your 'container_path' (the mount point in the
> container)?
> >>> If
> >>>> it is an
> >>>> absolute path, we require that dir to be pre-existed. If it is a
> relative
> >>>> path, we will
> >>>> mkdir for it.
> >>> It is an absolute path, but it does not exists in image (this is the
> >>> issue). Images are custom Docker images (images containing tools for
> batch
> >>> computing), and I want, for example, to mount some shared resources
> (user
> >>> home dir, common data, etc.) in the image. Of course those directories
> do
> >>> not pre-exists in container images as they are specific to the
> environment.
> >>> Requiring existence of the directory in the image is not issue as it
> >>> prevents using any existing image from a repo.
> >>>
> >>> When using Docker containerizer it works fine, I can mount any external
> >>> storage in the container.
> >>>
> >>> Olivie
> >>>
> >>>
> >>>> @Joshua,
> >>>> Thank for posting your workaround on mesos. As I mentioned above, in
> >>> 0.28.1
> >>>> or
> >>>> older, we only mkdir for container_path which is relative path (not
> >>>> starting with "/").
> >>>> Because if no rootfs specified for a mesos container, the container
> >>> shares
> >>>> the host
> >>>> root filesystem. Obviously we don't want any random files to be
> created
> >>>> implicitly
> >>>> on your host fs.
> >>>> From mesos 0.29 (release by the end of this month), we will mkdir the
> >>> mount
> >>>> point in the container except for the command task case that specify
> an
> >>>> absolute
> >>>> container_path without a rootfs. Because we simplify the mounting
> logic,
> >>> and
> >>>> sandbox bind mount will only be done in container mount namespace
> >>> instead of
> >>>> host mount namespace (what we did before). Please keep tuned.
> >>>>
> >>>> Cheers,
> >>>> Gilbert
> >>>>
> >>>> On Wed, May 18, 2016 at 8:14 AM, Joshua Cohen <[email protected]>
> wrote:
> >>>>
> >>>>> Hi Olivier,
> >>>>>
> >>>>> I touched on this issue as part of
> >>>>> https://issues.apache.org/jira/browse/MESOS-5229. It would be nice
> if
> >>>>> Mesos
> >>>>> automatically created container mount points if they don't already
> >>> exist.
> >>>>> In the meantime, as a workaround for this, I've updated my filesystem
> >>>>> images to include the path (e.g. in Dockerfile, add `RUN mkdir -p
> >>>>> /some/mount/point`). Not the best solution, but the only thing I've
> >>> seen
> >>>>> that works at the moment.
> >>>>>
> >>>>> Cheers,
> >>>>>
> >>>>> Joshua
> >>>>>
> >>>>> On Wed, May 18, 2016 at 7:36 AM, Guangya Liu <[email protected]>
> >>> wrote:
> >>>>>> It's pretty simple for you from scratch with source code
> >>>>>>
> >>>>>>
> >>>
> https://github.com/apache/mesos/blob/master/docs/getting-started.md#building-mesos
> >>>>>> ;-)
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Guangya
> >>>>>>
> >>>>>> On Wed, May 18, 2016 at 8:30 PM, Olivier Sallou <
> >>> [email protected]
> >>>>>> wrote:
> >>>>>>
> >>>>>>> On 05/18/2016 02:31 PM, Guangya Liu wrote:
> >>>>>>>> Just saw that you are working with 0.28.1, the "docker volume
> >>> driver"
> >>>>>>> code
> >>>>>>>> was not in 0.28.1, can you please have a try with mesos master
> >>> branch
> >>>>>> if
> >>>>>>>> you are only doing some test?
> >>>>>>> this is indeed test only for the moment. But I will have to
> >>>>>>> recompile/install mesos :-( (I used packages for install).
> >>>>>>>
> >>>>>>> I will try when possible, but thanks for the hint.
> >>>>>>>> Thanks,
> >>>>>>>>
> >>>>>>>> Guangya
> >>>>>>>>
> >>>>>>>> On Wed, May 18, 2016 at 8:28 PM, Guangya Liu <[email protected]
> >>>>>> wrote:
> >>>>>>>>> Hi Olivier,
> >>>>>>>>>
> >>>>>>>>> I think that you need to enable "docker volume isolator" if you
> >>> want
> >>>>>> use
> >>>>>>>>> external storage with unified container I was writing a document
> >>>>> here
> >>>>>>>>> https://reviews.apache.org/r/47511/, perhaps you can have a try
> >>>>>>> according
> >>>>>>>>> to the document and post some comments there if you find any
> >>> issues.
> >>>>>>>>> Also you can patch mesos-execute here
> >>>>>>> https://reviews.apache.org/r/46762/ to
> >>>>>>>>> have a try with mesos-execute.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>>
> >>>>>>>>> Guangya
> >>>>>>>>>
> >>>>>>>>> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou <
> >>>>>>> [email protected]>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Answering (partially) to myself.
> >>>>>>>>>>
> >>>>>>>>>> I seems issue is container_path does not exists inside
> >>> container.
> >>>>> On
> >>>>>>>>>> Docker, path is created and mounted. With pure mesos,
> >>>>> container_path
> >>>>>>>>>> must exists.
> >>>>>>>>>>
> >>>>>>>>>> mesos.proto says: "If the path is an absolute path, that path
> >>> must
> >>>>>>>>>> already exist."
> >>>>>>>>>>
> >>>>>>>>>> This is an issue however, using Docker images, the path I want
> >>> to
> >>>>>> mount
> >>>>>>>>>> does not exists, and it cannot be modified "on the fly".
> >>>>>>>>>>
> >>>>>>>>>> Is there a workaround for this ?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 05/18/2016 12:24 PM, Olivier Sallou wrote:
> >>>>>>>>>>> Hi,
> >>>>>>>>>>> I am trying unified containerizer on a single server
> >>>>> (master/slave)
> >>>>>> on
> >>>>>>>>>>> mesos 0.28.1, to switch from docker containerizer to
> >>> mesos+docker
> >>>>>>> image
> >>>>>>>>>>> container.
> >>>>>>>>>>>
> >>>>>>>>>>> I have setup slave config as suggested in documentation:
> >>>>>>>>>>>
> >>>>>>>>>>> containerizers=docker,mesos
> >>>>>>>>>>> image_providers=docker \
> >>>>>>>>>>> isolation=filesystem/linux,docker/runtime
> >>>>>>>>>>>
> >>>>>>>>>>> However, when I execute my task with a volume I have an error:
> >>>>>>>>>>>
> >>>>>>>>>>> ....
> >>>>>>>>>>> + mount -n --rbind
> >>>>>>>>>>>
> >>>
> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> >>>
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> >>>>>>>>>>> + mount -n --rbind
> >>>>>>>>>>>
> >>>
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> >>>
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> >>>>>>>>>>> mount: mount point
> >>>>>>>>>>>
> >>>
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> >>>>>>>>>>> does not exist
> >>>>>>>>>>> Failed to execute a preparation shell command
> >>>>>>>>>>>
> >>>>>>>>>>> Then, my task switches to FAILED.
> >>>>>>>>>>>
> >>>>>>>>>>> I define a local volume to bind mount in my "container"
> >>>>>>>>>>>
> >>>
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> >>>>>>>>>>> => /mnt/god-data
> >>>>>>>>>>> My directory exists on local server.
> >>>>>>>>>>> In mesos UI, I can see the .rootfs directory along stdout and
> >>>>> stderr
> >>>>>>>>>>> files, and inside .rootfs, I can see /mnt/god-data (empty).
> >>>>>>>>>>>
> >>>>>>>>>>> Running the same using Docker containerizer instead of mesos
> >>>>>>>>>>> containerizer (with a Docker image) works fine.
> >>>>>>>>>>>
> >>>>>>>>>>> It seems it fails to mount my local directory in the
> >>> container.
> >>>>> Any
> >>>>>>> idea
> >>>>>>>>>>> of what is going wrong or how to debug this?
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks
> >>>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Olivier Sallou
> >>>>>>>>>> IRISA / University of Rennes 1
> >>>>>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
> >>>>>>>>>> Tel: 02.99.84.71.95
> >>>>>>>>>>
> >>>>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
> >>>>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D
> >>>>> 8438
> >>>>>>> --
> >>>>>>> Olivier Sallou
> >>>>>>> IRISA / University of Rennes 1
> >>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
> >>>>>>> Tel: 02.99.84.71.95
> >>>>>>>
> >>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
> >>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D
> >>> 8438
>
> --
> Olivier Sallou
> IRISA / University of Rennes 1
> Campus de Beaulieu, 35000 RENNES - FRANCE
> Tel: 02.99.84.71.95
>
> gpg key id: 4096R/326D8438 (keyring.debian.org)
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>
>
