Dear Shuai and Tim, any news about the patch for MESOS-4743? Can I ask you when it will be officially released?
Thank you in advance Best regards, Marica Il giorno 24/feb/2016, alle ore 11:23, Shuai Lin <[email protected]> ha scritto: > ping @Tim, I think this bug also affects > https://issues.apache.org/jira/browse/MESOS-4743 . > > > > On Wed, Jan 20, 2016 at 10:20 PM, Shuai Lin <[email protected]> wrote: > The testing of this case requires to build a docker image for mesos-slave, so > it seems not practical to add a test case for it in the mesos tests. > > Anyway, here is the scripts I use for testing this issue: > https://gist.github.com/lins05/14455e92f37e91fd46ff > > On Wed, Jan 20, 2016 at 10:30 AM, Shuai Lin <[email protected]> wrote: > Hi Tim, > > The review is here: https://reviews.apache.org/r/42390/ , would you please > take a look? > > Regards, > Shuai > > On Sat, Jan 9, 2016 at 9:42 AM, Shuai Lin <[email protected]> wrote: > Hi Maria and Tim, > > I'm setting up a test case for this scenario that would fail, after which > I'll begin on fixing it. > > Is it feasible to include the fixing in the new release? > > I'm not sure, does 0.27 have an estimated release date now? > > > Regards, > Shuai > > On Sat, Jan 9, 2016 at 1:11 AM, Timothy Chen <[email protected]> wrote: > I can shepherd no problem. > > Tim > > On Dec 25, 2015, at 4:32 PM, Shuai Lin <[email protected]> wrote: > >> I'll work on it. @Tim could you shepherd it? >> >> On Sat, Dec 26, 2015 at 2:49 AM, Marica Antonacci >> <[email protected]> wrote: >> Hi Tim and Shuai, >> >> thank you very much for your reply. I have opened a JIRA issue on this: >> https://issues.apache.org/jira/browse/MESOS-4249 >> I hope it will be patched soon :) >> >> Best regards, >> Marica >> >> >> Il giorno 24/dic/2015, alle ore 17:54, Tim Chen <[email protected]> ha >> scritto: >> >>> Hi Marica/Shuai, >>> >>> Sorry haven't been able to spend the time to repro, but looks like Shuai >>> confirmed it. >>> >>> Can one of you file a JIRA? >>> >>> Thanks! >>> >>> Tim >>> >>> On Thu, Dec 24, 2015 at 6:16 AM, Shuai Lin <[email protected]> wrote: >>> Hi Marica, >>> >>> I can reproduce the problem exactly as you described in the first email of >>> this thread. Without `MESOS_DOCKER_MESOS_IMAGE` environment variable set, >>> the fetcher works just fine; With it, the fetcher steps seems skipped. This >>> looks like a bug to me. >>> >>> Regards, >>> Shuai >>> >>> On Tue, Dec 22, 2015 at 7:41 PM, Marica Antonacci >>> <[email protected]> wrote: >>> Dear all, >>> >>> I have not solved this issue yet. Please, can anyone run the same test and >>> let me know if the fetcher is correctly invoked? >>> The test is really simple, just try to start a dockerized app (see json >>> definition file below) through marathon on a mesos slave running in a >>> docker container started with the option —docker_mesos_image=<mesos slave >>> image>. >>> I would appreciate very much any feedback. >>> >>> Sample Marathon app: >>> { >>> "id": "test-app", >>> "container": { >>> "type": "DOCKER", >>> "docker": { >>> "image": "libmesos/ubuntu" >>> } >>> }, >>> "cpus": 1, >>> "mem": 512, >>> "uris": [ >>> "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv"; >>> ], >>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T; done" >>> } >>> >>> Docker run command to start dockerized mesos slave: >>> >>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e >>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e >>> MESOS_CONTAINERIZERS=docker,mesos -e >>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e >>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v >>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name >>> slave --net host --privileged --pid host mesos-slave >>> >>> Thank you very much in advance! >>> Best regards, >>> Marica >>> >>> Il giorno 19/dic/2015, alle ore 19:32, Marica Antonacci >>> <[email protected]> ha scritto: >>> >>>> Dear Tim, >>>> >>>> I have collected some information from my test environment, starting the >>>> slave container with and without the —docker_mesos_image startup flag. >>>> Please let me know if you need further input. Thank you very much for your >>>> support! >>>> >>>> Using the flag —docker_mesos_image: >>>> >>>> root@mesos-slave:~# docker ps >>>> CONTAINER ID IMAGE COMMAND CREATED >>>> STATUS PORTS NAMES >>>> b30cea22a07c libmesos/ubuntu "/bin/sh -c 'cd $MESO" 2 minutes >>>> ago Up 2 minutes >>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d >>>> da9c78ec5727 mesos-slave "/bin/sh -c '/usr/lib" 2 minutes >>>> ago Up 2 minutes >>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor >>>> 150f78fbf327 mesos-slave "/entrypoint.sh /usr/" 3 minutes >>>> ago Up 3 minutes slave >>>> >>>> root@mesos-slave:~# docker logs slave >>>> I1219 18:03:38.308544 19476 slave.cpp:1294] Got assigned task >>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework >>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>> I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task >>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework >>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>> I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown >>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' >>>> to user 'root' >>>> I1219 18:03:38.327221 19476 slave.cpp:4999] Launching executor >>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.1; >>>> mem(*):32 in work directory >>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' >>>> I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing task >>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor >>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework >>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>> I1219 18:03:38.335907 19480 docker.cpp:762] Starting container >>>> 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task >>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor >>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework >>>> '246b272b-d649-47c0-88ca-6b1ff35f437a-0000' >>>> I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid 19590 to >>>> '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pids/forked.pid' >>>> I1219 18:03:40.189172 19479 slave.cpp:2405] Got registration for executor >>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework >>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from >>>> executor(1)@90.147.170.246:39381 >>>> I1219 18:03:40.191141 19479 docker.cpp:1012] Ignoring updating container >>>> 'd965f59b-cc1a-4081-95d2-f3370214c84d' with resources passed to update is >>>> identical to existing resources >>>> I1219 18:03:40.192512 19479 slave.cpp:1793] Sending queued task >>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' to executor >>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework >>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at >>>> executor(1)@90.147.170.246:39381 >>>> I1219 18:03:42.425606 19483 slave.cpp:2762] Handling status update >>>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task >>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from >>>> executor(1)@90.147.170.246:39381 >>>> I1219 18:03:42.426415 19483 status_update_manager.cpp:322] Received status >>>> update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task >>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>> I1219 18:03:42.427276 19483 status_update_manager.cpp:826] Checkpointing >>>> UPDATE for status update TASK_RUNNING (UUID: >>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task >>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>> I1219 18:03:42.521395 19483 slave.cpp:3087] Forwarding the update >>>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task >>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to [email protected]:5050 >>>> I1219 18:03:42.521852 19483 slave.cpp:3011] Sending acknowledgement for >>>> status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) >>>> for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to >>>> executor(1)@90.147.170.246:39381 >>>> I1219 18:03:42.576373 19481 status_update_manager.cpp:394] Received status >>>> update acknowledgement (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for >>>> task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>> I1219 18:03:42.577263 19481 status_update_manager.cpp:826] Checkpointing >>>> ACK for status update TASK_RUNNING (UUID: >>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task >>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>> >>>> root@mesos-slave:~# docker logs >>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor >>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d" >>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" >>>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" >>>> --logbufsecs="0" --logging_level="INFO" >>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false" >>>> --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d" >>>> --stop_timeout="0ns" >>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d" >>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" >>>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" >>>> --logbufsecs="0" --logging_level="INFO" >>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false" >>>> --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d" >>>> --stop_timeout="0ns" >>>> I1219 18:03:40.177598 6 exec.cpp:136] Version: 0.26.0 >>>> I1219 18:03:40.192060 10 exec.cpp:210] Executor registered on slave >>>> db70e09f-f39d-491c-8480-73d9858c140b-S0 >>>> Registered docker executor on 90.147.170.246 >>>> Starting task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 >>>> >>>> root@mesos-slave:~# docker exec -it >>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor >>>> bash >>>> root@mesos-slave:/# ls -R /tmp/ >>>> hsperfdata_root/ mesos/ >>>> root@mesos-slave:/# ls -R /tmp/mesos/ >>>> /tmp/mesos/: >>>> slaves >>>> >>>> /tmp/mesos/slaves: >>>> db70e09f-f39d-491c-8480-73d9858c140b-S0 >>>> >>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0: >>>> frameworks >>>> >>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks: >>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>> >>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000: >>>> executors >>>> >>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors: >>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 >>>> >>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0: >>>> runs >>>> >>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs: >>>> d965f59b-cc1a-4081-95d2-f3370214c84d >>>> >>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d: >>>> stderr stdout >>>> >>>> >>>> root@mesos-slave:~# docker exec -it >>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d >>>> bash >>>> root@mesos-slave:/# env >>>> HOSTNAME=mesos-slave >>>> HOST=90.147.170.246 >>>> PORT0=31220 >>>> PORT_10000=31220 >>>> MESOS_TASK_ID=test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 >>>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin >>>> PWD=/ >>>> PORTS=31220 >>>> MESOS_CONTAINER_NAME=mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d >>>> SHLVL=1 >>>> HOME=/ >>>> MARATHON_APP_ID=/test-app >>>> MARATHON_APP_DOCKER_IMAGE=libmesos/ubuntu >>>> MARATHON_APP_VERSION=2015-12-19T18:03:37.542Z >>>> PORT=31220 >>>> MESOS_SANDBOX=/mnt/mesos/sandbox >>>> _=/usr/bin/env >>>> root@mesos-slave:/# ls -R $MESOS_SANDBOX >>>> /mnt/mesos/sandbox: >>>> stderr stdout >>>> >>>> >>>> root@mesos-slave:/# ls /var/log/mesos* >>>> /var/log/mesos-slave.INFO >>>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-182512.20647 >>>> >>>> /var/log/mesos: >>>> >>>> >>>> Disabling the flag —docker_mesos_image the fetcher is called, log is >>>> created, the file is downloaded in the sandbox: >>>> >>>> root@mesos-slave:~# docker exec -it slave bash >>>> root@mesos-slave:/# >>>> root@mesos-slave:/# >>>> root@mesos-slave:/# >>>> root@mesos-slave:/# ls /var/log/mesos* >>>> /var/log/mesos-fetcher.INFO >>>> /var/log/mesos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-181621.20157 >>>> /var/log/mesos-slave.INFO >>>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-181612.20124 >>>> >>>> /var/log/mesos: >>>> root@mesos-slave:/# >>>> root@mesos-slave:/# cat /var/log/mesos-fetcher.INFO >>>> Log file created at: 2015/12/19 18:16:21 >>>> Running on machine: mesos-slave >>>> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg >>>> I1219 18:16:21.127075 20157 logging.cpp:172] INFO level logging started! >>>> I1219 18:16:21.127499 20157 fetcher.cpp:422] Fetcher Info: >>>> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"value":"http:\/\/www.stat.cmu.edu\/~cshalizi\/402\/lectures\/16-glm-practicals\/snoqualmie.csv"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1\/frameworks\/246b272b-d649-47c0-88ca-6b1ff35f437a-0000\/executors\/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0\/runs\/1862ced9-3659-4429-8bc6-3f47f58970b4"} >>>> I1219 18:16:21.129547 20157 fetcher.cpp:377] Fetching URI >>>> 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' >>>> I1219 18:16:21.129576 20157 fetcher.cpp:248] Fetching directly into the >>>> sandbox directory >>>> I1219 18:16:21.129607 20157 fetcher.cpp:185] Fetching URI >>>> 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' >>>> I1219 18:16:21.129636 20157 fetcher.cpp:132] Downloading resource from >>>> 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' >>>> to >>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv' >>>> I1219 18:16:23.221889 20157 fetcher.cpp:454] Fetched >>>> 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' >>>> to >>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv' >>>> root@mesos-slave:/# >>>> >>>> root@mesos-slave:~# docker exec -it >>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8bc6-3f47f58970b4 >>>> bash >>>> root@mesos-slave:/# >>>> root@mesos-slave:/# >>>> root@mesos-slave:/# ls /mnt/mesos/sandbox/ >>>> snoqualmie.csv stderr stdout >>>> root@mesos-slave:/# >>>> >>>> >>>> Best Regards, >>>> Marica >>>> >>>> >>>> >>>> Il giorno 19/dic/2015, alle ore 02:12, Tim Chen <[email protected]> ha >>>> scritto: >>>> >>>>> HI Marica, >>>>> >>>>> Did you see the fetcher invoked at all from the Slave logs? Doesn't seem >>>>> possible we don't pass down the URI flags and if the fetcher failed the >>>>> container launch should have failed too. >>>>> >>>>> Also another possible situation is that web UI not really showing the >>>>> exact content of the sandbox, can you actually go into the directory and >>>>> see if the files are there? >>>>> >>>>> Thanks, >>>>> >>>>> Tim >>>>> >>>>> On Fri, Dec 18, 2015 at 4:23 PM, Marica Antonacci >>>>> <[email protected]> wrote: >>>>> Hi Tim, >>>>> >>>>> looking at the sandbox I can see only the stderr and stout file (see the >>>>> attached screenshot). If I remove —docker_mesos_image (and therefore the >>>>> executor is run inside the slave container) the file specified in the URI >>>>> field is shown in the sandbox. >>>>> Did you verify that the fetcher is called when using the >>>>> —docker_mesos_image flag ? >>>>> >>>>> Thanks a lot for your feedback >>>>> Marica >>>>> >>>>> <Schermata 2015-12-18 alle 10.15.58.png> >>>>> >>>>> Il giorno 19/dic/2015, alle ore 00:25, Tim Chen <[email protected]> ha >>>>> scritto: >>>>> >>>>>> Hi Marica, >>>>>> >>>>>> It should work as we fetch all the files before we launch the executor >>>>>> and place them in the sandbox, and we mount the sandbox into that >>>>>> container as well. >>>>>> >>>>>> How did you verify that the file is not downloaded? >>>>>> >>>>>> Tim >>>>>> >>>>>> On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci >>>>>> <[email protected]> wrote: >>>>>> Hi Grzegorz, >>>>>> >>>>>> I’m using this command line for docker run >>>>>> >>>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e >>>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e >>>>>> MESOS_CONTAINERIZERS=docker,mesos -e >>>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e >>>>>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v >>>>>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos >>>>>> --name slave --net host --privileged --pid host mesos-slave >>>>>> >>>>>> where mesos-slave is the image built from the docker file in this repo >>>>>> https://github.com/maricaantonacci/mesos-slave-dev >>>>>> >>>>>> I have tested successfully the deployment of dockerized applications >>>>>> through Marathon and dockerized jobs through Chronos and also the >>>>>> recovery seems to work fine with the flag docker_mesos_image. What is >>>>>> not working for me is the fetcher: it seems that when the executor is >>>>>> launched as separate container (thanks to the flag docker_mesos_image) >>>>>> the information about the URIs to be downloaded is lost…I hope someone >>>>>> can help to understand if this a bug or I’ m missing something. >>>>>> >>>>>> Cheers, >>>>>> Marica >>>>>> >>>>>> >>>>>> >>>>>> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk >>>>>> <[email protected]> ha scritto: >>>>>> >>>>>>> I've tried to use this flag, but cannot really run any container when >>>>>>> this flag is set. >>>>>>> I've raised this issue here: >>>>>>> https://www.mail-archive.com/[email protected]/msg04975.html and >>>>>>> here: >>>>>>> https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351 >>>>>>> but sadly no one was able to help me... >>>>>>> >>>>>>> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci >>>>>>> <[email protected]> napisał: >>>>>>> OK, the problem I spotted is related to the usage of the flag >>>>>>> —docker_mesos_image that allows the executor to >>>>>>> >>>>>>> >>>>>>> --docker_mesos_image=VALUE The docker image used to launch this >>>>>>> mesos slave instance. If an image is specified, the docker >>>>>>> containerizer assumes the slave is running in a docker container, and >>>>>>> launches executors with docker containers in order to recover them when >>>>>>> the slave restarts and recovers. >>>>>>> Has anyone used this flag and tested the behavior of the fetcher? >>>>>>> >>>>>>> Thank you >>>>>>> Marica >>>>>>> >>>>>>> >>>>>>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <[email protected]> ha >>>>>>> scritto: >>>>>>> >>>>>>>> no docker_mesos_image flag in my docker run, and the docker image is >>>>>>>> build by myself. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci >>>>>>>> <[email protected]>: >>>>>>> >>>>>>>> Yes, I did check inside the container and the csv file was not >>>>>>>> downloaded as shown also by the app details (see the screenshot below). >>>>>>>> >>>>>>>> Are you running your slave with the --docker_mesos_image flag? Can you >>>>>>>> please provide me the docker run command you are using to run your >>>>>>>> dockerized slave? >>>>>>>> >>>>>>>> Thank you very much >>>>>>> >>>>>>>> Marica >>>>>>>> >>>>>>>> >>>>>>>> <Schermata 2015-12-18 alle 10.15.58.png> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <[email protected]> >>>>>>>> ha scritto: >>>>>>>> >>>>>>>>> Hi Marica, >>>>>>>>> >>>>>>>>> use your test-app json, i can run it correctly, the csv is truely >>>>>>>>> download by mesos slave. please check mesos-master:5050 to check the >>>>>>>>> task detail download files. >>>>>>>>> >>>>>>>>> you describe the app container why not found the csv, because the csv >>>>>>>>> is download in slave container's folder, not in app container. so if >>>>>>>>> you run >>>>>>>>> >>>>>>>>> cd $MESOS_SANDBOX; >>>>>>>>> >>>>>>>>> the folder in app container is default value: >>>>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox >>>>>>>>> >>>>>>>>> but in real world, the sandbox is in slave container, not in app >>>>>>>>> container. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci >>>>>>>>> <[email protected]>: >>>>>>>>> Thank you very much, >>>>>>>>> >>>>>>>>> I’m using a sample application definition file, just for testing >>>>>>>>> purpose: >>>>>>>>> >>>>>>>>> { >>>>>>>>> "id": "test-app", >>>>>>>>> "container": { >>>>>>>>> "type": "DOCKER", >>>>>>>>> "docker": { >>>>>>>>> "image": "libmesos/ubuntu" >>>>>>>>> } >>>>>>>>> }, >>>>>>>>> "cpus": 1, >>>>>>>>> "mem": 512, >>>>>>>>> "uris": [ >>>>>>>>> "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv"; >>>>>>>>> ], >>>>>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T; >>>>>>>>> done" >>>>>>>>> } >>>>>>>>> >>>>>>>>> Here is the docker run command line: >>>>>>>>> >>>>>>>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> >>>>>>>>> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos >>>>>>>>> -e MESOS_CONTAINERIZERS=docker,mesos \ >>>>>>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e >>>>>>>>> MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave >>>>>>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v >>>>>>>>> /var/run/docker.sock:/var/run/docker.sock --name slave --net host >>>>>>>>> --privileged --pid host mesos-slave >>>>>>>>> >>>>>>>>> >>>>>>>>> As already mentioned, if I remove the environment variable >>>>>>>>> MESOS_docker_mesos_image the fetcher works fine and I can see the >>>>>>>>> file snoqualmie.csv inside the sandbox. >>>>>>>>> >>>>>>>>> Thank you again! I’m looking forward to hearing about your outcomes. >>>>>>>>> Best regards, >>>>>>>>> Marica >>>>>>>>> >>>>>>>>> >>>>>>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <[email protected]> >>>>>>>>> ha scritto: >>>>>>>>> >>>>>>>>>> Hi Marica, >>>>>>>>>> >>>>>>>>>> Could you please give a sample marathon json, i can test it asap. >>>>>>>>>> >>>>>>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci >>>>>>>>>> <[email protected]>: >>>>>>>>>> No, using the socket: >>>>>>>>>> >>>>>>>>>> -v /var/run/docker.sock:/var/run/docker.sock >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <[email protected]> >>>>>>>>>> ha scritto: >>>>>>>>>> >>>>>>>>>>> docker in docker mode? >>>>>>>>>>> >>>>>>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci >>>>>>>>>>> <[email protected]>: >>>>>>>>>>> Dear all, >>>>>>>>>>> >>>>>>>>>>> I'm testing the URIs fetching mechanism for both Marathon >>>>>>>>>>> applications and Chronos jobs and I have found that if the slave is >>>>>>>>>>> running inside a docker container (using docker_mesos_image startup >>>>>>>>>>> flag) and you submit the deployment of a dockerized application or >>>>>>>>>>> job the fetcher step is not performed. On the other hand, if I >>>>>>>>>>> request the deployment of a non-dockerized application, the URIs >>>>>>>>>>> are correctly fetched. Moreover, if I don’t provide the >>>>>>>>>>> docker_mesos_image flag, the fetcher works fine again for both >>>>>>>>>>> dockerized and non-dockerized applications. >>>>>>>>>>> >>>>>>>>>>> Therefore, it seems that the information about the URIs gets lost >>>>>>>>>>> when the dockerized mesos slave spawns the executor docker >>>>>>>>>>> container that in turn launches the application docker >>>>>>>>>>> container…Has anyone seen this problem before? I would like to know >>>>>>>>>>> if there is a workaround or a fixing. >>>>>>>>>>> >>>>>>>>>>> Thanks a lot in advance for you help >>>>>>>>>>> Best Regards, >>>>>>>>>>> Marica >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ---------------------------------------------------------- >>>>>>>>>>> Marica ANTONACCI >>>>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>>>> Via Orabona 4 >>>>>>>>>>> 70126 Bari - ITALY >>>>>>>>>>> Phone +39 080 5443244 >>>>>>>>>>> Skype: marica.antonacci >>>>>>>>>>> e-mail [email protected] >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Deshi Xiao >>>>>>>>>>> Twitter: xds2000 >>>>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>>> >>>>>>>>>> ---------------------------------------------------------- >>>>>>>>>> Marica ANTONACCI >>>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>>> Via Orabona 4 >>>>>>>>>> 70126 Bari - ITALY >>>>>>>>>> Phone +39 080 5443244 >>>>>>>>>> Skype: marica.antonacci >>>>>>>>>> e-mail [email protected] >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Deshi Xiao >>>>>>>>>> Twitter: xds2000 >>>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>> >>>>>>>>> ---------------------------------------------------------- >>>>>>>>> Marica ANTONACCI >>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>> Via Orabona 4 >>>>>>>>> 70126 Bari - ITALY >>>>>>>>> Phone +39 080 5443244 >>>>>>>>> Skype: marica.antonacci >>>>>>>>> e-mail [email protected] >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Deshi Xiao >>>>>>>>> Twitter: xds2000 >>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照 >>>>>>>>> 2015-12-18 下午4.55.38.png> >>>>>>>> >>>>>>>> ---------------------------------------------------------- >>>>>>>> Marica ANTONACCI >>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>> Via Orabona 4 >>>>>>>> 70126 Bari - ITALY >>>>>>>> Phone +39 080 5443244 >>>>>>>> Skype: marica.antonacci >>>>>>>> e-mail [email protected] >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Deshi Xiao >>>>>>>> Twitter: xds2000 >>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>> >>>>>>> >>>>>>> ---------------------------------------------------------- >>>>>>> Marica ANTONACCI >>>>>>> INFN - National Institute of Nuclear Physics >>>>>>> Via Orabona 4 >>>>>>> 70126 Bari - ITALY >>>>>>> Phone +39 080 5443244 >>>>>>> Skype: marica.antonacci >>>>>>> e-mail [email protected] >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> ---------------------------------------------------------- >>>>>> Marica ANTONACCI >>>>>> INFN - National Institute of Nuclear Physics >>>>>> Via Orabona 4 >>>>>> 70126 Bari - ITALY >>>>>> Phone +39 080 5443244 >>>>>> Skype: marica.antonacci >>>>>> e-mail [email protected] >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> ---------------------------------------------------------- >>>>> Marica ANTONACCI >>>>> INFN - National Institute of Nuclear Physics >>>>> Via Orabona 4 >>>>> 70126 Bari - ITALY >>>>> Phone +39 080 5443244 >>>>> Skype: marica.antonacci >>>>> e-mail [email protected] >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> ---------------------------------------------------------- >>>> Marica ANTONACCI >>>> INFN - National Institute of Nuclear Physics >>>> Via Orabona 4 >>>> 70126 Bari - ITALY >>>> Phone +39 080 5443244 >>>> Skype: marica.antonacci >>>> e-mail [email protected] >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> ---------------------------------------------------------- >>> Marica ANTONACCI >>> INFN - National Institute of Nuclear Physics >>> Via Orabona 4 >>> 70126 Bari - ITALY >>> Phone +39 080 5443244 >>> Skype: marica.antonacci >>> e-mail [email protected] >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >> >> ---------------------------------------------------------- >> Marica ANTONACCI >> INFN - National Institute of Nuclear Physics >> Via Orabona 4 >> 70126 Bari - ITALY >> Phone +39 080 5443244 >> Skype: marica.antonacci >> e-mail [email protected] >> >> >> >> >> >> >> >> >> >> > > > > ---------------------------------------------------------- Marica ANTONACCI INFN - National Institute of Nuclear Physics Via Orabona 4 70126 Bari - ITALY Phone +39 080 5443244 Skype: marica.antonacci e-mail [email protected]
