ping @Tim, I think this bug also affects https://issues.apache.org/jira/browse/MESOS-4743 .
On Wed, Jan 20, 2016 at 10:20 PM, Shuai Lin <[email protected]> wrote: > The testing of this case requires to build a docker image for mesos-slave, > so it seems not practical to add a test case for it in the mesos tests. > > Anyway, here is the scripts I use for testing this issue: > https://gist.github.com/lins05/14455e92f37e91fd46ff > > On Wed, Jan 20, 2016 at 10:30 AM, Shuai Lin <[email protected]> > wrote: > >> Hi Tim, >> >> The review is here: https://reviews.apache.org/r/42390/ , would you >> please take a look? >> >> Regards, >> Shuai >> >> On Sat, Jan 9, 2016 at 9:42 AM, Shuai Lin <[email protected]> wrote: >> >>> Hi Maria and Tim, >>> >>> I'm setting up a test case for this scenario that would fail, after >>> which I'll begin on fixing it. >>> >>> Is it feasible to include the fixing in the new release? >>> >>> >>> I'm not sure, does 0.27 have an estimated release date now? >>> >>> >>> Regards, >>> Shuai >>> >>> On Sat, Jan 9, 2016 at 1:11 AM, Timothy Chen <[email protected]> wrote: >>> >>>> I can shepherd no problem. >>>> >>>> Tim >>>> >>>> On Dec 25, 2015, at 4:32 PM, Shuai Lin <[email protected]> wrote: >>>> >>>> I'll work on it. @Tim could you shepherd it? >>>> >>>> On Sat, Dec 26, 2015 at 2:49 AM, Marica Antonacci < >>>> [email protected]> wrote: >>>> >>>>> Hi Tim and Shuai, >>>>> >>>>> thank you very much for your reply. I have opened a JIRA issue on >>>>> this: https://issues.apache.org/jira/browse/MESOS-4249 >>>>> I hope it will be patched soon :) >>>>> >>>>> Best regards, >>>>> Marica >>>>> >>>>> >>>>> Il giorno 24/dic/2015, alle ore 17:54, Tim Chen <[email protected]> >>>>> ha scritto: >>>>> >>>>> Hi Marica/Shuai, >>>>> >>>>> Sorry haven't been able to spend the time to repro, but looks like >>>>> Shuai confirmed it. >>>>> >>>>> Can one of you file a JIRA? >>>>> >>>>> Thanks! >>>>> >>>>> Tim >>>>> >>>>> On Thu, Dec 24, 2015 at 6:16 AM, Shuai Lin <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Marica, >>>>>> >>>>>> I can reproduce the problem exactly as you described in the first >>>>>> email of this thread. Without `MESOS_DOCKER_MESOS_IMAGE` environment >>>>>> variable set, the fetcher works just fine; With it, the fetcher steps >>>>>> seems >>>>>> skipped. This looks like a bug to me. >>>>>> >>>>>> Regards, >>>>>> Shuai >>>>>> >>>>>> On Tue, Dec 22, 2015 at 7:41 PM, Marica Antonacci < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Dear all, >>>>>>> >>>>>>> I have not solved this issue yet. Please, can anyone run the same >>>>>>> test and let me know if the fetcher is correctly invoked? >>>>>>> The test is really simple, just try to start a dockerized app (see >>>>>>> json definition file below) through marathon on a mesos slave running >>>>>>> in a >>>>>>> docker container started with the option —docker_mesos_image=<mesos >>>>>>> slave >>>>>>> image>. >>>>>>> I would appreciate very much any feedback. >>>>>>> >>>>>>> Sample Marathon app: >>>>>>> { >>>>>>> "id": "test-app", >>>>>>> "container": { >>>>>>> "type": "DOCKER", >>>>>>> "docker": { >>>>>>> "image": "libmesos/ubuntu" >>>>>>> } >>>>>>> }, >>>>>>> "cpus": 1, >>>>>>> "mem": 512, >>>>>>> "uris": [ " >>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv"; >>>>>>> ], >>>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u >>>>>>> +%T; done" >>>>>>> } >>>>>>> >>>>>>> Docker run command to start dockerized mesos slave: >>>>>>> >>>>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e >>>>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e >>>>>>> MESOS_CONTAINERIZERS=docker,mesos >>>>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log >>>>>>> -e >>>>>>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v >>>>>>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos >>>>>>> --name slave --net host --privileged --pid host mesos-slave >>>>>>> >>>>>>> Thank you very much in advance! >>>>>>> Best regards, >>>>>>> Marica >>>>>>> >>>>>>> Il giorno 19/dic/2015, alle ore 19:32, Marica Antonacci < >>>>>>> [email protected]> ha scritto: >>>>>>> >>>>>>> Dear Tim, >>>>>>> >>>>>>> I have collected some information from my test environment, starting >>>>>>> the slave container with and without the —docker_mesos_image startup >>>>>>> flag. >>>>>>> Please let me know if you need further input. Thank you very much for >>>>>>> your >>>>>>> support! >>>>>>> >>>>>>> Using the flag —docker_mesos_image: >>>>>>> >>>>>>> root@mesos-slave:~# docker ps >>>>>>> CONTAINER ID IMAGE COMMAND >>>>>>> CREATED STATUS PORTS NAMES >>>>>>> b30cea22a07c libmesos/ubuntu "/bin/sh -c 'cd $MESO" 2 >>>>>>> minutes ago Up 2 minutes >>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d >>>>>>> da9c78ec5727 mesos-slave "/bin/sh -c '/usr/lib" 2 >>>>>>> minutes ago Up 2 minutes >>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor >>>>>>> 150f78fbf327 mesos-slave "/entrypoint.sh /usr/" 3 >>>>>>> minutes ago Up 3 minutes slave >>>>>>> >>>>>>> *root@mesos-slave:~# docker logs slave* >>>>>>> I1219 18:03:38.308544 19476 slave.cpp:1294] Got assigned task >>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>>> I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task >>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>>> I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown >>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' >>>>>>> to user 'root' >>>>>>> I1219 18:03:38.327221 19476 slave.cpp:4999] Launching executor >>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.1; >>>>>>> mem(*):32 in work directory >>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' >>>>>>> I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing task >>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor >>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>>> I1219 18:03:38.335907 19480 docker.cpp:762] Starting container >>>>>>> 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task >>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor >>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework >>>>>>> '246b272b-d649-47c0-88ca-6b1ff35f437a-0000' >>>>>>> I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid 19590 >>>>>>> to >>>>>>> '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pids/forked.pid' >>>>>>> I1219 18:03:40.189172 19479 slave.cpp:2405] Got registration for >>>>>>> executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@ >>>>>>> 90.147.170.246:39381 >>>>>>> I1219 18:03:40.191141 19479 docker.cpp:1012] Ignoring updating >>>>>>> container 'd965f59b-cc1a-4081-95d2-f3370214c84d' with resources passed >>>>>>> to >>>>>>> update is identical to existing resources >>>>>>> I1219 18:03:40.192512 19479 slave.cpp:1793] Sending queued task >>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' to executor >>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at executor(1)@ >>>>>>> 90.147.170.246:39381 >>>>>>> I1219 18:03:42.425606 19483 slave.cpp:2762] Handling status update >>>>>>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task >>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@ >>>>>>> 90.147.170.246:39381 >>>>>>> I1219 18:03:42.426415 19483 status_update_manager.cpp:322] Received >>>>>>> status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) >>>>>>> for >>>>>>> task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>>> I1219 18:03:42.427276 19483 status_update_manager.cpp:826] >>>>>>> Checkpointing UPDATE for status update TASK_RUNNING (UUID: >>>>>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task >>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>>> I1219 18:03:42.521395 19483 slave.cpp:3087] Forwarding the update >>>>>>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task >>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to [email protected] >>>>>>> :5050 >>>>>>> I1219 18:03:42.521852 19483 slave.cpp:3011] Sending acknowledgement >>>>>>> for status update TASK_RUNNING (UUID: >>>>>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) >>>>>>> for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to executor(1)@ >>>>>>> 90.147.170.246:39381 >>>>>>> I1219 18:03:42.576373 19481 status_update_manager.cpp:394] Received >>>>>>> status update acknowledgement (UUID: >>>>>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) >>>>>>> for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>>> I1219 18:03:42.577263 19481 status_update_manager.cpp:826] >>>>>>> Checkpointing ACK for status update TASK_RUNNING (UUID: >>>>>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task >>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>>> >>>>>>> root@mesos-slave:~# *docker logs >>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor* >>>>>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d" >>>>>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" >>>>>>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" >>>>>>> --logbufsecs="0" --logging_level="INFO" >>>>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false" >>>>>>> --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d" >>>>>>> --stop_timeout="0ns" >>>>>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d" >>>>>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" >>>>>>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" >>>>>>> --logbufsecs="0" --logging_level="INFO" >>>>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false" >>>>>>> --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d" >>>>>>> --stop_timeout="0ns" >>>>>>> I1219 18:03:40.177598 6 exec.cpp:136] Version: 0.26.0 >>>>>>> I1219 18:03:40.192060 10 exec.cpp:210] Executor registered on >>>>>>> slave db70e09f-f39d-491c-8480-73d9858c140b-S0 >>>>>>> Registered docker executor on 90.147.170.246 >>>>>>> Starting task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 >>>>>>> >>>>>>> root@mesos-slave:~# *docker exec -it >>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor >>>>>>> bash* >>>>>>> root@mesos-slave:/# ls -R /tmp/ >>>>>>> hsperfdata_root/ mesos/ >>>>>>> root@mesos-slave:/# ls -R /tmp/mesos/ >>>>>>> /tmp/mesos/: >>>>>>> *slaves* >>>>>>> >>>>>>> /tmp/mesos/slaves: >>>>>>> *db70e09f-f39d-491c-8480-73d9858c140b-S0* >>>>>>> >>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0: >>>>>>> *frameworks* >>>>>>> >>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks: >>>>>>> *246b272b-d649-47c0-88ca-6b1ff35f437a-0000* >>>>>>> >>>>>>> >>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000: >>>>>>> *executors* >>>>>>> >>>>>>> >>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors: >>>>>>> *test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0* >>>>>>> >>>>>>> >>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0: >>>>>>> *runs* >>>>>>> >>>>>>> >>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs: >>>>>>> *d965f59b-cc1a-4081-95d2-f3370214c84d* >>>>>>> >>>>>>> >>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d: >>>>>>> stderr stdout >>>>>>> >>>>>>> >>>>>>> root@mesos-slave:~# *docker exec -it >>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d >>>>>>> bash* >>>>>>> root@mesos-slave:/# env >>>>>>> HOSTNAME=mesos-slave >>>>>>> HOST=90.147.170.246 >>>>>>> PORT0=31220 >>>>>>> PORT_10000=31220 >>>>>>> MESOS_TASK_ID=test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 >>>>>>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin >>>>>>> PWD=/ >>>>>>> PORTS=31220 >>>>>>> >>>>>>> MESOS_CONTAINER_NAME=mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d >>>>>>> SHLVL=1 >>>>>>> HOME=/ >>>>>>> MARATHON_APP_ID=/test-app >>>>>>> MARATHON_APP_DOCKER_IMAGE=libmesos/ubuntu >>>>>>> MARATHON_APP_VERSION=2015-12-19T18:03:37.542Z >>>>>>> PORT=31220 >>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox >>>>>>> _=/usr/bin/env >>>>>>> root@mesos-slave:/# ls -R $MESOS_SANDBOX >>>>>>> /mnt/mesos/sandbox: >>>>>>> stderr stdout >>>>>>> >>>>>>> >>>>>>> root@mesos-slave:/# ls /var/log/mesos* >>>>>>> */var/log/mesos-slave.INFO* >>>>>>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-182512.20647 >>>>>>> >>>>>>> /var/log/mesos: >>>>>>> >>>>>>> >>>>>>> Disabling the flag —docker_mesos_image the fetcher is called, log is >>>>>>> created, the file is downloaded in the sandbox: >>>>>>> >>>>>>> root@mesos-slave:~# docker exec -it slave bash >>>>>>> root@mesos-slave:/# >>>>>>> root@mesos-slave:/# >>>>>>> root@mesos-slave:/# >>>>>>> root@mesos-slave:/# ls /var/log/mesos* >>>>>>> */var/log/mesos-fetcher.INFO* >>>>>>> /var/log/mesos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-181621.20157 >>>>>>> */var/log/mesos-slave.INFO* >>>>>>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-181612.20124 >>>>>>> >>>>>>> /var/log/mesos: >>>>>>> root@mesos-slave:/# >>>>>>> root@mesos-slave:/# cat /var/log/mesos-fetcher.INFO >>>>>>> Log file created at: 2015/12/19 18:16:21 >>>>>>> Running on machine: mesos-slave >>>>>>> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg >>>>>>> I1219 18:16:21.127075 20157 logging.cpp:172] INFO level logging >>>>>>> started! >>>>>>> I1219 18:16:21.127499 20157 fetcher.cpp:422] Fetcher Info: >>>>>>> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"value":"http:\/\/ >>>>>>> www.stat.cmu.edu >>>>>>> \/~cshalizi\/402\/lectures\/16-glm-practicals\/snoqualmie.csv"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1\/frameworks\/246b272b-d649-47c0-88ca-6b1ff35f437a-0000\/executors\/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0\/runs\/1862ced9-3659-4429-8bc6-3f47f58970b4"} >>>>>>> I1219 18:16:21.129547 20157 fetcher.cpp:377] Fetching URI ' >>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' >>>>>>> I1219 18:16:21.129576 20157 fetcher.cpp:248] Fetching directly into >>>>>>> the sandbox directory >>>>>>> I1219 18:16:21.129607 20157 fetcher.cpp:185] Fetching URI ' >>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' >>>>>>> I1219 18:16:21.129636 20157 fetcher.cpp:132] Downloading resource >>>>>>> from ' >>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' >>>>>>> to >>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv' >>>>>>> I1219 18:16:23.221889 20157 fetcher.cpp:454] Fetched ' >>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' >>>>>>> to >>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv' >>>>>>> root@mesos-slave:/# >>>>>>> >>>>>>> root@mesos-slave:~# docker exec -it >>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8bc6-3f47f58970b4 >>>>>>> bash >>>>>>> root@mesos-slave:/# >>>>>>> root@mesos-slave:/# >>>>>>> root@mesos-slave:/# ls /mnt/mesos/sandbox/ >>>>>>> snoqualmie.csv stderr stdout >>>>>>> root@mesos-slave:/# >>>>>>> >>>>>>> >>>>>>> Best Regards, >>>>>>> Marica >>>>>>> >>>>>>> >>>>>>> >>>>>>> Il giorno 19/dic/2015, alle ore 02:12, Tim Chen <[email protected]> >>>>>>> ha scritto: >>>>>>> >>>>>>> HI Marica, >>>>>>> >>>>>>> Did you see the fetcher invoked at all from the Slave logs? Doesn't >>>>>>> seem possible we don't pass down the URI flags and if the fetcher failed >>>>>>> the container launch should have failed too. >>>>>>> >>>>>>> Also another possible situation is that web UI not really showing >>>>>>> the exact content of the sandbox, can you actually go into the directory >>>>>>> and see if the files are there? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Tim >>>>>>> >>>>>>> On Fri, Dec 18, 2015 at 4:23 PM, Marica Antonacci < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Tim, >>>>>>>> >>>>>>>> looking at the sandbox I can see only the stderr and stout file >>>>>>>> (see the attached screenshot). If I remove —docker_mesos_image (and >>>>>>>> therefore the executor is run inside the slave container) the file >>>>>>>> specified in the URI field is shown in the sandbox. >>>>>>>> Did you verify that the fetcher is called when using the >>>>>>>> —docker_mesos_image flag ? >>>>>>>> >>>>>>>> Thanks a lot for your feedback >>>>>>>> Marica >>>>>>>> >>>>>>>> <Schermata 2015-12-18 alle 10.15.58.png> >>>>>>>> >>>>>>>> Il giorno 19/dic/2015, alle ore 00:25, Tim Chen <[email protected]> >>>>>>>> ha scritto: >>>>>>>> >>>>>>>> Hi Marica, >>>>>>>> >>>>>>>> It should work as we fetch all the files before we launch the >>>>>>>> executor and place them in the sandbox, and we mount the sandbox into >>>>>>>> that >>>>>>>> container as well. >>>>>>>> >>>>>>>> How did you verify that the file is not downloaded? >>>>>>>> >>>>>>>> Tim >>>>>>>> >>>>>>>> On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Grzegorz, >>>>>>>>> >>>>>>>>> I’m using this command line for docker run >>>>>>>>> >>>>>>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> >>>>>>>>> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos >>>>>>>>> -e MESOS_CONTAINERIZERS=docker,mesos -e >>>>>>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e >>>>>>>>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup >>>>>>>>> -v >>>>>>>>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos >>>>>>>>> --name >>>>>>>>> slave --net host --privileged --pid host mesos-slave >>>>>>>>> >>>>>>>>> where mesos-slave is the image built from the docker file in this >>>>>>>>> repo https://github.com/maricaantonacci/mesos-slave-dev >>>>>>>>> >>>>>>>>> I have tested successfully the deployment of dockerized >>>>>>>>> applications through Marathon and dockerized jobs through Chronos and >>>>>>>>> also >>>>>>>>> the recovery seems to work fine with the flag docker_mesos_image. >>>>>>>>> What is >>>>>>>>> not working for me is the fetcher: it seems that when the executor is >>>>>>>>> launched as separate container (thanks to the flag >>>>>>>>> docker_mesos_image) the >>>>>>>>> information about the URIs to be downloaded is lost…I hope someone >>>>>>>>> can help >>>>>>>>> to understand if this a bug or I’ m missing something. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Marica >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk < >>>>>>>>> [email protected]> ha scritto: >>>>>>>>> >>>>>>>>> I've tried to use this flag, but cannot really run any container >>>>>>>>> when this flag is set. >>>>>>>>> I've raised this issue here: >>>>>>>>> https://www.mail-archive.com/[email protected]/msg04975.html and >>>>>>>>> here: >>>>>>>>> https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351 >>>>>>>>> but >>>>>>>>> sadly no one was able to help me... >>>>>>>>> >>>>>>>>> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci < >>>>>>>>> [email protected]> napisał: >>>>>>>>> >>>>>>>>>> OK, the problem I spotted is related to the usage of the >>>>>>>>>> flag —docker_mesos_image that allows the executor to >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> --docker_mesos_image=VALUE The docker image used to launch this >>>>>>>>>> mesos slave instance. If an image is specified, the docker >>>>>>>>>> containerizer >>>>>>>>>> assumes the slave is running in a docker container, and launches >>>>>>>>>> executors >>>>>>>>>> with docker containers in order to recover them when the slave >>>>>>>>>> restarts and >>>>>>>>>> recovers. >>>>>>>>>> Has anyone used this flag and tested the behavior of the fetcher? >>>>>>>>>> >>>>>>>>>> Thank you >>>>>>>>>> Marica >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao < >>>>>>>>>> [email protected]> ha scritto: >>>>>>>>>> >>>>>>>>>> no docker_mesos_image flag in my docker run, and the docker >>>>>>>>>> image is build by myself. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci < >>>>>>>>>> [email protected]>: >>>>>>>>>> >>>>>>>>>> Yes, I did check inside the container and the csv file was not >>>>>>>>>>> downloaded as shown also by the app details (see the screenshot >>>>>>>>>>> below). >>>>>>>>>>> >>>>>>>>>>> Are you running your slave with the --docker_mesos_image flag? >>>>>>>>>>> Can you please provide me the docker run command you are using to >>>>>>>>>>> run your >>>>>>>>>>> dockerized slave? >>>>>>>>>>> >>>>>>>>>>> Thank you very much >>>>>>>>>>> >>>>>>>>>> Marica >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> <Schermata 2015-12-18 alle 10.15.58.png> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao < >>>>>>>>>>> [email protected]> ha scritto: >>>>>>>>>>> >>>>>>>>>>> Hi Marica, >>>>>>>>>>> >>>>>>>>>>> use your test-app json, i can run it correctly, the csv is >>>>>>>>>>> truely download by mesos slave. please check mesos-master:5050 to >>>>>>>>>>> check the >>>>>>>>>>> task detail download files. >>>>>>>>>>> >>>>>>>>>>> you describe the app container why not found the csv, because >>>>>>>>>>> the csv is download in slave container's folder, not in app >>>>>>>>>>> container. so >>>>>>>>>>> if you run >>>>>>>>>>> >>>>>>>>>>> cd $MESOS_SANDBOX; >>>>>>>>>>> >>>>>>>>>>> the folder in app container is default value: >>>>>>>>>>> >>>>>>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox >>>>>>>>>>> but in real world, the sandbox is in slave container, not in app >>>>>>>>>>> container. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci < >>>>>>>>>>> [email protected]>: >>>>>>>>>>> >>>>>>>>>>>> Thank you very much, >>>>>>>>>>>> >>>>>>>>>>>> I’m using a sample application definition file, just for >>>>>>>>>>>> testing purpose: >>>>>>>>>>>> >>>>>>>>>>>> { >>>>>>>>>>>> "id": "test-app", >>>>>>>>>>>> "container": { >>>>>>>>>>>> "type": "DOCKER", >>>>>>>>>>>> "docker": { >>>>>>>>>>>> "image": "libmesos/ubuntu" >>>>>>>>>>>> } >>>>>>>>>>>> }, >>>>>>>>>>>> "cpus": 1, >>>>>>>>>>>> "mem": 512, >>>>>>>>>>>> *"uris": [ >>>>>>>>>>>> "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv >>>>>>>>>>>> <http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv>" >>>>>>>>>>>> ],* >>>>>>>>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date >>>>>>>>>>>> -u +%T; done" >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> Here is the docker run command line: >>>>>>>>>>>> >>>>>>>>>>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e >>>>>>>>>>>> MESOS_IP=<SLAVE_IP> -e >>>>>>>>>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos >>>>>>>>>>>> -e MESOS_CONTAINERIZERS=docker,mesos \ >>>>>>>>>>>> -e >>>>>>>>>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e >>>>>>>>>>>> MESOS_LOG_DIR=/var/log -e >>>>>>>>>>>> MESOS_docker_mesos_image=mesos-slave >>>>>>>>>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v >>>>>>>>>>>> /var/run/docker.sock:/var/run/docker.sock --name slave --net host >>>>>>>>>>>> --privileged --pid host mesos-slave >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> As already mentioned, if I remove the environment variable >>>>>>>>>>>> MESOS_docker_mesos_image the fetcher works fine and I can see the >>>>>>>>>>>> file >>>>>>>>>>>> snoqualmie.csv inside the sandbox. >>>>>>>>>>>> >>>>>>>>>>>> Thank you again! I’m looking forward to hearing about your >>>>>>>>>>>> outcomes. >>>>>>>>>>>> Best regards, >>>>>>>>>>>> Marica >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao < >>>>>>>>>>>> [email protected]> ha scritto: >>>>>>>>>>>> >>>>>>>>>>>> Hi Marica, >>>>>>>>>>>> >>>>>>>>>>>> Could you please give a sample marathon json, i can test it >>>>>>>>>>>> asap. >>>>>>>>>>>> >>>>>>>>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci < >>>>>>>>>>>> [email protected]>: >>>>>>>>>>>> >>>>>>>>>>>>> No, using the socket: >>>>>>>>>>>>> >>>>>>>>>>>>> -v /var/run/docker.sock:/var/run/docker.sock >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao < >>>>>>>>>>>>> [email protected]> ha scritto: >>>>>>>>>>>>> >>>>>>>>>>>>> docker in docker mode? >>>>>>>>>>>>> >>>>>>>>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci < >>>>>>>>>>>>> [email protected]>: >>>>>>>>>>>>> >>>>>>>>>>>>>> Dear all, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm testing the URIs fetching mechanism for both Marathon >>>>>>>>>>>>>> applications and Chronos jobs and I have found that if the slave >>>>>>>>>>>>>> is running >>>>>>>>>>>>>> inside a docker container (using *docker_mesos_image* >>>>>>>>>>>>>> startup flag) and you submit the deployment of a dockerized >>>>>>>>>>>>>> application or >>>>>>>>>>>>>> job the fetcher step is not performed. On the other hand, if I >>>>>>>>>>>>>> request the >>>>>>>>>>>>>> deployment of a non-dockerized application, the URIs are >>>>>>>>>>>>>> correctly fetched. >>>>>>>>>>>>>> Moreover, if I don’t provide the docker_mesos_image flag, the >>>>>>>>>>>>>> fetcher works >>>>>>>>>>>>>> fine again for both dockerized and non-dockerized applications. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Therefore, it seems that the information about the URIs gets >>>>>>>>>>>>>> lost when the dockerized mesos slave spawns the executor docker >>>>>>>>>>>>>> container >>>>>>>>>>>>>> that in turn launches the application docker container…Has >>>>>>>>>>>>>> anyone seen this >>>>>>>>>>>>>> problem before? I would like to know if there is a workaround or >>>>>>>>>>>>>> a fixing. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks a lot in advance for you help >>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>> Marica >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ---------------------------------------------------------- >>>>>>>>>>>>>> >>>>>>>>>>>>>> Marica ANTONACCI >>>>>>>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>>>>>>> Via Orabona 4 >>>>>>>>>>>>>> 70126 Bari - ITALY >>>>>>>>>>>>>> Phone +39 080 5443244 >>>>>>>>>>>>>> Skype: marica.antonacci >>>>>>>>>>>>>> e-mail [email protected] >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Deshi Xiao >>>>>>>>>>>>> Twitter: xds2000 >>>>>>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ---------------------------------------------------------- >>>>>>>>>>>>> >>>>>>>>>>>>> Marica ANTONACCI >>>>>>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>>>>>> Via Orabona 4 >>>>>>>>>>>>> 70126 Bari - ITALY >>>>>>>>>>>>> Phone +39 080 5443244 >>>>>>>>>>>>> Skype: marica.antonacci >>>>>>>>>>>>> e-mail [email protected] >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Deshi Xiao >>>>>>>>>>>> Twitter: xds2000 >>>>>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ---------------------------------------------------------- >>>>>>>>>>>> >>>>>>>>>>>> Marica ANTONACCI >>>>>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>>>>> Via Orabona 4 >>>>>>>>>>>> 70126 Bari - ITALY >>>>>>>>>>>> Phone +39 080 5443244 >>>>>>>>>>>> Skype: marica.antonacci >>>>>>>>>>>> e-mail [email protected] >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Deshi Xiao >>>>>>>>>>> Twitter: xds2000 >>>>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照 >>>>>>>>>>> 2015-12-18 下午4.55.38.png> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ---------------------------------------------------------- >>>>>>>>>>> >>>>>>>>>>> Marica ANTONACCI >>>>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>>>> Via Orabona 4 >>>>>>>>>>> 70126 Bari - ITALY >>>>>>>>>>> Phone +39 080 5443244 >>>>>>>>>>> Skype: marica.antonacci >>>>>>>>>>> e-mail [email protected] >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Deshi Xiao >>>>>>>>>> Twitter: xds2000 >>>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ---------------------------------------------------------- >>>>>>>>>> >>>>>>>>>> Marica ANTONACCI >>>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>>> Via Orabona 4 >>>>>>>>>> 70126 Bari - ITALY >>>>>>>>>> Phone +39 080 5443244 >>>>>>>>>> Skype: marica.antonacci >>>>>>>>>> e-mail [email protected] >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> ---------------------------------------------------------- >>>>>>>>> >>>>>>>>> Marica ANTONACCI >>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>> Via Orabona 4 >>>>>>>>> 70126 Bari - ITALY >>>>>>>>> Phone +39 080 5443244 >>>>>>>>> Skype: marica.antonacci >>>>>>>>> e-mail [email protected] >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> ---------------------------------------------------------- >>>>>>>> >>>>>>>> Marica ANTONACCI >>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>> Via Orabona 4 >>>>>>>> 70126 Bari - ITALY >>>>>>>> Phone +39 080 5443244 >>>>>>>> Skype: marica.antonacci >>>>>>>> e-mail [email protected] >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> ---------------------------------------------------------- >>>>>>> >>>>>>> Marica ANTONACCI >>>>>>> INFN - National Institute of Nuclear Physics >>>>>>> Via Orabona 4 >>>>>>> 70126 Bari - ITALY >>>>>>> Phone +39 080 5443244 >>>>>>> Skype: marica.antonacci >>>>>>> e-mail [email protected] >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> ---------------------------------------------------------- >>>>>>> >>>>>>> Marica ANTONACCI >>>>>>> INFN - National Institute of Nuclear Physics >>>>>>> Via Orabona 4 >>>>>>> 70126 Bari - ITALY >>>>>>> Phone +39 080 5443244 >>>>>>> Skype: marica.antonacci >>>>>>> e-mail [email protected] >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> ---------------------------------------------------------- >>>>> >>>>> Marica ANTONACCI >>>>> INFN - National Institute of Nuclear Physics >>>>> Via Orabona 4 >>>>> 70126 Bari - ITALY >>>>> Phone +39 080 5443244 >>>>> Skype: marica.antonacci >>>>> e-mail [email protected] >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>> >> >
