[jira] [Commented] (MESOS-4061) Flaky tests: docker containerizer tests on debian 8 VM
[
https://issues.apache.org/jira/browse/MESOS-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15214689#comment-15214689
]
Greg Mann commented on MESOS-4061:
--
I also observed the failure of
{{DockerContainerizerTest.ROOT_DOCKER_SkipRecoverNonDocker}} on the Mesosphere
internal CI, on both Ubuntu 14 and Ubuntu 12:
{code}
[16:56:58] : [Step 10/10] [ RUN ]
DockerContainerizerTest.ROOT_DOCKER_SkipRecoverNonDocker
[16:56:58]W: [Step 10/10] I0328 16:56:58.619204 25413 docker.cpp:740]
Recovering Docker containers
[16:56:58]W: [Step 10/10] I0328 16:56:58.619329 25413 docker.cpp:919]
Running docker -H unix:///var/run/docker.sock ps -a
[16:56:58]W: [Step 10/10] I0328 16:56:58.717419 25410 docker.cpp:800]
Running docker -H unix:///var/run/docker.sock inspect
mesos-1256102e-f3e5-43a0-ab64-c15db3bf243c-S0.fd72cfa3-2234-4a7b-ba09-9a761253c476
[16:56:58]W: [Step 10/10] I0328 16:56:58.822619 25409 docker.cpp:912]
Checking if Docker container named
'/mesos-1256102e-f3e5-43a0-ab64-c15db3bf243c-S0.fd72cfa3-2234-4a7b-ba09-9a761253c476'
was started by Mesos
[16:56:58]W: [Step 10/10] I0328 16:56:58.822669 25409 docker.cpp:922]
Checking if Mesos container with ID 'fd72cfa3-2234-4a7b-ba09-9a761253c476' has
been orphaned
[16:56:58]W: [Step 10/10] I0328 16:56:58.822715 25409 docker.cpp:712]
Running docker -H unix:///var/run/docker.sock stop -t 0
b02836190a3be54bbebbe5fe7ecd5d2984c35b9112678020a75221b808459742
[16:57:13] : [Step 10/10]
../../src/tests/containerizer/docker_containerizer_tests.cpp:1298: Failure
[16:57:13] : [Step 10/10] Failed to wait 15secs for recover
[16:57:13]W: [Step 10/10] I0328 16:57:13.621881 25393 docker.cpp:919]
Running docker -H unix:///var/run/docker.sock ps -a
[16:57:13]W: [Step 10/10] I0328 16:57:13.706827 25412 docker.cpp:800]
Running docker -H unix:///var/run/docker.sock inspect
mesos-1256102e-f3e5-43a0-ab64-c15db3bf243c-S0.fd72cfa3-2234-4a7b-ba09-9a761253c476
[16:57:13]W: [Step 10/10] I0328 16:57:13.812366 25393 docker.cpp:761]
Running docker -H unix:///var/run/docker.sock rm -f -v
b02836190a3be54bbebbe5fe7ecd5d2984c35b9112678020a75221b808459742
[16:57:13] : [Step 10/10] [ FAILED ]
DockerContainerizerTest.ROOT_DOCKER_SkipRecoverNonDocker (15297 ms)
{code}
> Flaky tests: docker containerizer tests on debian 8 VM
> --
>
> Key: MESOS-4061
> URL: https://issues.apache.org/jira/browse/MESOS-4061
> Project: Mesos
> Issue Type: Bug
> Environment: debian 8, vagrant, virtual box
>Reporter: Jojy Varghese
>
> Following tests were failing for 0.26 rc3:
> * DockerContainerizerTest.ROOT_DOCKER_NC_PortMapping
> * DockerContainerizerTest.ROOT_DOCKER_Recover
> * DockerContainerizerTest.ROOT_DOCKER_SlaveRecoveryTaskContainer
> * DockerContainerizerTest.ROOT_DOCKER_Launch_Executor
> * DockerContainerizerTest.ROOT_DOCKER_Launch
> * DockerContainerizerTest.ROOT_DOCKER_Usage
> * DockerContainerizerTest.ROOT_DOCKER_Update
> * DockerContainerizerTest.ROOT_DOCKER_SkipRecoverNonDocker
> Note that this is not a comprehensive list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-4061) Flaky tests: docker containerizer tests on debian 8 VM
[ https://issues.apache.org/jira/browse/MESOS-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096387#comment-15096387 ] Jojy Varghese commented on MESOS-4061: -- Moving to 0.28 .0 should be fine as we havent been able to reproduce this consistently. I am not sure if this is a real issue. > Flaky tests: docker containerizer tests on debian 8 VM > -- > > Key: MESOS-4061 > URL: https://issues.apache.org/jira/browse/MESOS-4061 > Project: Mesos > Issue Type: Bug > Environment: debian 8, vagrant, virtual box >Reporter: Jojy Varghese > > Following tests were failing for 0.26 rc3: > * DockerContainerizerTest.ROOT_DOCKER_NC_PortMapping > * DockerContainerizerTest.ROOT_DOCKER_Recover > * DockerContainerizerTest.ROOT_DOCKER_SlaveRecoveryTaskContainer > * DockerContainerizerTest.ROOT_DOCKER_Launch_Executor > * DockerContainerizerTest.ROOT_DOCKER_Launch > * DockerContainerizerTest.ROOT_DOCKER_Usage > * DockerContainerizerTest.ROOT_DOCKER_Update > * DockerContainerizerTest.ROOT_DOCKER_SkipRecoverNonDocker > Note that this is not a comprehensive list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-4061) Flaky tests: docker containerizer tests on debian 8 VM
[ https://issues.apache.org/jira/browse/MESOS-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095688#comment-15095688 ] Michael Park commented on MESOS-4061: - [~jojy] Is this a blocker for 0.27.0 or can we push it to 0.28.0? > Flaky tests: docker containerizer tests on debian 8 VM > -- > > Key: MESOS-4061 > URL: https://issues.apache.org/jira/browse/MESOS-4061 > Project: Mesos > Issue Type: Bug > Environment: debian 8, vagrant, virtual box >Reporter: Jojy Varghese > > Following tests were failing for 0.26 rc3: > * DockerContainerizerTest.ROOT_DOCKER_NC_PortMapping > * DockerContainerizerTest.ROOT_DOCKER_Recover > * DockerContainerizerTest.ROOT_DOCKER_SlaveRecoveryTaskContainer > * DockerContainerizerTest.ROOT_DOCKER_Launch_Executor > * DockerContainerizerTest.ROOT_DOCKER_Launch > * DockerContainerizerTest.ROOT_DOCKER_Usage > * DockerContainerizerTest.ROOT_DOCKER_Update > * DockerContainerizerTest.ROOT_DOCKER_SkipRecoverNonDocker > Note that this is not a comprehensive list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-4061) Flaky tests: docker containerizer tests on debian 8 VM
[
https://issues.apache.org/jira/browse/MESOS-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038533#comment-15038533
]
Jojy Varghese commented on MESOS-4061:
--
perf report:
{code}
34.34%34 docker [kernel.kallsyms] [k]
_raw_spin_unlock_irqrestore
10.10%10 docker docker [.] scanblock
8.08% 8 docker [kernel.kallsyms] [k] finish_task_switch
3.03% 3 docker [kernel.kallsyms] [k] smp_call_function_many
2.02% 2 docker [kernel.kallsyms] [k] __do_softirq
2.02% 2 docker [kernel.kallsyms] [k] flat_send_IPI_mask
2.02% 2 docker [kernel.kallsyms] [k] generic_exec_single
2.02% 2 docker docker [.] runtime.mallocgc
2.02% 2 docker docker [.] runtime.mapassign1
2.02% 2sudo [kernel.kallsyms] [k]
_raw_spin_unlock_irqrestore
1.01% 1 docker [kernel.kallsyms] [k] __do_page_fault
1.01% 1 docker [kernel.kallsyms] [k] __lru_cache_add
1.01% 1 docker [kernel.kallsyms] [k]
__mem_cgroup_commit_charge
1.01% 1 docker [kernel.kallsyms] [k] __wake_up_bit
1.01% 1 docker [kernel.kallsyms] [k] do_wp_page
1.01% 1 docker [kernel.kallsyms] [k] extract_buf
1.01% 1 docker [kernel.kallsyms] [k] page_add_new_anon_rmap
1.01% 1 docker [kernel.kallsyms] [k] perf_event_mmap
1.01% 1 docker docker [.] MHeap_FreeSpanLocked
1.01% 1 docker docker [.]
bytes.(*Buffer).WriteByte
1.01% 1 docker docker [.] encoding/json.init
1.01% 1 docker docker [.] fmt.(*pp).doPrintf
1.01% 1 docker docker [.]
math/rand.(*rngSource).Seed
[jira] [Commented] (MESOS-4061) Flaky tests: docker containerizer tests on debian 8 VM
[
https://issues.apache.org/jira/browse/MESOS-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038524#comment-15038524
]
Jojy Varghese commented on MESOS-4061:
--
Initial investigation shows that on vagrant + virtualbox + debian 8
environment, docker command run very slow. An example run :
{code}
vagrant@debian8:~$ time sudo docker -H unix:///var/run/docker.sock run -c 2048
-m 1073741824 -e MESOS_SANDBOX=/mnt/mesos/sandbox -e
MESOS_CONTAINER_NAME=mesos-aca33e44-bd72-4f35-ba7c-c97aa1d350c0-S0.10df81e8-fcef-4767-91bf-e61b9e95c8f3
-v
/tmp/DockerContainerizerTest_ROOT_DOCKER_Logs_dbSPqi/slaves/aca33e44-bd72-4f35-ba7c-c97aa1d350c0-S0/frameworks/aca33e44-bd72-4f35-ba7c-c97aa1d350c0-/executors/1/runs/10df81e8-fcef-4767-91bf-e61b9e95c8f3:/mnt/mesos/sandbox
--net host --entrypoint /bin/sh --name
mesos-aca33e44-bd72-4f35-ba7c-c97aa1d350c0-S0.10df81e8-fcef-4767-91bf-e61b9e95c8fC
--entrypoint /bin/sh busybox -c echo hello
Warning: '-c' is deprecated, it will be replaced by '--cpu-shares' soon. See
usage.
WARNING: Your kernel does not support swap limit capabilities, memory limited
without swap.
real1m25.964s
user0m0.020s
sys 0m0.020s
{code}
> Flaky tests: docker containerizer tests on debian 8 VM
> --
>
> Key: MESOS-4061
> URL: https://issues.apache.org/jira/browse/MESOS-4061
> Project: Mesos
> Issue Type: Bug
> Environment: debian 8, vagrant, virtual box
>Reporter: Jojy Varghese
>
> Following tests were failing for 0.26 rc3:
> * DockerContainerizerTest.ROOT_DOCKER_NC_PortMapping
> * DockerContainerizerTest.ROOT_DOCKER_Recover
> * DockerContainerizerTest.ROOT_DOCKER_SlaveRecoveryTaskContainer
> * DockerContainerizerTest.ROOT_DOCKER_Launch_Executor
> * DockerContainerizerTest.ROOT_DOCKER_Launch
> * DockerContainerizerTest.ROOT_DOCKER_Usage
> * DockerContainerizerTest.ROOT_DOCKER_Update
> * DockerContainerizerTest.ROOT_DOCKER_SkipRecoverNonDocker
> Note that this is not a comprehensive list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-4061) Flaky tests: docker containerizer tests on debian 8 VM
[
https://issues.apache.org/jira/browse/MESOS-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038556#comment-15038556
]
Till Toenshoff commented on MESOS-4061:
---
Just for the fun of it - same test on VMware Fusion:
{noformat}
real0m5.887s
user0m0.028s
sys 0m0.020s
{noformat}
> Flaky tests: docker containerizer tests on debian 8 VM
> --
>
> Key: MESOS-4061
> URL: https://issues.apache.org/jira/browse/MESOS-4061
> Project: Mesos
> Issue Type: Bug
> Environment: debian 8, vagrant, virtual box
>Reporter: Jojy Varghese
>
> Following tests were failing for 0.26 rc3:
> * DockerContainerizerTest.ROOT_DOCKER_NC_PortMapping
> * DockerContainerizerTest.ROOT_DOCKER_Recover
> * DockerContainerizerTest.ROOT_DOCKER_SlaveRecoveryTaskContainer
> * DockerContainerizerTest.ROOT_DOCKER_Launch_Executor
> * DockerContainerizerTest.ROOT_DOCKER_Launch
> * DockerContainerizerTest.ROOT_DOCKER_Usage
> * DockerContainerizerTest.ROOT_DOCKER_Update
> * DockerContainerizerTest.ROOT_DOCKER_SkipRecoverNonDocker
> Note that this is not a comprehensive list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-4061) Flaky tests: docker containerizer tests on debian 8 VM
[ https://issues.apache.org/jira/browse/MESOS-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038584#comment-15038584 ] Till Toenshoff commented on MESOS-4061: --- The more intersting test here actually would be re-running the same command (with a new name) as that would cut out the download of that busybox image. > Flaky tests: docker containerizer tests on debian 8 VM > -- > > Key: MESOS-4061 > URL: https://issues.apache.org/jira/browse/MESOS-4061 > Project: Mesos > Issue Type: Bug > Environment: debian 8, vagrant, virtual box >Reporter: Jojy Varghese > > Following tests were failing for 0.26 rc3: > * DockerContainerizerTest.ROOT_DOCKER_NC_PortMapping > * DockerContainerizerTest.ROOT_DOCKER_Recover > * DockerContainerizerTest.ROOT_DOCKER_SlaveRecoveryTaskContainer > * DockerContainerizerTest.ROOT_DOCKER_Launch_Executor > * DockerContainerizerTest.ROOT_DOCKER_Launch > * DockerContainerizerTest.ROOT_DOCKER_Usage > * DockerContainerizerTest.ROOT_DOCKER_Update > * DockerContainerizerTest.ROOT_DOCKER_SkipRecoverNonDocker > Note that this is not a comprehensive list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
