Jingang Li created MESOS-4423:
---------------------------------
Summary: LinuxFilesystemIsolatorTest.ROOT_MultipleContainers fails
as can't find sleep in container1
Key: MESOS-4423
URL: https://issues.apache.org/jira/browse/MESOS-4423
Project: Mesos
Issue Type: Bug
Affects Versions: 0.26.0
Environment: Suse SLE12 SP1
Reporter: Jingang Li
The test case LinuxFilesystemIsolatorTest.ROOT_MultipleContainers always fails
as below error. It should be container1 doesn't have the sleep command. In my
system, i have the command installed.
linux-vkq1:~/mesos/mesos-0.26.0/build # which sleep
/usr/bin/sleep
Case error:
linux-vkq1:~/mesos/mesos-0.26.0/build # ./bin/mesos-tests.sh
--gtest_filter="LinuxFilesystemIsolatorTest.ROOT_MultipleContainers" --verbose
Source directory: /root/mesos/mesos-0.26.0
Build directory: /root/mesos/mesos-0.26.0/build
/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us
-------------------------------------------------------------
We cannot run any cgroups tests that require mounting
hierarchies because you have the following hierarchies mounted:
/sys/fs/cgroup/blkio, /sys/fs/cgroup/cpu,cpuacct, /sys/fs/cgroup/cpuset,
/sys/fs/cgroup/devices, /sys/fs/cgroup/freezer, /sys/fs/cgroup/hugetlb,
/sys/fs/cgroup/memory, /sys/fs/cgroup/perf_event, /sys/fs/cgroup/systemd
We'll disable the CgroupsNoHierarchyTest test fixture for now.
-------------------------------------------------------------
/usr/bin/nc
Note: Google Test filter =
LinuxFilesystemIsolatorTest.ROOT_MultipleContainers-CgroupsNoHierarchyTest.ROOT_CGROUPS_NOHIERARCHY_MountUnmountHierarchy:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/0:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/1:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/2:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/3:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/4:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/5:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/6:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/7:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/8:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/9:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/10:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/11:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/12:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/13:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/14:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/15:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/16:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/17:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/18:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/19:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/20:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/21:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/22:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/23:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/24:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/25:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/26:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/27:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/28:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/29:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/30:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/31:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/32:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/33:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/34:SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/35:SlaveCount/Registrar_BENCHMARK_Test.Performance/0:SlaveCount/Registrar_BENCHMARK_Test.Performance/1:SlaveCount/Registrar_BENCHMARK_Test.Performance/2:SlaveCount/Registrar_BENCHMARK_Test.Performance/3
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from LinuxFilesystemIsolatorTest
[ RUN ] LinuxFilesystemIsolatorTest.ROOT_MultipleContainers
I0119 08:17:47.580173 17995 linux.cpp:82] Making '/tmp/7aEohd' a shared mount
I0119 08:17:47.598511 17995 linux_launcher.cpp:103] Using
/sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
I0119 08:17:47.602372 17995 systemd.cpp:128] systemd version `210` detected
W0119 08:17:47.602429 17995 systemd.cpp:136] Required functionality `Delegate`
was introduced in Version `218`. Your system may not function properly; however
since some distributions have patched systemd packages, your system may still
be functional. This is why we keep running. See MESOS-3352 for more information
I0119 08:17:47.607328 17995 systemd.cpp:210] Started systemd slice
`mesos_executors.slice`
I0119 08:17:47.610362 18011 containerizer.cpp:618] Starting container
'aee809bb-db8c-4044-8a1e-17fcc9e478ee' for executor 'test_executor1' of
framework ''
I0119 08:17:47.614701 18016 provisioner.cpp:289] Provisioning image rootfs
'/tmp/7aEohd/provisioner/containers/aee809bb-db8c-4044-8a1e-17fcc9e478ee/backends/copy/rootfses/af0c5832-35df-4920-ba6d-ed38d80a011a'
for container aee809bb-db8c-4044-8a1e-17fcc9e478ee
I0119 08:17:48.416606 18010 linux.cpp:391] Bind mounting work directory from
'/tmp/7aEohd/slaves/test_slave/frameworks/executors/test_executor1/runs/aee809bb-db8c-4044-8a1e-17fcc9e478ee'
to
'/tmp/7aEohd/provisioner/containers/aee809bb-db8c-4044-8a1e-17fcc9e478ee/backends/copy/rootfses/af0c5832-35df-4920-ba6d-ed38d80a011a/mnt/mesos/sandbox'
for container aee809bb-db8c-4044-8a1e-17fcc9e478ee
I0119 08:17:48.417196 18010 linux.cpp:722] Changing the ownership of the
persistent volume at '/tmp/7aEohd/volumes/roles/test_role/persistent_volume_id'
with uid 0 and gid 0
I0119 08:17:48.423754 18010 linux.cpp:762] Mounting
'/tmp/7aEohd/volumes/roles/test_role/persistent_volume_id' to
'/tmp/7aEohd/provisioner/containers/aee809bb-db8c-4044-8a1e-17fcc9e478ee/backends/copy/rootfses/af0c5832-35df-4920-ba6d-ed38d80a011a/mnt/mesos/sandbox/volume'
for persistent volume disk(test_role)[persistent_volume_id:volume]:32 of
container aee809bb-db8c-4044-8a1e-17fcc9e478ee
I0119 08:17:48.428866 18010 linux_launcher.cpp:365] Cloning child process with
flags = CLONE_NEWNS
I0119 08:17:48.431560 18010 linux_launcher.cpp:422] Assigned child process
'18053' to 'mesos_executors.slice'
I0119 08:17:48.435427 18011 containerizer.cpp:618] Starting container
'44249e16-6263-4a00-a737-2c13bd4fd405' for executor 'test_executor2' of
framework ''
I0119 08:17:48.437712 18011 provisioner.cpp:289] Provisioning image rootfs
'/tmp/7aEohd/provisioner/containers/44249e16-6263-4a00-a737-2c13bd4fd405/backends/copy/rootfses/4c9037b6-0e4f-4a30-8dee-29f3249b9898'
for container 44249e16-6263-4a00-a737-2c13bd4fd405
+ /root/mesos/mesos-0.26.0/build/src/mesos-containerizer mount --help=false
--operation=make-rslave --path=/
+ grep -E /tmp/7aEohd/.+ /proc/self/mountinfo
+ grep -v aee809bb-db8c-4044-8a1e-17fcc9e478ee
+ cut '-d ' -f5
+ xargs --no-run-if-empty umount -l
Changing root to
/tmp/7aEohd/provisioner/containers/aee809bb-db8c-4044-8a1e-17fcc9e478ee/backends/copy/rootfses/af0c5832-35df-4920-ba6d-ed38d80a011a
sh: sleep: command not found
I0119 08:17:48.724886 18012 containerizer.cpp:1257] Executor for container
'aee809bb-db8c-4044-8a1e-17fcc9e478ee' has exited
I0119 08:17:48.724957 18012 containerizer.cpp:1074] Destroying container
'aee809bb-db8c-4044-8a1e-17fcc9e478ee'
I0119 08:17:48.728327 18012 cgroups.cpp:2429] Freezing cgroup
/sys/fs/cgroup/freezer/mesos/aee809bb-db8c-4044-8a1e-17fcc9e478ee
I0119 08:17:48.731309 18015 cgroups.cpp:1411] Successfully froze cgroup
/sys/fs/cgroup/freezer/mesos/aee809bb-db8c-4044-8a1e-17fcc9e478ee after
2.92096ms
I0119 08:17:48.734146 18010 cgroups.cpp:2447] Thawing cgroup
/sys/fs/cgroup/freezer/mesos/aee809bb-db8c-4044-8a1e-17fcc9e478ee
I0119 08:17:48.736498 18010 cgroups.cpp:1440] Successfullly thawed cgroup
/sys/fs/cgroup/freezer/mesos/aee809bb-db8c-4044-8a1e-17fcc9e478ee after
2.292992ms
I0119 08:17:48.740923 18010 linux.cpp:835] Unmounting volume
'/tmp/7aEohd/provisioner/containers/aee809bb-db8c-4044-8a1e-17fcc9e478ee/backends/copy/rootfses/af0c5832-35df-4920-ba6d-ed38d80a011a/mnt/mesos/sandbox/volume'
for container aee809bb-db8c-4044-8a1e-17fcc9e478ee
I0119 08:17:48.741017 18010 linux.cpp:854] Unmounting sandbox/work directory
'/tmp/7aEohd/provisioner/containers/aee809bb-db8c-4044-8a1e-17fcc9e478ee/backends/copy/rootfses/af0c5832-35df-4920-ba6d-ed38d80a011a/mnt/mesos/sandbox'
for container aee809bb-db8c-4044-8a1e-17fcc9e478ee
I0119 08:17:48.741328 18010 provisioner.cpp:332] Destroying container rootfs at
'/tmp/7aEohd/provisioner/containers/aee809bb-db8c-4044-8a1e-17fcc9e478ee/backends/copy/rootfses/af0c5832-35df-4920-ba6d-ed38d80a011a'
for container aee809bb-db8c-4044-8a1e-17fcc9e478ee
I0119 08:17:49.635599 18011 linux.cpp:391] Bind mounting work directory from
'/tmp/7aEohd/slaves/test_slave/frameworks/executors/test_executor2/runs/44249e16-6263-4a00-a737-2c13bd4fd405'
to
'/tmp/7aEohd/provisioner/containers/44249e16-6263-4a00-a737-2c13bd4fd405/backends/copy/rootfses/4c9037b6-0e4f-4a30-8dee-29f3249b9898/mnt/mesos/sandbox'
for container 44249e16-6263-4a00-a737-2c13bd4fd405
I0119 08:17:49.639376 18011 linux_launcher.cpp:365] Cloning child process with
flags = CLONE_NEWNS
I0119 08:17:49.641855 18011 linux_launcher.cpp:422] Assigned child process
'18092' to 'mesos_executors.slice'
W0119 08:17:49.643654 18011 containerizer.cpp:1063] Ignoring destroy of unknown
container: aee809bb-db8c-4044-8a1e-17fcc9e478ee
../../src/tests/containerizer/filesystem_isolator_tests.cpp:926: Failure
(wait1).failure(): Unknown container: aee809bb-db8c-4044-8a1e-17fcc9e478ee
+ /root/mesos/mesos-0.26.0/build/src/mesos-containerizer mount --help=false
--operation=make-rslave --path=/
+ grep -E /tmp/7aEohd/.+ /proc/self/mountinfo
+ grep -v 44249e16-6263-4a00-a737-2c13bd4fd405
+ cut '-d ' -f5
+ xargs --no-run-if-empty umount -l
Changing root to
/tmp/7aEohd/provisioner/containers/44249e16-6263-4a00-a737-2c13bd4fd405/backends/copy/rootfses/4c9037b6-0e4f-4a30-8dee-29f3249b9898
umount:
/tmp/7aEohd/provisioner/containers/44249e16-6263-4a00-a737-2c13bd4fd405/backends/copy/rootfses/4c9037b6-0e4f-4a30-8dee-29f3249b9898/mnt/mesos/sandbox:
not mounted
E0119 08:17:50.303649 17995 shell.hpp:90] Command 'grep '/tmp/7aEohd'
/proc/mounts | cut -d' ' -f2 | xargs --no-run-if-empty umount -l' failed; this
is the output:
E0119 08:17:50.303710 17995 filesystem_isolator_tests.cpp:102] Failed to umount
for sandbox '/tmp/7aEohd': Failed to execute 'grep '/tmp/7aEohd' /proc/mounts |
cut -d' ' -f2 | xargs --no-run-if-empty umount -l'; the command was either not
found or exited with a non-zero exit status: 123
[ FAILED ] LinuxFilesystemIsolatorTest.ROOT_MultipleContainers (4210 ms)
[----------] 1 test from LinuxFilesystemIsolatorTest (4210 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (4227 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] LinuxFilesystemIsolatorTest.ROOT_MultipleContainers
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
