[
https://issues.apache.org/jira/browse/MESOS-6118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15516234#comment-15516234
]
Ian Babrou commented on MESOS-6118:
-----------------------------------
I've tried it and it didn't fix the issue for me:
{noformat}
Sep 23 11:52:01 myhost mesos-agent[10627]: F0923 11:52:01.524873 10648
fs.cpp:140] Check failed: !visitedParents.contains(parentId)
Sep 23 11:52:01 myhost mesos-agent[10627]: *** Check failure stack trace: ***
Sep 23 11:52:01 myhost mesos-agent[10627]: @ 0x7ff3bb5a253d
google::LogMessage::Fail()
Sep 23 11:52:01 myhost mesos-agent[10627]: @ 0x7ff3bb5a41bd
google::LogMessage::SendToLog()
Sep 23 11:52:01 myhost mesos-agent[10627]: @ 0x7ff3bb5a2102
google::LogMessage::Flush()
Sep 23 11:52:01 myhost mesos-agent[10627]: @ 0x7ff3bb5a4ba9
google::LogMessageFatal::~LogMessageFatal()
Sep 23 11:52:01 myhost mesos-agent[10627]: @ 0x7ff3bb07183d
_ZNSt17_Function_handlerIFviEZN5mesos8internal2fs14MountInfoTable4readERK6OptionIiEbEUliE_E9_M_invokeERKSt9_Any_datai
Sep 23 11:52:01 myhost mesos-agent[10627]: @ 0x7ff3bb0717a5
_ZNSt17_Function_handlerIFviEZN5mesos8internal2fs14MountInfoTable4readERK6OptionIiEbEUliE_E9_M_invokeERKSt9_Any_datai
Sep 23 11:52:01 myhost mesos-agent[10627]: @ 0x7ff3bb078c5a
mesos::internal::fs::MountInfoTable::read()
Sep 23 11:52:01 myhost mesos-agent[10627]: @ 0x7ff3bae2c346
mesos::internal::slave::DockerContainerizerProcess::unmountPersistentVolumes()
Sep 23 11:52:01 myhost mesos-agent[10627]: @ 0x7ff3bae48157
mesos::internal::slave::DockerContainerizerProcess::___destroy()
Sep 23 11:52:01 myhost mesos-agent[10627]: @ 0x7ff3bb546094
process::ProcessManager::resume()
Sep 23 11:52:01 myhost mesos-agent[10627]: @ 0x7ff3bb5463b7
_ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUt_vEEE6_M_runEv
Sep 23 11:52:01 myhost mesos-agent[10627]: @ 0x7ff3b9c20970 (unknown)
Sep 23 11:52:01 myhost mesos-agent[10627]: @ 0x7ff3b973f0a4 start_thread
Sep 23 11:52:01 myhost mesos-agent[10627]: @ 0x7ff3b947487d (unknown)
{noformat}
/proc/mounts:
{noformat}
rootfs / rootfs rw,size=65513288k,nr_inodes=16378322 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
devtmpfs /dev devtmpfs rw,nosuid,size=65513304k,nr_inodes=16378326,mode=755 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup
rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd
0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup
rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup
rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup
rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs
rw,relatime,fd=22,pgrp=1,timeout=300,minproto=5,maxproto=5,direct 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
/dev/md127 /state ext4 rw,relatime,stripe=384,data=ordered 0 0
10.10.14.18:/srv/hosts/myhost /srv nfs4
rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.10.23.25,local_lock=none,addr=10.10.14.18
0 0
10.10.14.18:/srv /srv-master nfs4
rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.10.23.25,local_lock=none,addr=10.10.14.18
0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
{noformat}
Build procedure changes:
{noformat}
build:
cd $(BUILDDIR)/mesos
&& \
+ patch -p1 < $(TOP)/rb51620.patch
&& \
autoreconf -f -i -Wall,no-obsolete
&& \
./bootstrap
&& \
./configure --enable-optimize --disable-python --prefix=/usr
&& \
{noformat}
> Agent would crash with docker container tasks due to host mount table read.
> ---------------------------------------------------------------------------
>
> Key: MESOS-6118
> URL: https://issues.apache.org/jira/browse/MESOS-6118
> Project: Mesos
> Issue Type: Bug
> Components: slave
> Affects Versions: 1.0.1
> Environment: Build: 2016-08-26 23:06:27 by centos
> Version: 1.0.1
> Git tag: 1.0.1
> Git SHA: 3611eb0b7eea8d144e9b2e840e0ba16f2f659ee3
> systemd version `219` detected
> Inializing systemd state
> Created systemd slice: `/run/systemd/system/mesos_executors.slice`
> Started systemd slice `mesos_executors.slice`
> Using isolation: posix/cpu,posix/mem,filesystem/posix,network/cni
> Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
> Linux ip-10-254-192-40 3.10.0-327.28.3.el7.x86_64 #1 SMP Thu Aug 18 19:05:49
> UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> Reporter: Jamie Briant
> Assignee: Kevin Klues
> Priority: Critical
> Labels: linux, slave
> Fix For: 1.1.0, 1.0.2
>
> Attachments: crashlogfull.log, cycle2.log, cycle3.log, cycle5.log,
> cycle6.log, slave-crash.log
>
>
> I have a framework which schedules thousands of short running (a few seconds
> to a few minutes) of tasks, over a period of several minutes. In 1.0.1, the
> slave process will crash every few minutes (with systemd restarting it).
> Crash is:
> Sep 01 20:52:23 ip-10-254-192-99 mesos-slave: F0901 20:52:23.905678 1232
> fs.cpp:140] Check failed: !visitedParents.contains(parentId)
> Sep 01 20:52:23 ip-10-254-192-99 mesos-slave: *** Check failure stack trace:
> ***
> Version 1.0.0 works without this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
