[
https://issues.apache.org/jira/browse/MESOS-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14902211#comment-14902211
]
Chengwei Yang commented on MESOS-3488:
--------------------------------------
I setup auditd and auditctl for /sys/fs/cgroup/memory/mesos like below
# auditctl -l
-w /sys/fs/cgroup/memory/mesos -p rwxa
And having mesos-slave running for about 1 hour
When I check again, /sys/fs/cgroup/memory/mesos gone and I see below logs in
/var/log/audit/audit.log
type=ANOM_PROMISCUOUS msg=audit(1442908163.322:924): dev=vethff752e0 prom=0
old_prom=256 auid=4294967295 uid=0 gid=0 ses=4294967295
type=SYSCALL msg=audit(1442908165.127:925): arch=c000003e syscall=84
success=yes exit=0 a0=7f59840088c8 a1=7f5993e65730 a2=0 a3=0 items=2 ppid=1
pid=12682 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0
fsgid=0 tty=(none) ses=4294967295 comm="mesos-slave"
exe="/usr/local/sbin/mesos-slave" key=(null)
type=CWD msg=audit(1442908165.127:925): cwd="/"
type=PATH msg=audit(1442908165.127:925): item=0
name="/sys/fs/cgroup/memory/mesos/" inode=112987 dev=00:18 mode=040755 ouid=0
ogid=0 rdev=00:00 objtype=PARENT
type=PATH msg=audit(1442908165.127:925): item=1
name="/sys/fs/cgroup/memory/mesos/be0aa404-c348-491b-ae58-518f306716ef"
inode=121617 dev=00:18 mode=040755 ouid=0 ogid=0 rdev=00:00 objtype=DELETE
type=USER_ACCT msg=audit(1442908201.173:926): pid=1352 uid=0 auid=4294967295
ses=4294967295 msg='op=PAM:accounting
grantors=pam_access,pam_unix,pam_localuser acct="root" exe="/usr/sbin/crond"
hostname=? addr=? terminal=cron res=success'
type=USER_ACCT msg=audit(1442908201.173:927): pid=1353 uid=0 auid=4294967295
ses=4294967295 msg='op=PAM:accounting
grantors=pam_access,pam_unix,pam_localuser acct="root" exe="/usr/sbin/crond"
hostname=? addr=? terminal=cron res=success'
type=CRED_ACQ msg=audit(1442908201.173:928): pid=1352 uid=0 auid=4294967295
ses=4294967295 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root"
exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=USER_ACCT msg=audit(1442908201.173:929): pid=1354 uid=0 auid=4294967295
ses=4294967295 msg='op=PAM:accounting
grantors=pam_access,pam_unix,pam_localuser acct="root" exe="/usr/sbin/crond"
hostname=? addr=? terminal=cron res=success'
type=LOGIN msg=audit(1442908201.173:930): pid=1352 uid=0 old-auid=4294967295
auid=0 old-ses=4294967295 ses=291 res=1
type=CRED_ACQ msg=audit(1442908201.174:931): pid=1353 uid=0 auid=4294967295
ses=4294967295 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root"
exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=CRED_ACQ msg=audit(1442908201.174:932): pid=1354 uid=0 auid=4294967295
ses=4294967295 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root"
exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=LOGIN msg=audit(1442908201.174:933): pid=1353 uid=0 old-auid=4294967295
auid=0 old-ses=4294967295 ses=292 res=1
type=LOGIN msg=audit(1442908201.174:934): pid=1354 uid=0 old-auid=4294967295
auid=0 old-ses=4294967295 ses=293 res=1
type=CONFIG_CHANGE msg=audit(1442908201.201:935): auid=4294967295
ses=4294967295 op="updated rules" path="/sys/fs/cgroup/memory/mesos" key=(null)
list=4 res=1
type=CONFIG_CHANGE msg=audit(1442908201.201:936): auid=4294967295
ses=4294967295 op="updated rules" path="/sys/fs/cgroup/memory/mesos" key=(null)
list=4 res=1
type=USER_START msg=audit(1442908201.209:937): pid=1352 uid=0 auid=0 ses=291
msg='op=PAM:session_open
grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct="root"
exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=USER_START msg=audit(1442908201.209:938): pid=1354 uid=0 auid=0 ses=293
msg='op=PAM:session_open
grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct="root"
exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=CRED_REFR msg=audit(1442908201.210:939): pid=1352 uid=0 auid=0 ses=291
msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond"
hostname=? addr=? terminal=cron res=success'
I didn't found any DELETE for /sys/fs/cgroup/memory/mesos in the log, doesn't
understand what the last two "CONFIG_CHANGE" for /sys/fs/cgroup/memory/mesos
means. And I did see *auditctl* says the rule is here.
> /sys/fs/cgroup/memory/mesos missing after running a while
> ---------------------------------------------------------
>
> Key: MESOS-3488
> URL: https://issues.apache.org/jira/browse/MESOS-3488
> Project: Mesos
> Issue Type: Bug
> Affects Versions: 0.21.0
> Environment: mesos 0.21.0 on CentOS 7.1
> Reporter: Chengwei Yang
>
> I setup mesos 0.21.0 on CentOS 7.1 with mesos-0.21.0 rpm downloaded from
> mesosphere.
> at first, it works fine, jobs are finished correctly, however, after running
> a while, all task goes to LOST.
> And there is nothing in **sandbox** and I see below from mesos-slave.ERROR
> ```
> E0922 14:02:31.329264 8336 slave.cpp:2787] Container
> '865c9a1c-3abe-4263-921e-8d0be2f0a56d' for executor
> '21bf1400-6456-45e5-8f28-fe5ade7e7bfd' of framework
> '20150401-105258-3755085578-5050-14676-0000' failed to start: Failed to
> prepare isolator: Failed to create directory
> '/sys/fs/cgroup/memory/mesos/865c9a1c-3abe-4263-921e-8d0be2f0a56d': No such
> file or directory
> ```
> And I checked that /sys/fs/cgroup/cpu/mesos does exist while
> /sys/fs/cgroup/memory/mesos is missing.
> Since at first mesos works fine (and I checked source code, that if it create
> /sys/fs/cgroup/memory/mesos failed at mesos-slave startup, it will log that
> error), so I'm curious when/which removed /sys/fs/cgroup/memory/mesos.
> Has anyone saw this issue before?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
