[
https://issues.apache.org/jira/browse/MESOS-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15590120#comment-15590120
]
Santhosh Shanmugham commented on MESOS-6420:
--------------------------------------------
I1019 22:50:45.495558 1117 slave.cpp:1495] Got assigned task
sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab for framework
201103282247-0000000019-0000
I1019 22:50:45.504261 1117 gc.cpp:83] Unscheduling
'/var/lib/mesos/slaves/7f9be60e-9bcf-47e0-8c6b-8b8ab278ecbe-S0/frameworks/201103282247-0000000019-0000'
from gc
I1019 22:50:45.505200 1117 gc.cpp:83] Unscheduling
'/var/lib/mesos/meta/slaves/7f9be60e-9bcf-47e0-8c6b-8b8ab278ecbe-S0/frameworks/201103282247-0000000019-0000'
from gc
I1019 22:50:45.506726 1116 slave.cpp:1614] Launching task
sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab for framework
201103282247-0000000019-0000
I1019 22:50:45.510138 1116 paths.cpp:528] Trying to chown
'/var/lib/mesos/slaves/7f9be60e-9bcf-47e0-8c6b-8b8ab278ecbe-S0/frameworks/201103282247-0000000019-0000/executors/thermos-sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab/runs/0f5238be-4dd6-4ffe-9250-ceddc904174f'
to user 'root'
==17403== Memcheck, a memory error detector
==17403== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==17403== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==17403== Command: /bin/sh -c chown\ -R\ 0:0\
'/var/lib/mesos/slaves/7f9be60e-9bcf-47e0-8c6b-8b8ab278ecbe-S0/frameworks/201103282247-0000000019-0000/executors/thermos-sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab/runs/0f5238be-4dd6-4ffe-9250-ceddc904174f'
==17403==
==17403== Memcheck, a memory error detector
==17403== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==17403== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==17403== Command: /bin/chown -R 0:0
/var/lib/mesos/slaves/7f9be60e-9bcf-47e0-8c6b-8b8ab278ecbe-S0/frameworks/201103282247-0000000019-0000/executors/thermos-sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab/runs/0f5238be-4dd6-4ffe-9250-ceddc904174f
==17403==
==17403==
==17403== FILE DESCRIPTORS: 23 open at exit.
==17403== Open file descriptor 33:
==17403== <inherited from parent>
==17403==
==17403== Open file descriptor 32:
==17403== <inherited from parent>
==17403==
==17403== Open file descriptor 31:
==17403== <inherited from parent>
==17403==
==17403== Open file descriptor 30:
==17403== <inherited from parent>
==17403==
==17403== Open file descriptor 29:
==17403== <inherited from parent>
==17403==
==17403== Open file descriptor 28:
==17403== <inherited from parent>
==17403==
==17403== Open file descriptor 27:
==17403== <inherited from parent>
==17403==
==17403== Open file descriptor 25:
==17403== <inherited from parent>
==17403==
==17403== Open file descriptor 23:
==17403== <inherited from parent>
==17403==
==17403== Open file descriptor 22:
==17403== <inherited from parent>
==17403==
==17403== Open file descriptor 21:
==17403== <inherited from parent>
==17403==
==17403== Open file descriptor 20:
==17403== <inherited from parent>
==17403==
==17403== Open file descriptor 19:
==17403== <inherited from parent>
==17403==
==17403== Open file descriptor 17:
==17403== <inherited from parent>
==17403==
==17403== Open file descriptor 16:
==17403== <inherited from parent>
==17403==
==17403== Open file descriptor 15:
==17403== <inherited from parent>
==17403==
==17403== Open file descriptor 14:
==17403== <inherited from parent>
==17403==
==17403== Open file descriptor 12:
==17403== <inherited from parent>
==17403==
==17403== Open AF_INET socket 10: 10.34.124.106:60994 <-> 10.35.95.111:2181
==17403== <inherited from parent>
==17403==
==17403== Open file descriptor 8:
==17403== <inherited from parent>
==17403==
==17403== Open file descriptor 7:
==17403== <inherited from parent>
==17403==
==17403== Open file descriptor 2:
==17403== <inherited from parent>
==17403==
==17403== Open file descriptor 0: /dev/null
==17403== <inherited from parent>
==17403==
==17403==
==17403== HEAP SUMMARY:
==17403== in use at exit: 4 bytes in 2 blocks
==17403== total heap usage: 106 allocs, 104 frees, 575,581 bytes allocated
==17403==
==17403== LEAK SUMMARY:
==17403== definitely lost: 0 bytes in 0 blocks
==17403== indirectly lost: 0 bytes in 0 blocks
==17403== possibly lost: 0 bytes in 0 blocks
==17403== still reachable: 4 bytes in 2 blocks
==17403== suppressed: 0 bytes in 0 blocks
==17403== Reachable blocks (those to which a pointer was found) are not shown.
==17403== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==17403==
==17403== For counts of detected and suppressed errors, rerun with: -v
==17403== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 4 from 4)
I1019 22:50:46.946646 1116 slave.cpp:5674] Launching executor
thermos-sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab of
framework 201103282247-0000000019-0000 with resources mem(*):128 in work
directory
'/var/lib/mesos/slaves/7f9be60e-9bcf-47e0-8c6b-8b8ab278ecbe-S0/frameworks/201103282247-0000000019-0000/executors/thermos-sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab/runs/0f5238be-4dd6-4ffe-9250-ceddc904174f'
I1019 22:50:46.952791 1117 containerizer.cpp:781] Starting container
'0f5238be-4dd6-4ffe-9250-ceddc904174f' for executor
'thermos-sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab' of
framework '201103282247-0000000019-0000'
I1019 22:50:46.972177 1116 slave.cpp:1840] Queuing task
'sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab' for executor
'thermos-sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab' of
framework 201103282247-0000000019-0000
I1019 22:50:46.991857 1117 cpushare.cpp:389] Updated 'cpu.shares' to 1024
(cpus 1) for container 0f5238be-4dd6-4ffe-9250-ceddc904174f
I1019 22:50:47.003842 1117 cpushare.cpp:411] Updated 'cpu.cfs_period_us' to
100ms and 'cpu.cfs_quota_us' to 100ms (cpus 1) for container
0f5238be-4dd6-4ffe-9250-ceddc904174f
I1019 22:50:47.022661 1117 mem.cpp:602] Started listening for OOM events for
container 0f5238be-4dd6-4ffe-9250-ceddc904174f
I1019 22:50:47.028313 1117 mem.cpp:722] Started listening on low memory
pressure events for container 0f5238be-4dd6-4ffe-9250-ceddc904174f
I1019 22:50:47.035648 1117 mem.cpp:722] Started listening on medium memory
pressure events for container 0f5238be-4dd6-4ffe-9250-ceddc904174f
I1019 22:50:47.040839 1117 mem.cpp:722] Started listening on critical memory
pressure events for container 0f5238be-4dd6-4ffe-9250-ceddc904174f
I1019 22:50:47.047967 1117 mem.cpp:353] Updated 'memory.soft_limit_in_bytes'
to 256MB for container 0f5238be-4dd6-4ffe-9250-ceddc904174f
I1019 22:50:47.058745 1117 mem.cpp:388] Updated 'memory.limit_in_bytes' to
256MB for container 0f5238be-4dd6-4ffe-9250-ceddc904174f
I1019 22:50:47.061167 1117 port_mapping.cpp:2511] Using non-ephemeral ports {}
and ephemeral ports [41984,43008) for container
0f5238be-4dd6-4ffe-9250-ceddc904174f of executor
'thermos-sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab'
I1019 22:50:47.067237 1117 perf_event.cpp:217] Preparing perf event cgroup for
0f5238be-4dd6-4ffe-9250-ceddc904174f
I1019 22:50:47.089234 1117 linux_launcher.cpp:281] Cloning child process with
flags = CLONE_NEWNS | CLONE_NEWNET
W1019 22:50:47.115586 1117 subprocess.hpp:422] Failed to execute
Subprocess::Hook in parent for child '17427': Failed to assign process to its
freezer cgroup: Failed to create freezer cgroup: Failed to create directory
'/sys/fs/cgroup/freezer/mesos/0f5238be-4dd6-4ffe-9250-ceddc904174f': File exists
E1019 22:50:47.121644 1117 slave.cpp:3976] Container
'0f5238be-4dd6-4ffe-9250-ceddc904174f' for executor
'thermos-sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab' of
framework 201103282247-0000000019-0000 failed to start: Failed to fork
executor: Failed to clone child process: Failed to execute Subprocess::Hook in
parent for child '17427': Failed to assign process to its freezer cgroup:
Failed to create freezer cgroup: Failed to create directory
'/sys/fs/cgroup/freezer/mesos/0f5238be-4dd6-4ffe-9250-ceddc904174f': File exists
I1019 22:50:47.122350 1117 containerizer.cpp:1622] Destroying container
'0f5238be-4dd6-4ffe-9250-ceddc904174f'
W1019 22:50:47.147097 1122 port_mapping.cpp:3335] The container has not been
isolated
I1019 22:50:47.220263 1122 slave.cpp:4082] Executor
'thermos-sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab' of
framework 201103282247-0000000019-0000 has terminated with unknown status
I1019 22:50:47.221346 1122 slave.cpp:3211] Handling status update TASK_FAILED
(UUID: ba27f178-1238-4304-9c27-31142feee177) for task
sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab of framework
201103282247-0000000019-0000 from @0.0.0.0:0
W1019 22:50:47.225955 1122 containerizer.cpp:1451] Ignoring update for unknown
container: 0f5238be-4dd6-4ffe-9250-ceddc904174f
I1019 22:50:47.228137 1122 status_update_manager.cpp:320] Received status
update TASK_FAILED (UUID: ba27f178-1238-4304-9c27-31142feee177) for task
sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab of framework
201103282247-0000000019-0000
I1019 22:50:47.233654 1122 status_update_manager.cpp:825] Checkpointing UPDATE
for status update TASK_FAILED (UUID: ba27f178-1238-4304-9c27-31142feee177) for
task sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab of
framework 201103282247-0000000019-0000
I1019 22:50:47.243929 1122 slave.cpp:3604] Forwarding the update TASK_FAILED
(UUID: ba27f178-1238-4304-9c27-31142feee177) for task
sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab of framework
201103282247-0000000019-0000 to master@10.52.191.133:5050
I1019 22:50:47.256544 1122 status_update_manager.cpp:392] Received status
update acknowledgement (UUID: ba27f178-1238-4304-9c27-31142feee177) for task
sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab of framework
201103282247-0000000019-0000
I1019 22:50:47.257736 1122 status_update_manager.cpp:825] Checkpointing ACK
for status update TASK_FAILED (UUID: ba27f178-1238-4304-9c27-31142feee177) for
task sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab of
framework 201103282247-0000000019-0000
I1019 22:50:47.263334 1122 slave.cpp:4193] Cleaning up executor
'thermos-sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab' of
framework 201103282247-0000000019-0000
I1019 22:50:47.267307 1117 gc.cpp:55] Scheduling
'/var/lib/mesos/slaves/7f9be60e-9bcf-47e0-8c6b-8b8ab278ecbe-S0/frameworks/201103282247-0000000019-0000/executors/thermos-sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab/runs/0f5238be-4dd6-4ffe-9250-ceddc904174f'
for gc 6.99999692708148days in the future
I1019 22:50:47.268810 1117 gc.cpp:55] Scheduling
'/var/lib/mesos/slaves/7f9be60e-9bcf-47e0-8c6b-8b8ab278ecbe-S0/frameworks/201103282247-0000000019-0000/executors/thermos-sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab'
for gc 6.99999691644741days in the future
I1019 22:50:47.268332 1122 slave.cpp:4281] Cleaning up framework
201103282247-0000000019-0000
I1019 22:50:47.269600 1117 gc.cpp:55] Scheduling
'/var/lib/mesos/meta/slaves/7f9be60e-9bcf-47e0-8c6b-8b8ab278ecbe-S0/frameworks/201103282247-0000000019-0000/executors/thermos-sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab/runs/0f5238be-4dd6-4ffe-9250-ceddc904174f'
for gc 6.99999690366815days in the future
I1019 22:50:47.270548 1117 gc.cpp:55] Scheduling
'/var/lib/mesos/meta/slaves/7f9be60e-9bcf-47e0-8c6b-8b8ab278ecbe-S0/frameworks/201103282247-0000000019-0000/executors/thermos-sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab'
for gc 6.99999689605333days in the future
I1019 22:50:47.272044 1121 status_update_manager.cpp:282] Closing status
update streams for framework 201103282247-0000000019-0000
I1019 22:50:47.273151 1112 gc.cpp:55] Scheduling
'/var/lib/mesos/slaves/7f9be60e-9bcf-47e0-8c6b-8b8ab278ecbe-S0/frameworks/201103282247-0000000019-0000'
for gc 6.99999684968889days in the future
I1019 22:50:47.274618 1119 gc.cpp:55] Scheduling
'/var/lib/mesos/meta/slaves/7f9be60e-9bcf-47e0-8c6b-8b8ab278ecbe-S0/frameworks/201103282247-0000000019-0000'
for gc 6.99999683078815days in the future
I1019 22:51:43.266806 1122 slave.cpp:4591] Current disk usage 27.77%. Max
allowed age: 4.356259166822419days
I1019 22:51:46.950247 1122 slave.cpp:4499] Framework
201103282247-0000000019-0000 seems to have exited. Ignoring registration
timeout for executor
'thermos-sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab'
I1019 22:52:43.269429 1125 slave.cpp:4591] Current disk usage 27.77%. Max
allowed age: 4.356256384320220days
I1019 22:53:43.271926 1114 slave.cpp:4591] Current disk usage 27.77%. Max
allowed age: 4.356254489850625days
> Mesos Agent leaking sockets when network isolation is ON
> --------------------------------------------------------
>
> Key: MESOS-6420
> URL: https://issues.apache.org/jira/browse/MESOS-6420
> Project: Mesos
> Issue Type: Bug
> Components: isolation, network, slave
> Affects Versions: 1.0.2
> Reporter: Santhosh Shanmugham
>
> Mesos Agent leaks one socket per task launched and eventually runs out of
> sockets. We were able to track it down to the network isolator
> (port_mapping.cpp). When we turned off the port mapping isolator no file
> descriptors where leaked. The leaked fd is a SOCK_STREAM socket.
> Leaked Sockets:
> sshanmugham[5]smf1-aeu-07-sr3(mesos.test.slave) ~ $ sudo lsof -p $(pgrep -u
> root -o -f /usr/local/sbin/mesos-slave) -nP | grep "can't"
> [sudo] password for sshanmugham:
> mesos-sla 57688 root 19u sock 0,6 0t0 2993216948 can't
> identify protocol
> mesos-sla 57688 root 27u sock 0,6 0t0 2993216468 can't
> identify protocol
> Extract from strace:
> ...
> [pid 57701] 19:14:02.493718 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.494395 close(19) = 0
> [pid 57701] 19:14:02.494448 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.494844 close(19) = 0
> [pid 57701] 19:14:02.494913 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.495565 close(19) = 0
> [pid 57701] 19:14:02.495617 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.496072 close(19) = 0
> [pid 57701] 19:14:02.496128 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.496758 close(19) = 0
> [pid 57701] 19:14:02.496812 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.497270 close(19) = 0
> [pid 57701] 19:14:02.497319 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.497698 close(19) = 0
> [pid 57701] 19:14:02.497750 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.498407 close(19) = 0
> [pid 57701] 19:14:02.498456 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.498899 close(19) = 0
> [pid 57701] 19:14:02.498963 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 63682] 19:14:02.499091 close(18 <unfinished ...>
> [pid 57701] 19:14:02.499634 close(19) = 0
> [pid 57701] 19:14:02.499689 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.500044 close(19) = 0
> [pid 57701] 19:14:02.500093 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.500734 close(19) = 0
> [pid 57701] 19:14:02.500782 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.501271 close(19) = 0
> [pid 57701] 19:14:02.501339 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.502030 close(19) = 0
> [pid 57701] 19:14:02.502101 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 19
> ...
> ...
> [pid 57691] 19:18:03.461022 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 6138] 19:18:03.461345 open("/etc/selinux/config", O_RDONLY <unfinished
> ...>
> [pid 57691] 19:18:03.461460 close(27) = 0
> [pid 57691] 19:18:03.461520 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 6138] 19:18:03.461632 close(3 <unfinished ...>
> [pid 6138] 19:18:03.461781 open("/proc/mounts", O_RDONLY <unfinished ...>
> [pid 6138] 19:18:03.462190 close(3 <unfinished ...>
> [pid 57691] 19:18:03.462374 close(27) = 0
> [pid 57691] 19:18:03.462430 socket(PF_NETLINK, SOCK_RAW, 0 <unfinished ...>
> [pid 6138] 19:18:03.462456 open("/proc/net/psched", O_RDONLY <unfinished ...>
> [pid 6138] 19:18:03.462678 close(3 <unfinished ...>
> [pid 6138] 19:18:03.462915 open("/etc/libnl/classid", O_RDONLY <unfinished
> ...>
> [pid 57691] 19:18:03.463046 close(27) = 0
> [pid 57691] 19:18:03.463111 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 6138] 19:18:03.463225 close(3 <unfinished ...>
> [pid 57691] 19:18:03.463845 close(27) = 0
> [pid 57691] 19:18:03.463911 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.464604 close(27) = 0
> [pid 57691] 19:18:03.464664 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.465074 close(27) = 0
> [pid 57691] 19:18:03.465132 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.465862 close(27) = 0
> [pid 57691] 19:18:03.465928 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.466713 close(27) = 0
> [pid 57691] 19:18:03.466780 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.467472 close(27) = 0
> [pid 57691] 19:18:03.467524 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.468012 close(27) = 0
> [pid 57691] 19:18:03.468075 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.468799 close(27) = 0
> [pid 57691] 19:18:03.468950 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.469505 close(27) = 0
> [pid 57691] 19:18:03.469578 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.470301 close(27) = 0
> [pid 57691] 19:18:03.470353 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 27
> ...
> The last socket the was created never has a corresponding close().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
