[jira] [Commented] (MESOS-6420) Mesos Agent leaking sockets when port mapping network isolator is ON
[
https://issues.apache.org/jira/browse/MESOS-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591253#comment-15591253
]
Santhosh Shanmugham commented on MESOS-6420:
https://reviews.apache.org/r/53049/
> Mesos Agent leaking sockets when port mapping network isolator is ON
>
>
> Key: MESOS-6420
> URL: https://issues.apache.org/jira/browse/MESOS-6420
> Project: Mesos
> Issue Type: Bug
> Components: isolation, network, slave
>Affects Versions: 1.0.2
>Reporter: Santhosh Shanmugham
>
> Mesos Agent leaks one socket per task launched and eventually runs out of
> sockets. We were able to track it down to the network isolator
> (port_mapping.cpp). When we turned off the port mapping isolator no file
> descriptors where leaked. The leaked fd is a SOCK_STREAM socket.
> Leaked Sockets:
> $ sudo lsof -p $(pgrep -u root -o -f /usr/local/sbin/mesos-slave) -nP | grep
> "can't"
> [sudo] password for sshanmugham:
> mesos-sla 57688 root 19u sock0,6 0t0 2993216948 can't
> identify protocol
> mesos-sla 57688 root 27u sock0,6 0t0 2993216468 can't
> identify protocol
> Extract from strace:
> ...
> [pid 57701] 19:14:02.493718 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.494395 close(19) = 0
> [pid 57701] 19:14:02.494448 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.494844 close(19) = 0
> [pid 57701] 19:14:02.494913 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.495565 close(19) = 0
> [pid 57701] 19:14:02.495617 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.496072 close(19) = 0
> [pid 57701] 19:14:02.496128 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.496758 close(19) = 0
> [pid 57701] 19:14:02.496812 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.497270 close(19) = 0
> [pid 57701] 19:14:02.497319 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.497698 close(19) = 0
> [pid 57701] 19:14:02.497750 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.498407 close(19) = 0
> [pid 57701] 19:14:02.498456 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.498899 close(19) = 0
> [pid 57701] 19:14:02.498963 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 63682] 19:14:02.499091 close(18
> [pid 57701] 19:14:02.499634 close(19) = 0
> [pid 57701] 19:14:02.499689 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.500044 close(19) = 0
> [pid 57701] 19:14:02.500093 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.500734 close(19) = 0
> [pid 57701] 19:14:02.500782 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.501271 close(19) = 0
> [pid 57701] 19:14:02.501339 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.502030 close(19) = 0
> [pid 57701] 19:14:02.502101 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 19
> ...
> ...
> [pid 57691] 19:18:03.461022 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 6138] 19:18:03.461345 open("/etc/selinux/config", O_RDONLY ...>
> [pid 57691] 19:18:03.461460 close(27) = 0
> [pid 57691] 19:18:03.461520 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 6138] 19:18:03.461632 close(3
> [pid 6138] 19:18:03.461781 open("/proc/mounts", O_RDONLY
> [pid 6138] 19:18:03.462190 close(3
> [pid 57691] 19:18:03.462374 close(27) = 0
> [pid 57691] 19:18:03.462430 socket(PF_NETLINK, SOCK_RAW, 0
> [pid 6138] 19:18:03.462456 open("/proc/net/psched", O_RDONLY
> [pid 6138] 19:18:03.462678 close(3
> [pid 6138] 19:18:03.462915 open("/etc/libnl/classid", O_RDONLY ...>
> [pid 57691] 19:18:03.463046 close(27) = 0
> [pid 57691] 19:18:03.463111 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 6138] 19:18:03.463225 close(3
> [pid 57691] 19:18:03.463845 close(27) = 0
> [pid 57691] 19:18:03.463911 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.464604 close(27) = 0
> [pid 57691] 19:18:03.464664 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.465074 close(27) = 0
> [pid 57691] 19:18:03.465132 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.465862 close(27) = 0
> [pid 57691] 19:18:03.465928 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.466713 close(27) = 0
> [pid 57691] 19:18:03.466780 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.467472 close(27) = 0
> [pid 57691] 19:18:03.467524 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.468012 close(27) = 0
> [pid 57691] 19:18:03.468075 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.468799 close(27) = 0
> [pid 57691] 19:18:03.468950 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.469505 close(27) = 0
> [pid 57691] 19:18:03.469578 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.470301 close(27) = 0
> [pid 57691] 19:18:03.470353
[jira] [Comment Edited] (MESOS-6420) Mesos Agent leaking sockets when port mapping network isolator is ON
[
https://issues.apache.org/jira/browse/MESOS-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591210#comment-15591210
]
Santhosh Shanmugham edited comment on MESOS-6420 at 10/20/16 8:39 AM:
--
The socket appears to be leaked at line 160 in
{{src/linux/routing/link/link.cpp}}
{code:title=src/slave/containerizer/mesos/isolators/network/port_mapping.cpp|borderStyle=solid}
2874 Try enable = link::setUp(veth(pid));
{code}
{code:title=src/linux/routing/link/link.cpp|borderStyle=solid}
243 Try setUp(const string& link)
244 {
245 return internal::set(link, IFF_UP);
246 }
{code}
{code:title=src/linux/routing/link/link.cpp|borderStyle=solid}
125 inline Try set(const std::string& _link, unsigned int flags)
126 {
...
144 int fd = ::socket(AF_INET, SOCK_STREAM, 0);
145 if (fd == -1) {
146 return ErrnoError();
147 }
148
149 if (ioctl(fd, SIOCSIFFLAGS, ) == -1) {
150 if (errno == ENODEV) {
151 os::close(fd);
152 return false;
153 } else {
154 // Save the error string as os::close may overwrite errno.
155 const std::string message = os::strerror(errno);
156 os::close(fd);
157 return Error(message);
158 }
159 }
160 // os::close(fd); << MISSING
161 return true;
162 }
{code}
was (Author: santhoshkuma...@gmail.com):
The socker appears to be leaked at line 160 in
{{src/linux/routing/link/link.cpp}}
{code:title=src/slave/containerizer/mesos/isolators/network/port_mapping.cpp|borderStyle=solid}
2874 Try enable = link::setUp(veth(pid));
{code}
{code:title=src/linux/routing/link/link.cpp|borderStyle=solid}
243 Try setUp(const string& link)
244 {
245 return internal::set(link, IFF_UP);
246 }
{code}
{code:title=src/linux/routing/link/link.cpp|borderStyle=solid}
125 inline Try set(const std::string& _link, unsigned int flags)
126 {
...
144 int fd = ::socket(AF_INET, SOCK_STREAM, 0);
145 if (fd == -1) {
146 return ErrnoError();
147 }
148
149 if (ioctl(fd, SIOCSIFFLAGS, ) == -1) {
150 if (errno == ENODEV) {
151 os::close(fd);
152 return false;
153 } else {
154 // Save the error string as os::close may overwrite errno.
155 const std::string message = os::strerror(errno);
156 os::close(fd);
157 return Error(message);
158 }
159 }
160 // os::close(fd); << MISSING
161 return true;
162 }
{code}
> Mesos Agent leaking sockets when port mapping network isolator is ON
>
>
> Key: MESOS-6420
> URL: https://issues.apache.org/jira/browse/MESOS-6420
> Project: Mesos
> Issue Type: Bug
> Components: isolation, network, slave
>Affects Versions: 1.0.2
>Reporter: Santhosh Shanmugham
>
> Mesos Agent leaks one socket per task launched and eventually runs out of
> sockets. We were able to track it down to the network isolator
> (port_mapping.cpp). When we turned off the port mapping isolator no file
> descriptors where leaked. The leaked fd is a SOCK_STREAM socket.
> Leaked Sockets:
> $ sudo lsof -p $(pgrep -u root -o -f /usr/local/sbin/mesos-slave) -nP | grep
> "can't"
> [sudo] password for sshanmugham:
> mesos-sla 57688 root 19u sock0,6 0t0 2993216948 can't
> identify protocol
> mesos-sla 57688 root 27u sock0,6 0t0 2993216468 can't
> identify protocol
> Extract from strace:
> ...
> [pid 57701] 19:14:02.493718 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.494395 close(19) = 0
> [pid 57701] 19:14:02.494448 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.494844 close(19) = 0
> [pid 57701] 19:14:02.494913 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.495565 close(19) = 0
> [pid 57701] 19:14:02.495617 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.496072 close(19) = 0
> [pid 57701] 19:14:02.496128 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.496758 close(19) = 0
> [pid 57701] 19:14:02.496812 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.497270 close(19) = 0
> [pid 57701] 19:14:02.497319 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.497698 close(19) = 0
> [pid 57701] 19:14:02.497750 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.498407 close(19) = 0
> [pid 57701] 19:14:02.498456 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.498899 close(19) = 0
> [pid 57701] 19:14:02.498963 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 63682] 19:14:02.499091 close(18
> [pid 57701] 19:14:02.499634 close(19) = 0
> [pid 57701] 19:14:02.499689 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.500044 close(19) = 0
> [pid 57701] 19:14:02.500093 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.500734 close(19) = 0
> [pid 57701] 19:14:02.500782 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[jira] [Comment Edited] (MESOS-6420) Mesos Agent leaking sockets when port mapping network isolator is ON
[
https://issues.apache.org/jira/browse/MESOS-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591210#comment-15591210
]
Santhosh Shanmugham edited comment on MESOS-6420 at 10/20/16 8:39 AM:
--
The socker appears to be leaked at line 160 in
{{src/linux/routing/link/link.cpp}}
{code:title=src/slave/containerizer/mesos/isolators/network/port_mapping.cpp|borderStyle=solid}
2874 Try enable = link::setUp(veth(pid));
{code}
{code:title=src/linux/routing/link/link.cpp|borderStyle=solid}
243 Try setUp(const string& link)
244 {
245 return internal::set(link, IFF_UP);
246 }
{code}
{code:title=src/linux/routing/link/link.cpp|borderStyle=solid}
125 inline Try set(const std::string& _link, unsigned int flags)
126 {
...
144 int fd = ::socket(AF_INET, SOCK_STREAM, 0);
145 if (fd == -1) {
146 return ErrnoError();
147 }
148
149 if (ioctl(fd, SIOCSIFFLAGS, ) == -1) {
150 if (errno == ENODEV) {
151 os::close(fd);
152 return false;
153 } else {
154 // Save the error string as os::close may overwrite errno.
155 const std::string message = os::strerror(errno);
156 os::close(fd);
157 return Error(message);
158 }
159 }
160 // os::close(fd); << MISSING
161 return true;
162 }
{code}
was (Author: santhoshkuma...@gmail.com):
The socker appears to be leaked inside {{src/linux/routing/link/link.cpp}}. See
line 160 in {{src/linux/routing/link/link.cpp}}
{code:title=src/slave/containerizer/mesos/isolators/network/port_mapping.cpp|borderStyle=solid}
2874 Try enable = link::setUp(veth(pid));
{code}
{code:title=src/linux/routing/link/link.cpp|borderStyle=solid}
243 Try setUp(const string& link)
244 {
245 return internal::set(link, IFF_UP);
246 }
{code}
{code:title=src/linux/routing/link/link.cpp|borderStyle=solid}
125 inline Try set(const std::string& _link, unsigned int flags)
126 {
...
144 int fd = ::socket(AF_INET, SOCK_STREAM, 0);
145 if (fd == -1) {
146 return ErrnoError();
147 }
148
149 if (ioctl(fd, SIOCSIFFLAGS, ) == -1) {
150 if (errno == ENODEV) {
151 os::close(fd);
152 return false;
153 } else {
154 // Save the error string as os::close may overwrite errno.
155 const std::string message = os::strerror(errno);
156 os::close(fd);
157 return Error(message);
158 }
159 }
160 // os::close(fd); << MISSING
161 return true;
162 }
{code}
> Mesos Agent leaking sockets when port mapping network isolator is ON
>
>
> Key: MESOS-6420
> URL: https://issues.apache.org/jira/browse/MESOS-6420
> Project: Mesos
> Issue Type: Bug
> Components: isolation, network, slave
>Affects Versions: 1.0.2
>Reporter: Santhosh Shanmugham
>
> Mesos Agent leaks one socket per task launched and eventually runs out of
> sockets. We were able to track it down to the network isolator
> (port_mapping.cpp). When we turned off the port mapping isolator no file
> descriptors where leaked. The leaked fd is a SOCK_STREAM socket.
> Leaked Sockets:
> $ sudo lsof -p $(pgrep -u root -o -f /usr/local/sbin/mesos-slave) -nP | grep
> "can't"
> [sudo] password for sshanmugham:
> mesos-sla 57688 root 19u sock0,6 0t0 2993216948 can't
> identify protocol
> mesos-sla 57688 root 27u sock0,6 0t0 2993216468 can't
> identify protocol
> Extract from strace:
> ...
> [pid 57701] 19:14:02.493718 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.494395 close(19) = 0
> [pid 57701] 19:14:02.494448 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.494844 close(19) = 0
> [pid 57701] 19:14:02.494913 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.495565 close(19) = 0
> [pid 57701] 19:14:02.495617 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.496072 close(19) = 0
> [pid 57701] 19:14:02.496128 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.496758 close(19) = 0
> [pid 57701] 19:14:02.496812 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.497270 close(19) = 0
> [pid 57701] 19:14:02.497319 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.497698 close(19) = 0
> [pid 57701] 19:14:02.497750 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.498407 close(19) = 0
> [pid 57701] 19:14:02.498456 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.498899 close(19) = 0
> [pid 57701] 19:14:02.498963 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 63682] 19:14:02.499091 close(18
> [pid 57701] 19:14:02.499634 close(19) = 0
> [pid 57701] 19:14:02.499689 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.500044 close(19) = 0
> [pid 57701] 19:14:02.500093 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.500734 close(19) = 0
> [pid 57701]
[jira] [Comment Edited] (MESOS-6420) Mesos Agent leaking sockets when port mapping network isolator is ON
[
https://issues.apache.org/jira/browse/MESOS-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591210#comment-15591210
]
Santhosh Shanmugham edited comment on MESOS-6420 at 10/20/16 8:39 AM:
--
The socker appears to be leaked inside {{src/linux/routing/link/link.cpp}}. See
line 160 in {{src/linux/routing/link/link.cpp}}
{code:title=src/slave/containerizer/mesos/isolators/network/port_mapping.cpp|borderStyle=solid}
2874 Try enable = link::setUp(veth(pid));
{code}
{code:title=src/linux/routing/link/link.cpp|borderStyle=solid}
243 Try setUp(const string& link)
244 {
245 return internal::set(link, IFF_UP);
246 }
{code}
{code:title=src/linux/routing/link/link.cpp|borderStyle=solid}
125 inline Try set(const std::string& _link, unsigned int flags)
126 {
...
144 int fd = ::socket(AF_INET, SOCK_STREAM, 0);
145 if (fd == -1) {
146 return ErrnoError();
147 }
148
149 if (ioctl(fd, SIOCSIFFLAGS, ) == -1) {
150 if (errno == ENODEV) {
151 os::close(fd);
152 return false;
153 } else {
154 // Save the error string as os::close may overwrite errno.
155 const std::string message = os::strerror(errno);
156 os::close(fd);
157 return Error(message);
158 }
159 }
160 // os::close(fd); << MISSING
161 return true;
162 }
{code}
was (Author: santhoshkuma...@gmail.com):
The socker appears to be leaked inside {{src/linux/routing/link/link.cpp}}.
{code:title=src/slave/containerizer/mesos/isolators/network/port_mapping.cpp|borderStyle=solid}
2874 Try enable = link::setUp(veth(pid));
{code}
{code:title=src/linux/routing/link/link.cpp|borderStyle=solid}
243 Try setUp(const string& link)
244 {
245 return internal::set(link, IFF_UP);
246 }
{code}
{code:title=src/linux/routing/link/link.cpp|borderStyle=solid}
125 inline Try set(const std::string& _link, unsigned int flags)
126 {
...
144 int fd = ::socket(AF_INET, SOCK_STREAM, 0);
145 if (fd == -1) {
146 return ErrnoError();
147 }
148
149 if (ioctl(fd, SIOCSIFFLAGS, ) == -1) {
150 if (errno == ENODEV) {
151 os::close(fd);
152 return false;
153 } else {
154 // Save the error string as os::close may overwrite errno.
155 const std::string message = os::strerror(errno);
156 os::close(fd);
157 return Error(message);
158 }
159 }
160 // os::close(fd); << MISSING
161 return true;
162 }
{code}
> Mesos Agent leaking sockets when port mapping network isolator is ON
>
>
> Key: MESOS-6420
> URL: https://issues.apache.org/jira/browse/MESOS-6420
> Project: Mesos
> Issue Type: Bug
> Components: isolation, network, slave
>Affects Versions: 1.0.2
>Reporter: Santhosh Shanmugham
>
> Mesos Agent leaks one socket per task launched and eventually runs out of
> sockets. We were able to track it down to the network isolator
> (port_mapping.cpp). When we turned off the port mapping isolator no file
> descriptors where leaked. The leaked fd is a SOCK_STREAM socket.
> Leaked Sockets:
> $ sudo lsof -p $(pgrep -u root -o -f /usr/local/sbin/mesos-slave) -nP | grep
> "can't"
> [sudo] password for sshanmugham:
> mesos-sla 57688 root 19u sock0,6 0t0 2993216948 can't
> identify protocol
> mesos-sla 57688 root 27u sock0,6 0t0 2993216468 can't
> identify protocol
> Extract from strace:
> ...
> [pid 57701] 19:14:02.493718 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.494395 close(19) = 0
> [pid 57701] 19:14:02.494448 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.494844 close(19) = 0
> [pid 57701] 19:14:02.494913 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.495565 close(19) = 0
> [pid 57701] 19:14:02.495617 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.496072 close(19) = 0
> [pid 57701] 19:14:02.496128 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.496758 close(19) = 0
> [pid 57701] 19:14:02.496812 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.497270 close(19) = 0
> [pid 57701] 19:14:02.497319 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.497698 close(19) = 0
> [pid 57701] 19:14:02.497750 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.498407 close(19) = 0
> [pid 57701] 19:14:02.498456 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.498899 close(19) = 0
> [pid 57701] 19:14:02.498963 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 63682] 19:14:02.499091 close(18
> [pid 57701] 19:14:02.499634 close(19) = 0
> [pid 57701] 19:14:02.499689 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.500044 close(19) = 0
> [pid 57701] 19:14:02.500093 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.500734 close(19) = 0
> [pid 57701] 19:14:02.500782
[jira] [Comment Edited] (MESOS-6420) Mesos Agent leaking sockets when port mapping network isolator is ON
[
https://issues.apache.org/jira/browse/MESOS-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591210#comment-15591210
]
Santhosh Shanmugham edited comment on MESOS-6420 at 10/20/16 8:38 AM:
--
The socker appears to be leaked inside {{src/linux/routing/link/link.cpp}}.
{code:title=src/slave/containerizer/mesos/isolators/network/port_mapping.cpp|borderStyle=solid}
2874 Try enable = link::setUp(veth(pid));
{code}
{code:title=src/linux/routing/link/link.cpp|borderStyle=solid}
243 Try setUp(const string& link)
244 {
245 return internal::set(link, IFF_UP);
246 }
{code}
{code:title=src/linux/routing/link/link.cpp|borderStyle=solid}
125 inline Try set(const std::string& _link, unsigned int flags)
126 {
...
144 int fd = ::socket(AF_INET, SOCK_STREAM, 0);
145 if (fd == -1) {
146 return ErrnoError();
147 }
148
149 if (ioctl(fd, SIOCSIFFLAGS, ) == -1) {
150 if (errno == ENODEV) {
151 os::close(fd);
152 return false;
153 } else {
154 // Save the error string as os::close may overwrite errno.
155 const std::string message = os::strerror(errno);
156 os::close(fd);
157 return Error(message);
158 }
159 }
160 // os::close(fd); << MISSING
161 return true;
162 }
{code}
was (Author: santhoshkuma...@gmail.com):
The appears to be happening inside {{src/linux/routing/link/link.cpp}}.
{code:title=src/slave/containerizer/mesos/isolators/network/port_mapping.cpp|borderStyle=solid}
2874 Try enable = link::setUp(veth(pid));
{code}
{code:title=src/linux/routing/link/link.cpp|borderStyle=solid}
243 Try setUp(const string& link)
244 {
245 return internal::set(link, IFF_UP);
246 }
{code}
{code:title=src/linux/routing/link/link.cpp|borderStyle=solid}
125 inline Try set(const std::string& _link, unsigned int flags)
126 {
...
144 int fd = ::socket(AF_INET, SOCK_STREAM, 0);
145 if (fd == -1) {
146 return ErrnoError();
147 }
148
149 if (ioctl(fd, SIOCSIFFLAGS, ) == -1) {
150 if (errno == ENODEV) {
151 os::close(fd);
152 return false;
153 } else {
154 // Save the error string as os::close may overwrite errno.
155 const std::string message = os::strerror(errno);
156 os::close(fd);
157 return Error(message);
158 }
159 }
160 // os::close(fd); << MISSING
161 return true;
162 }
{code}
> Mesos Agent leaking sockets when port mapping network isolator is ON
>
>
> Key: MESOS-6420
> URL: https://issues.apache.org/jira/browse/MESOS-6420
> Project: Mesos
> Issue Type: Bug
> Components: isolation, network, slave
>Affects Versions: 1.0.2
>Reporter: Santhosh Shanmugham
>
> Mesos Agent leaks one socket per task launched and eventually runs out of
> sockets. We were able to track it down to the network isolator
> (port_mapping.cpp). When we turned off the port mapping isolator no file
> descriptors where leaked. The leaked fd is a SOCK_STREAM socket.
> Leaked Sockets:
> $ sudo lsof -p $(pgrep -u root -o -f /usr/local/sbin/mesos-slave) -nP | grep
> "can't"
> [sudo] password for sshanmugham:
> mesos-sla 57688 root 19u sock0,6 0t0 2993216948 can't
> identify protocol
> mesos-sla 57688 root 27u sock0,6 0t0 2993216468 can't
> identify protocol
> Extract from strace:
> ...
> [pid 57701] 19:14:02.493718 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.494395 close(19) = 0
> [pid 57701] 19:14:02.494448 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.494844 close(19) = 0
> [pid 57701] 19:14:02.494913 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.495565 close(19) = 0
> [pid 57701] 19:14:02.495617 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.496072 close(19) = 0
> [pid 57701] 19:14:02.496128 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.496758 close(19) = 0
> [pid 57701] 19:14:02.496812 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.497270 close(19) = 0
> [pid 57701] 19:14:02.497319 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.497698 close(19) = 0
> [pid 57701] 19:14:02.497750 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.498407 close(19) = 0
> [pid 57701] 19:14:02.498456 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.498899 close(19) = 0
> [pid 57701] 19:14:02.498963 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 63682] 19:14:02.499091 close(18
> [pid 57701] 19:14:02.499634 close(19) = 0
> [pid 57701] 19:14:02.499689 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.500044 close(19) = 0
> [pid 57701] 19:14:02.500093 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.500734 close(19) = 0
> [pid 57701] 19:14:02.500782 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701]
[jira] [Commented] (MESOS-6420) Mesos Agent leaking sockets when port mapping network isolator is ON
[
https://issues.apache.org/jira/browse/MESOS-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591210#comment-15591210
]
Santhosh Shanmugham commented on MESOS-6420:
The appears to be happening inside {{src/linux/routing/link/link.cpp}}.
{code:title=src/slave/containerizer/mesos/isolators/network/port_mapping.cpp|borderStyle=solid}
2874 Try enable = link::setUp(veth(pid));
{code}
{code:title=src/linux/routing/link/link.cpp|borderStyle=solid}
243 Try setUp(const string& link)
244 {
245 return internal::set(link, IFF_UP);
246 }
{code}
{code:title=src/linux/routing/link/link.cpp|borderStyle=solid}
125 inline Try set(const std::string& _link, unsigned int flags)
126 {
...
144 int fd = ::socket(AF_INET, SOCK_STREAM, 0);
145 if (fd == -1) {
146 return ErrnoError();
147 }
148
149 if (ioctl(fd, SIOCSIFFLAGS, ) == -1) {
150 if (errno == ENODEV) {
151 os::close(fd);
152 return false;
153 } else {
154 // Save the error string as os::close may overwrite errno.
155 const std::string message = os::strerror(errno);
156 os::close(fd);
157 return Error(message);
158 }
159 }
160 // os::close(fd); << MISSING
161 return true;
162 }
{code}
> Mesos Agent leaking sockets when port mapping network isolator is ON
>
>
> Key: MESOS-6420
> URL: https://issues.apache.org/jira/browse/MESOS-6420
> Project: Mesos
> Issue Type: Bug
> Components: isolation, network, slave
>Affects Versions: 1.0.2
>Reporter: Santhosh Shanmugham
>
> Mesos Agent leaks one socket per task launched and eventually runs out of
> sockets. We were able to track it down to the network isolator
> (port_mapping.cpp). When we turned off the port mapping isolator no file
> descriptors where leaked. The leaked fd is a SOCK_STREAM socket.
> Leaked Sockets:
> $ sudo lsof -p $(pgrep -u root -o -f /usr/local/sbin/mesos-slave) -nP | grep
> "can't"
> [sudo] password for sshanmugham:
> mesos-sla 57688 root 19u sock0,6 0t0 2993216948 can't
> identify protocol
> mesos-sla 57688 root 27u sock0,6 0t0 2993216468 can't
> identify protocol
> Extract from strace:
> ...
> [pid 57701] 19:14:02.493718 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.494395 close(19) = 0
> [pid 57701] 19:14:02.494448 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.494844 close(19) = 0
> [pid 57701] 19:14:02.494913 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.495565 close(19) = 0
> [pid 57701] 19:14:02.495617 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.496072 close(19) = 0
> [pid 57701] 19:14:02.496128 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.496758 close(19) = 0
> [pid 57701] 19:14:02.496812 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.497270 close(19) = 0
> [pid 57701] 19:14:02.497319 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.497698 close(19) = 0
> [pid 57701] 19:14:02.497750 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.498407 close(19) = 0
> [pid 57701] 19:14:02.498456 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.498899 close(19) = 0
> [pid 57701] 19:14:02.498963 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 63682] 19:14:02.499091 close(18
> [pid 57701] 19:14:02.499634 close(19) = 0
> [pid 57701] 19:14:02.499689 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.500044 close(19) = 0
> [pid 57701] 19:14:02.500093 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.500734 close(19) = 0
> [pid 57701] 19:14:02.500782 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.501271 close(19) = 0
> [pid 57701] 19:14:02.501339 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.502030 close(19) = 0
> [pid 57701] 19:14:02.502101 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 19
> ...
> ...
> [pid 57691] 19:18:03.461022 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 6138] 19:18:03.461345 open("/etc/selinux/config", O_RDONLY ...>
> [pid 57691] 19:18:03.461460 close(27) = 0
> [pid 57691] 19:18:03.461520 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 6138] 19:18:03.461632 close(3
> [pid 6138] 19:18:03.461781 open("/proc/mounts", O_RDONLY
> [pid 6138] 19:18:03.462190 close(3
> [pid 57691] 19:18:03.462374 close(27) = 0
> [pid 57691] 19:18:03.462430 socket(PF_NETLINK, SOCK_RAW, 0
> [pid 6138] 19:18:03.462456 open("/proc/net/psched", O_RDONLY
> [pid 6138] 19:18:03.462678 close(3
> [pid 6138] 19:18:03.462915 open("/etc/libnl/classid", O_RDONLY ...>
> [pid 57691] 19:18:03.463046 close(27) = 0
> [pid 57691] 19:18:03.463111 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 6138] 19:18:03.463225 close(3
> [pid 57691] 19:18:03.463845 close(27) = 0
> [pid 57691] 19:18:03.463911
[jira] [Commented] (MESOS-6420) Mesos Agent leaking sockets when network isolation is ON
[
https://issues.apache.org/jira/browse/MESOS-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590126#comment-15590126
]
Santhosh Shanmugham commented on MESOS-6420:
Something about file
'/sys/fs/cgroup/freezer/mesos/0f5238be-4dd6-4ffe-9250-ceddc904174f' already
exists. See the full log for the life-time of the job below.
W1019 22:50:47.115586 1117 subprocess.hpp:422] Failed to execute
Subprocess::Hook in parent for child '17427': Failed to assign process to its
freezer cgroup: Failed to create freezer cgroup: Failed to create directory
'/sys/fs/cgroup/freezer/mesos/0f5238be-4dd6-4ffe-9250-ceddc904174f': File exists
E1019 22:50:47.121644 1117 slave.cpp:3976] Container
'0f5238be-4dd6-4ffe-9250-ceddc904174f' for executor
'thermos-sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab' of
framework 201103282247-19- failed to start: Failed to fork
executor: Failed to clone child process: Failed to execute Subprocess::Hook in
parent for child '17427': Failed to assign process to its freezer cgroup:
Failed to create freezer cgroup: Failed to create directory
'/sys/fs/cgroup/freezer/mesos/0f5238be-4dd6-4ffe-9250-ceddc904174f': File exists
I1019 22:50:47.122350 1117 containerizer.cpp:1622] Destroying container
'0f5238be-4dd6-4ffe-9250-ceddc904174f'
> Mesos Agent leaking sockets when network isolation is ON
>
>
> Key: MESOS-6420
> URL: https://issues.apache.org/jira/browse/MESOS-6420
> Project: Mesos
> Issue Type: Bug
> Components: isolation, network, slave
>Affects Versions: 1.0.2
>Reporter: Santhosh Shanmugham
>
> Mesos Agent leaks one socket per task launched and eventually runs out of
> sockets. We were able to track it down to the network isolator
> (port_mapping.cpp). When we turned off the port mapping isolator no file
> descriptors where leaked. The leaked fd is a SOCK_STREAM socket.
> Leaked Sockets:
> sshanmugham[5]smf1-aeu-07-sr3(mesos.test.slave) ~ $ sudo lsof -p $(pgrep -u
> root -o -f /usr/local/sbin/mesos-slave) -nP | grep "can't"
> [sudo] password for sshanmugham:
> mesos-sla 57688 root 19u sock0,6 0t0 2993216948 can't
> identify protocol
> mesos-sla 57688 root 27u sock0,6 0t0 2993216468 can't
> identify protocol
> Extract from strace:
> ...
> [pid 57701] 19:14:02.493718 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.494395 close(19) = 0
> [pid 57701] 19:14:02.494448 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.494844 close(19) = 0
> [pid 57701] 19:14:02.494913 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.495565 close(19) = 0
> [pid 57701] 19:14:02.495617 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.496072 close(19) = 0
> [pid 57701] 19:14:02.496128 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.496758 close(19) = 0
> [pid 57701] 19:14:02.496812 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.497270 close(19) = 0
> [pid 57701] 19:14:02.497319 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.497698 close(19) = 0
> [pid 57701] 19:14:02.497750 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.498407 close(19) = 0
> [pid 57701] 19:14:02.498456 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.498899 close(19) = 0
> [pid 57701] 19:14:02.498963 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 63682] 19:14:02.499091 close(18
> [pid 57701] 19:14:02.499634 close(19) = 0
> [pid 57701] 19:14:02.499689 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.500044 close(19) = 0
> [pid 57701] 19:14:02.500093 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.500734 close(19) = 0
> [pid 57701] 19:14:02.500782 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.501271 close(19) = 0
> [pid 57701] 19:14:02.501339 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.502030 close(19) = 0
> [pid 57701] 19:14:02.502101 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 19
> ...
> ...
> [pid 57691] 19:18:03.461022 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 6138] 19:18:03.461345 open("/etc/selinux/config", O_RDONLY ...>
> [pid 57691] 19:18:03.461460 close(27) = 0
> [pid 57691] 19:18:03.461520 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 6138] 19:18:03.461632 close(3
> [pid 6138] 19:18:03.461781 open("/proc/mounts", O_RDONLY
> [pid 6138] 19:18:03.462190 close(3
> [pid 57691] 19:18:03.462374 close(27) = 0
> [pid 57691] 19:18:03.462430 socket(PF_NETLINK, SOCK_RAW, 0
> [pid 6138] 19:18:03.462456 open("/proc/net/psched", O_RDONLY
> [pid 6138] 19:18:03.462678 close(3
> [pid 6138] 19:18:03.462915 open("/etc/libnl/classid", O_RDONLY ...>
> [pid 57691] 19:18:03.463046 close(27) = 0
> [pid 57691] 19:18:03.463111 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 6138]
[jira] [Commented] (MESOS-6420) Mesos Agent leaking sockets when network isolation is ON
[ https://issues.apache.org/jira/browse/MESOS-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590120#comment-15590120 ] Santhosh Shanmugham commented on MESOS-6420: I1019 22:50:45.495558 1117 slave.cpp:1495] Got assigned task sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab for framework 201103282247-19- I1019 22:50:45.504261 1117 gc.cpp:83] Unscheduling '/var/lib/mesos/slaves/7f9be60e-9bcf-47e0-8c6b-8b8ab278ecbe-S0/frameworks/201103282247-19-' from gc I1019 22:50:45.505200 1117 gc.cpp:83] Unscheduling '/var/lib/mesos/meta/slaves/7f9be60e-9bcf-47e0-8c6b-8b8ab278ecbe-S0/frameworks/201103282247-19-' from gc I1019 22:50:45.506726 1116 slave.cpp:1614] Launching task sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab for framework 201103282247-19- I1019 22:50:45.510138 1116 paths.cpp:528] Trying to chown '/var/lib/mesos/slaves/7f9be60e-9bcf-47e0-8c6b-8b8ab278ecbe-S0/frameworks/201103282247-19-/executors/thermos-sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab/runs/0f5238be-4dd6-4ffe-9250-ceddc904174f' to user 'root' ==17403== Memcheck, a memory error detector ==17403== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. ==17403== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info ==17403== Command: /bin/sh -c chown\ -R\ 0:0\ '/var/lib/mesos/slaves/7f9be60e-9bcf-47e0-8c6b-8b8ab278ecbe-S0/frameworks/201103282247-19-/executors/thermos-sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab/runs/0f5238be-4dd6-4ffe-9250-ceddc904174f' ==17403== ==17403== Memcheck, a memory error detector ==17403== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. ==17403== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info ==17403== Command: /bin/chown -R 0:0 /var/lib/mesos/slaves/7f9be60e-9bcf-47e0-8c6b-8b8ab278ecbe-S0/frameworks/201103282247-19-/executors/thermos-sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab/runs/0f5238be-4dd6-4ffe-9250-ceddc904174f ==17403== ==17403== ==17403== FILE DESCRIPTORS: 23 open at exit. ==17403== Open file descriptor 33: ==17403== ==17403== ==17403== Open file descriptor 32: ==17403== ==17403== ==17403== Open file descriptor 31: ==17403== ==17403== ==17403== Open file descriptor 30: ==17403== ==17403== ==17403== Open file descriptor 29: ==17403== ==17403== ==17403== Open file descriptor 28: ==17403== ==17403== ==17403== Open file descriptor 27: ==17403== ==17403== ==17403== Open file descriptor 25: ==17403== ==17403== ==17403== Open file descriptor 23: ==17403== ==17403== ==17403== Open file descriptor 22: ==17403== ==17403== ==17403== Open file descriptor 21: ==17403== ==17403== ==17403== Open file descriptor 20: ==17403== ==17403== ==17403== Open file descriptor 19: ==17403== ==17403== ==17403== Open file descriptor 17: ==17403== ==17403== ==17403== Open file descriptor 16: ==17403== ==17403== ==17403== Open file descriptor 15: ==17403== ==17403== ==17403== Open file descriptor 14: ==17403== ==17403== ==17403== Open file descriptor 12: ==17403== ==17403== ==17403== Open AF_INET socket 10: 10.34.124.106:60994 <-> 10.35.95.111:2181 ==17403== ==17403== ==17403== Open file descriptor 8: ==17403== ==17403== ==17403== Open file descriptor 7: ==17403== ==17403== ==17403== Open file descriptor 2: ==17403== ==17403== ==17403== Open file descriptor 0: /dev/null ==17403== ==17403== ==17403== ==17403== HEAP SUMMARY: ==17403== in use at exit: 4 bytes in 2 blocks ==17403== total heap usage: 106 allocs, 104 frees, 575,581 bytes allocated ==17403== ==17403== LEAK SUMMARY: ==17403==definitely lost: 0 bytes in 0 blocks ==17403==indirectly lost: 0 bytes in 0 blocks ==17403== possibly lost: 0 bytes in 0 blocks ==17403==still reachable: 4 bytes in 2 blocks ==17403== suppressed: 0 bytes in 0 blocks ==17403== Reachable blocks (those to which a pointer was found) are not shown. ==17403== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==17403== ==17403== For counts of detected and suppressed errors, rerun with: -v ==17403== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 4 from 4) I1019 22:50:46.946646 1116 slave.cpp:5674] Launching executor thermos-sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab of framework 201103282247-19- with resources mem(*):128 in work directory '/var/lib/mesos/slaves/7f9be60e-9bcf-47e0-8c6b-8b8ab278ecbe-S0/frameworks/201103282247-19-/executors/thermos-sshanmugham-devel-hello-0-f70e5cd5-d93c-45c5-85ec-434cb9c527ab/runs/0f5238be-4dd6-4ffe-9250-ceddc904174f' I1019 22:50:46.952791 1117 containerizer.cpp:781] Starting container '0f5238be-4dd6-4ffe-9250-ceddc904174f' for executor
[jira] [Commented] (MESOS-6420) Mesos Agent leaking sockets when network isolation is ON
[
https://issues.apache.org/jira/browse/MESOS-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590075#comment-15590075
]
Santhosh Shanmugham commented on MESOS-6420:
When running with Valgrind, the slave fails to prepare isolators for the
containers.
`/usr/local/bin/valgrind --trace-children=yes --tool=memcheck --leak-check=full
--track-fds=yes /usr/local/bin/mesos-slave.sh
/usr/local/mesos/conf/mesos-slave-config.sh`
Failed to launch container: Failed to fork executor: Failed to clone child
process: Failed to synchronize child process; Container destroyed while
preparing isolators
> Mesos Agent leaking sockets when network isolation is ON
>
>
> Key: MESOS-6420
> URL: https://issues.apache.org/jira/browse/MESOS-6420
> Project: Mesos
> Issue Type: Bug
> Components: isolation, network, slave
>Affects Versions: 1.0.2
>Reporter: Santhosh Shanmugham
>
> Mesos Agent leaks one socket per task launched and eventually runs out of
> sockets. We were able to track it down to the network isolator
> (port_mapping.cpp). When we turned off the port mapping isolator no file
> descriptors where leaked. The leaked fd is a SOCK_STREAM socket.
> Leaked Sockets:
> sshanmugham[5]smf1-aeu-07-sr3(mesos.test.slave) ~ $ sudo lsof -p $(pgrep -u
> root -o -f /usr/local/sbin/mesos-slave) -nP | grep "can't"
> [sudo] password for sshanmugham:
> mesos-sla 57688 root 19u sock0,6 0t0 2993216948 can't
> identify protocol
> mesos-sla 57688 root 27u sock0,6 0t0 2993216468 can't
> identify protocol
> Extract from strace:
> ...
> [pid 57701] 19:14:02.493718 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.494395 close(19) = 0
> [pid 57701] 19:14:02.494448 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.494844 close(19) = 0
> [pid 57701] 19:14:02.494913 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.495565 close(19) = 0
> [pid 57701] 19:14:02.495617 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.496072 close(19) = 0
> [pid 57701] 19:14:02.496128 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.496758 close(19) = 0
> [pid 57701] 19:14:02.496812 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.497270 close(19) = 0
> [pid 57701] 19:14:02.497319 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.497698 close(19) = 0
> [pid 57701] 19:14:02.497750 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.498407 close(19) = 0
> [pid 57701] 19:14:02.498456 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.498899 close(19) = 0
> [pid 57701] 19:14:02.498963 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 63682] 19:14:02.499091 close(18
> [pid 57701] 19:14:02.499634 close(19) = 0
> [pid 57701] 19:14:02.499689 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.500044 close(19) = 0
> [pid 57701] 19:14:02.500093 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.500734 close(19) = 0
> [pid 57701] 19:14:02.500782 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.501271 close(19) = 0
> [pid 57701] 19:14:02.501339 socket(PF_NETLINK, SOCK_RAW, 0) = 19
> [pid 57701] 19:14:02.502030 close(19) = 0
> [pid 57701] 19:14:02.502101 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 19
> ...
> ...
> [pid 57691] 19:18:03.461022 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 6138] 19:18:03.461345 open("/etc/selinux/config", O_RDONLY ...>
> [pid 57691] 19:18:03.461460 close(27) = 0
> [pid 57691] 19:18:03.461520 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 6138] 19:18:03.461632 close(3
> [pid 6138] 19:18:03.461781 open("/proc/mounts", O_RDONLY
> [pid 6138] 19:18:03.462190 close(3
> [pid 57691] 19:18:03.462374 close(27) = 0
> [pid 57691] 19:18:03.462430 socket(PF_NETLINK, SOCK_RAW, 0
> [pid 6138] 19:18:03.462456 open("/proc/net/psched", O_RDONLY
> [pid 6138] 19:18:03.462678 close(3
> [pid 6138] 19:18:03.462915 open("/etc/libnl/classid", O_RDONLY ...>
> [pid 57691] 19:18:03.463046 close(27) = 0
> [pid 57691] 19:18:03.463111 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 6138] 19:18:03.463225 close(3
> [pid 57691] 19:18:03.463845 close(27) = 0
> [pid 57691] 19:18:03.463911 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.464604 close(27) = 0
> [pid 57691] 19:18:03.464664 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.465074 close(27) = 0
> [pid 57691] 19:18:03.465132 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.465862 close(27) = 0
> [pid 57691] 19:18:03.465928 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.466713 close(27) = 0
> [pid 57691] 19:18:03.466780 socket(PF_NETLINK, SOCK_RAW, 0) = 27
> [pid 57691] 19:18:03.467472 close(27) = 0
> [pid 57691] 19:18:03.467524 socket(PF_NETLINK, SOCK_RAW, 0) = 27
>
[jira] [Created] (MESOS-6420) Mesos Agent leaking sockets when network isolation is ON
Santhosh Shanmugham created MESOS-6420:
--
Summary: Mesos Agent leaking sockets when network isolation is ON
Key: MESOS-6420
URL: https://issues.apache.org/jira/browse/MESOS-6420
Project: Mesos
Issue Type: Bug
Components: isolation, network, slave
Affects Versions: 1.0.2
Reporter: Santhosh Shanmugham
Mesos Agent leaks one socket per task launched and eventually runs out of
sockets. We were able to track it down to the network isolator
(port_mapping.cpp). When we turned off the port mapping isolator no file
descriptors where leaked. The leaked fd is a SOCK_STREAM socket.
Leaked Sockets:
sshanmugham[5]smf1-aeu-07-sr3(mesos.test.slave) ~ $ sudo lsof -p $(pgrep -u
root -o -f /usr/local/sbin/mesos-slave) -nP | grep "can't"
[sudo] password for sshanmugham:
mesos-sla 57688 root 19u sock0,6 0t0 2993216948 can't identify
protocol
mesos-sla 57688 root 27u sock0,6 0t0 2993216468 can't identify
protocol
Extract from strace:
...
[pid 57701] 19:14:02.493718 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.494395 close(19) = 0
[pid 57701] 19:14:02.494448 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.494844 close(19) = 0
[pid 57701] 19:14:02.494913 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.495565 close(19) = 0
[pid 57701] 19:14:02.495617 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.496072 close(19) = 0
[pid 57701] 19:14:02.496128 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.496758 close(19) = 0
[pid 57701] 19:14:02.496812 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.497270 close(19) = 0
[pid 57701] 19:14:02.497319 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.497698 close(19) = 0
[pid 57701] 19:14:02.497750 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.498407 close(19) = 0
[pid 57701] 19:14:02.498456 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.498899 close(19) = 0
[pid 57701] 19:14:02.498963 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 63682] 19:14:02.499091 close(18
[pid 57701] 19:14:02.499634 close(19) = 0
[pid 57701] 19:14:02.499689 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.500044 close(19) = 0
[pid 57701] 19:14:02.500093 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.500734 close(19) = 0
[pid 57701] 19:14:02.500782 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.501271 close(19) = 0
[pid 57701] 19:14:02.501339 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.502030 close(19) = 0
[pid 57701] 19:14:02.502101 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 19
...
...
[pid 57691] 19:18:03.461022 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 6138] 19:18:03.461345 open("/etc/selinux/config", O_RDONLY
[pid 57691] 19:18:03.461460 close(27) = 0
[pid 57691] 19:18:03.461520 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 6138] 19:18:03.461632 close(3
[pid 6138] 19:18:03.461781 open("/proc/mounts", O_RDONLY
[pid 6138] 19:18:03.462190 close(3
[pid 57691] 19:18:03.462374 close(27) = 0
[pid 57691] 19:18:03.462430 socket(PF_NETLINK, SOCK_RAW, 0
[pid 6138] 19:18:03.462456 open("/proc/net/psched", O_RDONLY
[pid 6138] 19:18:03.462678 close(3
[pid 6138] 19:18:03.462915 open("/etc/libnl/classid", O_RDONLY
[pid 57691] 19:18:03.463046 close(27) = 0
[pid 57691] 19:18:03.463111 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 6138] 19:18:03.463225 close(3
[pid 57691] 19:18:03.463845 close(27) = 0
[pid 57691] 19:18:03.463911 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 57691] 19:18:03.464604 close(27) = 0
[pid 57691] 19:18:03.464664 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 57691] 19:18:03.465074 close(27) = 0
[pid 57691] 19:18:03.465132 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 57691] 19:18:03.465862 close(27) = 0
[pid 57691] 19:18:03.465928 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 57691] 19:18:03.466713 close(27) = 0
[pid 57691] 19:18:03.466780 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 57691] 19:18:03.467472 close(27) = 0
[pid 57691] 19:18:03.467524 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 57691] 19:18:03.468012 close(27) = 0
[pid 57691] 19:18:03.468075 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 57691] 19:18:03.468799 close(27) = 0
[pid 57691] 19:18:03.468950 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 57691] 19:18:03.469505 close(27) = 0
[pid 57691] 19:18:03.469578 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 57691] 19:18:03.470301 close(27) = 0
[pid 57691] 19:18:03.470353 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 27
...
The last socket the was created never has a corresponding close().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
