Re: [ovs-discuss] ovs-vswitch Huge number of netlink file descriptors open
Can you provide your configuration database, that is, ovs-vswitchd.conf.db? On Sat, Oct 21, 2017 at 01:44:42PM +0800, quan_hp...@heetian.com wrote: > Hi,All, > The file descriptors are "netlink". > $sudo lsof -p $(pidof ovs-vswitchd) > ovs-vswit 2197 root *297u netlink 0t0 42309398 > GENERIC > ovs-vswit 2197 root *298u netlink 0t0 42309399 > GENERIC > ovs-vswit 2197 root *299u sock0,7 0t0 42309400 > protocol: NETLINK > ovs-vswit 2197 root *300u netlink 0t0 42309401 > GENERIC > ovs-vswit 2197 root *301u netlink 0t0 42309402 > GENERIC > ...More... > ovs-vswit 2197 root *351u netlink 0t0 39326247 > GENERIC > ovs-vswit 2197 root *352u netlink 0t0 39121841 > GENERIC > ovs-vswit 2197 root *354w REG 253,0 10086253 1342248977 > /var/log/openvswitch/ovs-vswitchd.log > $sudo reboot > When I restart the machine ,the file descriptor will increasing.Some time > more then 12K record. > > Recenty log about openvswitch. > ovsdb-server.log > 2017-10-21T04:01:02.559Z|01308|jsonrpc|WARN|Dropped 7 log messages in last 17 > seconds (most recently, 10 seconds ago) due to excessive rate > 2017-10-21T04:01:02.559Z|01309|jsonrpc|WARN|unix: receive error: Connection > reset by peer > 2017-10-21T04:01:02.560Z|01310|reconnect|WARN|unix: connection dropped > (Connection reset by peer) > 2017-10-21T04:01:03.282Z|01311|reconnect|WARN|unix: connection dropped > (Connection reset by peer) > 2017-10-21T04:01:10.075Z|01312|jsonrpc|WARN|Dropped 1 log messages in last 7 > seconds (most recently, 7 seconds ago) due to excessive rate > 2017-10-21T04:01:10.075Z|01313|jsonrpc|WARN|unix: send error: Broken pipe > 2017-10-21T04:01:10.076Z|01314|reconnect|WARN|unix: connection dropped > (Broken pipe) > 2017-10-21T04:01:14.907Z|01315|reconnect|WARN|unix: connection dropped > (Connection reset by peer) > 2017-10-21T04:01:20.134Z|01316|reconnect|WARN|unix: connection dropped > (Broken pipe) > 2017-10-21T04:01:27.499Z|01317|jsonrpc|WARN|Dropped 2 log messages in last 13 > seconds (most recently, 8 seconds ago) due to excessive rate > 2017-10-21T04:01:27.499Z|01318|jsonrpc|WARN|unix: receive error: Connection > reset by peer > 2017-10-21T04:01:27.499Z|01319|reconnect|WARN|unix: connection dropped > (Connection reset by peer) > 2017-10-21T04:01:33.758Z|01320|jsonrpc|WARN|unix: receive error: Connection > reset by peer > 2017-10-21T04:01:33.758Z|01321|reconnect|WARN|unix: connection dropped > (Connection reset by peer) > 2017-10-21T04:01:34.489Z|01322|reconnect|WARN|unix: connection dropped > (Connection reset by peer) > 2017-10-21T04:01:39.198Z|01323|reconnect|WARN|unix: connection dropped > (Connection reset by peer) > 2017-10-21T04:01:41.569Z|01324|reconnect|WARN|unix: connection dropped > (Broken pipe) > 2017-10-21T04:01:41.913Z|01325|reconnect|WARN|unix: connection dropped > (Broken pipe) > 2017-10-21T04:01:42.365Z|01326|reconnect|WARN|unix: connection dropped > (Connection reset by peer) > > ovs-vswitchd.log > 2017-10-21T03:56:10.511Z|422666|netlink_socket|ERR|connect(0): Argument list > too long > 2017-10-21T03:56:10.513Z|422667|netlink_socket|ERR|connect(0): Argument list > too long > 2017-10-21T03:56:10.516Z|422668|netlink_socket|ERR|connect(0): Argument list > too long > 2017-10-21T03:56:10.518Z|422669|netlink_socket|ERR|connect(0): Argument list > too long > 2017-10-21T03:56:10.520Z|422670|netlink_socket|ERR|connect(0): Argument list > too long > 2017-10-21T03:56:10.523Z|422671|netlink_socket|ERR|connect(0): Argument list > too long > 2017-10-21T03:56:10.525Z|422672|netlink_socket|ERR|connect(0): Argument list > too long > > > > 合天网安实验室-您身边的信息安全实验室 | quan_hp...@heetian.com > > From: Ben Pfaff > Date: 2017-10-21 01:41 > To: quan_hp...@heetian.com > CC: ovs-discuss > Subject: Re: [ovs-discuss] ovs-vswitch Huge number of netlink file > descriptors open > On Fri, Oct 20, 2017 at 06:56:26PM +0800, quan_hp...@heetian.com wrote: > > Hi All, > > I searched a mail list like my problem. > > https://mail.openvswitch.org/pipermail/ovs-discuss/2017-March/043817.html > > > > $sudo lsof -p $(pidof ovs-vswitchd) | wc -l > > 115536 > > $cat /proc/$(cat /var/run/openvswitch/ovs-vswitchd.pid)/limits | grep open > > Max open files9999files > > $ovs-vsctl show | grep -c Port > > 905 > > If anyone has any suggestions for how to solve it?Thanks. > > Can you find out what k
Re: [ovs-discuss] ovs-vswitch Huge number of netlink file descriptors open
Hi,All, The file descriptors are "netlink". $sudo lsof -p $(pidof ovs-vswitchd) ovs-vswit 2197 root *297u netlink 0t0 42309398 GENERIC ovs-vswit 2197 root *298u netlink 0t0 42309399 GENERIC ovs-vswit 2197 root *299u sock0,7 0t0 42309400 protocol: NETLINK ovs-vswit 2197 root *300u netlink 0t0 42309401 GENERIC ovs-vswit 2197 root *301u netlink 0t0 42309402 GENERIC ...More... ovs-vswit 2197 root *351u netlink 0t0 39326247 GENERIC ovs-vswit 2197 root *352u netlink 0t0 39121841 GENERIC ovs-vswit 2197 root *354w REG 253,0 10086253 1342248977 /var/log/openvswitch/ovs-vswitchd.log $sudo reboot When I restart the machine ,the file descriptor will increasing.Some time more then 12K record. Recenty log about openvswitch. ovsdb-server.log 2017-10-21T04:01:02.559Z|01308|jsonrpc|WARN|Dropped 7 log messages in last 17 seconds (most recently, 10 seconds ago) due to excessive rate 2017-10-21T04:01:02.559Z|01309|jsonrpc|WARN|unix: receive error: Connection reset by peer 2017-10-21T04:01:02.560Z|01310|reconnect|WARN|unix: connection dropped (Connection reset by peer) 2017-10-21T04:01:03.282Z|01311|reconnect|WARN|unix: connection dropped (Connection reset by peer) 2017-10-21T04:01:10.075Z|01312|jsonrpc|WARN|Dropped 1 log messages in last 7 seconds (most recently, 7 seconds ago) due to excessive rate 2017-10-21T04:01:10.075Z|01313|jsonrpc|WARN|unix: send error: Broken pipe 2017-10-21T04:01:10.076Z|01314|reconnect|WARN|unix: connection dropped (Broken pipe) 2017-10-21T04:01:14.907Z|01315|reconnect|WARN|unix: connection dropped (Connection reset by peer) 2017-10-21T04:01:20.134Z|01316|reconnect|WARN|unix: connection dropped (Broken pipe) 2017-10-21T04:01:27.499Z|01317|jsonrpc|WARN|Dropped 2 log messages in last 13 seconds (most recently, 8 seconds ago) due to excessive rate 2017-10-21T04:01:27.499Z|01318|jsonrpc|WARN|unix: receive error: Connection reset by peer 2017-10-21T04:01:27.499Z|01319|reconnect|WARN|unix: connection dropped (Connection reset by peer) 2017-10-21T04:01:33.758Z|01320|jsonrpc|WARN|unix: receive error: Connection reset by peer 2017-10-21T04:01:33.758Z|01321|reconnect|WARN|unix: connection dropped (Connection reset by peer) 2017-10-21T04:01:34.489Z|01322|reconnect|WARN|unix: connection dropped (Connection reset by peer) 2017-10-21T04:01:39.198Z|01323|reconnect|WARN|unix: connection dropped (Connection reset by peer) 2017-10-21T04:01:41.569Z|01324|reconnect|WARN|unix: connection dropped (Broken pipe) 2017-10-21T04:01:41.913Z|01325|reconnect|WARN|unix: connection dropped (Broken pipe) 2017-10-21T04:01:42.365Z|01326|reconnect|WARN|unix: connection dropped (Connection reset by peer) ovs-vswitchd.log 2017-10-21T03:56:10.511Z|422666|netlink_socket|ERR|connect(0): Argument list too long 2017-10-21T03:56:10.513Z|422667|netlink_socket|ERR|connect(0): Argument list too long 2017-10-21T03:56:10.516Z|422668|netlink_socket|ERR|connect(0): Argument list too long 2017-10-21T03:56:10.518Z|422669|netlink_socket|ERR|connect(0): Argument list too long 2017-10-21T03:56:10.520Z|422670|netlink_socket|ERR|connect(0): Argument list too long 2017-10-21T03:56:10.523Z|422671|netlink_socket|ERR|connect(0): Argument list too long 2017-10-21T03:56:10.525Z|422672|netlink_socket|ERR|connect(0): Argument list too long 合天网安实验室-您身边的信息安全实验室 | quan_hp...@heetian.com From: Ben Pfaff Date: 2017-10-21 01:41 To: quan_hp...@heetian.com CC: ovs-discuss Subject: Re: [ovs-discuss] ovs-vswitch Huge number of netlink file descriptors open On Fri, Oct 20, 2017 at 06:56:26PM +0800, quan_hp...@heetian.com wrote: > Hi All, > I searched a mail list like my problem. > https://mail.openvswitch.org/pipermail/ovs-discuss/2017-March/043817.html > > $sudo lsof -p $(pidof ovs-vswitchd) | wc -l > 115536 > $cat /proc/$(cat /var/run/openvswitch/ovs-vswitchd.pid)/limits | grep open > Max open files9999files > $ovs-vsctl show | grep -c Port > 905 > If anyone has any suggestions for how to solve it?Thanks. Can you find out what kinds of file descriptors are open? Are they sockets (and what kind), etc.? This could be a file descriptor leak of some kind. If it is, then restarting OVS would fix it; if it is not, then restarting OVS will not help, or at least not for long. Can you figure out whether restarting reduces the number of file descriptors, and by how much? ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] ovs-vswitch Huge number of netlink file descriptors open
On Fri, Oct 20, 2017 at 06:56:26PM +0800, quan_hp...@heetian.com wrote: > Hi All, > I searched a mail list like my problem. > https://mail.openvswitch.org/pipermail/ovs-discuss/2017-March/043817.html > > $sudo lsof -p $(pidof ovs-vswitchd) | wc -l > 115536 > $cat /proc/$(cat /var/run/openvswitch/ovs-vswitchd.pid)/limits | grep open > Max open files9999files > $ovs-vsctl show | grep -c Port > 905 > If anyone has any suggestions for how to solve it?Thanks. Can you find out what kinds of file descriptors are open? Are they sockets (and what kind), etc.? This could be a file descriptor leak of some kind. If it is, then restarting OVS would fix it; if it is not, then restarting OVS will not help, or at least not for long. Can you figure out whether restarting reduces the number of file descriptors, and by how much? ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
[ovs-discuss] ovs-vswitch Huge number of netlink file descriptors open
Hi All, I searched a mail list like my problem. https://mail.openvswitch.org/pipermail/ovs-discuss/2017-March/043817.html $sudo lsof -p $(pidof ovs-vswitchd) | wc -l 115536 $cat /proc/$(cat /var/run/openvswitch/ovs-vswitchd.pid)/limits | grep open Max open files9999files $ovs-vsctl show | grep -c Port 905 If anyone has any suggestions for how to solve it?Thanks. ovs-vswitchd.log 2017-10-17T12:33:38.811Z|18060|netlink_socket|ERR|fcntl: Too many open files 2017-10-17T12:33:38.813Z|18061|netlink_socket|ERR|fcntl: Too many open files 2017-10-17T12:33:38.815Z|18062|netlink_socket|ERR|fcntl: Too many open files 2017-10-17T12:33:38.980Z|18063|bridge|INFO|bridge br-int: added interface tap89a1ae90-54 on port 728 2017-10-17T12:33:38.982Z|18064|netlink_socket|ERR|fcntl: Too many open files 2017-10-17T12:33:38.982Z|18065|dpif|WARN|Dropped 137 log messages in last 1 seconds (most recently, 0 seconds ago) due to excessive rate 2017-10-17T12:33:38.982Z|18066|dpif|WARN|system@ovs-system: failed to add tapa6556974-45 as port: Too many open files 2017-10-17T12:33:38.985Z|18067|netlink_socket|ERR|fcntl: Too many open files 2017-10-17T12:33:38.987Z|18068|netlink_socket|ERR|fcntl: Too many open files 2017-10-17T12:33:38.989Z|18069|netlink_socket|ERR|fcntl: Too many open files 2017-10-17T12:33:38.991Z|18070|netlink_socket|ERR|fcntl: Too many open files 2017-10-17T12:33:38.994Z|18071|netlink_socket|ERR|fcntl: Too many open files 2017-10-17T12:33:38.996Z|18072|netlink_socket|ERR|fcntl: Too many open files 2017-10-17T12:33:38.998Z|18073|netlink_socket|ERR|fcntl: Too many open files ovsdb-server.log 2017-10-18T03:16:14.265Z|00209|ovsdb_file|INFO|/etc/openvswitch/conf.db: compacting database online (1508254105.699 seconds old, 1059 transactions, 10534268 bytes) 2017-10-18T03:16:14.368Z|00210|jsonrpc|WARN|unix: send error: Broken pipe 2017-10-18T03:16:14.368Z|00211|reconnect|WARN|unix: connection dropped (Broken pipe) 2017-10-18T03:16:29.375Z|00212|jsonrpc|WARN|unix: send error: Broken pipe 2017-10-18T03:16:29.375Z|00213|reconnect|WARN|unix: connection dropped (Broken pipe) 2017-10-18T03:16:37.765Z|00214|jsonrpc|WARN|unix: send error: Broken pipe 2017-10-18T03:16:37.765Z|00215|reconnect|WARN|unix: connection dropped (Broken pipe) 2017-10-18T03:16:43.657Z|00216|jsonrpc|WARN|unix: send error: Broken pipe 2017-10-18T03:16:43.658Z|00217|reconnect|WARN|unix: connection dropped (Broken pipe) 2017-10-18T03:17:04.876Z|00218|jsonrpc|WARN|unix: receive error: Connection reset by peer 2017-10-18T03:17:04.876Z|00219|reconnect|WARN|unix: connection dropped (Connection reset by peer) 2017-10-18T03:17:55.271Z|00220|jsonrpc|WARN|unix: receive error: Connection reset by peer 2017-10-18T03:17:55.272Z|00221|reconnect|WARN|unix: connection dropped (Connection reset by peer) ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
