Hello Klaus, sorry for my late repsonse.
root@swift1:~# netstat -lntpua | grep TIME_WAIT | grep 6000 | wc -l 40819 root@swift1:~# netstat -lntpua | grep TIME_WAIT | grep 6001 | wc -l 15136 Yeah all on the same host. I notice ~30 GET/s, so not much. @Clay, no i don't notice massiv client disconnects. Thanks Heiko Am 10.03.2015 um 10:39 schrieb Klaus Schürmann: > Hello Heiko, > > what kind of connections do you see in TIME_WAIT status? > All connections to the object-server (Port 6000) ? > Are the container- and account-server on the same host? > > How many PUTs and GETs do you receive per second? > > Best Regards > Klaus > > -----Ursprüngliche Nachricht----- > Von: Heiko Krämer [mailto:[email protected]] > Gesendet: Montag, 9. März 2015 14:08 > An: [email protected] > Betreff: [Openstack] Swift and TIME_WAIT network stack problem > > > Hi guys, > > we running in a described problem on our storage nodes. > The object auditor process throws errors because the system has no > available ports. > > Mar 9 13:05:37 swift2 object-replicator: Error syncing with node: > {'replication_port': 6000, 'zone': 1, 'weight': 100.0, 'ip': > '10.0.0.22', 'region': 1, 'port': 6000, 'replication_ip': '10.0.0.22', > 'meta': u'', 'device': 'sda5', 'id': 0}: #012Traceback (most recent call > last):#012 File > "/usr/lib/python2.7/dist-packages/swift/obj/replicator.py", line 282, in > update#012 '', headers=self.headers).getresponse()#012 File > "/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py", line > 157, in http_connect#012 ipaddr, port, method, path, headers, > query_string, ssl)#012 File > "/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py", line > 189, in http_connect_raw#012 conn.endheaders()#012 File > "/usr/lib/python2.7/httplib.py", line 954, in endheaders#012 > self._send_output(message_body)#012 File > "/usr/lib/python2.7/httplib.py", line 814, in _send_output#012 > self.send(msg)#012 File "/usr/lib/python2.7/httplib.py", line 776, in > send#012 self.connect()#012 File > "/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py", line > 108, in connect#012 return HTTPConnection.connect(self)#012 File > "/usr/lib/python2.7/httplib.py", line 757, in connect#012 > self.timeout, self.source_address)#012 File > "/usr/lib/python2.7/dist-packages/eventlet/green/socket.py", line 59, in > create_connection#012 raise error, msg#012error: [Errno 99] > EADDRNOTAVAIL > > > :~# netstat --inet | grep TIME_WAIT | wc -l > 63038 > > This value of used ports is on all nodes nearly the same and > fluctuates extremely. So i tuned the kernel and network stack of the > Linux kernel but without success. > > > # disable TIME_WAIT.. wait.. > net.ipv4.tcp_tw_recycle=1 > net.ipv4.tcp_tw_reuse=1 > > # disable syn cookies > net.ipv4.tcp_syncookies = 0 > > # double amount of allowed conntrack > net.ipv4.netfilter.ip_conntrack_max = 262144 > > net.ipv4.ip_local_port_range = 18000 65535 > net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 1 > net.netfilter.nf_conntrack_tcp_timeout_established=600 > net.netfilter.nf_conntrack_tcp_timeout_time_wait=30 > net.ipv4.tcp_fin_timeout=15 > net.ipv4.tcp_keepalive_intvl=30 > net.ipv4.tcp_keepalive_probes=5 > > > The object-server conf-file: > [object-replicator] > recon_enable = yes > concurrency = 2 > run_pause = 60 > reclaim_age = 259200 > interval = 60 > > [object-updater] > concurrency = 4 > recon_enable = yes > recon_cache_path = /var/cache/swift > slowdown = 0.1 > > [object-auditor] > bytes_per_second = 3000000 > files_per_second = 10 > concurrency = 4 > recon_enable = yes > recon_cache_path = /var/cache/swift > > > > Have anyone a hint for me ? > > > Greetings > Heiko > > -- > anynines.com > > > > _______________________________________________ > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : [email protected] > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack -- anynines.com
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
