Another possibility is a large number of client disconnects - which could happen serving mostly media content for progressive download; or maybe lots of X-Newest requests.
I've seen situations where the proxy could try harder to close down backend connections - https://bugs.launchpad.net/swift/+bug/667956 -Clay On Tue, Mar 10, 2015 at 2:39 AM, Klaus Schürmann < [email protected]> wrote: > Hello Heiko, > > what kind of connections do you see in TIME_WAIT status? > All connections to the object-server (Port 6000) ? > Are the container- and account-server on the same host? > > How many PUTs and GETs do you receive per second? > > Best Regards > Klaus > > -----Ursprüngliche Nachricht----- > Von: Heiko Krämer [mailto:[email protected]] > Gesendet: Montag, 9. März 2015 14:08 > An: [email protected] > Betreff: [Openstack] Swift and TIME_WAIT network stack problem > > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi guys, > > we running in a described problem on our storage nodes. > The object auditor process throws errors because the system has no > available ports. > > Mar 9 13:05:37 swift2 object-replicator: Error syncing with node: > {'replication_port': 6000, 'zone': 1, 'weight': 100.0, 'ip': > '10.0.0.22', 'region': 1, 'port': 6000, 'replication_ip': '10.0.0.22', > 'meta': u'', 'device': 'sda5', 'id': 0}: #012Traceback (most recent call > last):#012 File > "/usr/lib/python2.7/dist-packages/swift/obj/replicator.py", line 282, in > update#012 '', headers=self.headers).getresponse()#012 File > "/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py", line > 157, in http_connect#012 ipaddr, port, method, path, headers, > query_string, ssl)#012 File > "/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py", line > 189, in http_connect_raw#012 conn.endheaders()#012 File > "/usr/lib/python2.7/httplib.py", line 954, in endheaders#012 > self._send_output(message_body)#012 File > "/usr/lib/python2.7/httplib.py", line 814, in _send_output#012 > self.send(msg)#012 File "/usr/lib/python2.7/httplib.py", line 776, in > send#012 self.connect()#012 File > "/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py", line > 108, in connect#012 return HTTPConnection.connect(self)#012 File > "/usr/lib/python2.7/httplib.py", line 757, in connect#012 > self.timeout, self.source_address)#012 File > "/usr/lib/python2.7/dist-packages/eventlet/green/socket.py", line 59, in > create_connection#012 raise error, msg#012error: [Errno 99] > EADDRNOTAVAIL > > > :~# netstat --inet | grep TIME_WAIT | wc -l > 63038 > > This value of used ports is on all nodes nearly the same and fluctuates > extremely. So i tuned the kernel and network stack of the Linux kernel but > without success. > > > # disable TIME_WAIT.. wait.. > net.ipv4.tcp_tw_recycle=1 > net.ipv4.tcp_tw_reuse=1 > > # disable syn cookies > net.ipv4.tcp_syncookies = 0 > > # double amount of allowed conntrack > net.ipv4.netfilter.ip_conntrack_max = 262144 > > net.ipv4.ip_local_port_range = 18000 65535 > net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 1 > net.netfilter.nf_conntrack_tcp_timeout_established=600 > net.netfilter.nf_conntrack_tcp_timeout_time_wait=30 > net.ipv4.tcp_fin_timeout=15 > net.ipv4.tcp_keepalive_intvl=30 > net.ipv4.tcp_keepalive_probes=5 > > > The object-server conf-file: > [object-replicator] > recon_enable = yes > concurrency = 2 > run_pause = 60 > reclaim_age = 259200 > interval = 60 > > [object-updater] > concurrency = 4 > recon_enable = yes > recon_cache_path = /var/cache/swift > slowdown = 0.1 > > [object-auditor] > bytes_per_second = 3000000 > files_per_second = 10 > concurrency = 4 > recon_enable = yes > recon_cache_path = /var/cache/swift > > > > Have anyone a hint for me ? > > > Greetings > Heiko > > - -- > anynines.com > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1 > > iQEcBAEBAgAGBQJU/ZtBAAoJELxFogM4ixOFS88IANZZBzzFnFymWXRyuAQGjVpz > X7Os9Y9Jn41EOph4HHS9eablTc14YX4YiB/JvKj1KKJAUOkVoPfB5oC154hQ5Goa > i3f1qSWg3qEqv/lo5EvtX++B92Ut/68OSUblie1XGkivs6ZIfzeByzJqDgwdS2kV > UEMzyEw9K4oNFkyURts8vH4NX4FgqKIoaPaQh6qOe27YKEdWw9NJn3NbRzWncwVJ > R181jaerubZo8gYOVO9zYLHoPFLSxVft7zC6M0fHK6SqDUosA8zjperlvWChx2ZD > UnL3LAEs1BCSxnJw876AvH9nxwFwkZwioQeVW5inTtqxmvZRn0RnsCY/qzv51Oc= > =2fqn > -----END PGP SIGNATURE----- > > > > _______________________________________________ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : [email protected] > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > _______________________________________________ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : [email protected] > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
