Hi, I have a two node swift cluster receiving continuous traffic (mostly overwrites for existing objects) of 1GB files each.
Soon after the traffic started, I'm seeing the following traceback from some transactions... Traceback (most recent call last): File "/home/eightkpc/swift/swift/proxy/controllers/obj.py", line 692, in PUT chunk = next(data_source) File "/home/eightkpc/swift/swift/proxy/controllers/obj.py", line 559, in <lambda> data_source = iter(lambda: reader(self.app.client_chunk_size), '') File "/home/eightkpc/swift/swift/common/utils.py", line 2362, in read chunk = self.wsgi_input.read(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 147, in read return self._chunked_read(self.rfile, length) File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 137, in _chunked_read self.chunk_length = int(rfile.readline().split(";", 1)[0], 16) ValueError: invalid literal for int() with base 16: '' (txn: tx14e2df7680fd472fb92f0-00536ca4f0) (client_ip: 10.3.0.101) Seeing the following errors on storage logs... object-server: 10.3.0.102 - - [09/May/2014:01:36:49 +0000] "PUT /xvdg/492/AUTH_test/8kpc/30303A30323A30333A30343A30353A30396AEF6B53000000007B000000.2.data" 408 - "PUT http://10.3.0.102:8080/v1/AUTH_test/8kpc/30303A30323A30333A30343A30353A30396AEF6B53000000007B000000.2.data" "txf3b4e5f677004474bbd2f-00536c30d1" "proxy-server 12241" 95.6405 "-" It's success sometimes, but mostly 408 errors. I don't see any other logs for the transaction ID. or around these 408 errors in the log files. Is this a disk timeout issue? These are only 1GB files and normal writes to files on these disks are quite fast. The timeouts from the swift proxy files are... root@bulkstore-112:~# grep -R timeout /etc/swift/* /etc/swift/proxy-server.conf:client_timeout = 600 /etc/swift/proxy-server.conf:node_timeout = 600 /etc/swift/proxy-server.conf:recoverable_node_timeout = 600 Can someone help me troubleshoot this issue? -- -Shyam
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev