Hi, In my openstack swift cluster, I'm seeing a lot of 503 errors as a result of tracebacks in swift logs with "No such file or directory" exceptions... # grep -Rnw txdaba05e70c6b4dfaa5884-0059895aca /var/log/swift/* /var/log/swift/proxy.error:31030:Aug 7 23:31:39 BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c proxy-server: ERROR 500 Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/obj/server.py", line 1032, in __call__#012 res = method(req)#012 File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 1412, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/server.py", line 751, in PUT#012 writer.put(metadata)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 2451, in put#012 super(DiskFileWriter, self)._put(metadata, True)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1476, in _put#012 self._finalize_put, metadata, target_path, cleanup)#012 File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 3342, in force_run_in_thread#012 return self._run_in_eventlet_tpool(func, *args, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 3322, in _run_in_eventlet_tpool#012 raise result#012OSError: [Errno 2] No such file or directory#012 From Object Server re: /v1/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1 10.3.60.8:6010/xvdb1 (txn: txdaba05e70c6b4dfaa5884-0059895aca) (client_ip: 10.3.60.11) /var/log/swift/proxy.error:31031:Aug 7 23:31:39 BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c proxy-server: Object PUT returning 503 for [500, 201] (txn: txdaba05e70c6b4dfaa5884-0059895aca) (client_ip: 10.3.60.11) /var/log/swift/proxy.error:31032:Aug 7 23:31:39 BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c proxy-server: STDERR: 10.3.60.11 - - [08/Aug/2017 06:31:39] "PUT /v1/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1 HTTP/1.1" 503 346 1.553481 (txn: txdaba05e70c6b4dfaa5884-0059895aca) /var/log/swift/proxy.log:27701:Aug 7 23:31:39 BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c proxy-server: 10.3.60.11 10.3.60.11 08/Aug/2017/06/31/39 PUT /v1/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1 HTTP/1.0 503 - - AUTH_tke6014ecd5... 16777216 118 - txdaba05e70c6b4dfaa5884-0059895aca - 1.5526 - - 1502173898.383203983 1502173899.935844898 0 /var/log/swift/storage1.log:41634:Aug 7 23:31:39 BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c object-server: 10.3.60.8 - - [08/Aug/2017:06:31:39 +0000] "PUT /xvdb1/118/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1" 500 981 "PUT http://10.3.60.8:8080/v1/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1" "txdaba05e70c6b4dfaa5884-0059895aca" "proxy-server 2117" 1.0534 "-" 2127 0 /var/log/swift/storage2.log:128852:Aug 7 23:31:39 BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c container-server: 10.3.60.9 - - [08/Aug/2017:06:31:39 +0000] "PUT /xvdb2/972/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1" 201 - "PUT http://10.3.60.8:8080/xvdb2/118/AUTH_test/8kpc/data/37363A32353A33393A63353A36633A3566CA5588590000000000000000.73.0.1" "txdaba05e70c6b4dfaa5884-0059895aca" "object-server 1728" 0.0006 "-" 2099 0
I'm also seeing some errors removing tempfile errors in storage logs also... Aug 8 02:28:15 BulkStore-c2f99bd4-75ce-11e7-b536-02e7b943c03c object-server: Error removing tempfile: /srv/xvdb1/node/xvdb1/tmp/tmpFouKzU: #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 2396, in create#012 os.unlink(tmppath)#012OSError: [Errno 2] No such file or directory: '/srv/xvdb1/node/xvdb1/tmp/tmpFouKzU' (txn: tx860a415e4c454baeab4fc-005989842e) Can someone tell me what's going on? Thanks in advance for any help you can give me here. Regards, Shyam __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev