Hi Aravind,
There only seems to be container-server logs in your reply. So you have any
from the object server?
Also what ports are your object-server and container-servers listening on?
Your ring is saying the object-servers are listening on 6201. Just making
sure the port numbers aren't confused and the proxy isn't tryng to send the
object PUTs to the container servers. Because it's interesting that there
is no objects subfolder.
Matt
On Mon, Feb 12, 2018 at 10:12 PM, aRaviNd <ambadiarav...@gmail.com> wrote:
> Hi Clay, Hi All,
>
> Configured my cluster from ground up with one proxy node and one storage
> node.
>
>
>
> Now I am getting two types of errors.
>
> Large files:
>
> Feb 3 18:12:33 centos7-swift-proxy1 swift-proxy-server: ERROR with Object
> server 192.168.47.128:6201/sda re: Trying to write to
> /AUTH_admin/ara1/abc: #012Traceback (most recent call last):#012 File
> "/usr/lib/python2.7/site-packages/swift/proxy/controllers/obj.py", line
> 1617, in _send_file#012 self.conn.send(to_send)#012 File
> "/usr/lib64/python2.7/httplib.py", line 840, in send#012
> self.sock.sendall(data)#012 File "/usr/lib/python2.7/site-
> packages/eventlet/greenio/base.py", line 393, in sendall#012 tail +=
> self.send(data[tail:], flags)#012 File "/usr/lib/python2.7/site-
> packages/eventlet/greenio/base.py", line 384, in send#012 return
> self._send_loop(self.fd.send, data, flags)#012 File
> "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 371, in
> _send_loop#012 return send_method(data, *args)#012error: [Errno 32]
> Broken pipe
> Feb 3 18:12:33 centos7-swift-proxy1 swift-proxy-server: Object 1 PUT
> exceptions during send, 0/1 required connections (txn:
> tx0b01a226078f47b4b3593-005a7641e1) (client_ip: 192.168.47.132)
>
> Large file error on storage node:
>
> Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35624)
> Feb 4 19:15:51 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:15:51 +0000] "PUT /sda/89/AUTH_admin/ara1" 202 - "PUT
> http://192.168.47.132:8080/v1/AUTH_admin/ara1";
> "tx158875c238214209bdfc3-005a7646ab"
> "proxy-server 44255" 0.0063 "-" 112666 0
> Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:15:51] "PUT /sda/89/AUTH_admin/ara1
> HTTP/1.1" 202 252 0.006706 (txn: tx158875c238214209bdfc3-005a7646ab)
> Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35630)
> Feb 4 19:15:51 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:15:51 +0000] "HEAD /sda/89/AUTH_admin/ara1" 204 - "HEAD
> http://192.168.47.132:8080/v1/AUTH_admin/ara1";
> "txc23a59c70e8940bf85101-005a7646ab"
> "proxy-server 44253" 0.0011 "-" 112666 0
> Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:15:51] "HEAD /sda/89/AUTH_admin/ara1
> HTTP/1.1" 204 521 0.001422 (txn: txc23a59c70e8940bf85101-005a7646ab)
> Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35632)
> Feb 4 19:15:51 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:15:51 +0000] "HEAD /sda/75/AUTH_admin/ara1/abc" 404 - "HEAD
> http://192.168.47.132:8080/v1/AUTH_admin/ara1/abc";
> "txc23a59c70e8940bf85101-005a7646ab" "proxy-server 44253" 0.0003 "-"
> 112666 0
> Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:15:51] "HEAD /sda/75/AUTH_admin/ara1/abc
> HTTP/1.1" 404 351 0.000549 (txn: txc23a59c70e8940bf85101-005a7646ab)
> Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35634)
> Feb 4 19:15:51 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:15:51 +0000] "PUT /sda/75/AUTH_admin/ara1/abc" 404 - "PUT
> http://192.168.47.132:8080/v1/AUTH_admin/ara1/abc";
> "tx5138df5cf1f840808d2b2-005a7646ab" "proxy-server 44253" 0.0002 "-"
> 112666 0
> Feb 4 19:15:51 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:15:51] "PUT /sda/75/AUTH_admin/ara1/abc
> HTTP/1.1" 404 212 0.000506 (txn: tx5138df5cf1f840808d2b2-005a7646ab)
> Feb 4 19:15:52 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35636)
> Feb 4 19:15:52 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:15:52 +0000] "PUT /sda/75/AUTH_admin/ara1/abc" 404 - "PUT
> http://192.168.47.132:8080/v1/AUTH_admin/ara1/abc";
> "txeb1c76b896854e4885009-005a7646ac" "proxy-server 44253" 0.0002 "-"
> 112666 0
> Feb 4 19:15:52 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:15:52] "PUT /sda/75/AUTH_admin/ara1/abc
> HTTP/1.1" 404 212 0.000589 (txn: txeb1c76b896854e4885009-005a7646ac)
> Feb 4 19:15:54 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35638)
> Feb 4 19:15:54 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:15:54 +0000] "PUT /sda/75/AUTH_admin/ara1/abc" 404 - "PUT
> http://192.168.47.132:8080/v1/AUTH_admin/ara1/abc";
> "tx1a011b125608447ca36a8-005a7646ae" "proxy-server 44253" 0.0002 "-"
> 112666 0
> Feb 4 19:15:54 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:15:54] "PUT /sda/75/AUTH_admin/ara1/abc
> HTTP/1.1" 404 212 0.000569 (txn: tx1a011b125608447ca36a8-005a7646ae)
>
> or
>
> Smaller files:
>
> Feb 3 18:30:40 centos7-swift-proxy1 swift-proxy-server: ERROR Unhandled
> exception in request: #012Traceback (most recent call last):#012 File
> "/usr/lib/python2.7/site-packages/swift/proxy/server.py", line 521, in
> handle_request#012 return handler(req)#012 File
> "/usr/lib/python2.7/site-packages/swift/proxy/controllers/base.py", line
> 283, in wrapped#012 return func(*a, **kw)#012 File
> "/usr/lib/python2.7/site-packages/swift/proxy/controllers/obj.py", line
> 745, in PUT#012 req, data_source, nodes, partition,
> outgoing_headers)#012 File
> "/usr/lib/python2.7/site-packages/swift/proxy/controllers/obj.py",
> line 949, in _store_object#012 self._get_put_responses(req, putters,
> len(nodes))#012 File
> "/usr/lib/python2.7/site-packages/swift/proxy/controllers/obj.py",
> line 406, in _get_put_responses#012 _handle_response(putter,
> response)#012 File
> "/usr/lib/python2.7/site-packages/swift/proxy/controllers/obj.py",
> line 402, in _handle_response#012 etags.add(response.getheader('
> etag').strip('"'))#012AttributeError: 'NoneType' object has no attribute
> 'strip' (txn: tx029f271d16a84de09d4fb-005a764620) (client_ip:
> 192.168.47.132)
>
> Smaller file error on storage node:
>
> Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35696)
> Feb 4 19:17:34 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:17:34 +0000] "PUT /sda/89/AUTH_admin/ara1" 202 - "PUT
> http://192.168.47.132:8080/v1/AUTH_admin/ara1";
> "txb8d792ec40844162978cc-005a764712"
> "proxy-server 44253" 0.0077 "-" 112666 0
> Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:17:34] "PUT /sda/89/AUTH_admin/ara1
> HTTP/1.1" 202 252 0.008124 (txn: txb8d792ec40844162978cc-005a764712)
> Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35702)
> Feb 4 19:17:34 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:17:34 +0000] "HEAD /sda/89/AUTH_admin/ara1" 204 - "HEAD
> http://192.168.47.132:8080/v1/AUTH_admin/ara1";
> "tx672029e079574e78b1c1b-005a764712"
> "proxy-server 44254" 0.0011 "-" 112666 0
> Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:17:34] "HEAD /sda/89/AUTH_admin/ara1
> HTTP/1.1" 204 521 0.001465 (txn: tx672029e079574e78b1c1b-005a764712)
> Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35704)
> Feb 4 19:17:34 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:17:34 +0000] "HEAD /sda/26/AUTH_admin/ara1/swift.conf"
> 404 - "HEAD http://192.168.47.132:8080/v1/AUTH_admin/ara1/swift.conf";
> "tx672029e079574e78b1c1b-005a764712" "proxy-server 44254" 0.0003 "-"
> 112666 0
> Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:17:34] "HEAD
> /sda/26/AUTH_admin/ara1/swift.conf HTTP/1.1" 404 351 0.000547 (txn:
> tx672029e079574e78b1c1b-005a764712)
> Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35706)
> Feb 4 19:17:34 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:17:34 +0000] "PUT /sda/26/AUTH_admin/ara1/swift.conf" 404
> - "PUT http://192.168.47.132:8080/v1/AUTH_admin/ara1/swift.conf";
> "tx0ec94217b5584b7ea4e21-005a764712" "proxy-server 44254" 0.0002 "-"
> 112666 0
> Feb 4 19:17:34 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:17:34] "PUT
> /sda/26/AUTH_admin/ara1/swift.conf HTTP/1.1" 404 212 0.000513 (txn:
> tx0ec94217b5584b7ea4e21-005a764712)
> Feb 4 19:17:35 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35708)
> Feb 4 19:17:35 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:17:35 +0000] "PUT /sda/26/AUTH_admin/ara1/swift.conf" 404
> - "PUT http://192.168.47.132:8080/v1/AUTH_admin/ara1/swift.conf";
> "tx8e93f98453804972b39d8-005a764713" "proxy-server 44254" 0.0004 "-"
> 112666 0
> Feb 4 19:17:35 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:17:35] "PUT
> /sda/26/AUTH_admin/ara1/swift.conf HTTP/1.1" 404 212 0.000791 (txn:
> tx8e93f98453804972b39d8-005a764713)
> Feb 4 19:17:37 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35710)
> Feb 4 19:17:37 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:17:37 +0000] "PUT /sda/26/AUTH_admin/ara1/swift.conf" 404
> - "PUT http://192.168.47.132:8080/v1/AUTH_admin/ara1/swift.conf";
> "txd44faaca5b5b408da4fea-005a764715" "proxy-server 44254" 0.0004 "-"
> 112666 0
> Feb 4 19:17:37 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:17:37] "PUT
> /sda/26/AUTH_admin/ara1/swift.conf HTTP/1.1" 404 212 0.000716 (txn:
> txd44faaca5b5b408da4fea-005a764715)
> Feb 4 19:17:41 centos7-swift-node1 container-server: STDERR: (112666)
> accepted ('192.168.47.132', 35712)
> Feb 4 19:17:41 centos7-swift-node1 container-server: 192.168.47.132 - -
> [05/Feb/2018:00:17:41 +0000] "PUT /sda/26/AUTH_admin/ara1/swift.conf" 404
> - "PUT http://192.168.47.132:8080/v1/AUTH_admin/ara1/swift.conf";
> "tx6e5c49ba6d54489aa1de5-005a764719" "proxy-server 44254" 0.0002 "-"
> 112666 0
> Feb 4 19:17:41 centos7-swift-node1 container-server: STDERR:
> 192.168.47.132 - - [05/Feb/2018 00:17:41] "PUT
> /sda/26/AUTH_admin/ara1/swift.conf HTTP/1.1" 404 212 0.000590 (txn:
> tx6e5c49ba6d54489aa1de5-005a764719)
>
> Directories created on storage node are
>
>
>
> We are really stuck on the issue and not able to use swift in our
> environment. Any help will be really appreciated.
>
> Aravind M D
>
> On Thu, Feb 8, 2018 at 2:43 AM, Clay Gerrard <clay.gerr...@gmail.com>
> wrote:
>
>> One replica is a little strange. Do the uploads *always* fail - in the
>> same way? Or is this just one example of a PUT that returned 503? Are you
>> doing a lot of concurrent PUTs to the same object/name/disk?
>>
>> The error from the log (EPIPE) means the object-server closed the
>> connection as the proxy was writing to it... which is a little strange.
>> There should be a corresponding exception/error from the object-server
>> service - you should make sure the object-servers are running and find
>> where they are logging - then grep all the logs for the transaction-id to
>> get a better picture of the whole distributed transaction.
>>
>> If you keep digging I know you can find the problem. Let us know what
>> you find.
>>
>> Good luck,
>>
>> -Clay
>>
>>
>> On Wed, Feb 7, 2018 at 1:58 AM, aRaviNd <ambadiarav...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> We have created an openstack cluster with one proxy server and three
>>> storage nodes. Configuration consist of two regions and three zones.
>>>
>>> [image: enter image description here]
>>> <https://i.stack.imgur.com/w9EWG.png>
>>>
>>> We are able to create containers
>>>
>>> [image: enter image description here]
>>> <https://i.stack.imgur.com/QgQQM.png>
>>>
>>> But while trying to upload files we are getting 503 service unavailable
>>> and seeing below logs in swift.log
>>>
>>> [image: enter image description here]
>>> <https://i.stack.imgur.com/7ROMm.png>
>>> - Aravind
>>>
>>> _______________________________________________
>>> Mailing list: http://lists.openstack.org/cgi
>>> -bin/mailman/listinfo/openstack
>>> Post to : openstack@lists.openstack.org
>>> Unsubscribe : http://lists.openstack.org/cgi
>>> -bin/mailman/listinfo/openstack
>>>
>>>
>>
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
>
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
