Re: [Openstack] [Swift] PUT Object performance problem
Hi clay I just using the command like `swift-ring-builder object.builder set_weight d234 2220`. Then execute `swift-ring-builder object.builder rebalance` and push the rings all object node. Here is more detail information. I/O scheduler : deadline OS : CentOS 7.4 Kernel version : 3.10.0-693.11.1.el7.x86_64 I will try decrease the node_timeout from 3 to 1. (Do you have any recommend value?) On Tue, Oct 30, 2018 at 1:42 AM Clay Gerrard wrote: > Obviously a re-balance will cost some IO, but it's normally perceptible to > the client unless you were already on a razor thin line. > > Two config options seem obvious to think about experimenting with: > > You could decrease the node_timeout and let the proxy try to write more to > handoffs > You could try to use some of the ionice options (or other tuning options) > to make the replicators hammer the disks a little less > > It might having something to do with how you're organizing your ring > weight changes - could you describe how you're managing rings? > Could also be a io scheduling issue - are you using noop/deadline or cfq? > What kernel version? > > -Clay > > > On Mon, Oct 29, 2018 at 9:55 AM SeongSoo Cho wrote: > >> For more information, I use ocata version. >> >> 2018년 10월 29일 (월) 오후 10:07, SeongSoo Cho 님이 작성: >> >>> Hello, All >>> >>> I have a terrible problem with object server. >>> Here is the case. >>> 1. User upload an object to proxy-server >>> 2. Proxy server try to connect with object-server >>> 3. If one of object-server is slow to respond, proxy-server is waiting >>> for response. >>> 3.1 While waiting for response, proxy-server can't do anything >>> 4. So, The response of client request will be delayed. >>> >>> In my opinion, this code seems to be a problem >>> ( >>> https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L1734 >>> ) >>> >>> ``` >>> with ResponseTimeout(node_timeout): >>> resp = conn.getexpect() >>> ``` >>> >>> If node_timeout's value is 3 and object-server respond after 2 seconds, >>> proxy-server wait 2 seconds. >>> >>> Because proxy-server wait for the above response, the execution of the >>> following code is delayed. >>> ( >>> https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L627 >>> ) >>> >>> ``` >>> for node in nodes: >>> try: >>> putter = self._make_putter(node, part, req, headers) >>> self.app.set_node_timing(node, putter.connect_duration) >>> return putter >>> ``` >>> >>> This problem occurs when i do a ring rebalance. >>> When object-replicator delete a partition directory that are no longer >>> mine, the disk becomes very busy (Because of xfsaild daemon) >>> Because the disk are busy, object-server can't create diskfile during >>> PUT operation. >>> >>> Is there anyone who is having problems like me? >>> How can I solve this problem? >>> >>> I need everyone's help. >>> Thanks. >>> >>> Best Regards >>> SeongSoo Cho >>> >>> -- >>> SeongSoo Cho (South Korea) >>> >>> ___ >> Mailing list: >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> Post to : openstack@lists.openstack.org >> Unsubscribe : >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [Swift] PUT Object performance problem
Obviously a re-balance will cost some IO, but it's normally perceptible to the client unless you were already on a razor thin line. Two config options seem obvious to think about experimenting with: You could decrease the node_timeout and let the proxy try to write more to handoffs You could try to use some of the ionice options (or other tuning options) to make the replicators hammer the disks a little less It might having something to do with how you're organizing your ring weight changes - could you describe how you're managing rings? Could also be a io scheduling issue - are you using noop/deadline or cfq? What kernel version? -Clay On Mon, Oct 29, 2018 at 9:55 AM SeongSoo Cho wrote: > For more information, I use ocata version. > > 2018년 10월 29일 (월) 오후 10:07, SeongSoo Cho 님이 작성: > >> Hello, All >> >> I have a terrible problem with object server. >> Here is the case. >> 1. User upload an object to proxy-server >> 2. Proxy server try to connect with object-server >> 3. If one of object-server is slow to respond, proxy-server is waiting >> for response. >> 3.1 While waiting for response, proxy-server can't do anything >> 4. So, The response of client request will be delayed. >> >> In my opinion, this code seems to be a problem >> ( >> https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L1734 >> ) >> >> ``` >> with ResponseTimeout(node_timeout): >> resp = conn.getexpect() >> ``` >> >> If node_timeout's value is 3 and object-server respond after 2 seconds, >> proxy-server wait 2 seconds. >> >> Because proxy-server wait for the above response, the execution of the >> following code is delayed. >> ( >> https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L627 >> ) >> >> ``` >> for node in nodes: >> try: >> putter = self._make_putter(node, part, req, headers) >> self.app.set_node_timing(node, putter.connect_duration) >> return putter >> ``` >> >> This problem occurs when i do a ring rebalance. >> When object-replicator delete a partition directory that are no longer >> mine, the disk becomes very busy (Because of xfsaild daemon) >> Because the disk are busy, object-server can't create diskfile during PUT >> operation. >> >> Is there anyone who is having problems like me? >> How can I solve this problem? >> >> I need everyone's help. >> Thanks. >> >> Best Regards >> SeongSoo Cho >> >> -- >> SeongSoo Cho (South Korea) >> >> ___ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack@lists.openstack.org > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [Swift] PUT Object performance problem
For more information, I use ocata version. 2018년 10월 29일 (월) 오후 10:07, SeongSoo Cho 님이 작성: > Hello, All > > I have a terrible problem with object server. > Here is the case. > 1. User upload an object to proxy-server > 2. Proxy server try to connect with object-server > 3. If one of object-server is slow to respond, proxy-server is waiting for > response. > 3.1 While waiting for response, proxy-server can't do anything > 4. So, The response of client request will be delayed. > > In my opinion, this code seems to be a problem > ( > https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L1734 > ) > > ``` > with ResponseTimeout(node_timeout): > resp = conn.getexpect() > ``` > > If node_timeout's value is 3 and object-server respond after 2 seconds, > proxy-server wait 2 seconds. > > Because proxy-server wait for the above response, the execution of the > following code is delayed. > ( > https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L627 > ) > > ``` > for node in nodes: > try: > putter = self._make_putter(node, part, req, headers) > self.app.set_node_timing(node, putter.connect_duration) > return putter > ``` > > This problem occurs when i do a ring rebalance. > When object-replicator delete a partition directory that are no longer > mine, the disk becomes very busy (Because of xfsaild daemon) > Because the disk are busy, object-server can't create diskfile during PUT > operation. > > Is there anyone who is having problems like me? > How can I solve this problem? > > I need everyone's help. > Thanks. > > Best Regards > SeongSoo Cho > > -- > SeongSoo Cho (South Korea) > > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack