Re: [openstack-dev] Swift debugging / performance - large latencies seen.
On Tue, Jul 9, 2013 at 2:40 PM, Snider, Tim tim.sni...@netapp.com wrote: I have 2 openstack clusters running the Folsom release with multiple Swift nodes. I also have a small setup that is running only Swift with a single node. I’m noticing very large Swift I/O latencies (seconds long) on the openstack clusters – ssbench output snippet is below. Performance is approximately identical on the openstack clusters. The Swift only cluster performs much better. Keystone performance can be pretty awful unless you are using something else than the default WSGI container configuration (single process eventlet I think). I would suggest you try to run it under apache with multiple process. See the dicussion at last summit about Keystone performance here : https://etherpad.openstack.org/havana-keystone-performance Chmouel. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Swift debugging / performance - large latencies seen.
Thanks for the hint. Is there documentation on how to run keystone using apache and multiple processes? I'm pretty raw with python, ruby, apache ... Thx. From: Chmouel Boudjnah [mailto:chmo...@enovance.com] Sent: Tuesday, July 09, 2013 8:22 AM To: OpenStack Development Mailing List Subject: Re: [openstack-dev] Swift debugging / performance - large latencies seen. On Tue, Jul 9, 2013 at 2:40 PM, Snider, Tim tim.sni...@netapp.commailto:tim.sni...@netapp.com wrote: I have 2 openstack clusters running the Folsom release with multiple Swift nodes. I also have a small setup that is running only Swift with a single node. I'm noticing very large Swift I/O latencies (seconds long) on the openstack clusters - ssbench output snippet is below. Performance is approximately identical on the openstack clusters. The Swift only cluster performs much better. Keystone performance can be pretty awful unless you are using something else than the default WSGI container configuration (single process eventlet I think). I would suggest you try to run it under apache with multiple process. See the dicussion at last summit about Keystone performance here : https://etherpad.openstack.org/havana-keystone-performance Chmouel. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Swift debugging / performance - large latencies seen.
Hi Tim, it's the first link on google for apache keystone : http://docs.openstack.org/developer/keystone/apache-httpd.html we probably want to move this to openstack-operators@ Chmouel. On Tue, Jul 9, 2013 at 3:31 PM, Snider, Tim tim.sni...@netapp.com wrote: Thanks for the hint. Is there documentation on how to run keystone using apache and multiple processes? I’m pretty raw with python, ruby, apache … Thx. From: Chmouel Boudjnah [mailto:chmo...@enovance.com] Sent: Tuesday, July 09, 2013 8:22 AM To: OpenStack Development Mailing List Subject: Re: [openstack-dev] Swift debugging / performance - large latencies seen. On Tue, Jul 9, 2013 at 2:40 PM, Snider, Tim tim.sni...@netapp.com wrote: I have 2 openstack clusters running the Folsom release with multiple Swift nodes. I also have a small setup that is running only Swift with a single node. I’m noticing very large Swift I/O latencies (seconds long) on the openstack clusters – ssbench output snippet is below. Performance is approximately identical on the openstack clusters. The Swift only cluster performs much better. Keystone performance can be pretty awful unless you are using something else than the default WSGI container configuration (single process eventlet I think). I would suggest you try to run it under apache with multiple process. See the dicussion at last summit about Keystone performance here : https://etherpad.openstack.org/havana-keystone-performance Chmouel. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Swift debugging / performance - large latencies seen.
That helped -- using tempauth instead of keystone improved performance significantly on the openStack cluster. However performance is still slower compared to my Swift only setup where commands are sent directly to the swift node instead of going thru the controller node in the openstack cluster. How does controller and swift node communications affect swift performance? I've also noticed that all objects are stored on the 2nd swift node and not the 1st on the openstack cluster. I'm wondering if that could also be a factor in slow performance. Keystone: TOTAL Count:50 Average requests per second: 9.2 min max avg std_dev 95%-ile Worst latency TX ID First-byte latency: 0.067 - 2.5130.390 ( 0.604)1.948 (all obj sizes) txae75691d37d544b4ac0cfe3b8cba7f38 Last-byte latency: 0.067 - 3.3370.430 ( 0.695)1.997 (all obj sizes) txdcedb82227654b338daa85751f6d1232 First-byte latency: 0.070 - 2.5130.542 ( 0.749)2.255 ( tiny objs) txae75691d37d544b4ac0cfe3b8cba7f38 Last-byte latency: 0.070 - 2.5140.468 ( 0.659)1.997 ( tiny objs) txae75691d37d544b4ac0cfe3b8cba7f38 First-byte latency: 0.067 - 1.8840.251 ( 0.382)0.695 ( small objs) tx2ceec827f3304530b01a0d5993eea2e8 Last-byte latency: 0.067 - 3.3370.385 ( 0.732)1.884 ( small objs) txdcedb82227654b338daa85751f6d1232 Tempauth: Count:50 Average requests per second: 65.7 min max avg std_dev 95%-ile Worst latency TX ID First-byte latency: 0.006 - 0.0730.014 ( 0.015)0.055 (all obj sizes) tx69bf033a246645808b2c6a280e334f15 Last-byte latency: 0.006 - 0.2480.047 ( 0.070)0.198 (all obj sizes) txb8cf5dc0ce264eb08a1e05edbbf5a40f First-byte latency: 0.006 - 0.0730.017 ( 0.020)0.072 ( tiny objs) tx69bf033a246645808b2c6a280e334f15 Last-byte latency: 0.006 - 0.2480.053 ( 0.072)0.195 ( tiny objs) txb8cf5dc0ce264eb08a1e05edbbf5a40f First-byte latency: 0.006 - 0.0260.010 ( 0.005)0.026 ( small objs) tx65d1fd4b6ae049bb902442ac4c28ffe9 Last-byte latency: 0.006 - 0.2180.040 ( 0.066)0.198 ( small objs) txbfd6ebc74ed04068affd17c123572a44 Swift Only: TOTAL Count:50 Average requests per second: 397.0 min max avg std_dev 95%-ile Worst latency TX ID First-byte latency: 0.003 - 0.0070.005 ( 0.001)0.006 (all obj sizes) None Last-byte latency: 0.003 - 0.0460.008 ( 0.009)0.029 (all obj sizes) None First-byte latency: 0.003 - 0.0070.005 ( 0.001)0.007 ( tiny objs) None Last-byte latency: 0.003 - 0.0460.008 ( 0.010)0.027 ( tiny objs) None First-byte latency: 0.004 - 0.0060.005 ( 0.001)0.006 ( small objs) None Last-byte latency: 0.004 - 0.0430.008 ( 0.009)0.029 ( small objs) None From: Chmouel Boudjnah [mailto:chmo...@enovance.com] Sent: Tuesday, July 09, 2013 8:22 AM To: OpenStack Development Mailing List Subject: Re: [openstack-dev] Swift debugging / performance - large latencies seen. On Tue, Jul 9, 2013 at 2:40 PM, Snider, Tim tim.sni...@netapp.commailto:tim.sni...@netapp.com wrote: I have 2 openstack clusters running the Folsom release with multiple Swift nodes. I also have a small setup that is running only Swift with a single node. I'm noticing very large Swift I/O latencies (seconds long) on the openstack clusters - ssbench output snippet is below. Performance is approximately identical on the openstack clusters. The Swift only cluster performs much better. Keystone performance can be pretty awful unless you are using something else than the default WSGI container configuration (single process eventlet I think). I would suggest you try to run it under apache with multiple process. See the dicussion at last summit about Keystone performance here : https://etherpad.openstack.org/havana-keystone-performance Chmouel. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
