Re: [openstack-dev] Swift debugging / performance - large latencies seen.

2013-07-09 Thread Snider, Tim 
That helped -- using tempauth instead of keystone improved performance 
significantly on the openStack cluster.
However performance is still slower compared to my Swift only setup where 
commands are sent directly to the swift node instead of going thru the 
controller node in the openstack  cluster.
How does controller and swift node communications affect swift performance?
I've also noticed that all objects are stored on the 2nd swift node and not the 
1st on the openstack cluster. I'm wondering if that could also be a factor in 
slow performance.

Keystone:
TOTAL
   Count:50  Average requests per second:   9.2
min   max  avg  std_dev  95%-ile
   Worst latency TX ID
   First-byte latency:  0.067 -   2.5130.390  (  0.604)1.948  (all 
obj sizes)  txae75691d37d544b4ac0cfe3b8cba7f38
   Last-byte  latency:  0.067 -   3.3370.430  (  0.695)1.997  (all 
obj sizes)  txdcedb82227654b338daa85751f6d1232
   First-byte latency:  0.070 -   2.5130.542  (  0.749)2.255  (
tiny objs)  txae75691d37d544b4ac0cfe3b8cba7f38
   Last-byte  latency:  0.070 -   2.5140.468  (  0.659)1.997  (
tiny objs)  txae75691d37d544b4ac0cfe3b8cba7f38
   First-byte latency:  0.067 -   1.8840.251  (  0.382)0.695  (   
small objs)  tx2ceec827f3304530b01a0d5993eea2e8
   Last-byte  latency:  0.067 -   3.3370.385  (  0.732)1.884  (   
small objs)  txdcedb82227654b338daa85751f6d1232

Tempauth:
   Count:50  Average requests per second:  65.7
min   max  avg  std_dev  95%-ile
   Worst latency TX ID
   First-byte latency:  0.006 -   0.0730.014  (  0.015)0.055  (all 
obj sizes)  tx69bf033a246645808b2c6a280e334f15
   Last-byte  latency:  0.006 -   0.2480.047  (  0.070)0.198  (all 
obj sizes)  txb8cf5dc0ce264eb08a1e05edbbf5a40f
   First-byte latency:  0.006 -   0.0730.017  (  0.020)0.072  (
tiny objs)  tx69bf033a246645808b2c6a280e334f15
   Last-byte  latency:  0.006 -   0.2480.053  (  0.072)0.195  (
tiny objs)  txb8cf5dc0ce264eb08a1e05edbbf5a40f
   First-byte latency:  0.006 -   0.0260.010  (  0.005)0.026  (   
small objs)  tx65d1fd4b6ae049bb902442ac4c28ffe9
   Last-byte  latency:  0.006 -   0.2180.040  (  0.066)0.198  (   
small objs)  txbfd6ebc74ed04068affd17c123572a44

Swift Only:
TOTAL
   Count:50  Average requests per second: 397.0
min   max  avg  std_dev  95%-ile
   Worst latency TX ID
   First-byte latency:  0.003 -   0.0070.005  (  0.001)0.006  (all 
obj sizes)  None
   Last-byte  latency:  0.003 -   0.0460.008  (  0.009)0.029  (all 
obj sizes)  None
   First-byte latency:  0.003 -   0.0070.005  (  0.001)0.007  (
tiny objs)  None
   Last-byte  latency:  0.003 -   0.0460.008  (  0.010)0.027  (
tiny objs)  None
   First-byte latency:  0.004 -   0.0060.005  (  0.001)0.006  (   
small objs)  None
   Last-byte  latency:  0.004 -   0.0430.008  (  0.009)0.029  (   
small objs)  None

From: Chmouel Boudjnah [mailto:chmo...@enovance.com]
Sent: Tuesday, July 09, 2013 8:22 AM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] Swift debugging / performance - large latencies 
seen.

On Tue, Jul 9, 2013 at 2:40 PM, Snider, Tim 
mailto:tim.sni...@netapp.com>> wrote:
I have 2 openstack clusters running the Folsom release with multiple Swift 
nodes. I also have a small setup that is running only Swift with a single node. 
 I'm noticing very large Swift I/O latencies (seconds long) on the openstack 
clusters - ssbench output snippet is below. Performance is approximately 
identical on the openstack clusters. The Swift only cluster performs much 
better.

Keystone performance can be pretty awful unless you are using something else 
than the default WSGI container configuration (single process eventlet I 
think). I would suggest you try to run it under apache with multiple process.

See the dicussion at last summit about Keystone performance here :

https://etherpad.openstack.org/havana-keystone-performance

Chmouel.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Swift debugging / performance - large latencies seen.

2013-07-09 Thread Chmouel Boudjnah 
Hi Tim,

it's the first link on google for "apache keystone" :

http://docs.openstack.org/developer/keystone/apache-httpd.html

 we probably want to move this to openstack-operators@

Chmouel.

On Tue, Jul 9, 2013 at 3:31 PM, Snider, Tim  wrote:
> Thanks for the hint. Is there documentation on how to run keystone using
> apache and multiple processes?
>
> I’m pretty raw with python, ruby, apache …
>
> Thx.
>
>
>
> From: Chmouel Boudjnah [mailto:chmo...@enovance.com]
> Sent: Tuesday, July 09, 2013 8:22 AM
> To: OpenStack Development Mailing List
> Subject: Re: [openstack-dev] Swift debugging / performance - large latencies
> seen.
>
>
>
> On Tue, Jul 9, 2013 at 2:40 PM, Snider, Tim  wrote:
>
> I have 2 openstack clusters running the Folsom release with multiple Swift
> nodes. I also have a small setup that is running only Swift with a single
> node.  I’m noticing very large Swift I/O latencies (seconds long) on the
> openstack clusters – ssbench output snippet is below. Performance is
> approximately identical on the openstack clusters. The Swift only cluster
> performs much better.
>
>
> Keystone performance can be pretty awful unless you are using something else
> than the default WSGI container configuration (single process eventlet I
> think). I would suggest you try to run it under apache with multiple
> process.
>
> See the dicussion at last summit about Keystone performance here :
>
> https://etherpad.openstack.org/havana-keystone-performance
>
> Chmouel.
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Swift debugging / performance - large latencies seen.

2013-07-09 Thread Snider, Tim 
Thanks for the hint. Is there documentation on how to run keystone using apache 
and multiple processes?
I'm pretty raw with python, ruby, apache ...
Thx.

From: Chmouel Boudjnah [mailto:chmo...@enovance.com]
Sent: Tuesday, July 09, 2013 8:22 AM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] Swift debugging / performance - large latencies 
seen.

On Tue, Jul 9, 2013 at 2:40 PM, Snider, Tim 
mailto:tim.sni...@netapp.com>> wrote:
I have 2 openstack clusters running the Folsom release with multiple Swift 
nodes. I also have a small setup that is running only Swift with a single node. 
 I'm noticing very large Swift I/O latencies (seconds long) on the openstack 
clusters - ssbench output snippet is below. Performance is approximately 
identical on the openstack clusters. The Swift only cluster performs much 
better.

Keystone performance can be pretty awful unless you are using something else 
than the default WSGI container configuration (single process eventlet I 
think). I would suggest you try to run it under apache with multiple process.

See the dicussion at last summit about Keystone performance here :

https://etherpad.openstack.org/havana-keystone-performance

Chmouel.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Swift debugging / performance - large latencies seen.

2013-07-09 Thread Chmouel Boudjnah 
On Tue, Jul 9, 2013 at 2:40 PM, Snider, Tim  wrote:

> I have 2 openstack clusters running the Folsom release with multiple Swift
> nodes. I also have a small setup that is running only Swift with a single
> node.  I’m noticing very large Swift I/O latencies (seconds long) on the
> openstack clusters – ssbench output snippet is below. Performance is
> approximately identical on the openstack clusters. The Swift only cluster
> performs much better. 
>

Keystone performance can be pretty awful unless you are using something
else than the default WSGI container configuration (single process eventlet
I think). I would suggest you try to run it under apache with multiple
process.

See the dicussion at last summit about Keystone performance here :

https://etherpad.openstack.org/havana-keystone-performance

Chmouel.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Swift debugging / performance - large latencies seen.

2013-07-09 Thread Snider, Tim 
I have 2 openstack clusters running the Folsom release with multiple Swift 
nodes. I also have a small setup that is running only Swift with a single node. 
 I'm noticing very large Swift I/O latencies (seconds long) on the openstack 
clusters - ssbench output snippet is below. Performance is approximately 
identical on the openstack clusters. The Swift only cluster performs much 
better.
Setup differences:
Openstack clusters are using Keystone authentication - Swift only setup uses 
temp auth.
Multiple Swift nodes on openstack clusters - Swift only has single node.

I also see that all object I/Os are being sent to the 2nd Swift node, there are 
no objects on the 1st swift node. Both nodes are running swift-proxy services.
I have LOG_LOCAL0 set in the environment and also logging enabled in 
/etc/swift/swift.conf, but haven't seen any log entries made.

What can I look at to debug the cause of the excessive latencies on both of 
these stacks?
I'd also like to determine why all objects are on the 2nd swift node and none 
on the 1st node.

I'm supposed to do performance evaluations but need to fix the latency problem 
first.

Ssbench output snippet:
TOTAL
   Count:50  Average requests per second:   9.2
min   max  avg  std_dev  95%-ile
   Worst latency TX ID
   First-byte latency:  0.067 -   2.5130.390  (  0.604)1.948  (all 
obj sizes)  txae75691d37d544b4ac0cfe3b8cba7f38
   Last-byte  latency:  0.067 -   3.3370.430  (  0.695)1.997  (all 
obj sizes)  txdcedb82227654b338daa85751f6d1232
   First-byte latency:  0.070 -   2.5130.542  (  0.749)2.255  (
tiny objs)  txae75691d37d544b4ac0cfe3b8cba7f38
   Last-byte  latency:  0.070 -   2.5140.468  (  0.659)1.997  (
tiny objs)  txae75691d37d544b4ac0cfe3b8cba7f38
   First-byte latency:  0.067 -   1.8840.251  (  0.382)0.695  (   
small objs)  tx2ceec827f3304530b01a0d5993eea2e8
   Last-byte  latency:  0.067 -   3.3370.385  (  0.732)1.884  (   
small objs)  txdcedb82227654b338daa85751f6d1232

Thanks,
Tim

Timothy Snider
Strategic Planning & Architecture - Advanced Development
NetApp
316-636-8736 Direct Phone
316-213-0223 Mobile Phone
tim.sni...@netapp.com
netapp.com
 [Description: http://media.netapp.com/images/netapp-logo-sig-5.gif]


<>___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev