I have 2 openstack clusters running the Folsom release with multiple Swift nodes. I also have a small setup that is running only Swift with a single node. I'm noticing very large Swift I/O latencies (seconds long) on the openstack clusters - ssbench output snippet is below. Performance is approximately identical on the openstack clusters. The Swift only cluster performs much better. Setup differences: Openstack clusters are using Keystone authentication - Swift only setup uses temp auth. Multiple Swift nodes on openstack clusters - Swift only has single node.
I also see that all object I/Os are being sent to the 2nd Swift node, there are
no objects on the 1st swift node. Both nodes are running swift-proxy services.
I have LOG_LOCAL0 set in the environment and also logging enabled in
/etc/swift/swift.conf, but haven't seen any log entries made.
What can I look at to debug the cause of the excessive latencies on both of
these stacks?
I'd also like to determine why all objects are on the 2nd swift node and none
on the 1st node.
I'm supposed to do performance evaluations but need to fix the latency problem
first.
Ssbench output snippet:
TOTAL
Count: 50 Average requests per second: 9.2
min max avg std_dev 95%-ile
Worst latency TX ID
First-byte latency: 0.067 - 2.513 0.390 ( 0.604) 1.948 (all
obj sizes) txae75691d37d544b4ac0cfe3b8cba7f38
Last-byte latency: 0.067 - 3.337 0.430 ( 0.695) 1.997 (all
obj sizes) txdcedb82227654b338daa85751f6d1232
First-byte latency: 0.070 - 2.513 0.542 ( 0.749) 2.255 (
tiny objs) txae75691d37d544b4ac0cfe3b8cba7f38
Last-byte latency: 0.070 - 2.514 0.468 ( 0.659) 1.997 (
tiny objs) txae75691d37d544b4ac0cfe3b8cba7f38
First-byte latency: 0.067 - 1.884 0.251 ( 0.382) 0.695 (
small objs) tx2ceec827f3304530b01a0d5993eea2e8
Last-byte latency: 0.067 - 3.337 0.385 ( 0.732) 1.884 (
small objs) txdcedb82227654b338daa85751f6d1232
Thanks,
Tim
Timothy Snider
Strategic Planning & Architecture - Advanced Development
NetApp
316-636-8736 Direct Phone
316-213-0223 Mobile Phone
tim.sni...@netapp.com
netapp.com<http://www.netapp.com/?ref_source=eSig>
[Description: http://media.netapp.com/images/netapp-logo-sig-5.gif]
<<inline: image001.gif>>
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
