Re: [Openstack] [SWIFT] Change the partition power to recreate the RING

2013-01-14 Thread Alejandro Comisario
Chuck / John. We are having 50.000 request per minute ( where 10.000+ are put from small objects, from 10KB to 150KB ) We are using swift 1.7.4 with keystone token caching so no latency over there. We are having 12 proxyes and 24 datanodes divided in 4 zones ( each datanode has 48gb of ram, 2

Re: [Openstack] [SWIFT] Change the partition power to recreate the RING

2013-01-14 Thread Tong Li
John and swifters, I see this problem as a big problem and I think that the scenario described by Alejandro is a very common scenario. I am thinking if it is possible to have like two rings (one with the newer extended power, one with the existing ring power), when significant changes

Re: [Openstack] [SWIFT] Change the partition power to recreate the RING

2013-01-14 Thread John Dickinson
Yes, I think it would be a great topic for the summit. --John On Jan 14, 2013, at 7:54 AM, Tong Li liton...@us.ibm.com wrote: John and swifters, I see this problem as a big problem and I think that the scenario described by Alejandro is a very common scenario. I am thinking if it is

Re: [Openstack] [SWIFT] Change the partition power to recreate the RING

2013-01-14 Thread Chuck Thier
Hi Alejandro, I really doubt that partition size is causing these issues. It can be difficult to debug these types of issues without access to the cluster, but I can think of a couple of things to look at. 1. Check your disk io usage and io wait on the storage nodes. If that seems abnormally

Re: [Openstack] [SWIFT] Change the partition power to recreate the RING

2013-01-14 Thread Alejandro Comisario
Chuck et All. Let me go through the point one by one. #1 Even seeing that object-auditor allways runs and never stops, we stoped the swift-*-auditor and didnt see any improvements, from all the datanodes we have an average of 8% IO-WAIT (using iostat), the only thing that we see is the pid

Re: [Openstack] [SWIFT] Change the partition power to recreate the RING

2013-01-14 Thread Chuck Thier
Hey Alejandro, Those were the most common issues that people run into when they are having performance issues with swift. The other thing to check is to look at the logs to make sure there are no major issues (like bad drives, misconfigured nodes, etc.), which could add latency to the requests.

[Openstack] [SWIFT] Change the partition power to recreate the RING

2013-01-11 Thread Alejandro Comisario
Hi guys. We've created a swift cluster several months ago, the things is that righ now we cant add hardware and we configured lots of partitions thinking about the final picture of the cluster. Today each datanodes is having 2500+ partitions per device, and even tuning the background processes (

Re: [Openstack] [SWIFT] Change the partition power to recreate the RING

2013-01-11 Thread John Dickinson
If effect, this would be a complete replacement of your rings, and that is essentially a whole new cluster. All of the existing data would need to be rehashed into the new ring before it is available. There is no process that rehashes the data to ensure that it is still in the correct