Using swift stat probably isn't the best way to determine cluster performance, as those stats are updated async, and could be delayed quite a bit as you are heavily loading the cluster. It also might be worthwhile to use a tool like swift-bench to test your cluster to make sure it is properly setup before loading data into the system.
-- Chuck On Mon, Jan 14, 2013 at 10:38 AM, Leander Bessa Beernaert <[email protected]> wrote: > I'm getting around 5-6.5 GB a day of bytes written on Swift. I calculated > this by calling "swift stat && sleep 60s && swift stat". I did some > calculation based on those values to get to the end result. > > Currently I'm resetting swift with a node size of 64, since 90% of the files > are less than 70KB in size. I think that might help. > > > On Mon, Jan 14, 2013 at 4:34 PM, Chuck Thier <[email protected]> wrote: >> >> Hey Leander, >> >> Can you post what performance you are getting? If they are all >> sharing the same GigE network, you might also check that the links >> aren't being saturated, as it is pretty easy to saturate pushing 200k >> files around. >> >> -- >> Chuck >> >> On Mon, Jan 14, 2013 at 10:15 AM, Leander Bessa Beernaert >> <[email protected]> wrote: >> > Well, I've fixed the node size and disabled the all the replicator and >> > auditor processes. However, it is even slower now than it was before :/. >> > Any >> > suggestions? >> > >> > >> > On Mon, Jan 14, 2013 at 3:23 PM, Leander Bessa Beernaert >> > <[email protected]> wrote: >> >> >> >> Ok, thanks for all the tips/help. >> >> >> >> Regards, >> >> >> >> Leander >> >> >> >> >> >> On Mon, Jan 14, 2013 at 3:21 PM, Robert van Leeuwen >> >> <[email protected]> wrote: >> >>> >> >>> > Allow me to rephrase. >> >>> > I've read somewhere (can't remember where) that it would be faster >> >>> > to >> >>> > upload files if they would be uploaded to separate containeres. >> >>> > This was suggested for a standard swift installation with a certain >> >>> > replication factor. >> >>> > Since I'll be uploading the files with the replicators turned off, >> >>> > does >> >>> > it really matter if I insert a group of them in separate >> >>> > containeres? >> >>> >> >>> My guess is this concerns the SQLite database load distribution. >> >>> So yes, it still matters. >> >>> >> >>> Just to be clear: turning replicators off does not matter at ALL when >> >>> putting files in a healthy cluster. >> >>> Files will be "replicated" / put on all required nodes at the moment >> >>> the >> >>> put request is done. >> >>> The put request will only give an OK when there is quorum writing the >> >>> file (the file is stored on more than half of the required object >> >>> nodes) >> >>> The replicator daemons do not have anything to do with this. >> >>> >> >>> Cheers, >> >>> Robert >> >>> >> >>> _______________________________________________ >> >>> Mailing list: https://launchpad.net/~openstack >> >>> Post to : [email protected] >> >>> Unsubscribe : https://launchpad.net/~openstack >> >>> More help : https://help.launchpad.net/ListHelp >> >>> >> >> >> > >> > >> > _______________________________________________ >> > Mailing list: https://launchpad.net/~openstack >> > Post to : [email protected] >> > Unsubscribe : https://launchpad.net/~openstack >> > More help : https://help.launchpad.net/ListHelp >> > > > _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : [email protected] Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
