Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
as Chuck explain you usually would see that on the server, having said that if you use latest swiftclient from github you will be able to see the requests that swiftclient make via keystoneclient to keystone to get a token. If you go on the swift proxy server and only if you use a recent checked out version of swift you will be able to see the auth_token debug messages (the authentication middleware for validating a token to keystone) in your proxy-server log. I would advise to take a look at a devstack, running it with swift and see how it work on the swift proxy server console (in screen). On Mon, Jan 14, 2013 at 8:00 PM, Leander Bessa Beernaert leande...@gmail.com wrote: Neither keystone nor swift proxy are producing any logs. I'm not sure what to do :S On Mon, Jan 14, 2013 at 6:50 PM, Chuck Thier cth...@gmail.com wrote: You would have to look at the proxy log to see if a request is being made. The results from the swift command line are just the calls that the client makes. The server still haves to validate the token on every request. -- Chuck On Mon, Jan 14, 2013 at 12:37 PM, Leander Bessa Beernaert leande...@gmail.com wrote: Below is an output from Swift stat, since I don't see any requests to keystone, I'm assuming that memcache is being used right? REQ: curl -i http://192.168.111.215:8080/v1/AUTH_[ID] -X HEAD -H X-Auth-Token: [TOKEN] DEBUG:swiftclient:REQ: curl -i http://192.168.111.215:8080/v1/AUTH_[ID]-X HEAD -H X-Auth-Token: [TOKEN] RESP STATUS: 204 DEBUG:swiftclient:RESP STATUS: 204 Account: AUTH_[ID] Containers: 44 Objects: 4818 Bytes: 112284450 Accept-Ranges: bytes X-Timestamp: 1358184925.20885 X-Trans-Id: tx8cffb469c9c542be830db10a2b90d901 On Mon, Jan 14, 2013 at 6:31 PM, Dolph Mathews dolph.math...@gmail.com wrote: If memcache is being utilized by your keystone middleware, you should see keystone attaching to it on the first incoming request, e.g.: keystoneclient.middleware.auth_token [INFO]: Using Keystone memcache for caching token You may also want to use auth_token from keystoneclient = v0.2.0 if you're not already (instead of from keystone itself). -Dolph On Mon, Jan 14, 2013 at 11:43 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Are you by any chance referring to this topic https://lists.launchpad.net/openstack/msg08639.html regarding the keystone token cache? If so I've already added the configuration line and have not noticed any speedup :/ On Mon, Jan 14, 2013 at 5:19 PM, Leander Bessa Beernaert leande...@gmail.com wrote: I'm using the ubuntu 12.04 packages of the folsom repository by the way. On Mon, Jan 14, 2013 at 5:18 PM, Chuck Thier cth...@gmail.com wrote: On Mon, Jan 14, 2013 at 11:03 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Also, I'm unable to run the swift-bench with keystone. Hrm... That was supposed to be fixed with this bug: https://bugs.launchpad.net/swift/+bug/1011727 My keystone dev instance isn't working at the moment, but I'll see if I can get one of the team to take a look at it. -- Chuck ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
I've updated the keystoneclient to the lastest version available in GitHub and I'm still not getting any speedups. I've switched to the tempauth system and noticed an immediate increase in throughput (6.4 GB/day to 53.5 GB/day). Since I'm time-restrained, I'll stick with the tempauth system for now. I'll check back on the keystone token cache later. Thanks for all the help. On Tue, Jan 15, 2013 at 8:36 AM, Chmouel Boudjnah chmo...@chmouel.comwrote: as Chuck explain you usually would see that on the server, having said that if you use latest swiftclient from github you will be able to see the requests that swiftclient make via keystoneclient to keystone to get a token. If you go on the swift proxy server and only if you use a recent checked out version of swift you will be able to see the auth_token debug messages (the authentication middleware for validating a token to keystone) in your proxy-server log. I would advise to take a look at a devstack, running it with swift and see how it work on the swift proxy server console (in screen). On Mon, Jan 14, 2013 at 8:00 PM, Leander Bessa Beernaert leande...@gmail.com wrote: Neither keystone nor swift proxy are producing any logs. I'm not sure what to do :S On Mon, Jan 14, 2013 at 6:50 PM, Chuck Thier cth...@gmail.com wrote: You would have to look at the proxy log to see if a request is being made. The results from the swift command line are just the calls that the client makes. The server still haves to validate the token on every request. -- Chuck On Mon, Jan 14, 2013 at 12:37 PM, Leander Bessa Beernaert leande...@gmail.com wrote: Below is an output from Swift stat, since I don't see any requests to keystone, I'm assuming that memcache is being used right? REQ: curl -i http://192.168.111.215:8080/v1/AUTH_[ID] -X HEAD -H X-Auth-Token: [TOKEN] DEBUG:swiftclient:REQ: curl -i http://192.168.111.215:8080/v1/AUTH_[ID] -X HEAD -H X-Auth-Token: [TOKEN] RESP STATUS: 204 DEBUG:swiftclient:RESP STATUS: 204 Account: AUTH_[ID] Containers: 44 Objects: 4818 Bytes: 112284450 Accept-Ranges: bytes X-Timestamp: 1358184925.20885 X-Trans-Id: tx8cffb469c9c542be830db10a2b90d901 On Mon, Jan 14, 2013 at 6:31 PM, Dolph Mathews dolph.math...@gmail.com wrote: If memcache is being utilized by your keystone middleware, you should see keystone attaching to it on the first incoming request, e.g.: keystoneclient.middleware.auth_token [INFO]: Using Keystone memcache for caching token You may also want to use auth_token from keystoneclient = v0.2.0 if you're not already (instead of from keystone itself). -Dolph On Mon, Jan 14, 2013 at 11:43 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Are you by any chance referring to this topic https://lists.launchpad.net/openstack/msg08639.html regarding the keystone token cache? If so I've already added the configuration line and have not noticed any speedup :/ On Mon, Jan 14, 2013 at 5:19 PM, Leander Bessa Beernaert leande...@gmail.com wrote: I'm using the ubuntu 12.04 packages of the folsom repository by the way. On Mon, Jan 14, 2013 at 5:18 PM, Chuck Thier cth...@gmail.com wrote: On Mon, Jan 14, 2013 at 11:03 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Also, I'm unable to run the swift-bench with keystone. Hrm... That was supposed to be fixed with this bug: https://bugs.launchpad.net/swift/+bug/1011727 My keystone dev instance isn't working at the moment, but I'll see if I can get one of the team to take a look at it. -- Chuck ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
any chances you can try with latest swift as well and set : log_level = DEBUG in swift proxy-server.conf and what what the authtoken middleware is doing in the /var/log/syslog (or wherever syslog log on your distro). Chmouel. On Tue, Jan 15, 2013 at 12:09 PM, Leander Bessa Beernaert leande...@gmail.com wrote: I've updated the keystoneclient to the lastest version available in GitHub and I'm still not getting any speedups. I've switched to the tempauth system and noticed an immediate increase in throughput (6.4 GB/day to 53.5 GB/day). Since I'm time-restrained, I'll stick with the tempauth system for now. I'll check back on the keystone token cache later. Thanks for all the help. On Tue, Jan 15, 2013 at 8:36 AM, Chmouel Boudjnah chmo...@chmouel.comwrote: as Chuck explain you usually would see that on the server, having said that if you use latest swiftclient from github you will be able to see the requests that swiftclient make via keystoneclient to keystone to get a token. If you go on the swift proxy server and only if you use a recent checked out version of swift you will be able to see the auth_token debug messages (the authentication middleware for validating a token to keystone) in your proxy-server log. I would advise to take a look at a devstack, running it with swift and see how it work on the swift proxy server console (in screen). On Mon, Jan 14, 2013 at 8:00 PM, Leander Bessa Beernaert leande...@gmail.com wrote: Neither keystone nor swift proxy are producing any logs. I'm not sure what to do :S On Mon, Jan 14, 2013 at 6:50 PM, Chuck Thier cth...@gmail.com wrote: You would have to look at the proxy log to see if a request is being made. The results from the swift command line are just the calls that the client makes. The server still haves to validate the token on every request. -- Chuck On Mon, Jan 14, 2013 at 12:37 PM, Leander Bessa Beernaert leande...@gmail.com wrote: Below is an output from Swift stat, since I don't see any requests to keystone, I'm assuming that memcache is being used right? REQ: curl -i http://192.168.111.215:8080/v1/AUTH_[ID] -X HEAD -H X-Auth-Token: [TOKEN] DEBUG:swiftclient:REQ: curl -i http://192.168.111.215:8080/v1/AUTH_[ID] -X HEAD -H X-Auth-Token: [TOKEN] RESP STATUS: 204 DEBUG:swiftclient:RESP STATUS: 204 Account: AUTH_[ID] Containers: 44 Objects: 4818 Bytes: 112284450 Accept-Ranges: bytes X-Timestamp: 1358184925.20885 X-Trans-Id: tx8cffb469c9c542be830db10a2b90d901 On Mon, Jan 14, 2013 at 6:31 PM, Dolph Mathews dolph.math...@gmail.com wrote: If memcache is being utilized by your keystone middleware, you should see keystone attaching to it on the first incoming request, e.g.: keystoneclient.middleware.auth_token [INFO]: Using Keystone memcache for caching token You may also want to use auth_token from keystoneclient = v0.2.0 if you're not already (instead of from keystone itself). -Dolph On Mon, Jan 14, 2013 at 11:43 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Are you by any chance referring to this topic https://lists.launchpad.net/openstack/msg08639.html regarding the keystone token cache? If so I've already added the configuration line and have not noticed any speedup :/ On Mon, Jan 14, 2013 at 5:19 PM, Leander Bessa Beernaert leande...@gmail.com wrote: I'm using the ubuntu 12.04 packages of the folsom repository by the way. On Mon, Jan 14, 2013 at 5:18 PM, Chuck Thier cth...@gmail.com wrote: On Mon, Jan 14, 2013 at 11:03 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Also, I'm unable to run the swift-bench with keystone. Hrm... That was supposed to be fixed with this bug: https://bugs.launchpad.net/swift/+bug/1011727 My keystone dev instance isn't working at the moment, but I'll see if I can get one of the team to take a look at it. -- Chuck ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
Sorry, At the moment i'm unable to fetch the latest version of swift :s On Tue, Jan 15, 2013 at 12:14 PM, Chmouel Boudjnah chmo...@chmouel.comwrote: any chances you can try with latest swift as well and set : log_level = DEBUG in swift proxy-server.conf and what what the authtoken middleware is doing in the /var/log/syslog (or wherever syslog log on your distro). Chmouel. On Tue, Jan 15, 2013 at 12:09 PM, Leander Bessa Beernaert leande...@gmail.com wrote: I've updated the keystoneclient to the lastest version available in GitHub and I'm still not getting any speedups. I've switched to the tempauth system and noticed an immediate increase in throughput (6.4 GB/day to 53.5 GB/day). Since I'm time-restrained, I'll stick with the tempauth system for now. I'll check back on the keystone token cache later. Thanks for all the help. On Tue, Jan 15, 2013 at 8:36 AM, Chmouel Boudjnah chmo...@chmouel.comwrote: as Chuck explain you usually would see that on the server, having said that if you use latest swiftclient from github you will be able to see the requests that swiftclient make via keystoneclient to keystone to get a token. If you go on the swift proxy server and only if you use a recent checked out version of swift you will be able to see the auth_token debug messages (the authentication middleware for validating a token to keystone) in your proxy-server log. I would advise to take a look at a devstack, running it with swift and see how it work on the swift proxy server console (in screen). On Mon, Jan 14, 2013 at 8:00 PM, Leander Bessa Beernaert leande...@gmail.com wrote: Neither keystone nor swift proxy are producing any logs. I'm not sure what to do :S On Mon, Jan 14, 2013 at 6:50 PM, Chuck Thier cth...@gmail.com wrote: You would have to look at the proxy log to see if a request is being made. The results from the swift command line are just the calls that the client makes. The server still haves to validate the token on every request. -- Chuck On Mon, Jan 14, 2013 at 12:37 PM, Leander Bessa Beernaert leande...@gmail.com wrote: Below is an output from Swift stat, since I don't see any requests to keystone, I'm assuming that memcache is being used right? REQ: curl -i http://192.168.111.215:8080/v1/AUTH_[ID] -X HEAD -H X-Auth-Token: [TOKEN] DEBUG:swiftclient:REQ: curl -i http://192.168.111.215:8080/v1/AUTH_[ID] -X HEAD -H X-Auth-Token: [TOKEN] RESP STATUS: 204 DEBUG:swiftclient:RESP STATUS: 204 Account: AUTH_[ID] Containers: 44 Objects: 4818 Bytes: 112284450 Accept-Ranges: bytes X-Timestamp: 1358184925.20885 X-Trans-Id: tx8cffb469c9c542be830db10a2b90d901 On Mon, Jan 14, 2013 at 6:31 PM, Dolph Mathews dolph.math...@gmail.com wrote: If memcache is being utilized by your keystone middleware, you should see keystone attaching to it on the first incoming request, e.g.: keystoneclient.middleware.auth_token [INFO]: Using Keystone memcache for caching token You may also want to use auth_token from keystoneclient = v0.2.0 if you're not already (instead of from keystone itself). -Dolph On Mon, Jan 14, 2013 at 11:43 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Are you by any chance referring to this topic https://lists.launchpad.net/openstack/msg08639.html regarding the keystone token cache? If so I've already added the configuration line and have not noticed any speedup :/ On Mon, Jan 14, 2013 at 5:19 PM, Leander Bessa Beernaert leande...@gmail.com wrote: I'm using the ubuntu 12.04 packages of the folsom repository by the way. On Mon, Jan 14, 2013 at 5:18 PM, Chuck Thier cth...@gmail.com wrote: On Mon, Jan 14, 2013 at 11:03 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Also, I'm unable to run the swift-bench with keystone. Hrm... That was supposed to be fixed with this bug: https://bugs.launchpad.net/swift/+bug/1011727 My keystone dev instance isn't working at the moment, but I'll see if I can get one of the team to take a look at it. -- Chuck ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
Hello all, I'm trying to upload 200GB of 200KB files to Swift. I'm using 4 clients (each hosted on a different machine) with 10 threads each uploading files using the official python-swiftclient. Each thread is uploading to a separate container. I have 5 storage nodes and 1 proxy node. The nodes are all running with a replication factor of 3. Each node has a quad-core i3 processor, 4GB of RAM and a gigabit network interface. Is there any way I can speed up this process? At the moment it takes about 20 seconds per file or more. Regards, Leander ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
I forgot to mention that I'm also using the suggestions mentioned here: http://docs.openstack.org/developer/swift/deployment_guide.html#general-system-tuning On Mon, Jan 14, 2013 at 11:02 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Hello all, I'm trying to upload 200GB of 200KB files to Swift. I'm using 4 clients (each hosted on a different machine) with 10 threads each uploading files using the official python-swiftclient. Each thread is uploading to a separate container. I have 5 storage nodes and 1 proxy node. The nodes are all running with a replication factor of 3. Each node has a quad-core i3 processor, 4GB of RAM and a gigabit network interface. Is there any way I can speed up this process? At the moment it takes about 20 seconds per file or more. Regards, Leander ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
On Mon, Jan 14, 2013 at 11:02 AM, Leander Bessa Beernaert leande...@gmail.commailto:leande...@gmail.com wrote: Hello all, I'm trying to upload 200GB of 200KB files to Swift. I'm using 4 clients (each hosted on a different machine) with 10 threads each uploading files using the official python-swiftclient. Each thread is uploading to a separate container. I have 5 storage nodes and 1 proxy node. The nodes are all running with a replication factor of 3. Each node has a quad-core i3 processor, 4GB of RAM and a gigabit network interface. Is there any way I can speed up this process? At the moment it takes about 20 seconds per file or more. It is very likely the system is starved on IO's. As a temporary workaround you can stop the object-replicator and object-auditor during the import to have less daemons competing for IO's. Some general troubleshooting tips: Use iotop to look for the processes consuming io's Assuming you use XFS: Make sure the filesystem is created with the appropriate inode size as described in the docs. (e.g. mkfs.xfs -i size=1024) Also with lots of files you need quite a bit of memory to cache the inodes into memory. Use the xfs runtime stats to get some indication about the cache: http://xfs.org/index.php/Runtime_Stats xs_dir_lookup and xs_ig_missed will give some indication how much IO's are spend on the inode lookups You can look at slabtop to see how much memory is used by the inode cache. Cheers, Robert ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
By stopping, do you mean halt the service (kill the process) or is it a change in the configuration file? On Mon, Jan 14, 2013 at 1:20 PM, Robert van Leeuwen robert.vanleeu...@spilgames.com wrote: On Mon, Jan 14, 2013 at 11:02 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Hello all, I'm trying to upload 200GB of 200KB files to Swift. I'm using 4 clients (each hosted on a different machine) with 10 threads each uploading files using the official python-swiftclient. Each thread is uploading to a separate container. I have 5 storage nodes and 1 proxy node. The nodes are all running with a replication factor of 3. Each node has a quad-core i3 processor, 4GB of RAM and a gigabit network interface. Is there any way I can speed up this process? At the moment it takes about 20 seconds per file or more. It is very likely the system is starved on IO's. As a temporary workaround you can stop the object-replicator and object-auditor during the import to have less daemons competing for IO's. Some general troubleshooting tips: Use iotop to look for the processes consuming io's Assuming you use XFS: Make sure the filesystem is created with the appropriate inode size as described in the docs. (e.g. mkfs.xfs -i size=1024) Also with lots of files you need quite a bit of memory to cache the inodes into memory. Use the xfs runtime stats to get some indication about the cache: http://xfs.org/index.php/Runtime_Stats xs_dir_lookup and xs_ig_missed will give some indication how much IO's are spend on the inode lookups You can look at slabtop to see how much memory is used by the inode cache. Cheers, Robert ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
By stopping, do you mean halt the service (kill the process) or is it a change in the configuration file? Just halt the service. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
According to the info below, i think the current size is 256 right? If I format the storage partition, will that automatically clear all the contents from the storage or do I need to clean something else as well? Output from xfs_info: meta-data=/dev/sda3 isize=256agcount=4, agsize=13309312 blks Yes, this is the wrong inode size. When you format the disk on one node it will start to sync the removed data back from the other nodes to this machine (as long as the object-replicator is running on the other nodes). Note that this can take a pretty long time. (our nodes with millions of files more then a week to sync) If you want to throw away everything and start with a empty cluster my guess is you would need to stop everything and also remove the account and container databases. I've never done this so I can't tell you for sure if that does not break anything or if it needs any manual intervention to re-create the databases. Cheers, Robert ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
I see. With replication switched off during upload, does inserting into various containers speed up the process or is it irrelevant? On Mon, Jan 14, 2013 at 1:49 PM, Robert van Leeuwen robert.vanleeu...@spilgames.com wrote: According to the info below, i think the current size is 256 right? If I format the storage partition, will that automatically clear all the contents from the storage or do I need to clean something else as well? Output from xfs_info: meta-data=/dev/sda3 isize=256agcount=4, agsize=13309312 blks Yes, this is the wrong inode size. When you format the disk on one node it will start to sync the removed data back from the other nodes to this machine (as long as the object-replicator is running on the other nodes). Note that this can take a pretty long time. (our nodes with millions of files more then a week to sync) If you want to throw away everything and start with a empty cluster my guess is you would need to stop everything and also remove the account and container databases. I've never done this so I can't tell you for sure if that does not break anything or if it needs any manual intervention to re-create the databases. Cheers, Robert ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
I see. With replication switched off during upload, does inserting into various containers speed up the process or is it irrelevant? I'm not sure what's your question but maybe this helps: In short: The replication daemon is walking across your files to check if any files need to be replicated to other nodes (e.g. when a node was broken or a new node is added to the cluster). Because it is scanning your filesystem it eats up io's. That is why it speeds up the system if you turn this off. The number off objects in a container impacts the sqlite databases. Each container is a SQLite database. The SQLite database will get slower with more objects in them. I think the recommendations for spinning disks is max a few million for one container, when using SSD's for the databases you can go quite a bit higher though. Cheers, Robert ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
Allow me to rephrase. I've read somewhere (can't remember where) that it would be faster to upload files if they would be uploaded to separate containeres. This was suggested for a standard swift installation with a certain replication factor. Since I'll be uploading the files with the replicators turned off, does it really matter if I insert a group of them in separate containeres? On Mon, Jan 14, 2013 at 2:27 PM, Robert van Leeuwen robert.vanleeu...@spilgames.com wrote: I see. With replication switched off during upload, does inserting into various containers speed up the process or is it irrelevant? I'm not sure what's your question but maybe this helps: In short: The replication daemon is walking across your files to check if any files need to be replicated to other nodes (e.g. when a node was broken or a new node is added to the cluster). Because it is scanning your filesystem it eats up io's. That is why it speeds up the system if you turn this off. The number off objects in a container impacts the sqlite databases. Each container is a SQLite database. The SQLite database will get slower with more objects in them. I think the recommendations for spinning disks is max a few million for one container, when using SSD's for the databases you can go quite a bit higher though. Cheers, Robert ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
Ok, thanks for all the tips/help. Regards, Leander On Mon, Jan 14, 2013 at 3:21 PM, Robert van Leeuwen robert.vanleeu...@spilgames.com wrote: Allow me to rephrase. I've read somewhere (can't remember where) that it would be faster to upload files if they would be uploaded to separate containeres. This was suggested for a standard swift installation with a certain replication factor. Since I'll be uploading the files with the replicators turned off, does it really matter if I insert a group of them in separate containeres? My guess is this concerns the SQLite database load distribution. So yes, it still matters. Just to be clear: turning replicators off does not matter at ALL when putting files in a healthy cluster. Files will be replicated / put on all required nodes at the moment the put request is done. The put request will only give an OK when there is quorum writing the file (the file is stored on more than half of the required object nodes) The replicator daemons do not have anything to do with this. Cheers, Robert ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
Allow me to rephrase. I've read somewhere (can't remember where) that it would be faster to upload files if they would be uploaded to separate containeres. This was suggested for a standard swift installation with a certain replication factor. Since I'll be uploading the files with the replicators turned off, does it really matter if I insert a group of them in separate containeres? My guess is this concerns the SQLite database load distribution. So yes, it still matters. Just to be clear: turning replicators off does not matter at ALL when putting files in a healthy cluster. Files will be replicated / put on all required nodes at the moment the put request is done. The put request will only give an OK when there is quorum writing the file (the file is stored on more than half of the required object nodes) The replicator daemons do not have anything to do with this. Cheers, Robert ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
Well, I've fixed the node size and disabled the all the replicator and auditor processes. However, it is even slower now than it was before :/. Any suggestions? On Mon, Jan 14, 2013 at 3:23 PM, Leander Bessa Beernaert leande...@gmail.com wrote: Ok, thanks for all the tips/help. Regards, Leander On Mon, Jan 14, 2013 at 3:21 PM, Robert van Leeuwen robert.vanleeu...@spilgames.com wrote: Allow me to rephrase. I've read somewhere (can't remember where) that it would be faster to upload files if they would be uploaded to separate containeres. This was suggested for a standard swift installation with a certain replication factor. Since I'll be uploading the files with the replicators turned off, does it really matter if I insert a group of them in separate containeres? My guess is this concerns the SQLite database load distribution. So yes, it still matters. Just to be clear: turning replicators off does not matter at ALL when putting files in a healthy cluster. Files will be replicated / put on all required nodes at the moment the put request is done. The put request will only give an OK when there is quorum writing the file (the file is stored on more than half of the required object nodes) The replicator daemons do not have anything to do with this. Cheers, Robert ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
Hey Leander, Can you post what performance you are getting? If they are all sharing the same GigE network, you might also check that the links aren't being saturated, as it is pretty easy to saturate pushing 200k files around. -- Chuck On Mon, Jan 14, 2013 at 10:15 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Well, I've fixed the node size and disabled the all the replicator and auditor processes. However, it is even slower now than it was before :/. Any suggestions? On Mon, Jan 14, 2013 at 3:23 PM, Leander Bessa Beernaert leande...@gmail.com wrote: Ok, thanks for all the tips/help. Regards, Leander On Mon, Jan 14, 2013 at 3:21 PM, Robert van Leeuwen robert.vanleeu...@spilgames.com wrote: Allow me to rephrase. I've read somewhere (can't remember where) that it would be faster to upload files if they would be uploaded to separate containeres. This was suggested for a standard swift installation with a certain replication factor. Since I'll be uploading the files with the replicators turned off, does it really matter if I insert a group of them in separate containeres? My guess is this concerns the SQLite database load distribution. So yes, it still matters. Just to be clear: turning replicators off does not matter at ALL when putting files in a healthy cluster. Files will be replicated / put on all required nodes at the moment the put request is done. The put request will only give an OK when there is quorum writing the file (the file is stored on more than half of the required object nodes) The replicator daemons do not have anything to do with this. Cheers, Robert ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
I'm getting around 5-6.5 GB a day of bytes written on Swift. I calculated this by calling swift stat sleep 60s swift stat. I did some calculation based on those values to get to the end result. Currently I'm resetting swift with a node size of 64, since 90% of the files are less than 70KB in size. I think that might help. On Mon, Jan 14, 2013 at 4:34 PM, Chuck Thier cth...@gmail.com wrote: Hey Leander, Can you post what performance you are getting? If they are all sharing the same GigE network, you might also check that the links aren't being saturated, as it is pretty easy to saturate pushing 200k files around. -- Chuck On Mon, Jan 14, 2013 at 10:15 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Well, I've fixed the node size and disabled the all the replicator and auditor processes. However, it is even slower now than it was before :/. Any suggestions? On Mon, Jan 14, 2013 at 3:23 PM, Leander Bessa Beernaert leande...@gmail.com wrote: Ok, thanks for all the tips/help. Regards, Leander On Mon, Jan 14, 2013 at 3:21 PM, Robert van Leeuwen robert.vanleeu...@spilgames.com wrote: Allow me to rephrase. I've read somewhere (can't remember where) that it would be faster to upload files if they would be uploaded to separate containeres. This was suggested for a standard swift installation with a certain replication factor. Since I'll be uploading the files with the replicators turned off, does it really matter if I insert a group of them in separate containeres? My guess is this concerns the SQLite database load distribution. So yes, it still matters. Just to be clear: turning replicators off does not matter at ALL when putting files in a healthy cluster. Files will be replicated / put on all required nodes at the moment the put request is done. The put request will only give an OK when there is quorum writing the file (the file is stored on more than half of the required object nodes) The replicator daemons do not have anything to do with this. Cheers, Robert ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
Using swift stat probably isn't the best way to determine cluster performance, as those stats are updated async, and could be delayed quite a bit as you are heavily loading the cluster. It also might be worthwhile to use a tool like swift-bench to test your cluster to make sure it is properly setup before loading data into the system. -- Chuck On Mon, Jan 14, 2013 at 10:38 AM, Leander Bessa Beernaert leande...@gmail.com wrote: I'm getting around 5-6.5 GB a day of bytes written on Swift. I calculated this by calling swift stat sleep 60s swift stat. I did some calculation based on those values to get to the end result. Currently I'm resetting swift with a node size of 64, since 90% of the files are less than 70KB in size. I think that might help. On Mon, Jan 14, 2013 at 4:34 PM, Chuck Thier cth...@gmail.com wrote: Hey Leander, Can you post what performance you are getting? If they are all sharing the same GigE network, you might also check that the links aren't being saturated, as it is pretty easy to saturate pushing 200k files around. -- Chuck On Mon, Jan 14, 2013 at 10:15 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Well, I've fixed the node size and disabled the all the replicator and auditor processes. However, it is even slower now than it was before :/. Any suggestions? On Mon, Jan 14, 2013 at 3:23 PM, Leander Bessa Beernaert leande...@gmail.com wrote: Ok, thanks for all the tips/help. Regards, Leander On Mon, Jan 14, 2013 at 3:21 PM, Robert van Leeuwen robert.vanleeu...@spilgames.com wrote: Allow me to rephrase. I've read somewhere (can't remember where) that it would be faster to upload files if they would be uploaded to separate containeres. This was suggested for a standard swift installation with a certain replication factor. Since I'll be uploading the files with the replicators turned off, does it really matter if I insert a group of them in separate containeres? My guess is this concerns the SQLite database load distribution. So yes, it still matters. Just to be clear: turning replicators off does not matter at ALL when putting files in a healthy cluster. Files will be replicated / put on all required nodes at the moment the put request is done. The put request will only give an OK when there is quorum writing the file (the file is stored on more than half of the required object nodes) The replicator daemons do not have anything to do with this. Cheers, Robert ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
I'm currently using the swift client to upload files, would you recommend another approach? On Mon, Jan 14, 2013 at 4:43 PM, Chuck Thier cth...@gmail.com wrote: Using swift stat probably isn't the best way to determine cluster performance, as those stats are updated async, and could be delayed quite a bit as you are heavily loading the cluster. It also might be worthwhile to use a tool like swift-bench to test your cluster to make sure it is properly setup before loading data into the system. -- Chuck On Mon, Jan 14, 2013 at 10:38 AM, Leander Bessa Beernaert leande...@gmail.com wrote: I'm getting around 5-6.5 GB a day of bytes written on Swift. I calculated this by calling swift stat sleep 60s swift stat. I did some calculation based on those values to get to the end result. Currently I'm resetting swift with a node size of 64, since 90% of the files are less than 70KB in size. I think that might help. On Mon, Jan 14, 2013 at 4:34 PM, Chuck Thier cth...@gmail.com wrote: Hey Leander, Can you post what performance you are getting? If they are all sharing the same GigE network, you might also check that the links aren't being saturated, as it is pretty easy to saturate pushing 200k files around. -- Chuck On Mon, Jan 14, 2013 at 10:15 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Well, I've fixed the node size and disabled the all the replicator and auditor processes. However, it is even slower now than it was before :/. Any suggestions? On Mon, Jan 14, 2013 at 3:23 PM, Leander Bessa Beernaert leande...@gmail.com wrote: Ok, thanks for all the tips/help. Regards, Leander On Mon, Jan 14, 2013 at 3:21 PM, Robert van Leeuwen robert.vanleeu...@spilgames.com wrote: Allow me to rephrase. I've read somewhere (can't remember where) that it would be faster to upload files if they would be uploaded to separate containeres. This was suggested for a standard swift installation with a certain replication factor. Since I'll be uploading the files with the replicators turned off, does it really matter if I insert a group of them in separate containeres? My guess is this concerns the SQLite database load distribution. So yes, it still matters. Just to be clear: turning replicators off does not matter at ALL when putting files in a healthy cluster. Files will be replicated / put on all required nodes at the moment the put request is done. The put request will only give an OK when there is quorum writing the file (the file is stored on more than half of the required object nodes) The replicator daemons do not have anything to do with this. Cheers, Robert ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
I currently have 4 machines running 10 clients each uploading 1/40th of the data. More than 40 simultaneous clientes starts to severely affect Keystone's ability to handle these operations. On Mon, Jan 14, 2013 at 4:58 PM, Chuck Thier cth...@gmail.com wrote: That should be fine, but it doesn't have any way of reporting stats currently. You could use tools like ifstat to look at how much bandwidth you are using. You can also look at how much cpu the swift tool is using. Depending on how your data is setup, you could run several swift-client processes in parallel until you max either your network or cpu. I would start with one client first, until you max it out, then move on to the next. -- Chuck On Mon, Jan 14, 2013 at 10:45 AM, Leander Bessa Beernaert leande...@gmail.com wrote: I'm currently using the swift client to upload files, would you recommend another approach? On Mon, Jan 14, 2013 at 4:43 PM, Chuck Thier cth...@gmail.com wrote: Using swift stat probably isn't the best way to determine cluster performance, as those stats are updated async, and could be delayed quite a bit as you are heavily loading the cluster. It also might be worthwhile to use a tool like swift-bench to test your cluster to make sure it is properly setup before loading data into the system. -- Chuck On Mon, Jan 14, 2013 at 10:38 AM, Leander Bessa Beernaert leande...@gmail.com wrote: I'm getting around 5-6.5 GB a day of bytes written on Swift. I calculated this by calling swift stat sleep 60s swift stat. I did some calculation based on those values to get to the end result. Currently I'm resetting swift with a node size of 64, since 90% of the files are less than 70KB in size. I think that might help. On Mon, Jan 14, 2013 at 4:34 PM, Chuck Thier cth...@gmail.com wrote: Hey Leander, Can you post what performance you are getting? If they are all sharing the same GigE network, you might also check that the links aren't being saturated, as it is pretty easy to saturate pushing 200k files around. -- Chuck On Mon, Jan 14, 2013 at 10:15 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Well, I've fixed the node size and disabled the all the replicator and auditor processes. However, it is even slower now than it was before :/. Any suggestions? On Mon, Jan 14, 2013 at 3:23 PM, Leander Bessa Beernaert leande...@gmail.com wrote: Ok, thanks for all the tips/help. Regards, Leander On Mon, Jan 14, 2013 at 3:21 PM, Robert van Leeuwen robert.vanleeu...@spilgames.com wrote: Allow me to rephrase. I've read somewhere (can't remember where) that it would be faster to upload files if they would be uploaded to separate containeres. This was suggested for a standard swift installation with a certain replication factor. Since I'll be uploading the files with the replicators turned off, does it really matter if I insert a group of them in separate containeres? My guess is this concerns the SQLite database load distribution. So yes, it still matters. Just to be clear: turning replicators off does not matter at ALL when putting files in a healthy cluster. Files will be replicated / put on all required nodes at the moment the put request is done. The put request will only give an OK when there is quorum writing the file (the file is stored on more than half of the required object nodes) The replicator daemons do not have anything to do with this. Cheers, Robert ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
Also, I'm unable to run the swift-bench with keystone.
I always get this error:
Traceback (most recent call last):
File /usr/bin/swift-bench, line 149, in module
controller.run()
File /usr/lib/python2.7/dist-packages/swift/common/bench.py, line 159,
in run
puts = BenchPUT(self.logger, self.conf, self.names)
File /usr/lib/python2.7/dist-packages/swift/common/bench.py, line 241,
in __init__
Bench.__init__(self, logger, conf, names)
File /usr/lib/python2.7/dist-packages/swift/common/bench.py, line 55,
in __init__
auth_version=self.auth_version)
File /usr/lib/python2.7/dist-packages/swiftclient/client.py, line 281,
in get_auth
if (kwargs['os_options'].get('object_storage_url') and
KeyError: 'os_options'
Here's my config file:
[bench]
auth = http://192.168.111.202:5000/v2.0
user = [tenant]:[user]
key = [key]
auth_version = 2
log-level = INFO
timeout = 10
put_concurrency = 10
get_concurrency = 10
lower_object_size = 20
upper_object_size = 2
num_objects = 1000
num_gets = 1
num_containers = 20
delete = yes
On Mon, Jan 14, 2013 at 5:01 PM, Leander Bessa Beernaert
leande...@gmail.com wrote:
I currently have 4 machines running 10 clients each uploading 1/40th of
the data. More than 40 simultaneous clientes starts to severely affect
Keystone's ability to handle these operations.
On Mon, Jan 14, 2013 at 4:58 PM, Chuck Thier cth...@gmail.com wrote:
That should be fine, but it doesn't have any way of reporting stats
currently. You could use tools like ifstat to look at how much
bandwidth you are using. You can also look at how much cpu the swift
tool is using. Depending on how your data is setup, you could run
several swift-client processes in parallel until you max either your
network or cpu. I would start with one client first, until you max it
out, then move on to the next.
--
Chuck
On Mon, Jan 14, 2013 at 10:45 AM, Leander Bessa Beernaert
leande...@gmail.com wrote:
I'm currently using the swift client to upload files, would you
recommend
another approach?
On Mon, Jan 14, 2013 at 4:43 PM, Chuck Thier cth...@gmail.com wrote:
Using swift stat probably isn't the best way to determine cluster
performance, as those stats are updated async, and could be delayed
quite a bit as you are heavily loading the cluster. It also might be
worthwhile to use a tool like swift-bench to test your cluster to make
sure it is properly setup before loading data into the system.
--
Chuck
On Mon, Jan 14, 2013 at 10:38 AM, Leander Bessa Beernaert
leande...@gmail.com wrote:
I'm getting around 5-6.5 GB a day of bytes written on Swift. I
calculated
this by calling swift stat sleep 60s swift stat. I did some
calculation based on those values to get to the end result.
Currently I'm resetting swift with a node size of 64, since 90% of
the
files
are less than 70KB in size. I think that might help.
On Mon, Jan 14, 2013 at 4:34 PM, Chuck Thier cth...@gmail.com
wrote:
Hey Leander,
Can you post what performance you are getting? If they are all
sharing the same GigE network, you might also check that the links
aren't being saturated, as it is pretty easy to saturate pushing
200k
files around.
--
Chuck
On Mon, Jan 14, 2013 at 10:15 AM, Leander Bessa Beernaert
leande...@gmail.com wrote:
Well, I've fixed the node size and disabled the all the replicator
and
auditor processes. However, it is even slower now than it was
before
:/.
Any
suggestions?
On Mon, Jan 14, 2013 at 3:23 PM, Leander Bessa Beernaert
leande...@gmail.com wrote:
Ok, thanks for all the tips/help.
Regards,
Leander
On Mon, Jan 14, 2013 at 3:21 PM, Robert van Leeuwen
robert.vanleeu...@spilgames.com wrote:
Allow me to rephrase.
I've read somewhere (can't remember where) that it would be
faster
to
upload files if they would be uploaded to separate
containeres.
This was suggested for a standard swift installation with a
certain
replication factor.
Since I'll be uploading the files with the replicators turned
off,
does
it really matter if I insert a group of them in separate
containeres?
My guess is this concerns the SQLite database load distribution.
So yes, it still matters.
Just to be clear: turning replicators off does not matter at ALL
when
putting files in a healthy cluster.
Files will be replicated / put on all required nodes at the
moment
the
put request is done.
The put request will only give an OK when there is quorum
writing
the
file (the file is stored on more than half of the required
object
nodes)
The replicator daemons do not have anything to do with this.
Cheers,
Robert
___
Mailing list:
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
That should be fine, but it doesn't have any way of reporting stats currently. You could use tools like ifstat to look at how much bandwidth you are using. You can also look at how much cpu the swift tool is using. Depending on how your data is setup, you could run several swift-client processes in parallel until you max either your network or cpu. I would start with one client first, until you max it out, then move on to the next. -- Chuck On Mon, Jan 14, 2013 at 10:45 AM, Leander Bessa Beernaert leande...@gmail.com wrote: I'm currently using the swift client to upload files, would you recommend another approach? On Mon, Jan 14, 2013 at 4:43 PM, Chuck Thier cth...@gmail.com wrote: Using swift stat probably isn't the best way to determine cluster performance, as those stats are updated async, and could be delayed quite a bit as you are heavily loading the cluster. It also might be worthwhile to use a tool like swift-bench to test your cluster to make sure it is properly setup before loading data into the system. -- Chuck On Mon, Jan 14, 2013 at 10:38 AM, Leander Bessa Beernaert leande...@gmail.com wrote: I'm getting around 5-6.5 GB a day of bytes written on Swift. I calculated this by calling swift stat sleep 60s swift stat. I did some calculation based on those values to get to the end result. Currently I'm resetting swift with a node size of 64, since 90% of the files are less than 70KB in size. I think that might help. On Mon, Jan 14, 2013 at 4:34 PM, Chuck Thier cth...@gmail.com wrote: Hey Leander, Can you post what performance you are getting? If they are all sharing the same GigE network, you might also check that the links aren't being saturated, as it is pretty easy to saturate pushing 200k files around. -- Chuck On Mon, Jan 14, 2013 at 10:15 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Well, I've fixed the node size and disabled the all the replicator and auditor processes. However, it is even slower now than it was before :/. Any suggestions? On Mon, Jan 14, 2013 at 3:23 PM, Leander Bessa Beernaert leande...@gmail.com wrote: Ok, thanks for all the tips/help. Regards, Leander On Mon, Jan 14, 2013 at 3:21 PM, Robert van Leeuwen robert.vanleeu...@spilgames.com wrote: Allow me to rephrase. I've read somewhere (can't remember where) that it would be faster to upload files if they would be uploaded to separate containeres. This was suggested for a standard swift installation with a certain replication factor. Since I'll be uploading the files with the replicators turned off, does it really matter if I insert a group of them in separate containeres? My guess is this concerns the SQLite database load distribution. So yes, it still matters. Just to be clear: turning replicators off does not matter at ALL when putting files in a healthy cluster. Files will be replicated / put on all required nodes at the moment the put request is done. The put request will only give an OK when there is quorum writing the file (the file is stored on more than half of the required object nodes) The replicator daemons do not have anything to do with this. Cheers, Robert ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
On Mon, Jan 14, 2013 at 11:01 AM, Leander Bessa Beernaert leande...@gmail.com wrote: I currently have 4 machines running 10 clients each uploading 1/40th of the data. More than 40 simultaneous clientes starts to severely affect Keystone's ability to handle these operations. You might also double check that you are running a very recent version of keystone that includes the update to use the swift memcache servers and the correct configuration. This will cache tokens and prevent having to make a call to keystone for every single request. If that is an issue, that is likely causing a lot of latency to each request. -- Chuck ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
On Mon, Jan 14, 2013 at 11:03 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Also, I'm unable to run the swift-bench with keystone. Hrm... That was supposed to be fixed with this bug: https://bugs.launchpad.net/swift/+bug/1011727 My keystone dev instance isn't working at the moment, but I'll see if I can get one of the team to take a look at it. -- Chuck ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
I'm using the ubuntu 12.04 packages of the folsom repository by the way. On Mon, Jan 14, 2013 at 5:18 PM, Chuck Thier cth...@gmail.com wrote: On Mon, Jan 14, 2013 at 11:03 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Also, I'm unable to run the swift-bench with keystone. Hrm... That was supposed to be fixed with this bug: https://bugs.launchpad.net/swift/+bug/1011727 My keystone dev instance isn't working at the moment, but I'll see if I can get one of the team to take a look at it. -- Chuck ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
Are you by any chance referring to this topic https://lists.launchpad.net/openstack/msg08639.html regarding the keystone token cache? If so I've already added the configuration line and have not noticed any speedup :/ On Mon, Jan 14, 2013 at 5:19 PM, Leander Bessa Beernaert leande...@gmail.com wrote: I'm using the ubuntu 12.04 packages of the folsom repository by the way. On Mon, Jan 14, 2013 at 5:18 PM, Chuck Thier cth...@gmail.com wrote: On Mon, Jan 14, 2013 at 11:03 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Also, I'm unable to run the swift-bench with keystone. Hrm... That was supposed to be fixed with this bug: https://bugs.launchpad.net/swift/+bug/1011727 My keystone dev instance isn't working at the moment, but I'll see if I can get one of the team to take a look at it. -- Chuck ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
If memcache is being utilized by your keystone middleware, you should see keystone attaching to it on the first incoming request, e.g.: keystoneclient.middleware.auth_token [INFO]: Using Keystone memcache for caching token You may also want to use auth_token from keystoneclient = v0.2.0 if you're not already (instead of from keystone itself). -Dolph On Mon, Jan 14, 2013 at 11:43 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Are you by any chance referring to this topic https://lists.launchpad.net/openstack/msg08639.html regarding the keystone token cache? If so I've already added the configuration line and have not noticed any speedup :/ On Mon, Jan 14, 2013 at 5:19 PM, Leander Bessa Beernaert leande...@gmail.com wrote: I'm using the ubuntu 12.04 packages of the folsom repository by the way. On Mon, Jan 14, 2013 at 5:18 PM, Chuck Thier cth...@gmail.com wrote: On Mon, Jan 14, 2013 at 11:03 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Also, I'm unable to run the swift-bench with keystone. Hrm... That was supposed to be fixed with this bug: https://bugs.launchpad.net/swift/+bug/1011727 My keystone dev instance isn't working at the moment, but I'll see if I can get one of the team to take a look at it. -- Chuck ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
You would have to look at the proxy log to see if a request is being made. The results from the swift command line are just the calls that the client makes. The server still haves to validate the token on every request. -- Chuck On Mon, Jan 14, 2013 at 12:37 PM, Leander Bessa Beernaert leande...@gmail.com wrote: Below is an output from Swift stat, since I don't see any requests to keystone, I'm assuming that memcache is being used right? REQ: curl -i http://192.168.111.215:8080/v1/AUTH_[ID] -X HEAD -H X-Auth-Token: [TOKEN] DEBUG:swiftclient:REQ: curl -i http://192.168.111.215:8080/v1/AUTH_[ID] -X HEAD -H X-Auth-Token: [TOKEN] RESP STATUS: 204 DEBUG:swiftclient:RESP STATUS: 204 Account: AUTH_[ID] Containers: 44 Objects: 4818 Bytes: 112284450 Accept-Ranges: bytes X-Timestamp: 1358184925.20885 X-Trans-Id: tx8cffb469c9c542be830db10a2b90d901 On Mon, Jan 14, 2013 at 6:31 PM, Dolph Mathews dolph.math...@gmail.com wrote: If memcache is being utilized by your keystone middleware, you should see keystone attaching to it on the first incoming request, e.g.: keystoneclient.middleware.auth_token [INFO]: Using Keystone memcache for caching token You may also want to use auth_token from keystoneclient = v0.2.0 if you're not already (instead of from keystone itself). -Dolph On Mon, Jan 14, 2013 at 11:43 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Are you by any chance referring to this topic https://lists.launchpad.net/openstack/msg08639.html regarding the keystone token cache? If so I've already added the configuration line and have not noticed any speedup :/ On Mon, Jan 14, 2013 at 5:19 PM, Leander Bessa Beernaert leande...@gmail.com wrote: I'm using the ubuntu 12.04 packages of the folsom repository by the way. On Mon, Jan 14, 2013 at 5:18 PM, Chuck Thier cth...@gmail.com wrote: On Mon, Jan 14, 2013 at 11:03 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Also, I'm unable to run the swift-bench with keystone. Hrm... That was supposed to be fixed with this bug: https://bugs.launchpad.net/swift/+bug/1011727 My keystone dev instance isn't working at the moment, but I'll see if I can get one of the team to take a look at it. -- Chuck ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
Neither keystone nor swift proxy are producing any logs. I'm not sure what to do :S On Mon, Jan 14, 2013 at 6:50 PM, Chuck Thier cth...@gmail.com wrote: You would have to look at the proxy log to see if a request is being made. The results from the swift command line are just the calls that the client makes. The server still haves to validate the token on every request. -- Chuck On Mon, Jan 14, 2013 at 12:37 PM, Leander Bessa Beernaert leande...@gmail.com wrote: Below is an output from Swift stat, since I don't see any requests to keystone, I'm assuming that memcache is being used right? REQ: curl -i http://192.168.111.215:8080/v1/AUTH_[ID] -X HEAD -H X-Auth-Token: [TOKEN] DEBUG:swiftclient:REQ: curl -i http://192.168.111.215:8080/v1/AUTH_[ID]-X HEAD -H X-Auth-Token: [TOKEN] RESP STATUS: 204 DEBUG:swiftclient:RESP STATUS: 204 Account: AUTH_[ID] Containers: 44 Objects: 4818 Bytes: 112284450 Accept-Ranges: bytes X-Timestamp: 1358184925.20885 X-Trans-Id: tx8cffb469c9c542be830db10a2b90d901 On Mon, Jan 14, 2013 at 6:31 PM, Dolph Mathews dolph.math...@gmail.com wrote: If memcache is being utilized by your keystone middleware, you should see keystone attaching to it on the first incoming request, e.g.: keystoneclient.middleware.auth_token [INFO]: Using Keystone memcache for caching token You may also want to use auth_token from keystoneclient = v0.2.0 if you're not already (instead of from keystone itself). -Dolph On Mon, Jan 14, 2013 at 11:43 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Are you by any chance referring to this topic https://lists.launchpad.net/openstack/msg08639.html regarding the keystone token cache? If so I've already added the configuration line and have not noticed any speedup :/ On Mon, Jan 14, 2013 at 5:19 PM, Leander Bessa Beernaert leande...@gmail.com wrote: I'm using the ubuntu 12.04 packages of the folsom repository by the way. On Mon, Jan 14, 2013 at 5:18 PM, Chuck Thier cth...@gmail.com wrote: On Mon, Jan 14, 2013 at 11:03 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Also, I'm unable to run the swift-bench with keystone. Hrm... That was supposed to be fixed with this bug: https://bugs.launchpad.net/swift/+bug/1011727 My keystone dev instance isn't working at the moment, but I'll see if I can get one of the team to take a look at it. -- Chuck ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift
Rather than ping-ponging emails back and forth on this list, it would be easier if you could hop on to the #openstack-swift IRC channel on freenode to discuss further. -- Chuck On Mon, Jan 14, 2013 at 1:00 PM, Leander Bessa Beernaert leande...@gmail.com wrote: Neither keystone nor swift proxy are producing any logs. I'm not sure what to do :S On Mon, Jan 14, 2013 at 6:50 PM, Chuck Thier cth...@gmail.com wrote: You would have to look at the proxy log to see if a request is being made. The results from the swift command line are just the calls that the client makes. The server still haves to validate the token on every request. -- Chuck On Mon, Jan 14, 2013 at 12:37 PM, Leander Bessa Beernaert leande...@gmail.com wrote: Below is an output from Swift stat, since I don't see any requests to keystone, I'm assuming that memcache is being used right? REQ: curl -i http://192.168.111.215:8080/v1/AUTH_[ID] -X HEAD -H X-Auth-Token: [TOKEN] DEBUG:swiftclient:REQ: curl -i http://192.168.111.215:8080/v1/AUTH_[ID] -X HEAD -H X-Auth-Token: [TOKEN] RESP STATUS: 204 DEBUG:swiftclient:RESP STATUS: 204 Account: AUTH_[ID] Containers: 44 Objects: 4818 Bytes: 112284450 Accept-Ranges: bytes X-Timestamp: 1358184925.20885 X-Trans-Id: tx8cffb469c9c542be830db10a2b90d901 On Mon, Jan 14, 2013 at 6:31 PM, Dolph Mathews dolph.math...@gmail.com wrote: If memcache is being utilized by your keystone middleware, you should see keystone attaching to it on the first incoming request, e.g.: keystoneclient.middleware.auth_token [INFO]: Using Keystone memcache for caching token You may also want to use auth_token from keystoneclient = v0.2.0 if you're not already (instead of from keystone itself). -Dolph On Mon, Jan 14, 2013 at 11:43 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Are you by any chance referring to this topic https://lists.launchpad.net/openstack/msg08639.html regarding the keystone token cache? If so I've already added the configuration line and have not noticed any speedup :/ On Mon, Jan 14, 2013 at 5:19 PM, Leander Bessa Beernaert leande...@gmail.com wrote: I'm using the ubuntu 12.04 packages of the folsom repository by the way. On Mon, Jan 14, 2013 at 5:18 PM, Chuck Thier cth...@gmail.com wrote: On Mon, Jan 14, 2013 at 11:03 AM, Leander Bessa Beernaert leande...@gmail.com wrote: Also, I'm unable to run the swift-bench with keystone. Hrm... That was supposed to be fixed with this bug: https://bugs.launchpad.net/swift/+bug/1011727 My keystone dev instance isn't working at the moment, but I'll see if I can get one of the team to take a look at it. -- Chuck ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
