Author: tomaz
Date: Sat May 26 04:10:23 2012
New Revision: 1342849
URL: http://svn.apache.org/viewvc?rev=1342849&view=rev
Log:
Update "Using Libcloud in async and multi threaded environments" page and add
"Efficiently download multiple files using gevent" example.
Modified:
libcloud/site/trunk/content/docs/storage-examples.mdtext
libcloud/site/trunk/content/docs/using-libcloud-in-multi-threaded-and-async-environments.mdtext
Modified: libcloud/site/trunk/content/docs/storage-examples.mdtext
URL:
http://svn.apache.org/viewvc/libcloud/site/trunk/content/docs/storage-examples.mdtext?rev=1342849&r1=1342848&r2=1342849&view=diff
==============================================================================
--- libcloud/site/trunk/content/docs/storage-examples.mdtext (original)
+++ libcloud/site/trunk/content/docs/storage-examples.mdtext Sat May 26
04:10:23 2012
@@ -3,6 +3,7 @@ title: Storage -> Examples
## Examples
* [Create a backup of a directory and directly stream it to
CloudFiles](#example-1-streaming-upload)
+* [Efficiently download multiple files using
gevent](#efficiently-download-multiple-files-using-gevent)
<h3 id="example-1-streaming-upload">Create a backup of a directory and
directly stream it to CloudFiles</h3>
@@ -39,3 +40,44 @@ title: Storage -> Examples
return_code = pipe.poll()
print 'Upload complete, transferred: %s KB' % ((obj.size / 1024))
+
+<h3 id="efficiently-download-multiple-files-using-gevent">Efficiently download
multiple files using gevent</h3>
+
+ ::python
+ import os.path
+
+ from gevent import monkey
+ from gevent.pool import Pool
+ monkey.patch_all()
+
+ from libcloud.storage.providers import get_driver
+ from libcloud.storage.types import Provider
+
+ USERNAME = 'username'
+ API_KEY = 'api key'
+
+ cls = get_driver(Provider.CLOUDFILES_US)
+ driver = cls(USERNAME, API_KEY)
+
+ def download_obj(container, obj):
+ driver = cls(USERNAME, API_KEY)
+ obj = driver.get_object(container_name=container.name,
+ object_name=obj.name)
+ filename = os.path.basename(obj.name)
+ path = os.path.join(os.path.expanduser('~/Downloads'), filename)
+ print 'Downloading: %s to %s' % (obj.name, path)
+ obj.download(destination_path=path)
+
+ containers = driver.list_containers()
+
+ jobs = []
+ pool = Pool(20)
+
+ for index, container in enumerate(containers):
+ objects = container.list_objects()
+
+ for obj in objects:
+ pool.spawn(download_obj, container, obj)
+
+ pool.join()
+ print 'Done'
Modified:
libcloud/site/trunk/content/docs/using-libcloud-in-multi-threaded-and-async-environments.mdtext
URL:
http://svn.apache.org/viewvc/libcloud/site/trunk/content/docs/using-libcloud-in-multi-threaded-and-async-environments.mdtext?rev=1342849&r1=1342848&r2=1342849&view=diff
==============================================================================
---
libcloud/site/trunk/content/docs/using-libcloud-in-multi-threaded-and-async-environments.mdtext
(original)
+++
libcloud/site/trunk/content/docs/using-libcloud-in-multi-threaded-and-async-environments.mdtext
Sat May 26 04:10:23 2012
@@ -26,15 +26,21 @@ easiest solution is to create a new driv
### Using Libcloud with gevent ###
gevent has an ability to monkey patch and replace functions in the Python
`socket`
-and `time` module with its own functions which don't block.
+`urllib2`, `httplib` and `time` module with its own functions which don't
block.
-The only thing you need to use to make Libcloud efficiently work with gevent is
-to enable this monkey-patching:
+You need to do two things when you want to use Libcloud with gevent:
+
+1. Enable monkey patching
::python
from gevent import monkey
monkey.patch_all()
+2. Create a separate driver instance for each Greenlet. This is necessary
+because a driver instance reuses the same `Connection` class.
+
+For example see [Efficiently download multiple files using
gevent](/docs/storage-examples.html#efficiently-download-multiple-files-using-gevent).
+
### Using Libcloud with Twisted ###
Libcloud has no Twisted support included in the core which means you need
