Author: buildbot
Date: Sat May 26 04:10:32 2012
New Revision: 819146
Log:
Staging update by buildbot for libcloud
Modified:
websites/staging/libcloud/trunk/content/ (props changed)
websites/staging/libcloud/trunk/content/docs/storage-examples.html
websites/staging/libcloud/trunk/content/docs/using-libcloud-in-multi-threaded-and-async-environments.html
Propchange: websites/staging/libcloud/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Sat May 26 04:10:32 2012
@@ -1 +1 @@
-1342848
+1342849
Modified: websites/staging/libcloud/trunk/content/docs/storage-examples.html
==============================================================================
--- websites/staging/libcloud/trunk/content/docs/storage-examples.html
(original)
+++ websites/staging/libcloud/trunk/content/docs/storage-examples.html Sat May
26 04:10:32 2012
@@ -96,6 +96,7 @@
<h2 id="examples">Examples</h2>
<ul>
<li><a href="#example-1-streaming-upload">Create a backup of a directory and
directly stream it to CloudFiles</a></li>
+<li><a href="#efficiently-download-multiple-files-using-gevent">Efficiently
download multiple files using gevent</a></li>
</ul>
<h3 id="example-1-streaming-upload">Create a backup of a directory and
directly stream it to CloudFiles</h3>
@@ -132,6 +133,48 @@
<span class="k">print</span> <span class="s">'Upload complete,
transferred: </span><span class="si">%s</span><span class="s"> KB'</span>
<span class="o">%</span> <span class="p">((</span><span
class="n">obj</span><span class="o">.</span><span class="n">size</span> <span
class="o">/</span> <span class="mi">1024</span><span class="p">))</span>
</pre></div>
+
+
+<h3 id="efficiently-download-multiple-files-using-gevent">Efficiently download
multiple files using gevent</h3>
+
+<div class="codehilite"><pre><span class="kn">import</span> <span
class="nn">os.path</span>
+
+<span class="kn">from</span> <span class="nn">gevent</span> <span
class="kn">import</span> <span class="n">monkey</span>
+<span class="kn">from</span> <span class="nn">gevent.pool</span> <span
class="kn">import</span> <span class="n">Pool</span>
+<span class="n">monkey</span><span class="o">.</span><span
class="n">patch_all</span><span class="p">()</span>
+
+<span class="kn">from</span> <span
class="nn">libcloud.storage.providers</span> <span class="kn">import</span>
<span class="n">get_driver</span>
+<span class="kn">from</span> <span class="nn">libcloud.storage.types</span>
<span class="kn">import</span> <span class="n">Provider</span>
+
+<span class="n">USERNAME</span> <span class="o">=</span> <span
class="s">'username'</span>
+<span class="n">API_KEY</span> <span class="o">=</span> <span
class="s">'api key'</span>
+
+<span class="n">cls</span> <span class="o">=</span> <span
class="n">get_driver</span><span class="p">(</span><span
class="n">Provider</span><span class="o">.</span><span
class="n">CLOUDFILES_US</span><span class="p">)</span>
+<span class="n">driver</span> <span class="o">=</span> <span
class="n">cls</span><span class="p">(</span><span
class="n">USERNAME</span><span class="p">,</span> <span
class="n">API_KEY</span><span class="p">)</span>
+
+<span class="k">def</span> <span class="nf">download_obj</span><span
class="p">(</span><span class="n">container</span><span class="p">,</span>
<span class="n">obj</span><span class="p">):</span>
+ <span class="n">driver</span> <span class="o">=</span> <span
class="n">cls</span><span class="p">(</span><span
class="n">USERNAME</span><span class="p">,</span> <span
class="n">API_KEY</span><span class="p">)</span>
+ <span class="n">obj</span> <span class="o">=</span> <span
class="n">driver</span><span class="o">.</span><span
class="n">get_object</span><span class="p">(</span><span
class="n">container_name</span><span class="o">=</span><span
class="n">container</span><span class="o">.</span><span
class="n">name</span><span class="p">,</span>
+ <span class="n">object_name</span><span
class="o">=</span><span class="n">obj</span><span class="o">.</span><span
class="n">name</span><span class="p">)</span>
+ <span class="n">filename</span> <span class="o">=</span> <span
class="n">os</span><span class="o">.</span><span class="n">path</span><span
class="o">.</span><span class="n">basename</span><span class="p">(</span><span
class="n">obj</span><span class="o">.</span><span class="n">name</span><span
class="p">)</span>
+ <span class="n">path</span> <span class="o">=</span> <span
class="n">os</span><span class="o">.</span><span class="n">path</span><span
class="o">.</span><span class="n">join</span><span class="p">(</span><span
class="n">os</span><span class="o">.</span><span class="n">path</span><span
class="o">.</span><span class="n">expanduser</span><span
class="p">(</span><span class="s">'~/Downloads'</span><span
class="p">),</span> <span class="n">filename</span><span class="p">)</span>
+ <span class="k">print</span> <span class="s">'Downloading:
</span><span class="si">%s</span><span class="s"> to </span><span
class="si">%s</span><span class="s">'</span> <span class="o">%</span> <span
class="p">(</span><span class="n">obj</span><span class="o">.</span><span
class="n">name</span><span class="p">,</span> <span class="n">path</span><span
class="p">)</span>
+ <span class="n">obj</span><span class="o">.</span><span
class="n">download</span><span class="p">(</span><span
class="n">destination_path</span><span class="o">=</span><span
class="n">path</span><span class="p">)</span>
+
+<span class="n">containers</span> <span class="o">=</span> <span
class="n">driver</span><span class="o">.</span><span
class="n">list_containers</span><span class="p">()</span>
+
+<span class="n">jobs</span> <span class="o">=</span> <span class="p">[]</span>
+<span class="n">pool</span> <span class="o">=</span> <span
class="n">Pool</span><span class="p">(</span><span class="mi">20</span><span
class="p">)</span>
+
+<span class="k">for</span> <span class="n">index</span><span
class="p">,</span> <span class="n">container</span> <span class="ow">in</span>
<span class="nb">enumerate</span><span class="p">(</span><span
class="n">containers</span><span class="p">):</span>
+ <span class="n">objects</span> <span class="o">=</span> <span
class="n">container</span><span class="o">.</span><span
class="n">list_objects</span><span class="p">()</span>
+
+ <span class="k">for</span> <span class="n">obj</span> <span
class="ow">in</span> <span class="n">objects</span><span class="p">:</span>
+ <span class="n">pool</span><span class="o">.</span><span
class="n">spawn</span><span class="p">(</span><span
class="n">download_obj</span><span class="p">,</span> <span
class="n">container</span><span class="p">,</span> <span
class="n">obj</span><span class="p">)</span>
+
+<span class="n">pool</span><span class="o">.</span><span
class="n">join</span><span class="p">()</span>
+<span class="k">print</span> <span class="s">'Done'</span>
+</pre></div>
</div>
Modified:
websites/staging/libcloud/trunk/content/docs/using-libcloud-in-multi-threaded-and-async-environments.html
==============================================================================
---
websites/staging/libcloud/trunk/content/docs/using-libcloud-in-multi-threaded-and-async-environments.html
(original)
+++
websites/staging/libcloud/trunk/content/docs/using-libcloud-in-multi-threaded-and-async-environments.html
Sat May 26 04:10:32 2012
@@ -111,14 +111,21 @@ you don't want to deal with complex (and
easiest solution is to create a new driver instance inside each thread.</p>
<h3 id="using-libcloud-with-gevent">Using Libcloud with gevent</h3>
<p>gevent has an ability to monkey patch and replace functions in the Python
<code>socket</code>
-and <code>time</code> module with its own functions which don't block.</p>
-<p>The only thing you need to use to make Libcloud efficiently work with
gevent is
-to enable this monkey-patching:</p>
-<div class="codehilite"><pre><span class="kn">from</span> <span
class="nn">gevent</span> <span class="kn">import</span> <span
class="n">monkey</span>
-<span class="n">monkey</span><span class="o">.</span><span
class="n">patch_all</span><span class="p">()</span>
-</pre></div>
-
-
+<code>urllib2</code>, <code>httplib</code> and <code>time</code> module with
its own functions which don't block.</p>
+<p>You need to do two things when you want to use Libcloud with gevent:</p>
+<ol>
+<li>
+<p>Enable monkey patching</p>
+<p>::python
+from gevent import monkey
+monkey.patch_all()</p>
+</li>
+<li>
+<p>Create a separate driver instance for each Greenlet. This is necessary
+because a driver instance reuses the same <code>Connection</code> class.</p>
+</li>
+</ol>
+<p>For example see <a
href="/docs/storage-examples.html#efficiently-download-multiple-files-using-gevent">Efficiently
download multiple files using gevent</a>.</p>
<h3 id="using-libcloud-with-twisted">Using Libcloud with Twisted</h3>
<p>Libcloud has no Twisted support included in the core which means you need
to be careful when you use it with Twisted and some other async frameworks.</p>
