Modified: libcloud/site/trunk/generated/blog/archives/2015/11/index.html URL: http://svn.apache.org/viewvc/libcloud/site/trunk/generated/blog/archives/2015/11/index.html?rev=1797582&r1=1797581&r2=1797582&view=diff ============================================================================== --- libcloud/site/trunk/generated/blog/archives/2015/11/index.html (original) +++ libcloud/site/trunk/generated/blog/archives/2015/11/index.html Sun Jun 4 14:37:45 2017 @@ -51,7 +51,7 @@ - <link href='/assets/global-eb5319c938d664847c83b2674bfd22c2.css' rel='stylesheet' type='text/css' /> + <link href='/assets/global-57b0092d353d0369071e1021782bb38d.css' rel='stylesheet' type='text/css' /> </head> @@ -193,7 +193,7 @@ participation and contributions.</p> </div><!-- /.container --> <!-- JavaScript --> - <script src='/assets/global-c6949aebbeb9952ae7967e9bc1aedfbc.js' type='text/javascript'></script> + <script src='/assets/global-14c8e0124e6d563413005cb909342e2a.js' type='text/javascript'></script>
Modified: libcloud/site/trunk/generated/blog/archives/2015/12/index.html URL: http://svn.apache.org/viewvc/libcloud/site/trunk/generated/blog/archives/2015/12/index.html?rev=1797582&r1=1797581&r2=1797582&view=diff ============================================================================== --- libcloud/site/trunk/generated/blog/archives/2015/12/index.html (original) +++ libcloud/site/trunk/generated/blog/archives/2015/12/index.html Sun Jun 4 14:37:45 2017 @@ -51,7 +51,7 @@ - <link href='/assets/global-eb5319c938d664847c83b2674bfd22c2.css' rel='stylesheet' type='text/css' /> + <link href='/assets/global-57b0092d353d0369071e1021782bb38d.css' rel='stylesheet' type='text/css' /> </head> @@ -211,7 +211,7 @@ list of people who contributed to this r </div><!-- /.container --> <!-- JavaScript --> - <script src='/assets/global-c6949aebbeb9952ae7967e9bc1aedfbc.js' type='text/javascript'></script> + <script src='/assets/global-14c8e0124e6d563413005cb909342e2a.js' type='text/javascript'></script> Modified: libcloud/site/trunk/generated/blog/archives/2016/01/index.html URL: http://svn.apache.org/viewvc/libcloud/site/trunk/generated/blog/archives/2016/01/index.html?rev=1797582&r1=1797581&r2=1797582&view=diff ============================================================================== --- libcloud/site/trunk/generated/blog/archives/2016/01/index.html (original) +++ libcloud/site/trunk/generated/blog/archives/2016/01/index.html Sun Jun 4 14:37:45 2017 @@ -51,7 +51,7 @@ - <link href='/assets/global-eb5319c938d664847c83b2674bfd22c2.css' rel='stylesheet' type='text/css' /> + <link href='/assets/global-57b0092d353d0369071e1021782bb38d.css' rel='stylesheet' type='text/css' /> </head> @@ -566,7 +566,7 @@ and compatibility reasons. SSL v3.0 is c </div><!-- /.container --> <!-- JavaScript --> - <script src='/assets/global-c6949aebbeb9952ae7967e9bc1aedfbc.js' type='text/javascript'></script> + <script src='/assets/global-14c8e0124e6d563413005cb909342e2a.js' type='text/javascript'></script> Modified: libcloud/site/trunk/generated/blog/archives/2016/02/index.html URL: http://svn.apache.org/viewvc/libcloud/site/trunk/generated/blog/archives/2016/02/index.html?rev=1797582&r1=1797581&r2=1797582&view=diff ============================================================================== --- libcloud/site/trunk/generated/blog/archives/2016/02/index.html (original) +++ libcloud/site/trunk/generated/blog/archives/2016/02/index.html Sun Jun 4 14:37:45 2017 @@ -51,7 +51,7 @@ - <link href='/assets/global-eb5319c938d664847c83b2674bfd22c2.css' rel='stylesheet' type='text/css' /> + <link href='/assets/global-57b0092d353d0369071e1021782bb38d.css' rel='stylesheet' type='text/css' /> </head> @@ -364,7 +364,7 @@ swap this out for any of the Amazon publ </div><!-- /.container --> <!-- JavaScript --> - <script src='/assets/global-c6949aebbeb9952ae7967e9bc1aedfbc.js' type='text/javascript'></script> + <script src='/assets/global-14c8e0124e6d563413005cb909342e2a.js' type='text/javascript'></script> Modified: libcloud/site/trunk/generated/blog/archives/2016/04/index.html URL: http://svn.apache.org/viewvc/libcloud/site/trunk/generated/blog/archives/2016/04/index.html?rev=1797582&r1=1797581&r2=1797582&view=diff ============================================================================== --- libcloud/site/trunk/generated/blog/archives/2016/04/index.html (original) +++ libcloud/site/trunk/generated/blog/archives/2016/04/index.html Sun Jun 4 14:37:45 2017 @@ -51,7 +51,7 @@ - <link href='/assets/global-eb5319c938d664847c83b2674bfd22c2.css' rel='stylesheet' type='text/css' /> + <link href='/assets/global-57b0092d353d0369071e1021782bb38d.css' rel='stylesheet' type='text/css' /> </head> @@ -504,7 +504,7 @@ that there are no regression bugs with t </div><!-- /.container --> <!-- JavaScript --> - <script src='/assets/global-c6949aebbeb9952ae7967e9bc1aedfbc.js' type='text/javascript'></script> + <script src='/assets/global-14c8e0124e6d563413005cb909342e2a.js' type='text/javascript'></script> Modified: libcloud/site/trunk/generated/blog/archives/2016/06/index.html URL: http://svn.apache.org/viewvc/libcloud/site/trunk/generated/blog/archives/2016/06/index.html?rev=1797582&r1=1797581&r2=1797582&view=diff ============================================================================== --- libcloud/site/trunk/generated/blog/archives/2016/06/index.html (original) +++ libcloud/site/trunk/generated/blog/archives/2016/06/index.html Sun Jun 4 14:37:45 2017 @@ -51,7 +51,7 @@ - <link href='/assets/global-eb5319c938d664847c83b2674bfd22c2.css' rel='stylesheet' type='text/css' /> + <link href='/assets/global-57b0092d353d0369071e1021782bb38d.css' rel='stylesheet' type='text/css' /> </head> @@ -428,7 +428,7 @@ list of people who contributed to this r </div><!-- /.container --> <!-- JavaScript --> - <script src='/assets/global-c6949aebbeb9952ae7967e9bc1aedfbc.js' type='text/javascript'></script> + <script src='/assets/global-14c8e0124e6d563413005cb909342e2a.js' type='text/javascript'></script> Modified: libcloud/site/trunk/generated/blog/archives/2016/07/index.html URL: http://svn.apache.org/viewvc/libcloud/site/trunk/generated/blog/archives/2016/07/index.html?rev=1797582&r1=1797581&r2=1797582&view=diff ============================================================================== --- libcloud/site/trunk/generated/blog/archives/2016/07/index.html (original) +++ libcloud/site/trunk/generated/blog/archives/2016/07/index.html Sun Jun 4 14:37:45 2017 @@ -51,7 +51,7 @@ - <link href='/assets/global-eb5319c938d664847c83b2674bfd22c2.css' rel='stylesheet' type='text/css' /> + <link href='/assets/global-57b0092d353d0369071e1021782bb38d.css' rel='stylesheet' type='text/css' /> </head> @@ -205,7 +205,7 @@ list of people who contributed to this r </div><!-- /.container --> <!-- JavaScript --> - <script src='/assets/global-c6949aebbeb9952ae7967e9bc1aedfbc.js' type='text/javascript'></script> + <script src='/assets/global-14c8e0124e6d563413005cb909342e2a.js' type='text/javascript'></script> Modified: libcloud/site/trunk/generated/blog/archives/2016/09/index.html URL: http://svn.apache.org/viewvc/libcloud/site/trunk/generated/blog/archives/2016/09/index.html?rev=1797582&r1=1797581&r2=1797582&view=diff ============================================================================== --- libcloud/site/trunk/generated/blog/archives/2016/09/index.html (original) +++ libcloud/site/trunk/generated/blog/archives/2016/09/index.html Sun Jun 4 14:37:45 2017 @@ -51,7 +51,7 @@ - <link href='/assets/global-eb5319c938d664847c83b2674bfd22c2.css' rel='stylesheet' type='text/css' /> + <link href='/assets/global-57b0092d353d0369071e1021782bb38d.css' rel='stylesheet' type='text/css' /> </head> @@ -261,7 +261,7 @@ list of people who contributed to this r </div><!-- /.container --> <!-- JavaScript --> - <script src='/assets/global-c6949aebbeb9952ae7967e9bc1aedfbc.js' type='text/javascript'></script> + <script src='/assets/global-14c8e0124e6d563413005cb909342e2a.js' type='text/javascript'></script> Modified: libcloud/site/trunk/generated/blog/archives/2016/10/index.html URL: http://svn.apache.org/viewvc/libcloud/site/trunk/generated/blog/archives/2016/10/index.html?rev=1797582&r1=1797581&r2=1797582&view=diff ============================================================================== --- libcloud/site/trunk/generated/blog/archives/2016/10/index.html (original) +++ libcloud/site/trunk/generated/blog/archives/2016/10/index.html Sun Jun 4 14:37:45 2017 @@ -51,7 +51,7 @@ - <link href='/assets/global-eb5319c938d664847c83b2674bfd22c2.css' rel='stylesheet' type='text/css' /> + <link href='/assets/global-57b0092d353d0369071e1021782bb38d.css' rel='stylesheet' type='text/css' /> </head> @@ -248,7 +248,7 @@ list of people who contributed to this r </div><!-- /.container --> <!-- JavaScript --> - <script src='/assets/global-c6949aebbeb9952ae7967e9bc1aedfbc.js' type='text/javascript'></script> + <script src='/assets/global-14c8e0124e6d563413005cb909342e2a.js' type='text/javascript'></script> Modified: libcloud/site/trunk/generated/blog/archives/2016/11/index.html URL: http://svn.apache.org/viewvc/libcloud/site/trunk/generated/blog/archives/2016/11/index.html?rev=1797582&r1=1797581&r2=1797582&view=diff ============================================================================== --- libcloud/site/trunk/generated/blog/archives/2016/11/index.html (original) +++ libcloud/site/trunk/generated/blog/archives/2016/11/index.html Sun Jun 4 14:37:45 2017 @@ -51,7 +51,7 @@ - <link href='/assets/global-eb5319c938d664847c83b2674bfd22c2.css' rel='stylesheet' type='text/css' /> + <link href='/assets/global-57b0092d353d0369071e1021782bb38d.css' rel='stylesheet' type='text/css' /> </head> @@ -204,7 +204,7 @@ list of people who contributed to this r </div><!-- /.container --> <!-- JavaScript --> - <script src='/assets/global-c6949aebbeb9952ae7967e9bc1aedfbc.js' type='text/javascript'></script> + <script src='/assets/global-14c8e0124e6d563413005cb909342e2a.js' type='text/javascript'></script> Modified: libcloud/site/trunk/generated/blog/archives/2016/12/index.html URL: http://svn.apache.org/viewvc/libcloud/site/trunk/generated/blog/archives/2016/12/index.html?rev=1797582&r1=1797581&r2=1797582&view=diff ============================================================================== --- libcloud/site/trunk/generated/blog/archives/2016/12/index.html (original) +++ libcloud/site/trunk/generated/blog/archives/2016/12/index.html Sun Jun 4 14:37:45 2017 @@ -51,7 +51,7 @@ - <link href='/assets/global-eb5319c938d664847c83b2674bfd22c2.css' rel='stylesheet' type='text/css' /> + <link href='/assets/global-57b0092d353d0369071e1021782bb38d.css' rel='stylesheet' type='text/css' /> </head> @@ -212,7 +212,7 @@ list of people who contributed to this r </div><!-- /.container --> <!-- JavaScript --> - <script src='/assets/global-c6949aebbeb9952ae7967e9bc1aedfbc.js' type='text/javascript'></script> + <script src='/assets/global-14c8e0124e6d563413005cb909342e2a.js' type='text/javascript'></script> Modified: libcloud/site/trunk/generated/blog/archives/2017/04/index.html URL: http://svn.apache.org/viewvc/libcloud/site/trunk/generated/blog/archives/2017/04/index.html?rev=1797582&r1=1797581&r2=1797582&view=diff ============================================================================== --- libcloud/site/trunk/generated/blog/archives/2017/04/index.html (original) +++ libcloud/site/trunk/generated/blog/archives/2017/04/index.html Sun Jun 4 14:37:45 2017 @@ -51,7 +51,7 @@ - <link href='/assets/global-eb5319c938d664847c83b2674bfd22c2.css' rel='stylesheet' type='text/css' /> + <link href='/assets/global-57b0092d353d0369071e1021782bb38d.css' rel='stylesheet' type='text/css' /> </head> @@ -95,381 +95,6 @@ <div class="post"> - <h2><a href="/blog/2017/04/25/libcloud-2-0-0-released.html">Libcloud 2.0.0 released</a></h2> - - - - - - <span class="post-date-author">By Anthony Shaw on Apr 25, 2017</span> - - <div class="post-content"> - <p>We are pleased to announce the release of Libcloud 2.0.0.</p> - -<p>This major release brings many new features, improvements, bug-fixes, and drivers.</p> - -<h2>Release highlights</h2> - -<ul> -<li>Apache Libcloud 2.0 series replaces the use of Python httplib with a hard dependency on the <code>requests</code> package. Users no longer -have to specific Certificate Authority bundles when using Apache Libcloud</li> -<li>10% performance improvement through the use of HTTP sessions</li> -<li>Support for buffered IO streams for storage drivers</li> -<li>Support for Python 3.6, deprecation of Python 3.2</li> -<li>3 new drivers - <a href="http://libcloud.readthedocs.io/en/latest/compute/drivers/onapp.html";>OnApp Compute</a>, <a href="http://libcloud.readthedocs.io/en/latest/dns/drivers/onapp.html";>OnApp DNS</a>, <a href="http://libcloud.readthedocs.io/en/latest/compute/drivers/oneandone.html";>1&1 Compute</a></li> -<li>Lots of improvements to our Azure ARM support</li> -<li>Continuing udates to the Amazon drivers</li> -</ul> - -<p>A detailed description of the 2.0 HTTP API is documented <a href="http://libcloud.readthedocs.io/en/latest/other/changes_in_2_0.html";>here</a></p> - -<h3>Compute API Changes</h3> - -<ul> -<li>Outscale SAS doc improvements and logo update</li> -</ul> - -<h4>OnApp Changes</h4> - -<ul> -<li>Add list images support for OnApp driver</li> -<li>Add keypair management to OnApp driver</li> -</ul> - -<h4>Amazon EC2 Changes</h4> - -<ul> -<li>Add r4 instance types for AWS</li> -<li>Add support for AWS eu-west-2 and ca-central-1 regions</li> -<li>Add P2 GPU instance types</li> -<li>Add method to modify snapshot attribute for EC2</li> -<li>Add ENA support for EC2 compute images</li> -<li>Add support for forcing detachment of EBS volumes to EC2 driver</li> -<li>Add support for ModifyVolume and DescribeVolumesModifications</li> -<li>Added Import Snapshot and Describe Import Snapshot to EC2 compute driver</li> -<li>Add missing regions in AWS storage and compute drivers</li> -<li>Add SR-IOV net support to images in EC2 compute driver</li> -<li>Fix - update t2.small image size from 11 CPU to 1</li> -<li>Added Billing Product for image in EC2 compute driver</li> -</ul> - -<h4>Linode changes</h4> - -<ul> -<li>Add start, stop instance methods and fix incorrect state TERMINATED to STOPPED</li> -</ul> - -<h4>Azure ARM changes</h4> - -<ul> -<li>Fix typeerror on ex<em>list</em>nics</li> -<li>Add support for Azure Cloud Environments as well as Locations</li> -<li>Fix string representation of the VhdImage type and fix listing of Public IP addresses</li> -<li>Add network security groups to azure ARM</li> -<li>Add the ability to list resource groups</li> -<li>Fix Azure ARM driver condition for ex<em>list</em>publishers where location is specified</li> -</ul> - -<h4>Google Cloud changes</h4> - -<ul> -<li>Allow delete instances from managed group</li> -<li>Allow preemptible instances to be created</li> -<li>Remove validation checks for guestOsFeatures</li> -</ul> - -<h4>Alibaba Aliyun changes</h4> - -<ul> -<li>Add aliyun ecs instance join leave security group</li> -<li>Fix Aliyun ECS, Load balancer and storage adapters when using unicode UTF-8 characters in the names of resources</li> -</ul> - -<h4>DigitalOcean changes</h4> - -<ul> -<li>Add price_monthly extra param to digitalocean sizes</li> -</ul> - -<h4>VMWare vSphere changes</h4> - -<ul> -<li>Fix issue with authentication methods crashing</li> -</ul> - -<h3>Storage</h3> - -<ul> -<li>Reintroduce S3 multipart upload support with signature v4</li> -</ul> - -<h3>DNS</h3> - -<ul> -<li>Add <a href="http://libcloud.readthedocs.io/en/latest/dns/drivers/onapp.html";>OnApp driver</a></li> -</ul> - -<h2>Minor changes</h2> - -<h3>Common</h3> - -<ul> -<li>Added an integration test API and a test suite for validating functionality -without mocking any libcloud subsystems</li> -<li>Change Cloudscale to cloudscale.ch.</li> -</ul> - -<h2>Bug Fixes</h2> - -<h3>Compute</h3> - -<h4>2.0.0</h4> - -<ul> -<li><p>Fix OpenStack drivers not correctly setting URLs when used with identity API, would default to 127.0.0.1 and service -catalog URLs were not adhered to.</p></li> -<li><p>Fix Aliyun ECS, Load balancer and storage adapters when using unicode UTF-8 characters in the names of resources -in 2.0.0rc2 < it would fail as a MalformedResponseError, Python 2.7 element tree was raising a unicode error</p></li> -<li><p>Refactor the test classes to use the full libcloud.http and libcloud.common.base modules, with Connection, -Response all used with requests<em>mock. This increases our test coverages and catches bugs in drivers' custom -parse</em>body and auth modules</p></li> -<li><p>Rename libcloud.httplib_ssl to libcloud.http now that we don't use httplib</p></li> -</ul> - -<h4>2.0.0rc2</h4> - -<ul> -<li>Fix Public IP not assigned when creating NIC on Azure ARM</li> -<li>Fix a bug in profitbricks driver where listing snapshots would request a malformed URL</li> -<li>Fix LIBCLOUD-806 bug where vsphere driver cannot be instantiated</li> -<li>[google compute] Improve performance of list nodes by caching volume information.</li> -</ul> - -<p>Full change log can be found at <a href="https://libcloud.readthedocs.org/en/latest/changelog.html";>here</a>.</p> - -<h3>Special thank you</h3> - -<p>I would like to wish a special thank you to all of our community contributors -for their ongoing support to the project.</p> - -<ul> -<li>Tinu Cleatus for the OnApp driver</li> -<li>Alex Misstear for the EC2 improvements</li> -<li>Jie Ren for the Aliyun improvements</li> -<li>Francisco Ros for the DigitalOcean improvements</li> -<li>Peter Amstutz and Joseph Hall for the Azure ARM API updates</li> -<li>All of our community for their ongoing support and contributions...</li> -</ul> - -<h3>Download</h3> - -<p>The release can can be downloaded from -<a href="https://libcloud.apache.org/downloads.html";>https://libcloud.apache.org/downloads.html</a> or installed using pip:</p> - -<pre> -pip install apache-libcloud==2.0.0 -</pre> - -<h3>Upgrading</h3> - -<p>If you have installed Libcloud using pip you can also use it to upgrade it:</p> - -<pre> -pip install --upgrade apache-libcloud==2.0.0 -</pre> - -<h3>Upgrade notes</h3> - -<p>A page which describes backward incompatible or semi-incompatible -changes and how to preserve the old behavior when this is possible -can be found at <a href="https://libcloud.readthedocs.org/en/latest/upgrade_notes.html";>https://libcloud.readthedocs.org/en/latest/upgrade_notes.html</a></p> - -<h3>Documentation</h3> - -<p>Regular and API documentation is available at <a href="https://libcloud.readthedocs.org/en/latest/";>https://libcloud.readthedocs.org/en/latest/</a></p> - -<h3>Bugs / Issues</h3> - -<p>If you find any bug or issue, please report it on our issue tracker -<a href="https://issues.apache.org/jira/browse/LIBCLOUD";>https://issues.apache.org/jira/browse/LIBCLOUD</a>. -Don't forget to attach an example and / or test which reproduces your -problem.</p> - -<h3>Thanks</h3> - -<p>Thanks to everyone who contributed and made this release possible! Full -list of people who contributed to this release can be found in the -<a href="https://libcloud.readthedocs.org/en/latest/changelog.html";>CHANGES file</a>.</p> - - </div> - - <div class="row section post-meta"> - <div class="col-md-12 post-tags"> - <p>Tags: <a href="/blog/tags/news.html" rel="tag">news</a>, <a href="/blog/tags/release%20announcement.html" rel="tag">release announcement</a></p> - </div> - </div> -</div> - - - <div class="post"> - - <h2><a href="/blog/2017/04/09/async-rfc.html">Have your say - async support in Apache Libcloud</a></h2> - - - - - - <span class="post-date-author">By Anthony Shaw on Apr 09, 2017</span> - - <div class="post-content"> - <p>One of the big requests whilst we were replacing <code>httplib</code> with the <code>requests</code> package in 2.0 was why didn't -we use a HTTP library that supports <em>asynchronous</em> API calls.</p> - -<p>The intention for 2.0 and replacing the HTTP backend classes was to improve the usability of the project, by making SSL -certificates easier to manage, improving the maintainability of our source code by using an active 3rd party package and -also improving performance and stability.</p> - -<p>Apache Libcloud already has documentation on threaded libraries like gevent and callback-based libraries like Twisted, see -<a href="https://libcloud.readthedocs.io/en/latest/other/using-libcloud-in-multithreaded-and-async-environments.html";>using libcloud in multithreaded environments</a> -for examples.</p> - -<p><a href="https://www.python.org/dev/peps/pep-0492/#";>PEP 492</a>, implemented in Python 3.5 provides a new coroutine protocol using methods, -<code>__await__</code> for classes, a coroutine method wrapper, or a method that returns a coroutine object. -Also async <a href="https://www.python.org/dev/peps/pep-0492/#asynchronous-iterators-and-async-for";>iterators</a> and <a href="https://www.python.org/dev/peps/pep-0492/#asynchronous-context-managers-and-async-with";>context managers</a> -have been introduced.</p> - -<p>We would like to take advantage of the new language features by offering APIs in Apache Libcloud without breaking backward compatibility and -compatibility for users of <Python 3.5.</p> - -<p>Use cases for this would be:</p> - -<ul> -<li>Being able to fetch <code>Node</code> or <code>StorageObject</code>s from multiple geographies or drivers simultaneously.</li> -<li>Being able to quickly upload or download storage objects by parallelizing operations on the <code>StorageDriver</code>.</li> -<li>Being able to call a long-running API method (e.g. generate report), whilst running other code.</li> -</ul> - -<h2>Design 1 - async context managers <a href="https://github.com/apache/libcloud/pull/1016";>PR 1016</a></h2> - -<p>This design would allow drivers to operate in 2 modes, the first is for synchronous method calls, they return list or object -data as per usual. The second mode, API methods like <code>NodeDriver.list_nodes</code> would return a <a href="https://www.python.org/dev/peps/pep-0492/#coroutine-objects";>coroutine object</a> -and could be awaited or gathered using an event loop.</p> -<div class="highlight"><pre><code class="python language-python" data-lang="python"><span class="kn">import</span> <span class="nn">asyncio</span> - -<span class="kn">from</span> <span class="nn">integration.driver.test</span> <span class="kn">import</span> <span class="n">TestNodeDriver</span> -<span class="kn">from</span> <span class="nn">libcloud.async_util</span> <span class="kn">import</span> <span class="n">AsyncSession</span> - -<span class="n">driver</span> <span class="o">=</span> <span class="n">TestNodeDriver</span><span class="p">(</span><span class="s">'apache'</span><span class="p">,</span> <span class="s">'libcloud'</span><span class="p">)</span> - -<span class="n">async</span> <span class="k">def</span> <span class="nf">run</span><span class="p">():</span> - <span class="c"># regular API call</span> - <span class="n">nodes</span> <span class="o">=</span> <span class="n">driver</span><span class="o">.</span><span class="n">list_nodes</span><span class="p">()</span> - - <span class="n">async</span> <span class="k">with</span> <span class="n">AsyncSession</span><span class="p">(</span><span class="n">driver</span><span class="p">)</span> <span class="k">as</span> <span class="n">async_instance</span><span class="p">:</span> - <span class="n">nodes</span> <span class="o">=</span> <span class="n">await</span> <span class="n">async_instance</span><span class="o">.</span><span class="n">list_nodes</span><span class="p">()</span> - - <span class="k">assert</span> <span class="nb">len</span><span class="p">(</span><span class="n">nodes</span><span class="p">)</span> <span class="o">==</span> <span class="mi">2</span> - -<span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span> -<span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">run</span><span class="p">())</span> -<span class="n">loop</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> -</code></pre></div> -<h2>Design 2 - Additional methods in each driver for coroutines <a href="https://github.com/apache/libcloud/pull/1027";>PR 1027</a></h2> - -<p>This is the second design concept for async support in Libcloud.</p> - -<p>The concept here is to have Asynchronous Mixins, <code>LibcloudConnection</code> uses requests and <code>LibcloudAsyncConnection</code> uses aiohttp for async transport <a href="https://github.com/tonybaloney/libcloud/blob/d4fe097476d2f02941e17d5e1b1d405fcf44c0f7/libcloud/connection_async.py#L22-L42";>see</a></p> - -<p>The LibcloudAsyncConnection is an implementation detail of AsyncConnection, which is the API for the drivers to consume <a href="https://github.com/tonybaloney/libcloud/blob/d4fe097476d2f02941e17d5e1b1d405fcf44c0f7/libcloud/common/base.py#L742-L778";>see</a></p> - -<p>The drivers then use this mixin for their custom connection classes, e.g.</p> -<div class="highlight"><pre><code class="python language-python" data-lang="python"><span class="k">class</span> <span class="nc">GoogleStorageConnection</span><span class="p">(</span><span class="n">ConnectionUserAndKey</span><span class="p">,</span> <span class="n">AsyncConnection</span><span class="p">):</span> - <span class="o">...</span> -</code></pre></div> -<p>They then inherit from <code>libcloud.storage.base.StorageAsyncDriver</code>, which uses a new set of base methods, e.g. <code>iterate_containers_async</code> and can be implemented like this:</p> -<div class="highlight"><pre><code class="python language-python" data-lang="python"> <span class="n">async</span> <span class="k">def</span> <span class="nf">iterate_containers_async</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> - <span class="n">response</span> <span class="o">=</span> <span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">request_async</span><span class="p">(</span><span class="s">'/'</span><span class="p">)</span> - <span class="k">if</span> <span class="n">response</span><span class="o">.</span><span class="n">status</span> <span class="o">==</span> <span class="n">httplib</span><span class="o">.</span><span class="n">OK</span><span class="p">:</span> - <span class="n">containers</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_to_containers</span><span class="p">(</span><span class="n">obj</span><span class="o">=</span><span class="n">response</span><span class="o">.</span><span class="n">object</span><span class="p">,</span> - <span class="n">xpath</span><span class="o">=</span><span class="s">'Buckets/Bucket'</span><span class="p">)</span> - <span class="k">return</span> <span class="n">containers</span> - - <span class="k">raise</span> <span class="n">LibcloudError</span><span class="p">(</span><span class="s">'Unexpected status code: </span><span class="si">%s</span><span class="s">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">status</span><span class="p">),</span> - <span class="n">driver</span><span class="o">=</span><span class="bp">self</span><span class="p">)</span> -</code></pre></div> -<p>Now the consumer can more or less do this:</p> -<div class="highlight"><pre><code class="python language-python" data-lang="python"><span class="kn">from</span> <span class="nn">libcloud.storage.providers</span> <span class="kn">import</span> <span class="n">get_driver</span> -<span class="kn">from</span> <span class="nn">libcloud.storage.types</span> <span class="kn">import</span> <span class="n">Provider</span> - -<span class="kn">import</span> <span class="nn">asyncio</span> - -<span class="n">GoogleStorageDriver</span> <span class="o">=</span> <span class="n">get_driver</span><span class="p">(</span><span class="n">Provider</span><span class="o">.</span><span class="n">GOOGLE_STORAGE</span><span class="p">)</span> -<span class="n">driver</span> <span class="o">=</span> <span class="n">GoogleStorageDriver</span><span class="p">(</span><span class="n">key</span><span class="o">=</span><span class="n">KEY</span><span class="p">,</span> <span class="n">secret</span><span class="o">=</span><span class="n">SECRET</span><span class="p">)</span> - -<span class="k">def</span> <span class="nf">do_stuff_with_object</span><span class="p">(</span><span class="n">obj</span><span class="p">):</span> - <span class="k">print</span><span class="p">(</span><span class="n">obj</span><span class="p">)</span> - -<span class="n">async</span> <span class="k">def</span> <span class="nf">run</span><span class="p">():</span> - <span class="n">tasks</span> <span class="o">=</span> <span class="p">[]</span> - <span class="n">async</span> <span class="k">for</span> <span class="n">container</span> <span class="ow">in</span> <span class="n">driver</span><span class="o">.</span><span class="n">iterate_containers_async</span><span class="p">():</span> - <span class="n">async</span> <span class="k">for</span> <span class="n">obj</span> <span class="ow">in</span> <span class="n">driver</span><span class="o">.</span><span class="n">iterate_container_objects_async</span><span class="p">(</span><span class="n">container</span><span class="p">):</span> - <span class="n">tasks</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">asyncio</span><span class="o">.</span><span class="n">ensure_future</span><span class="p">(</span><span class="n">do_stuff_with_object</span><span class="p">(</span><span class="n">obj</span><span class="p">)))</span> - <span class="n">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">gather</span><span class="p">(</span><span class="o">*</span><span class="n">tasks</span><span class="p">)</span> - -<span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span> -<span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">run</span><span class="p">())</span> -<span class="n">loop</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> -</code></pre></div> -<h2>Design 3 - Initializer with "<em>async</em>" mode</h2> - -<p>This option is similar to 2, except that if a driver is instantiated with "<code>async=True</code>", -then all driver class methods would return coroutine objects. Internally, it would -patch the Connection class with the AsyncConnection class.</p> - -<p>The downside of this is that all method calls to a driver would need to be awaited or used -by an event loop.</p> -<div class="highlight"><pre><code class="python language-python" data-lang="python"><span class="kn">from</span> <span class="nn">libcloud.storage.providers</span> <span class="kn">import</span> <span class="n">get_driver</span> -<span class="kn">from</span> <span class="nn">libcloud.storage.types</span> <span class="kn">import</span> <span class="n">Provider</span> - -<span class="kn">import</span> <span class="nn">asyncio</span> - -<span class="n">GoogleStorageDriver</span> <span class="o">=</span> <span class="n">get_driver</span><span class="p">(</span><span class="n">Provider</span><span class="o">.</span><span class="n">GOOGLE_STORAGE</span><span class="p">)</span> -<span class="n">driver</span> <span class="o">=</span> <span class="n">GoogleStorageDriver</span><span class="p">(</span><span class="n">key</span><span class="o">=</span><span class="n">KEY</span><span class="p">,</span> <span class="n">secret</span><span class="o">=</span><span class="n">SECRET</span><span class="p">,</span> <span class="n">async</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> - -<span class="k">def</span> <span class="nf">do_stuff_with_object</span><span class="p">(</span><span class="n">obj</span><span class="p">):</span> - <span class="k">print</span><span class="p">(</span><span class="n">obj</span><span class="p">)</span> - -<span class="n">async</span> <span class="k">def</span> <span class="nf">run</span><span class="p">():</span> - <span class="n">tasks</span> <span class="o">=</span> <span class="p">[]</span> - <span class="n">async</span> <span class="k">for</span> <span class="n">container</span> <span class="ow">in</span> <span class="n">driver</span><span class="o">.</span><span class="n">iterate_containers</span><span class="p">():</span> - <span class="n">async</span> <span class="k">for</span> <span class="n">obj</span> <span class="ow">in</span> <span class="n">driver</span><span class="o">.</span><span class="n">iterate_container_objects</span><span class="p">(</span><span class="n">container</span><span class="p">):</span> - <span class="n">tasks</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">asyncio</span><span class="o">.</span><span class="n">ensure_future</span><span class="p">(</span><span class="n">do_stuff_with_object</span><span class="p">(</span><span class="n">obj</span><span class="p">)))</span> - <span class="n">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">gather</span><span class="p">(</span><span class="o">*</span><span class="n">tasks</span><span class="p">)</span> - -<span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span> -<span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">run</span><span class="p">())</span> -<span class="n">loop</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> -</code></pre></div> -<h1>Give us feedback</h1> - -<p>Got a better idea? Have an API or design, the question we're asking is -"if you wanted to use Libcloud for an async application, what would the code look like?" This helps us design -the API and the implementation details can follow.</p> - -<p>Feel free to comment on the mailing list or on the pull requests, or raise your own pull-request with an API design.</p> - - </div> - - <div class="row section post-meta"> - <div class="col-md-12 post-tags"> - <p>Tags: <a href="/blog/tags/news.html" rel="tag">news</a>, <a href="/blog/tags/tutorial.html" rel="tag">tutorial</a></p> - </div> - </div> -</div> - - - <div class="post"> - <h2><a href="/blog/2017/04/07/libcloud-2-0-0-rc2-released.html">Libcloud 2.0.0rc2 released</a></h2> @@ -656,7 +281,7 @@ list of people who contributed to this r </div><!-- /.container --> <!-- JavaScript --> - <script src='/assets/global-c6949aebbeb9952ae7967e9bc1aedfbc.js' type='text/javascript'></script> + <script src='/assets/global-14c8e0124e6d563413005cb909342e2a.js' type='text/javascript'></script> Modified: libcloud/site/trunk/generated/blog/atom.xml URL: http://svn.apache.org/viewvc/libcloud/site/trunk/generated/blog/atom.xml?rev=1797582&r1=1797581&r2=1797582&view=diff ============================================================================== --- libcloud/site/trunk/generated/blog/atom.xml (original) +++ libcloud/site/trunk/generated/blog/atom.xml Sun Jun 4 14:37:45 2017 @@ -5,370 +5,17 @@ <link href="https://libcloud.apache.org"; rel="self" /> <link href="https://libcloud.apache.org"; /> <id>https://libcloud.apache.org</id> - <updated>2017-04-26T10:14:30+10:00</updated> + <updated>2017-06-04T16:33:45+02:00</updated> <author> <name>The Apache Software Foundation</name> </author> <entry> - <title>Libcloud 2.0.0 released</title> - <link href="https://libcloud.apache.org/blog/2017/04/25/libcloud-2-0-0-released.html"; /> - <id>https://libcloud.apache.org/blog/2017/04/25/libcloud-2-0-0-released.html</id> - <updated>2017-04-25T00:00:00+10:00</updated> - <author> - <name>Anthony Shaw</name> - </author> - <content type="html"><p>We are pleased to announce the release of Libcloud 2.0.0.</p> - -<p>This major release brings many new features, improvements, bug-fixes, and drivers.</p> - -<h2>Release highlights</h2> - -<ul> -<li>Apache Libcloud 2.0 series replaces the use of Python httplib with a hard dependency on the <code>requests</code> package. Users no longer -have to specific Certificate Authority bundles when using Apache Libcloud</li> -<li>10% performance improvement through the use of HTTP sessions</li> -<li>Support for buffered IO streams for storage drivers</li> -<li>Support for Python 3.6, deprecation of Python 3.2</li> -<li>3 new drivers - <a href="http://libcloud.readthedocs.io/en/latest/compute/drivers/onapp.html">OnApp Compute</a>, <a href="http://libcloud.readthedocs.io/en/latest/dns/drivers/onapp.html">OnApp DNS</a>, <a href="http://libcloud.readthedocs.io/en/latest/compute/drivers/oneandone.html">1&amp;1 Compute</a></li> -<li>Lots of improvements to our Azure ARM support</li> -<li>Continuing udates to the Amazon drivers</li> -</ul> - -<p>A detailed description of the 2.0 HTTP API is documented <a href="http://libcloud.readthedocs.io/en/latest/other/changes_in_2_0.html">here</a></p> - -<h3>Compute API Changes</h3> - -<ul> -<li>Outscale SAS doc improvements and logo update</li> -</ul> - -<h4>OnApp Changes</h4> - -<ul> -<li>Add list images support for OnApp driver</li> -<li>Add keypair management to OnApp driver</li> -</ul> - -<h4>Amazon EC2 Changes</h4> - -<ul> -<li>Add r4 instance types for AWS</li> -<li>Add support for AWS eu-west-2 and ca-central-1 regions</li> -<li>Add P2 GPU instance types</li> -<li>Add method to modify snapshot attribute for EC2</li> -<li>Add ENA support for EC2 compute images</li> -<li>Add support for forcing detachment of EBS volumes to EC2 driver</li> -<li>Add support for ModifyVolume and DescribeVolumesModifications</li> -<li>Added Import Snapshot and Describe Import Snapshot to EC2 compute driver</li> -<li>Add missing regions in AWS storage and compute drivers</li> -<li>Add SR-IOV net support to images in EC2 compute driver</li> -<li>Fix - update t2.small image size from 11 CPU to 1</li> -<li>Added Billing Product for image in EC2 compute driver</li> -</ul> - -<h4>Linode changes</h4> - -<ul> -<li>Add start, stop instance methods and fix incorrect state TERMINATED to STOPPED</li> -</ul> - -<h4>Azure ARM changes</h4> - -<ul> -<li>Fix typeerror on ex<em>list</em>nics</li> -<li>Add support for Azure Cloud Environments as well as Locations</li> -<li>Fix string representation of the VhdImage type and fix listing of Public IP addresses</li> -<li>Add network security groups to azure ARM</li> -<li>Add the ability to list resource groups</li> -<li>Fix Azure ARM driver condition for ex<em>list</em>publishers where location is specified</li> -</ul> - -<h4>Google Cloud changes</h4> - -<ul> -<li>Allow delete instances from managed group</li> -<li>Allow preemptible instances to be created</li> -<li>Remove validation checks for guestOsFeatures</li> -</ul> - -<h4>Alibaba Aliyun changes</h4> - -<ul> -<li>Add aliyun ecs instance join leave security group</li> -<li>Fix Aliyun ECS, Load balancer and storage adapters when using unicode UTF-8 characters in the names of resources</li> -</ul> - -<h4>DigitalOcean changes</h4> - -<ul> -<li>Add price_monthly extra param to digitalocean sizes</li> -</ul> - -<h4>VMWare vSphere changes</h4> - -<ul> -<li>Fix issue with authentication methods crashing</li> -</ul> - -<h3>Storage</h3> - -<ul> -<li>Reintroduce S3 multipart upload support with signature v4</li> -</ul> - -<h3>DNS</h3> - -<ul> -<li>Add <a href="http://libcloud.readthedocs.io/en/latest/dns/drivers/onapp.html">OnApp driver</a></li> -</ul> - -<h2>Minor changes</h2> - -<h3>Common</h3> - -<ul> -<li>Added an integration test API and a test suite for validating functionality -without mocking any libcloud subsystems</li> -<li>Change Cloudscale to cloudscale.ch.</li> -</ul> - -<h2>Bug Fixes</h2> - -<h3>Compute</h3> - -<h4>2.0.0</h4> - -<ul> -<li><p>Fix OpenStack drivers not correctly setting URLs when used with identity API, would default to 127.0.0.1 and service -catalog URLs were not adhered to.</p></li> -<li><p>Fix Aliyun ECS, Load balancer and storage adapters when using unicode UTF-8 characters in the names of resources -in 2.0.0rc2 &lt; it would fail as a MalformedResponseError, Python 2.7 element tree was raising a unicode error</p></li> -<li><p>Refactor the test classes to use the full libcloud.http and libcloud.common.base modules, with Connection, -Response all used with requests<em>mock. This increases our test coverages and catches bugs in drivers&#39; custom -parse</em>body and auth modules</p></li> -<li><p>Rename libcloud.httplib_ssl to libcloud.http now that we don&#39;t use httplib</p></li> -</ul> - -<h4>2.0.0rc2</h4> - -<ul> -<li>Fix Public IP not assigned when creating NIC on Azure ARM</li> -<li>Fix a bug in profitbricks driver where listing snapshots would request a malformed URL</li> -<li>Fix LIBCLOUD-806 bug where vsphere driver cannot be instantiated</li> -<li>[google compute] Improve performance of list nodes by caching volume information.</li> -</ul> - -<p>Full change log can be found at <a href="https://libcloud.readthedocs.org/en/latest/changelog.html">here</a>.</p> - -<h3>Special thank you</h3> - -<p>I would like to wish a special thank you to all of our community contributors -for their ongoing support to the project.</p> - -<ul> -<li>Tinu Cleatus for the OnApp driver</li> -<li>Alex Misstear for the EC2 improvements</li> -<li>Jie Ren for the Aliyun improvements</li> -<li>Francisco Ros for the DigitalOcean improvements</li> -<li>Peter Amstutz and Joseph Hall for the Azure ARM API updates</li> -<li>All of our community for their ongoing support and contributions...</li> -</ul> - -<h3>Download</h3> - -<p>The release can can be downloaded from -<a href="https://libcloud.apache.org/downloads.html">https://libcloud.apache.org/downloads.html</a> or installed using pip:</p> - -<pre> -pip install apache-libcloud==2.0.0 -</pre> - -<h3>Upgrading</h3> - -<p>If you have installed Libcloud using pip you can also use it to upgrade it:</p> - -<pre> -pip install --upgrade apache-libcloud==2.0.0 -</pre> - -<h3>Upgrade notes</h3> - -<p>A page which describes backward incompatible or semi-incompatible -changes and how to preserve the old behavior when this is possible -can be found at <a href="https://libcloud.readthedocs.org/en/latest/upgrade_notes.html">https://libcloud.readthedocs.org/en/latest/upgrade_notes.html</a></p> - -<h3>Documentation</h3> - -<p>Regular and API documentation is available at <a href="https://libcloud.readthedocs.org/en/latest/">https://libcloud.readthedocs.org/en/latest/</a></p> - -<h3>Bugs / Issues</h3> - -<p>If you find any bug or issue, please report it on our issue tracker -<a href="https://issues.apache.org/jira/browse/LIBCLOUD">https://issues.apache.org/jira/browse/LIBCLOUD</a>. -Don&#39;t forget to attach an example and / or test which reproduces your -problem.</p> - -<h3>Thanks</h3> - -<p>Thanks to everyone who contributed and made this release possible! Full -list of people who contributed to this release can be found in the -<a href="https://libcloud.readthedocs.org/en/latest/changelog.html">CHANGES file</a>.</p> -</content> - </entry> - - <entry> - <title>Have your say - async support in Apache Libcloud</title> - <link href="https://libcloud.apache.org/blog/2017/04/09/async-rfc.html"; /> - <id>https://libcloud.apache.org/blog/2017/04/09/async-rfc.html</id> - <updated>2017-04-09T00:00:00+10:00</updated> - <author> - <name>Anthony Shaw</name> - </author> - <content type="html"><p>One of the big requests whilst we were replacing <code>httplib</code> with the <code>requests</code> package in 2.0 was why didn&#39;t -we use a HTTP library that supports <em>asynchronous</em> API calls.</p> - -<p>The intention for 2.0 and replacing the HTTP backend classes was to improve the usability of the project, by making SSL -certificates easier to manage, improving the maintainability of our source code by using an active 3rd party package and -also improving performance and stability.</p> - -<p>Apache Libcloud already has documentation on threaded libraries like gevent and callback-based libraries like Twisted, see -<a href="https://libcloud.readthedocs.io/en/latest/other/using-libcloud-in-multithreaded-and-async-environments.html">using libcloud in multithreaded environments</a> -for examples.</p> - -<p><a href="https://www.python.org/dev/peps/pep-0492/#">PEP 492</a>, implemented in Python 3.5 provides a new coroutine protocol using methods, -<code>__await__</code> for classes, a coroutine method wrapper, or a method that returns a coroutine object. -Also async <a href="https://www.python.org/dev/peps/pep-0492/#asynchronous-iterators-and-async-for">iterators</a> and <a href="https://www.python.org/dev/peps/pep-0492/#asynchronous-context-managers-and-async-with">context managers</a> -have been introduced.</p> - -<p>We would like to take advantage of the new language features by offering APIs in Apache Libcloud without breaking backward compatibility and -compatibility for users of &lt;Python 3.5.</p> - -<p>Use cases for this would be:</p> - -<ul> -<li>Being able to fetch <code>Node</code> or <code>StorageObject</code>s from multiple geographies or drivers simultaneously.</li> -<li>Being able to quickly upload or download storage objects by parallelizing operations on the <code>StorageDriver</code>.</li> -<li>Being able to call a long-running API method (e.g. generate report), whilst running other code.</li> -</ul> - -<h2>Design 1 - async context managers <a href="https://github.com/apache/libcloud/pull/1016">PR 1016</a></h2> - -<p>This design would allow drivers to operate in 2 modes, the first is for synchronous method calls, they return list or object -data as per usual. The second mode, API methods like <code>NodeDriver.list_nodes</code> would return a <a href="https://www.python.org/dev/peps/pep-0492/#coroutine-objects">coroutine object</a> -and could be awaited or gathered using an event loop.</p> -<div class="highlight"><pre><code class="python language-python" data-lang="python"><span class="kn">import</span> <span class="nn">asyncio</span> - -<span class="kn">from</span> <span class="nn">integration.driver.test</span> <span class="kn">import</span> <span class="n">TestNodeDriver</span> -<span class="kn">from</span> <span class="nn">libcloud.async_util</span> <span class="kn">import</span> <span class="n">AsyncSession</span> - -<span class="n">driver</span> <span class="o">=</span> <span class="n">TestNodeDriver</span><span class="p">(</span><span class="s">&#39;apache&#39;</span><span class="p">,</span> <span class="s">&#39;libcloud&#39;</span><span class="p">)</span> - -<span class="n">async</span> <span class="k">def</span> <span class="nf">run</span><span class="p">():</span> - <span class="c"># regular API call</span> - <span class="n">nodes</span> <span class="o">=</span> <span class="n">driver</span><span class="o">.</span><span class="n">list_nodes</span><span class="p">()</span> - - <span class="n">async</span> <span class="k">with</span> <span class="n">AsyncSession</span><span class="p">(</span><span class="n">driver</span><span class="p">)</span> <span class="k">as</span> <span class="n">async_instance</span><span class="p">:</span> - <span class="n">nodes</span> <span class="o">=</span> <span class="n">await</span> <span class="n">async_instance</span><span class="o">.</span><span class="n">list_nodes</span><span class="p">()</span> - - <span class="k">assert</span> <span class="nb">len</span><span class="p">(</span><span class="n">nodes</span><span class="p">)</span> <span class="o">==</span> <span class="mi">2</span> - -<span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span> -<span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">run</span><span class="p">())</span> -<span class="n">loop</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> -</code></pre></div> -<h2>Design 2 - Additional methods in each driver for coroutines <a href="https://github.com/apache/libcloud/pull/1027">PR 1027</a></h2> - -<p>This is the second design concept for async support in Libcloud.</p> - -<p>The concept here is to have Asynchronous Mixins, <code>LibcloudConnection</code> uses requests and <code>LibcloudAsyncConnection</code> uses aiohttp for async transport <a href="https://github.com/tonybaloney/libcloud/blob/d4fe097476d2f02941e17d5e1b1d405fcf44c0f7/libcloud/connection_async.py#L22-L42">see</a></p> - -<p>The LibcloudAsyncConnection is an implementation detail of AsyncConnection, which is the API for the drivers to consume <a href="https://github.com/tonybaloney/libcloud/blob/d4fe097476d2f02941e17d5e1b1d405fcf44c0f7/libcloud/common/base.py#L742-L778">see</a></p> - -<p>The drivers then use this mixin for their custom connection classes, e.g.</p> -<div class="highlight"><pre><code class="python language-python" data-lang="python"><span class="k">class</span> <span class="nc">GoogleStorageConnection</span><span class="p">(</span><span class="n">ConnectionUserAndKey</span><span class="p">,</span> <span class="n">AsyncConnection</span><span class="p">):</span> - <span class="o">...</span> -</code></pre></div> -<p>They then inherit from <code>libcloud.storage.base.StorageAsyncDriver</code>, which uses a new set of base methods, e.g. <code>iterate_containers_async</code> and can be implemented like this:</p> -<div class="highlight"><pre><code class="python language-python" data-lang="python"> <span class="n">async</span> <span class="k">def</span> <span class="nf">iterate_containers_async</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> - <span class="n">response</span> <span class="o">=</span> <span class="n">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">request_async</span><span class="p">(</span><span class="s">&#39;/&#39;</span><span class="p">)</span> - <span class="k">if</span> <span class="n">response</span><span class="o">.</span><span class="n">status</span> <span class="o">==</span> <span class="n">httplib</span><span class="o">.</span><span class="n">OK</span><span class="p">:</span> - <span class="n">containers</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_to_containers</span><span class="p">(</span><span class="n">obj</span><span class="o">=</span><span class="n">response</span><span class="o">.</span><span class="n">object</span><span class="p">,</span> - <span class="n">xpath</span><span class="o">=</span><span class="s">&#39;Buckets/Bucket&#39;</span><span class="p">)</span> - <span class="k">return</span> <span class="n">containers</span> - - <span class="k">raise</span> <span class="n">LibcloudError</span><span class="p">(</span><span class="s">&#39;Unexpected status code: </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">status</span><span class="p">),</span> - <span class="n">driver</span><span class="o">=</span><span class="bp">self</span><span class="p">)</span> -</code></pre></div> -<p>Now the consumer can more or less do this:</p> -<div class="highlight"><pre><code class="python language-python" data-lang="python"><span class="kn">from</span> <span class="nn">libcloud.storage.providers</span> <span class="kn">import</span> <span class="n">get_driver</span> -<span class="kn">from</span> <span class="nn">libcloud.storage.types</span> <span class="kn">import</span> <span class="n">Provider</span> - -<span class="kn">import</span> <span class="nn">asyncio</span> - -<span class="n">GoogleStorageDriver</span> <span class="o">=</span> <span class="n">get_driver</span><span class="p">(</span><span class="n">Provider</span><span class="o">.</span><span class="n">GOOGLE_STORAGE</span><span class="p">)</span> -<span class="n">driver</span> <span class="o">=</span> <span class="n">GoogleStorageDriver</span><span class="p">(</span><span class="n">key</span><span class="o">=</span><span class="n">KEY</span><span class="p">,</span> <span class="n">secret</span><span class="o">=</span><span class="n">SECRET</span><span class="p">)</span> - -<span class="k">def</span> <span class="nf">do_stuff_with_object</span><span class="p">(</span><span class="n">obj</span><span class="p">):</span> - <span class="k">print</span><span class="p">(</span><span class="n">obj</span><span class="p">)</span> - -<span class="n">async</span> <span class="k">def</span> <span class="nf">run</span><span class="p">():</span> - <span class="n">tasks</span> <span class="o">=</span> <span class="p">[]</span> - <span class="n">async</span> <span class="k">for</span> <span class="n">container</span> <span class="ow">in</span> <span class="n">driver</span><span class="o">.</span><span class="n">iterate_containers_async</span><span class="p">():</span> - <span class="n">async</span> <span class="k">for</span> <span class="n">obj</span> <span class="ow">in</span> <span class="n">driver</span><span class="o">.</span><span class="n">iterate_container_objects_async</span><span class="p">(</span><span class="n">container</span><span class="p">):</span> - <span class="n">tasks</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">asyncio</span><span class="o">.</span><span class="n">ensure_future</span><span class="p">(</span><span class="n">do_stuff_with_object</span><span class="p">(</span><span class="n">obj</span><span class="p">)))</span> - <span class="n">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">gather</span><span class="p">(</span><span class="o">*</span><span class="n">tasks</span><span class="p">)</span> - -<span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span> -<span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">run</span><span class="p">())</span> -<span class="n">loop</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> -</code></pre></div> -<h2>Design 3 - Initializer with &quot;<em>async</em>&quot; mode</h2> - -<p>This option is similar to 2, except that if a driver is instantiated with &quot;<code>async=True</code>&quot;, -then all driver class methods would return coroutine objects. Internally, it would -patch the Connection class with the AsyncConnection class.</p> - -<p>The downside of this is that all method calls to a driver would need to be awaited or used -by an event loop.</p> -<div class="highlight"><pre><code class="python language-python" data-lang="python"><span class="kn">from</span> <span class="nn">libcloud.storage.providers</span> <span class="kn">import</span> <span class="n">get_driver</span> -<span class="kn">from</span> <span class="nn">libcloud.storage.types</span> <span class="kn">import</span> <span class="n">Provider</span> - -<span class="kn">import</span> <span class="nn">asyncio</span> - -<span class="n">GoogleStorageDriver</span> <span class="o">=</span> <span class="n">get_driver</span><span class="p">(</span><span class="n">Provider</span><span class="o">.</span><span class="n">GOOGLE_STORAGE</span><span class="p">)</span> -<span class="n">driver</span> <span class="o">=</span> <span class="n">GoogleStorageDriver</span><span class="p">(</span><span class="n">key</span><span class="o">=</span><span class="n">KEY</span><span class="p">,</span> <span class="n">secret</span><span class="o">=</span><span class="n">SECRET</span><span class="p">,</span> <span class="n">async</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> - -<span class="k">def</span> <span class="nf">do_stuff_with_object</span><span class="p">(</span><span class="n">obj</span><span class="p">):</span> - <span class="k">print</span><span class="p">(</span><span class="n">obj</span><span class="p">)</span> - -<span class="n">async</span> <span class="k">def</span> <span class="nf">run</span><span class="p">():</span> - <span class="n">tasks</span> <span class="o">=</span> <span class="p">[]</span> - <span class="n">async</span> <span class="k">for</span> <span class="n">container</span> <span class="ow">in</span> <span class="n">driver</span><span class="o">.</span><span class="n">iterate_containers</span><span class="p">():</span> - <span class="n">async</span> <span class="k">for</span> <span class="n">obj</span> <span class="ow">in</span> <span class="n">driver</span><span class="o">.</span><span class="n">iterate_container_objects</span><span class="p">(</span><span class="n">container</span><span class="p">):</span> - <span class="n">tasks</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">asyncio</span><span class="o">.</span><span class="n">ensure_future</span><span class="p">(</span><span class="n">do_stuff_with_object</span><span class="p">(</span><span class="n">obj</span><span class="p">)))</span> - <span class="n">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">gather</span><span class="p">(</span><span class="o">*</span><span class="n">tasks</span><span class="p">)</span> - -<span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span> -<span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">run</span><span class="p">())</span> -<span class="n">loop</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> -</code></pre></div> -<h1>Give us feedback</h1> - -<p>Got a better idea? Have an API or design, the question we&#39;re asking is -&quot;if you wanted to use Libcloud for an async application, what would the code look like?&quot; This helps us design -the API and the implementation details can follow.</p> - -<p>Feel free to comment on the mailing list or on the pull requests, or raise your own pull-request with an API design.</p> -</content> - </entry> - - <entry> <title>Libcloud 2.0.0rc2 released</title> <link href="https://libcloud.apache.org/blog/2017/04/07/libcloud-2-0-0-rc2-released.html"; /> <id>https://libcloud.apache.org/blog/2017/04/07/libcloud-2-0-0-rc2-released.html</id> - <updated>2017-04-07T00:00:00+10:00</updated> + <updated>2017-04-07T00:00:00+02:00</updated> <author> <name>Anthony Shaw</name> </author> @@ -511,7 +158,7 @@ list of people who contributed to this r <title>Libcloud 1.5.0 released</title> <link href="https://libcloud.apache.org/blog/2016/12/29/libcloud-1-5-0-released.html"; /> <id>https://libcloud.apache.org/blog/2016/12/29/libcloud-1-5-0-released.html</id> - <updated>2016-12-29T00:00:00+11:00</updated> + <updated>2016-12-29T00:00:00+01:00</updated> <author> <name>Anthony Shaw</name> </author> @@ -585,7 +232,7 @@ list of people who contributed to this r <title>Libcloud 1.4.0 released</title> <link href="https://libcloud.apache.org/blog/2016/11/27/libcloud-1-4-0-released.html"; /> <id>https://libcloud.apache.org/blog/2016/11/27/libcloud-1-4-0-released.html</id> - <updated>2016-11-27T00:00:00+11:00</updated> + <updated>2016-11-27T00:00:00+01:00</updated> <author> <name>Anthony Shaw</name> </author> @@ -651,7 +298,7 @@ list of people who contributed to this r <title>Libcloud 1.3.0 released</title> <link href="https://libcloud.apache.org/blog/2016/10/14/libcloud-1-3-0-released.html"; /> <id>https://libcloud.apache.org/blog/2016/10/14/libcloud-1-3-0-released.html</id> - <updated>2016-10-14T00:00:00+11:00</updated> + <updated>2016-10-14T00:00:00+02:00</updated> <author> <name>Anthony Shaw</name> </author> @@ -761,7 +408,7 @@ list of people who contributed to this r <title>Libcloud 1.2.1 released</title> <link href="https://libcloud.apache.org/blog/2016/09/23/libcloud-1-2-1-released.html"; /> <id>https://libcloud.apache.org/blog/2016/09/23/libcloud-1-2-1-released.html</id> - <updated>2016-09-23T00:00:00+10:00</updated> + <updated>2016-09-23T00:00:00+02:00</updated> <author> <name>Anthony Shaw</name> </author> @@ -884,7 +531,7 @@ list of people who contributed to this r <title>Libcloud 1.1.0 released</title> <link href="https://libcloud.apache.org/blog/2016/07/07/libcloud-1-1-0-released.html"; /> <id>https://libcloud.apache.org/blog/2016/07/07/libcloud-1-1-0-released.html</id> - <updated>2016-07-07T00:00:00+10:00</updated> + <updated>2016-07-07T00:00:00+02:00</updated> <author> <name>Tomaz Muraus</name> </author> @@ -951,7 +598,7 @@ list of people who contributed to this r <title>Libcloud 1.0.0 released</title> <link href="https://libcloud.apache.org/blog/2016/06/22/libcloud-1-0-0-released.html"; /> <id>https://libcloud.apache.org/blog/2016/06/22/libcloud-1-0-0-released.html</id> - <updated>2016-06-22T00:00:00+10:00</updated> + <updated>2016-06-22T00:00:00+02:00</updated> <author> <name>Anthony Shaw</name> </author> @@ -1241,7 +888,7 @@ list of people who contributed to this r <title>Libcloud 1.0.0-rc2 released</title> <link href="https://libcloud.apache.org/blog/2016/04/11/libcloud-1-0-0-rc2-released.html"; /> <id>https://libcloud.apache.org/blog/2016/04/11/libcloud-1-0-0-rc2-released.html</id> - <updated>2016-04-11T00:00:00+10:00</updated> + <updated>2016-04-11T00:00:00+02:00</updated> <author> <name>Anthony Shaw</name> </author> @@ -1504,4 +1151,163 @@ list of people who contributed to this r </content> </entry> + <entry> + <title>Experimental support for the requests package</title> + <link href="https://libcloud.apache.org/blog/2016/04/06/requests-support.html"; /> + <id>https://libcloud.apache.org/blog/2016/04/06/requests-support.html</id> + <updated>2016-04-06T00:00:00+02:00</updated> + <author> + <name>Anthony Shaw</name> + </author> + <content type="html"><h2>Background</h2> + +<p>I&#39;ve just pushed a branch of the latest version of libcloud using the popular <code>requests</code> package by Kenneth Reitz instead of our home-rolled HTTP client library.</p> + +<p>This article is for both users and developers of libcloud. If you want to give feedback, please join the developer mailing list.</p> + +<h2>Why?</h2> + +<ul> +<li>requests is the defacto standard - it would be in the standard library but agreed against to allow it to develop faster https://github.com/kennethreitz/requests/issues/2424</li> +<li>it works with python 2.6-&gt;3.5</li> +<li>Our SSL experience has a lot to be desired for Windows users, having to download the CA cert package and setting environment variables just to get SSL working</li> +<li>Developers can use requests_mock for deeper integration testing</li> +<li>less code to maintain</li> +<li>the role of libcloud is for cloud abstraction, we provide no value in writing and maintaining our own HTTP client library</li> +</ul> + +<h2>Benefits of requests</h2> + +<p>There are a number of benefits to having a requests package</p> + +<ul> +<li>The client library code is smaller, leaner and simpler.</li> +<li>Requests has built in decompression support, we no longer need to support this</li> +<li>Requests has built in RAW download, upload support, helping with our storage drivers</li> +</ul> + +<h2>Implications of the change</h2> + +<ul> +<li>There are no longer 2 classes (<code>LibcloudHTTPSConnection</code> and <code>LibcloudHTTPConnection</code>) to be provided to each driver, they are now 1 class - <code>LibcloudConnection</code>. You probably won&#39;t notice this because it is a property of the <code>Connection</code> class, but +if you are developing or extending functionality then it is implicated.</li> +<li>Unit tests will look slightly different (see below)</li> +<li>This change broke 4200 unit tests (out of 6340)! I&#39;ve since fixed them all since they were coupled to the original implementation, but now I don&#39;t know if all of tests are valid.</li> +</ul> + +<h2>Testing with requests</h2> + +<p>Unit tests that were written like this:</p> +<div class="highlight"><pre><code class="python language-python" data-lang="python"><span class="k">class</span> <span class="nc">DigitalOceanTests</span><span class="p">(</span><span class="n">LibcloudTestCase</span><span class="p">):</span> + + <span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> + <span class="n">DigitalOceanBaseDriver</span><span class="o">.</span><span class="n">connectionCls</span><span class="o">.</span><span class="n">conn_classes</span> <span class="o">=</span> \ + <span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="n">DigitalOceanMockHttp</span><span class="p">)</span> + <span class="n">DigitalOceanMockHttp</span><span class="o">.</span><span class="n">type</span> <span class="o">=</span> <span class="bp">None</span> + <span class="bp">self</span><span class="o">.</span><span class="n">driver</span> <span class="o">=</span> <span class="n">DigitalOceanBaseDriver</span><span class="p">(</span><span class="o">*</span><span class="n">DIGITALOCEAN_v1_PARAMS</span><span class="p">)</span> +</code></pre></div> +<p>Because of the change have been modified to (I updated all of them - so this is just for future reference)</p> +<div class="highlight"><pre><code class="python language-python" data-lang="python"><span class="k">class</span> <span class="nc">DigitalOceanTests</span><span class="p">(</span><span class="n">LibcloudTestCase</span><span class="p">):</span> + + <span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> + <span class="n">DigitalOceanBaseDriver</span><span class="o">.</span><span class="n">connectionCls</span><span class="o">.</span><span class="n">conn_class</span> <span class="o">=</span> <span class="n">DigitalOceanMockHttp</span> + <span class="n">DigitalOceanMockHttp</span><span class="o">.</span><span class="n">type</span> <span class="o">=</span> <span class="bp">None</span> + <span class="bp">self</span><span class="o">.</span><span class="n">driver</span> <span class="o">=</span> <span class="n">DigitalOceanBaseDriver</span><span class="p">(</span><span class="o">*</span><span class="n">DIGITALOCEAN_v1_PARAMS</span><span class="p">)</span> +</code></pre></div> +<h1>Check it out!</h1> + +<p>The package is on my personal apache site, you can download it and install it in a virtualenv for testing.</p> + +<p><code>pip install -e http://people.apache.org/~anthonyshaw/libcloud/1.0.0-rc2-requests/apache-libcloud-1.0.0-rc2-requests.zip@feature#egg=apache-libcloud</code></p> + +<p>The hashes are my <a href="http://people.apache.org/%7Eanthonyshaw/libcloud/1.0.0-rc2-requests/">apache space</a></p> + +<p>Have a look at the <a href="https://github.com/apache/libcloud/pull/728/files">PR and the change set</a> for a list of changes</p> + +<h1>What might break?</h1> + +<p>What I&#39;m really looking for is for users of Libcloud to take 15 minutes, an existing (working) libcloud script, install this package in a virtualenv and just validate +that there are no regression bugs with this change.</p> + +<p>I&#39;m particularly sceptical about the storage drivers.</p> + +<p>Once we have enough community feedback, we will propose a vote to merge this into trunk for future release.</p> + +<h2>Credit</h2> + +<p>Credit to dz0ny on IRC for contributing some of the requests patch.</p> +</content> + </entry> + + <entry> + <title>New compute drivers and deprecated drivers in 1.0</title> + <link href="https://libcloud.apache.org/blog/2016/02/16/new-drivers-deprecated-drivers.html"; /> + <id>https://libcloud.apache.org/blog/2016/02/16/new-drivers-deprecated-drivers.html</id> + <updated>2016-02-16T00:00:00+01:00</updated> + <author> + <name>Anthony Shaw</name> + </author> + <content type="html"><p>With Libcloud 1.0.0 around the corner, it&#39;s time to have a spring clean of the compute drivers. Granted, it&#39;s not spring everywhere -actually +I&#39;m writing from Sydney, Australia where it&#39;s definitely summer.</p> + +<p>Looking at the 52 providers in the 0.21.0 release, I have identified 5 providers that are no longer available or open.</p> + +<ul> +<li>CloudFrames + +<ul> +<li>Looks dead - website doesn&#39;t work, can&#39;t see any references to this online.</li> +<li><code>libcloud.compute.drivers.cloudframes</code></li> +</ul></li> +<li>HP Public Cloud (Helion) + +<ul> +<li><a href="https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2566">Shut down on 31st of January 2016</a></li> +<li><code>libcloud.compute.drivers.hpcloud</code></li> +</ul></li> +<li>IBM SmartCloud Enterprise + +<ul> +<li><a href="http://www.theregister.co.uk/2013/10/31/ibm_to_shutter_smartcloud_enterprise_moves_customers_to_softlayer/">Shut down 2014, customers instructed to move to SoftLayer</a></li> +<li><code>libcloud.compute.drivers.ibm_sce</code></li> +</ul></li> +<li>Ninefold + +<ul> +<li><a href="http://www.itnews.com.au/news/ninefold-to-shut-down-411312">Ninefold shut down in January 2016</a></li> +<li><code>libcloud.compute.drivers.ninefold</code></li> +</ul></li> +<li>Opsource + +<ul> +<li>Dimension Data acquired OpSource in 2012, the Opsource driver is succeeded by the Dimension Data driver.</li> +<li><code>libcloud.compute.drivers.opsource</code></li> +</ul></li> +</ul> + +<h3>Handling deprecated drivers</h3> + +<p>For 1.0.0, we need a clean and user-friendly way of handling deprecated drivers as well as keeping the repository clean from legacy code.</p> + +<p>The most obvious implementation is that calls to <code>get_driver(Provider.NINEFOLD)</code> as an example will return a user error message saying +this provider is no longer supported with a link to a new article and an alternative solution.</p> + +<p>Currently, users trying to instantiate a HPE public cloud driver for example will get a connection error, which is not user friendly.</p> + +<h3>New compute drivers in 1.0.0-pre2</h3> + +<p>The upcoming release, so currently available in trunk contains some new compute drivers.</p> + +<ul> +<li><a href="http://libcloud.readthedocs.org/en/latest/compute/drivers/ntta.html">NTT America Public Cloud</a></li> +<li><a href="http://libcloud.readthedocs.org/en/latest/compute/drivers/internetsolutions.html">Internet Solutions Public Cloud</a></li> +<li><a href="http://libcloud.readthedocs.org/en/latest/compute/drivers/medone.html">Med-1 Public Cloud</a></li> +<li><a href="http://libcloud.readthedocs.org/en/latest/compute/drivers/indosat.html">Indosat Cloud</a></li> +<li><a href="http://libcloud.readthedocs.org/en/latest/compute/drivers/bsnl.html">BSNL IDC Cloud</a></li> +</ul> + +<p>Full change log can be found at <a href="https://github.com/apache/libcloud/blob/trunk/CHANGES.rst">here</a>.</p> +</content> + </entry> + </feed>
