[GitHub] [libcloud] michaelperel commented on a change in pull request #1278: [LIBCLOUD-1037] Add Azurite support for Azure Blob Storage driver
michaelperel commented on a change in pull request #1278: [LIBCLOUD-1037] Add
Azurite support for Azure Blob Storage driver
URL: https://github.com/apache/libcloud/pull/1278#discussion_r288337437
##
File path: libcloud/storage/drivers/azure_blobs.py
##
@@ -214,8 +237,9 @@ def _xml_to_container(self, node):
'meta_data': {}
}
-for meta in list(metadata):
-extra['meta_data'][meta.tag] = meta.text
+if metadata:
Review comment:
Same as above, consider "is not None"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [libcloud] michaelperel commented on a change in pull request #1278: [LIBCLOUD-1037] Add Azurite support for Azure Blob Storage driver
michaelperel commented on a change in pull request #1278: [LIBCLOUD-1037] Add Azurite support for Azure Blob Storage driver URL: https://github.com/apache/libcloud/pull/1278#discussion_r288337351 ## File path: libcloud/storage/drivers/azure_blobs.py ## @@ -184,6 +205,8 @@ def _ex_connection_class_kwargs(self): # host argument has precedence if not self._host_argument_set: result['host'] = '%s.%s' % (self.key, AZURE_STORAGE_HOST_SUFFIX) +else: +result['account_prefix'] = self.key Review comment: consider documenting the keys of the dictionary, as [per the contributing guide](https://libcloud.readthedocs.io/en/latest/development.html#when-returning-a-dictionary-document-its-structure) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [libcloud] michaelperel commented on a change in pull request #1278: [LIBCLOUD-1037] Add Azurite support for Azure Blob Storage driver
michaelperel commented on a change in pull request #1278: [LIBCLOUD-1037] Add
Azurite support for Azure Blob Storage driver
URL: https://github.com/apache/libcloud/pull/1278#discussion_r288337598
##
File path: libcloud/storage/drivers/azure_blobs.py
##
@@ -214,8 +237,9 @@ def _xml_to_container(self, node):
'meta_data': {}
}
-for meta in list(metadata):
-extra['meta_data'][meta.tag] = meta.text
+if metadata:
+for meta in list(metadata):
+extra['meta_data'][meta.tag] = meta.text
Review comment:
is meta.tag guaranteed to be not None?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [libcloud] michaelperel commented on a change in pull request #1278: [LIBCLOUD-1037] Add Azurite support for Azure Blob Storage driver
michaelperel commented on a change in pull request #1278: [LIBCLOUD-1037] Add
Azurite support for Azure Blob Storage driver
URL: https://github.com/apache/libcloud/pull/1278#discussion_r288336908
##
File path: libcloud/storage/drivers/azure_blobs.py
##
@@ -156,6 +156,27 @@ class AzureBlobsConnection(AzureConnection):
"""
Represents a single connection to Azure Blobs
"""
+def __init__(self, *args, **kwargs):
+self.account_prefix = kwargs.pop('account_prefix', None)
+super(AzureBlobsConnection, self).__init__(*args, **kwargs)
+
+def morph_action_hook(self, action):
+action = super(AzureBlobsConnection, self).morph_action_hook(action)
+
+# The main Azure Blob Storage service uses the hostname to distinguish
+# between accounts, e.g. `theaccount.blob.core.windows.net`.
+# However, some custom deployments of the service such as the official
+# emulator (https://github.com/Azure/Azurite) instead use a URL prefix
+# such as `/theaccount`. The use-case is drivers instantiated via
+# `host=somewhere.tld` and `key='theaccount'`.
+# If a user wants to specify a custom host without an account prefix,
+# e.g. for use-cases where the custom host implements an auditing
+# proxy or similar, they can instantiate the driver with
+# `host=theaccount.somewhere.tld` and `key=''`.
+if self.account_prefix:
Review comment:
consider "is not None", as [per the contributing
guide](https://libcloud.readthedocs.io/en/latest/development.html#prefer-to-use-is-not-none-when-checking-if-a-variable-is-provided-or-defined)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [libcloud] michaelperel commented on a change in pull request #1278: [LIBCLOUD-1037] Add Azurite support for Azure Blob Storage driver
michaelperel commented on a change in pull request #1278: [LIBCLOUD-1037] Add
Azurite support for Azure Blob Storage driver
URL: https://github.com/apache/libcloud/pull/1278#discussion_r288337851
##
File path: libcloud/storage/drivers/azure_blobs.py
##
@@ -299,8 +323,9 @@ def _xml_to_object(self, container, blob):
extra['md5_hash'] = value
meta_data = {}
-for meta in list(metadata):
-meta_data[meta.tag] = meta.text
+if metadata:
+for meta in list(metadata):
+meta_data[meta.tag] = meta.text
Review comment:
same as above, is meta.tag guaranteed to be not None?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [libcloud] michaelperel commented on a change in pull request #1278: [LIBCLOUD-1037] Add Azurite support for Azure Blob Storage driver
michaelperel commented on a change in pull request #1278: [LIBCLOUD-1037] Add
Azurite support for Azure Blob Storage driver
URL: https://github.com/apache/libcloud/pull/1278#discussion_r288337744
##
File path: libcloud/storage/drivers/azure_blobs.py
##
@@ -299,8 +323,9 @@ def _xml_to_object(self, container, blob):
extra['md5_hash'] = value
meta_data = {}
-for meta in list(metadata):
-meta_data[meta.tag] = meta.text
+if metadata:
Review comment:
Same as above, consider "is not None"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[libcloud] annotated tag v2.5.0-tentative created (now c1146c7)
This is an automated email from the ASF dual-hosted git repository. tomaz pushed a change to annotated tag v2.5.0-tentative in repository https://gitbox.apache.org/repos/asf/libcloud.git. at c1146c7 (tag) tagging 51fe8e7564d75a84d06268529855d0d20406ccf8 (commit) replaces v2.4.0 by Tomaz Muraus on Tue May 28 22:50:15 2019 +0200 - Log - Tag for v2.5.0-tentative release. -BEGIN PGP SIGNATURE- iQKTBAABCgB9FiEEmXgo3GL3Wc6hidZeLAdUss4GkvMFAlztnypfFIAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDk5 NzgyOERDNjJGNzU5Q0VBMTg5RDY1RTJDMDc1NEIyQ0UwNjkyRjMACgkQLAdUss4G kvPRwg/+KI46ypCHZmy17auoFm+Xtp5IqNgVoWkfVVrTB/81D1jLDg7+2rXxFtxz AJzFEtFN/8zTxL6DIo/rMLWNwNcd3QBGDfhAKX2J154G9E7hKat1QT3fA82a5liF nhqxRK0VMJPUQy6iZgDTekOFQwNZAL2r7v2aUnF+pVeD8TeIQChpxcKclrFrop5a uz7NPcvMejsKuKQHBAqSMHzOmtYdiMhr5St4Nmi7u1CE6hh1m/UEqFrvea5YGpWq xgln1dq1eHJ9l8z8sY1ddU3369GSJSeI+n5d2CJ4HiDL4okLECyhuXefbHexkzp1 wqJ9tkcKbDyPrHkq3QsorS48I2CMqtESD9BEXwypW8x2OCgNZUeoYACndnu1Lt/2 +RI0+/JNHm+iySOS7qAlVb7dAcgdxi/nNoCOmMwHf2r3uNR2gFWXixiBQwEfxIMq j4GFmOMGzsbeaYuK1yMwoyFrPoxXDmOayNNmnGSVhsYo5SY+5OunOOzeyhO3tiBY /PnFd5eCbB3YI4fA57sQFJaxtQf/e0U4CA4Q+TPBM+K7+l8TRxWBmaFQHaa95ENT ngEUSu/iz0ME52yYHX+JlhsCbqfnYwmKa2UaBe0BxbDJfAmyCE/Uqf+uiFV0vq1P qkGvnyS2eWwvCeZ6CbDrQ3mSq7tJmjl65ZL1RSWqNLgN+k0PL5A= =4i/c -END PGP SIGNATURE- --- No new revisions were added by this update.
[libcloud] 02/04: Update committer guide.
This is an automated email from the ASF dual-hosted git repository. tomaz pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/libcloud.git commit bbfa4186703c9d63b7ef4326ab784f5470091464 Author: Tomaz Muraus AuthorDate: Tue May 28 22:32:57 2019 +0200 Update committer guide. --- docs/committer_guide.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/committer_guide.rst b/docs/committer_guide.rst index 52a39df..c1c3bbc 100644 --- a/docs/committer_guide.rst +++ b/docs/committer_guide.rst @@ -105,7 +105,7 @@ preparing a release. * Make sure tests pass on all the supported Python versions (``tox``) * Remove the ``tox`` directory with ``rm -rf .tox`` * Remove the _secrets_ file with ``rm libcloud/test/secrets.py`` -* Remove leftover builds from previous releases. ``rm -f dist/apache*`` +* Remove leftover builds from previous releases. ``rm -f dist/apache* ; rm -rf apache_libcloud.egg-info`` 2. Update JIRA ~~
[libcloud] 04/04: Update version for v2.5.0 release.
This is an automated email from the ASF dual-hosted git repository. tomaz pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/libcloud.git commit 51fe8e7564d75a84d06268529855d0d20406ccf8 Author: Tomaz Muraus AuthorDate: Tue May 28 22:35:09 2019 +0200 Update version for v2.5.0 release. --- CHANGES.rst | 4 ++-- docs/conf.py | 4 ++-- libcloud/__init__.py | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/CHANGES.rst b/CHANGES.rst index 2e72c55..88dba31 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -1,8 +1,8 @@ Changelog = -Changes in Apache Libcloud in development -- +Changes in Apache Libcloud 2.5.0 + General ~~~ diff --git a/docs/conf.py b/docs/conf.py index 4fea7bb..e3457fd 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -73,9 +73,9 @@ html_show_sphinx = False # built documents. # # The short X.Y version. -version = '2.4.0' +version = '2.5.0' # The full version, including alpha/beta/rc tags. -release = '2.4.0' +release = '2.5.0' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. diff --git a/libcloud/__init__.py b/libcloud/__init__.py index e1c4064..412de83 100644 --- a/libcloud/__init__.py +++ b/libcloud/__init__.py @@ -40,7 +40,7 @@ __all__ = [ 'enable_debug' ] -__version__ = '2.4.1-dev' +__version__ = '2.5.0' def enable_debug(fo):
[libcloud] 01/04: Fix typo.
This is an automated email from the ASF dual-hosted git repository. tomaz pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/libcloud.git commit 2b878955cf36d17dbac29ba5dcc39be16633fa44 Author: Tomaz Muraus AuthorDate: Tue May 28 22:29:01 2019 +0200 Fix typo. --- docs/committer_guide.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/committer_guide.rst b/docs/committer_guide.rst index d6ae14e..52a39df 100644 --- a/docs/committer_guide.rst +++ b/docs/committer_guide.rst @@ -97,7 +97,7 @@ preparing a release. * Make sure ``CHANGES`` file is up to date * Make sure ``__version__`` string in ``libcloud/__init__.py`` is up to date * Make sure ``version`` and ``release`` in ``docs/conf.py`` are up to date -* Update constants: ``python contrib/scrap-ec2-sizes.py > libcloud/compute/constants.py`` +* Update constants: ``python contrib/scrape-ec2-sizes.py > libcloud/compute/constants.py`` 1. Pre-release check list ~
[libcloud] branch trunk updated (44b4f2b -> 51fe8e7)
This is an automated email from the ASF dual-hosted git repository. tomaz pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/libcloud.git. from 44b4f2b Merge pull request #1291 from Kami/enable_pypy_builds new 2b87895 Fix typo. new bbfa418 Update committer guide. new b36efbe Update EC2 related instance constants. new 51fe8e7 Update version for v2.5.0 release. The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.rst | 4 +- docs/committer_guide.rst | 4 +- docs/conf.py | 4 +- libcloud/__init__.py | 2 +- libcloud/compute/constants.py | 11776 +++- 5 files changed, 6850 insertions(+), 4940 deletions(-)
[GitHub] [libcloud] codecov-io edited a comment on issue #1278: [LIBCLOUD-1037] Add Azurite support for Azure Blob Storage driver
codecov-io edited a comment on issue #1278: [LIBCLOUD-1037] Add Azurite support for Azure Blob Storage driver URL: https://github.com/apache/libcloud/pull/1278#issuecomment-46762 # [Codecov](https://codecov.io/gh/apache/libcloud/pull/1278?src=pr=h1) Report > Merging [#1278](https://codecov.io/gh/apache/libcloud/pull/1278?src=pr=desc) into [trunk](https://codecov.io/gh/apache/libcloud/commit/44b4f2beaebee7a478eab40a73fc73506a75702d?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `84.61%`. [](https://codecov.io/gh/apache/libcloud/pull/1278?src=pr=tree) ```diff @@Coverage Diff @@ ##trunk#1278 +/- ## == + Coverage 85.95% 85.95% +<.01% == Files 359 359 Lines 7391173932 +21 Branches 6705 6709 +4 == + Hits6352863548 +20 - Misses 7700 7701 +1 Partials 2683 2683 ``` | [Impacted Files](https://codecov.io/gh/apache/libcloud/pull/1278?src=pr=tree) | Coverage Δ | | |---|---|---| | [libcloud/storage/drivers/azure\_blobs.py](https://codecov.io/gh/apache/libcloud/pull/1278/diff?src=pr=tree#diff-bGliY2xvdWQvc3RvcmFnZS9kcml2ZXJzL2F6dXJlX2Jsb2JzLnB5) | `69.42% <81.25%> (+0.99%)` | :arrow_up: | | [libcloud/test/storage/test\_azure\_blobs.py](https://codecov.io/gh/apache/libcloud/pull/1278/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9zdG9yYWdlL3Rlc3RfYXp1cmVfYmxvYnMucHk=) | `92.04% <90%> (-0.04%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/libcloud/pull/1278?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/libcloud/pull/1278?src=pr=footer). Last update [44b4f2b...4200af8](https://codecov.io/gh/apache/libcloud/pull/1278?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [libcloud] Kami commented on issue #1291: Re-enable PyPy builds
Kami commented on issue #1291: Re-enable PyPy builds URL: https://github.com/apache/libcloud/pull/1291#issuecomment-496646937 Confirmed it's working - https://travis-ci.org/apache/libcloud/jobs/538387079, https://travis-ci.org/apache/libcloud/jobs/538387080. It does increase overall / aggregate build time, but it's still shorter than it was in the past because of the timing improvements in #1290. Will go ahead and merge it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [libcloud] Kami merged pull request #1291: Re-enable PyPy builds
Kami merged pull request #1291: Re-enable PyPy builds URL: https://github.com/apache/libcloud/pull/1291 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[libcloud] branch trunk updated (b23b236 -> 44b4f2b)
This is an automated email from the ASF dual-hosted git repository. tomaz pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/libcloud.git. from b23b236 Merge pull request #1290 from Kami/test_warning_fixes new d3adba7 Re-enable PyPy builds. new 3395de3 Also run pypy3 tests using python setup.py test for consistency with other test targets. new 44b4f2b Merge pull request #1291 from Kami/enable_pypy_builds The 6368 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .travis.yml | 4 ++-- tox.ini | 4 2 files changed, 2 insertions(+), 6 deletions(-)
[GitHub] [libcloud] codecov-io edited a comment on issue #1291: Re-enable PyPy builds
codecov-io edited a comment on issue #1291: Re-enable PyPy builds URL: https://github.com/apache/libcloud/pull/1291#issuecomment-496643309 # [Codecov](https://codecov.io/gh/apache/libcloud/pull/1291?src=pr=h1) Report > Merging [#1291](https://codecov.io/gh/apache/libcloud/pull/1291?src=pr=desc) into [trunk](https://codecov.io/gh/apache/libcloud/commit/b23b236c3247a632572b00953666f6c2c3da079f?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/libcloud/pull/1291?src=pr=tree) ```diff @@Coverage Diff @@ ##trunk#1291 +/- ## == - Coverage 85.95% 85.95% -0.01% == Files 359 359 Lines 7391173911 Branches 6705 6705 == - Hits6353263530 -2 - Misses 7697 7698 +1 - Partials 2682 2683 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/libcloud/pull/1291?src=pr=tree) | Coverage Δ | | |---|---|---| | [libcloud/test/compute/test\_upcloud.py](https://codecov.io/gh/apache/libcloud/pull/1291/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3RfdXBjbG91ZC5weQ==) | `90.06% <0%> (-1.33%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/libcloud/pull/1291?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/libcloud/pull/1291?src=pr=footer). Last update [b23b236...3395de3](https://codecov.io/gh/apache/libcloud/pull/1291?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [libcloud] codecov-io commented on issue #1291: Re-enable PyPy builds
codecov-io commented on issue #1291: Re-enable PyPy builds URL: https://github.com/apache/libcloud/pull/1291#issuecomment-496643309 # [Codecov](https://codecov.io/gh/apache/libcloud/pull/1291?src=pr=h1) Report > Merging [#1291](https://codecov.io/gh/apache/libcloud/pull/1291?src=pr=desc) into [trunk](https://codecov.io/gh/apache/libcloud/commit/b23b236c3247a632572b00953666f6c2c3da079f?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/libcloud/pull/1291?src=pr=tree) ```diff @@Coverage Diff @@ ##trunk#1291 +/- ## == - Coverage 85.95% 85.95% -0.01% == Files 359 359 Lines 7391173911 Branches 6705 6705 == - Hits6353263530 -2 - Misses 7697 7699 +2 Partials 2682 2682 ``` | [Impacted Files](https://codecov.io/gh/apache/libcloud/pull/1291?src=pr=tree) | Coverage Δ | | |---|---|---| | [libcloud/test/compute/test\_ec2.py](https://codecov.io/gh/apache/libcloud/pull/1291/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3RfZWMyLnB5) | `97.74% <0%> (-0.17%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/libcloud/pull/1291?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/libcloud/pull/1291?src=pr=footer). Last update [b23b236...3395de3](https://codecov.io/gh/apache/libcloud/pull/1291?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [libcloud] Kami opened a new pull request #1291: Re-enable PyPy builds
Kami opened a new pull request #1291: Re-enable PyPy builds URL: https://github.com/apache/libcloud/pull/1291 This pull request will try to re-enable builds under PyPy. It looks like they were disabled at some point. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [libcloud] Kami edited a comment on issue #1290: Various code fixes and improvements
Kami edited a comment on issue #1290: Various code fixes and improvements URL: https://github.com/apache/libcloud/pull/1290#issuecomment-496620751 Tests run time is now down from ~160 to ~110 seconds. Those small sleep optimizations added up. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[libcloud] branch trunk updated (f40b2c4 -> b23b236)
This is an automated email from the ASF dual-hosted git repository. tomaz pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/libcloud.git. from f40b2c4 Fix Python 3 compatibility. new b561cc5 Don't use deprecated method name. new 558b283 Make sure the file handles are closed. new aa1de2c Make sure that ChunkStreamReader used by the CloudFiles driver correctly closes the file description if the iterator is not fully exhausted or if the iterator is never read from. new d868b3f Add changelog entry. new b22da71 Update more code to ensure that open file handles are correctly closed. new d890f81 Remove deprecation warning which appear when running tests under Python 3 by using Python 3 recommended unit test assertion methods. new b409a47 Importlib under Python >= 3.5. new 64f514f Use newer non-deprecated assertion and other methods on newer Python 3 versions to avoid deprecation warnings during test runs. new b3fec66 Use list(elem) instead of elem.getchildren(). new cce310c Ignore UserWarning warnings when running tests. new 15fc7fa Fix lint. new 71e8d61 Print 10 slowest tests during pytest test runs. new afbcba0 Speed up various tests by using smaller poll / sleep intervals. new b4c2c92 Speed up more tests. new 868de3d Add changelog entries. new 972cd7a Speed up more tests. new dc16186 Fix more warnings. new f110711 Try caching a tox directory. new 5d5bdc3 Use -vvv flag so we see when dependency cache is reused, etc. new 40f8281 Speed up more tests. new b99cd17 Reduce verbosity. new 5b0b64a Reduce timeout to speed up tests. new b23b236 Merge pull request #1290 from Kami/test_warning_fixes The 6365 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .travis.yml | 1 + CHANGES.rst | 15 +++ libcloud/__init__.py | 11 + libcloud/common/brightbox.py | 5 +-- libcloud/common/durabledns.py| 26 +-- libcloud/common/zonomi.py| 6 +-- libcloud/compute/drivers/ec2.py | 2 +- libcloud/compute/drivers/ecp.py | 4 +- libcloud/compute/drivers/vcloud.py | 2 +- libcloud/compute/drivers/voxel.py| 4 +- libcloud/compute/drivers/vultr.py| 2 +- libcloud/dns/drivers/durabledns.py | 20 - libcloud/loadbalancer/drivers/nttcis.py | 15 +-- libcloud/storage/drivers/azure_blobs.py | 4 +- libcloud/storage/drivers/cloudfiles.py | 13 +- libcloud/test/common/test_openstack_identity.py | 39 + libcloud/test/common/test_retry_limit.py | 4 +- libcloud/test/common/test_upcloud.py | 8 ++-- libcloud/test/compute/test_cloudsigma_v2_0.py| 45 +-- libcloud/test/compute/test_cloudstack.py | 11 ++--- libcloud/test/compute/test_deployment.py | 37 libcloud/test/compute/test_digitalocean_v2.py| 9 ++-- libcloud/test/compute/test_dimensiondata_v2_3.py | 9 ++-- libcloud/test/compute/test_dimensiondata_v2_4.py | 9 ++-- libcloud/test/compute/test_ec2.py| 4 +- libcloud/test/compute/test_elasticstack.py | 5 ++- libcloud/test/compute/test_gce.py| 9 +++- libcloud/test/compute/test_joyent.py | 9 ++-- libcloud/test/compute/test_nttcis.py | 9 ++-- libcloud/test/compute/test_oneandone.py | 2 +- libcloud/test/compute/test_openstack.py | 13 +- libcloud/test/compute/test_scaleway.py | 13 +++--- libcloud/test/compute/test_ssh_client.py | 9 ++-- libcloud/test/compute/test_upcloud.py| 18 libcloud/test/dns/test_auroradns.py | 30 ++--- libcloud/test/dns/test_base.py | 24 +- libcloud/test/dns/test_linode.py | 8 ++-- libcloud/test/dns/test_nfsn.py | 22 +- libcloud/test/loadbalancer/test_alb.py | 4 +- libcloud/test/loadbalancer/test_cloudstack.py| 5 ++- libcloud/test/loadbalancer/test_elb.py | 4 +- libcloud/test/storage/test_backblaze_b2.py | 13 +++--- libcloud/test/storage/test_base.py | 17 +++ libcloud/test/test_connection.py | 37 libcloud/test/test_http.py | 5 ++- libcloud/utils/publickey.py | 4 +- libcloud/utils/py3.py| 56 +++- pytest.ini
[GitHub] [libcloud] Kami merged pull request #1290: Various code fixes and improvements
Kami merged pull request #1290: Various code fixes and improvements URL: https://github.com/apache/libcloud/pull/1290 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [libcloud] c-w commented on issue #1278: [LIBCLOUD-1037] Add Azurite support for Azure Blob Storage driver
c-w commented on issue #1278: [LIBCLOUD-1037] Add Azurite support for Azure Blob Storage driver URL: https://github.com/apache/libcloud/pull/1278#issuecomment-496639381 @Kami Any chance you could review this? It would be great to land Azurite and IoT Edge Storage support in libcloud since this has been a requirement on a couple of my recent projects. To increase confidence in the change, you can take a look at the [CI run against Azure Storage](https://clewolff.visualstudio.com/libcloud-tests/_build/results?buildId=51) which proves that there are no regressions and the [CI run against IoT Edge Storage and Azurite](https://travis-ci.org/CatalystCode/appinsights-on-premises/builds/524048143) which proves the new functionality. Also adding @michaelperel and @cicorias to review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [libcloud] codecov-io edited a comment on issue #1290: Various code fixes and improvements
codecov-io edited a comment on issue #1290: Various code fixes and improvements URL: https://github.com/apache/libcloud/pull/1290#issuecomment-496611671 # [Codecov](https://codecov.io/gh/apache/libcloud/pull/1290?src=pr=h1) Report > Merging [#1290](https://codecov.io/gh/apache/libcloud/pull/1290?src=pr=desc) into [trunk](https://codecov.io/gh/apache/libcloud/commit/f40b2c4d2e6531cc4b52972b028f56b17f90cc27?src=pr=desc) will **decrease** coverage by `0.03%`. > The diff coverage is `79.77%`. [](https://codecov.io/gh/apache/libcloud/pull/1290?src=pr=tree) ```diff @@Coverage Diff @@ ##trunk#1290 +/- ## == - Coverage 85.98% 85.95% -0.04% == Files 359 359 Lines 7386773911 +44 Branches 6700 6705 +5 == + Hits6351863530 +12 - Misses 7670 7699 +29 - Partials 2679 2682 +3 ``` | [Impacted Files](https://codecov.io/gh/apache/libcloud/pull/1290?src=pr=tree) | Coverage Δ | | |---|---|---| | [libcloud/compute/drivers/vcloud.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvY29tcHV0ZS9kcml2ZXJzL3ZjbG91ZC5weQ==) | `82.12% <ø> (ø)` | :arrow_up: | | [libcloud/test/compute/test\_openstack.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3Rfb3BlbnN0YWNrLnB5) | `94.89% <ø> (ø)` | :arrow_up: | | [libcloud/test/compute/test\_dimensiondata\_v2\_4.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3RfZGltZW5zaW9uZGF0YV92Ml80LnB5) | `88.02% <ø> (ø)` | :arrow_up: | | [libcloud/test/compute/test\_gce.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3RfZ2NlLnB5) | `97.74% <ø> (ø)` | :arrow_up: | | [libcloud/test/compute/test\_dimensiondata\_v2\_3.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3RfZGltZW5zaW9uZGF0YV92Ml8zLnB5) | `88.01% <ø> (ø)` | :arrow_up: | | [libcloud/test/compute/test\_nttcis.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3RfbnR0Y2lzLnB5) | `86.18% <ø> (ø)` | :arrow_up: | | [libcloud/test/compute/test\_ssh\_client.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3Rfc3NoX2NsaWVudC5weQ==) | `30.89% <0%> (ø)` | :arrow_up: | | [libcloud/common/brightbox.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvY29tbW9uL2JyaWdodGJveC5weQ==) | `53.65% <0%> (ø)` | :arrow_up: | | [libcloud/compute/drivers/voxel.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvY29tcHV0ZS9kcml2ZXJzL3ZveGVsLnB5) | `84.12% <0%> (-3.18%)` | :arrow_down: | | [libcloud/test/test\_connection.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC90ZXN0X2Nvbm5lY3Rpb24ucHk=) | `99.59% <100%> (ø)` | :arrow_up: | | ... and [34 more](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/libcloud/pull/1290?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/libcloud/pull/1290?src=pr=footer). Last update [f40b2c4...f110711](https://codecov.io/gh/apache/libcloud/pull/1290?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [libcloud] Kami commented on issue #1290: Various code fixes and improvements
Kami commented on issue #1290: Various code fixes and improvements URL: https://github.com/apache/libcloud/pull/1290#issuecomment-496620751 Test run time is now down from ~160 to ~120 seconds. Those small sleep optimizations added up. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [libcloud] codecov-io edited a comment on issue #1290: Various code fixes and improvements
codecov-io edited a comment on issue #1290: Various code fixes and improvements URL: https://github.com/apache/libcloud/pull/1290#issuecomment-496611671 # [Codecov](https://codecov.io/gh/apache/libcloud/pull/1290?src=pr=h1) Report > Merging [#1290](https://codecov.io/gh/apache/libcloud/pull/1290?src=pr=desc) into [trunk](https://codecov.io/gh/apache/libcloud/commit/f40b2c4d2e6531cc4b52972b028f56b17f90cc27?src=pr=desc) will **decrease** coverage by `0.03%`. > The diff coverage is `80.11%`. [](https://codecov.io/gh/apache/libcloud/pull/1290?src=pr=tree) ```diff @@Coverage Diff @@ ##trunk#1290 +/- ## == - Coverage 85.98% 85.95% -0.04% == Files 359 359 Lines 7386773911 +44 Branches 6700 6705 +5 == + Hits6351863532 +14 - Misses 7670 7698 +28 - Partials 2679 2681 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/libcloud/pull/1290?src=pr=tree) | Coverage Δ | | |---|---|---| | [libcloud/test/compute/test\_gce.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3RfZ2NlLnB5) | `97.74% <ø> (ø)` | :arrow_up: | | [libcloud/test/compute/test\_openstack.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3Rfb3BlbnN0YWNrLnB5) | `94.89% <ø> (ø)` | :arrow_up: | | [libcloud/test/compute/test\_dimensiondata\_v2\_4.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3RfZGltZW5zaW9uZGF0YV92Ml80LnB5) | `88.02% <ø> (ø)` | :arrow_up: | | [libcloud/test/compute/test\_dimensiondata\_v2\_3.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3RfZGltZW5zaW9uZGF0YV92Ml8zLnB5) | `88.01% <ø> (ø)` | :arrow_up: | | [libcloud/common/brightbox.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvY29tbW9uL2JyaWdodGJveC5weQ==) | `53.65% <0%> (ø)` | :arrow_up: | | [libcloud/test/compute/test\_ssh\_client.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3Rfc3NoX2NsaWVudC5weQ==) | `30.89% <0%> (ø)` | :arrow_up: | | [libcloud/test/common/test\_upcloud.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21tb24vdGVzdF91cGNsb3VkLnB5) | `100% <100%> (ø)` | :arrow_up: | | [libcloud/test/compute/test\_digitalocean\_v2.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3RfZGlnaXRhbG9jZWFuX3YyLnB5) | `99.45% <100%> (ø)` | :arrow_up: | | [libcloud/test/test\_connection.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC90ZXN0X2Nvbm5lY3Rpb24ucHk=) | `99.59% <100%> (ø)` | :arrow_up: | | [libcloud/test/loadbalancer/test\_elb.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9sb2FkYmFsYW5jZXIvdGVzdF9lbGIucHk=) | `99.21% <100%> (ø)` | :arrow_up: | | ... and [29 more](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/libcloud/pull/1290?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/libcloud/pull/1290?src=pr=footer). Last update [f40b2c4...972cd7a](https://codecov.io/gh/apache/libcloud/pull/1290?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [libcloud] codecov-io commented on issue #1290: Various code fixes and improvements
codecov-io commented on issue #1290: Various code fixes and improvements URL: https://github.com/apache/libcloud/pull/1290#issuecomment-496611671 # [Codecov](https://codecov.io/gh/apache/libcloud/pull/1290?src=pr=h1) Report > Merging [#1290](https://codecov.io/gh/apache/libcloud/pull/1290?src=pr=desc) into [trunk](https://codecov.io/gh/apache/libcloud/commit/f40b2c4d2e6531cc4b52972b028f56b17f90cc27?src=pr=desc) will **decrease** coverage by `0.02%`. > The diff coverage is `80.11%`. [](https://codecov.io/gh/apache/libcloud/pull/1290?src=pr=tree) ```diff @@Coverage Diff @@ ##trunk#1290 +/- ## == - Coverage 85.98% 85.96% -0.03% == Files 359 359 Lines 7386773911 +44 Branches 6700 6705 +5 == + Hits6351863534 +16 - Misses 7670 7696 +26 - Partials 2679 2681 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/libcloud/pull/1290?src=pr=tree) | Coverage Δ | | |---|---|---| | [libcloud/test/compute/test\_gce.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3RfZ2NlLnB5) | `97.74% <ø> (ø)` | :arrow_up: | | [libcloud/test/compute/test\_openstack.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3Rfb3BlbnN0YWNrLnB5) | `94.89% <ø> (ø)` | :arrow_up: | | [libcloud/test/compute/test\_dimensiondata\_v2\_3.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3RfZGltZW5zaW9uZGF0YV92Ml8zLnB5) | `88.01% <ø> (ø)` | :arrow_up: | | [libcloud/common/brightbox.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvY29tbW9uL2JyaWdodGJveC5weQ==) | `53.65% <0%> (ø)` | :arrow_up: | | [libcloud/test/compute/test\_ssh\_client.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3Rfc3NoX2NsaWVudC5weQ==) | `30.89% <0%> (ø)` | :arrow_up: | | [libcloud/test/common/test\_upcloud.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21tb24vdGVzdF91cGNsb3VkLnB5) | `100% <100%> (ø)` | :arrow_up: | | [libcloud/test/compute/test\_digitalocean\_v2.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3RfZGlnaXRhbG9jZWFuX3YyLnB5) | `99.45% <100%> (ø)` | :arrow_up: | | [libcloud/test/test\_connection.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC90ZXN0X2Nvbm5lY3Rpb24ucHk=) | `99.59% <100%> (ø)` | :arrow_up: | | [libcloud/test/loadbalancer/test\_elb.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9sb2FkYmFsYW5jZXIvdGVzdF9lbGIucHk=) | `99.21% <100%> (ø)` | :arrow_up: | | [libcloud/test/compute/test\_joyent.py](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3Rfam95ZW50LnB5) | `98.68% <100%> (ø)` | :arrow_up: | | ... and [29 more](https://codecov.io/gh/apache/libcloud/pull/1290/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/libcloud/pull/1290?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/libcloud/pull/1290?src=pr=footer). Last update [f40b2c4...868de3d](https://codecov.io/gh/apache/libcloud/pull/1290?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [libcloud] c-w commented on issue #1202: Fix hash value in azure blob store
c-w commented on issue #1202: Fix hash value in azure blob store URL: https://github.com/apache/libcloud/pull/1202#issuecomment-496580356 @daviskirk I'm seeing some authentication errors with your branch running against a live storage account: ``` Traceback (most recent call last): File "/home/vsts/work/1/s/tests/test_storage.py", line 120, in test_objects_stream_io self._test_objects(do_upload, do_download) File "/home/vsts/work/1/s/tests/test_storage.py", line 67, in _test_objects container = self.driver.create_container(_random_container_name()) File "/opt/hostedtoolcache/Python/2.7.15/x64/lib/python2.7/site-packages/libcloud/storage/drivers/azure_blobs.py", line 516, in create_container method='PUT') File "/opt/hostedtoolcache/Python/2.7.15/x64/lib/python2.7/site-packages/libcloud/common/base.py", line 637, in request response = responseCls(**kwargs) File "/opt/hostedtoolcache/Python/2.7.15/x64/lib/python2.7/site-packages/libcloud/common/base.py", line 152, in __init__ message=self.parse_error(), File "/opt/hostedtoolcache/Python/2.7.15/x64/lib/python2.7/site-packages/libcloud/common/azure.py", line 84, in parse_error raise InvalidCredsError(error_msg) InvalidCredsError: 'AuthenticationFailed: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.' ``` You can see the full test run on [Azure Pipelines](https://clewolff.visualstudio.com/libcloud-tests/_build/results?buildId=40) and the code for the tests at [c-w/libcloud-tests](https://github.com/c-w/libcloud-tests). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [libcloud] Kami opened a new pull request #1290: Various code fixes and improvements
Kami opened a new pull request #1290: Various code fixes and improvements URL: https://github.com/apache/libcloud/pull/1290 This pull request fixes various issues in the code which were identified when running tests under newer versions of Python 3 (newer versions print various deprecation warning and also report open file leakage): 1. Fix various tests which didn't correctly cleanup and close open files 2. Fix ``libcloud.enable_debug`` debug function so it correctly closes the file handle on exit 3. Update test code to use non-deprecated assertion methods on newer versions of Python 3 4. Various other deprecation error fixes 5. Ignore user warnings when running tests using tox (this way we silence non-important warnings) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [libcloud] Kami commented on issue #1283: Fixes LIBCLOUD-1039 - cloudfiles download_object_as_stream
Kami commented on issue #1283: Fixes LIBCLOUD-1039 - cloudfiles download_object_as_stream URL: https://github.com/apache/libcloud/pull/1283#issuecomment-496520209 I've confirmed this bug fix. I also added a test case and merged this into trunk. Thanks again for your contribution. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (LIBCLOUD-1039) Cloudfiles storage download_object_as_stream results in HttpLibResponseProxy error.
[ https://issues.apache.org/jira/browse/LIBCLOUD-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849729#comment-16849729 ] ASF subversion and git services commented on LIBCLOUD-1039: --- Commit 49b2206729ff4bacfef6803a1424e2ea6fa9045b in libcloud's branch refs/heads/trunk from Tomaz Muraus [ https://gitbox.apache.org/repos/asf?p=libcloud.git;h=49b2206 ] Merge branch 'fix/LIBCLOUD-1039-fix-download-object-as-stream-cloudfiles' of https://github.com/mattseymour/libcloud into mattseymour-fix/LIBCLOUD-1039-fix-download-object-as-stream-cloudfiles > Cloudfiles storage download_object_as_stream results in HttpLibResponseProxy > error. > --- > > Key: LIBCLOUD-1039 > URL: https://issues.apache.org/jira/browse/LIBCLOUD-1039 > Project: Libcloud > Issue Type: Bug > Environment: All, > > Specific instance: Linux (ubuntu), Python3.5 and Python3.6, libcloud version > 2.4.0 >Reporter: matt seymour >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > When calling the download_object_as_stream method in the CLoudfiles driver a > HttpLibResponseProxy error is raised as the resulting (returning) object is > not a generator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIBCLOUD-1039) Cloudfiles storage download_object_as_stream results in HttpLibResponseProxy error.
[ https://issues.apache.org/jira/browse/LIBCLOUD-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849730#comment-16849730 ] ASF subversion and git services commented on LIBCLOUD-1039: --- Commit 49b2206729ff4bacfef6803a1424e2ea6fa9045b in libcloud's branch refs/heads/trunk from Tomaz Muraus [ https://gitbox.apache.org/repos/asf?p=libcloud.git;h=49b2206 ] Merge branch 'fix/LIBCLOUD-1039-fix-download-object-as-stream-cloudfiles' of https://github.com/mattseymour/libcloud into mattseymour-fix/LIBCLOUD-1039-fix-download-object-as-stream-cloudfiles > Cloudfiles storage download_object_as_stream results in HttpLibResponseProxy > error. > --- > > Key: LIBCLOUD-1039 > URL: https://issues.apache.org/jira/browse/LIBCLOUD-1039 > Project: Libcloud > Issue Type: Bug > Environment: All, > > Specific instance: Linux (ubuntu), Python3.5 and Python3.6, libcloud version > 2.4.0 >Reporter: matt seymour >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > When calling the download_object_as_stream method in the CLoudfiles driver a > HttpLibResponseProxy error is raised as the resulting (returning) object is > not a generator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIBCLOUD-1039) Cloudfiles storage download_object_as_stream results in HttpLibResponseProxy error.
[ https://issues.apache.org/jira/browse/LIBCLOUD-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849728#comment-16849728 ] ASF subversion and git services commented on LIBCLOUD-1039: --- Commit 494cac548d9f92338c403c2cb76458bdf79530da in libcloud's branch refs/heads/trunk from Matt Seymour [ https://gitbox.apache.org/repos/asf?p=libcloud.git;h=494cac5 ] Fixes LIBCLOUD-1039 - cloudfiles download_object_as_stream When downloading an object as a stream `download_object_as_stream` in storage/drivers/cloudfiles.py a generator object should be returned. Currently a rerquest/response object is returned in resulting in a HttpLibResponseProxy error. This commit alters the callback_kwargs to use a requests/response.iter_content resulting in an iterator object being returned. > Cloudfiles storage download_object_as_stream results in HttpLibResponseProxy > error. > --- > > Key: LIBCLOUD-1039 > URL: https://issues.apache.org/jira/browse/LIBCLOUD-1039 > Project: Libcloud > Issue Type: Bug > Environment: All, > > Specific instance: Linux (ubuntu), Python3.5 and Python3.6, libcloud version > 2.4.0 >Reporter: matt seymour >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > When calling the download_object_as_stream method in the CLoudfiles driver a > HttpLibResponseProxy error is raised as the resulting (returning) object is > not a generator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [libcloud] asfgit merged pull request #1283: Fixes LIBCLOUD-1039 - cloudfiles download_object_as_stream
asfgit merged pull request #1283: Fixes LIBCLOUD-1039 - cloudfiles download_object_as_stream URL: https://github.com/apache/libcloud/pull/1283 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[libcloud] branch trunk updated (2adabe4 -> f40b2c4)
This is an automated email from the ASF dual-hosted git repository. tomaz pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/libcloud.git. from 2adabe4 Wrap lines in CHANGES.rst at 80 characters. add 494cac5 Fixes LIBCLOUD-1039 - cloudfiles download_object_as_stream new 49b2206 Merge branch 'fix/LIBCLOUD-1039-fix-download-object-as-stream-cloudfiles' of https://github.com/mattseymour/libcloud into mattseymour-fix/LIBCLOUD-1039-fix-download-object-as-stream-cloudfiles new 6bb395d Add a test case for it. new a634635 Add changelog entry for it. new 54dbe43 Fix and improve tests which were broken (not testing the correct thing) since the migration to requests. new f40b2c4 Fix Python 3 compatibility. The 5 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.rst | 5 +++ libcloud/storage/drivers/cloudfiles.py | 8 +++-- libcloud/test/storage/test_cloudfiles.py | 55 +++- libcloud/test/storage/test_s3.py | 38 +- 4 files changed, 102 insertions(+), 4 deletions(-)
[libcloud] 02/05: Add a test case for it.
This is an automated email from the ASF dual-hosted git repository.
tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git
commit 6bb395dcbbefde09bcdd70c3151abce193162b25
Author: Tomaz Muraus
AuthorDate: Tue May 28 15:05:57 2019 +0200
Add a test case for it.
---
libcloud/test/storage/test_cloudfiles.py | 49 +++-
1 file changed, 48 insertions(+), 1 deletion(-)
diff --git a/libcloud/test/storage/test_cloudfiles.py
b/libcloud/test/storage/test_cloudfiles.py
index 05ac82f..a157408 100644
--- a/libcloud/test/storage/test_cloudfiles.py
+++ b/libcloud/test/storage/test_cloudfiles.py
@@ -13,7 +13,7 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-from hashlib import sha1
+
import hmac
import os
import os.path # pylint: disable-msg=W0404
@@ -21,13 +21,18 @@ import math
import sys
import copy
from io import BytesIO
+from hashlib import sha1
+
import mock
+from mock import Mock
+from mock import PropertyMock
import libcloud.utils.files
from libcloud.utils.py3 import b
from libcloud.utils.py3 import httplib
from libcloud.utils.py3 import urlquote
+from libcloud.utils.py3 import StringIO
from libcloud.common.types import MalformedResponseError
from libcloud.storage.base import CHUNK_SIZE, Container, Object
@@ -360,6 +365,35 @@ class CloudFilesTests(unittest.TestCase):
obj=obj, chunk_size=None)
self.assertTrue(hasattr(stream, '__iter__'))
+def test_download_object_data_is_not_buffered_in_memory(self):
+# Test case which verifies that response.body attribute is not accessed
+# and as such, whole body response is not buffered into RAM
+
+# If content is consumed and response.content attribute accessed
execption
+# will be thrown and test will fail
+mock_response = Mock(name='mock response')
+mock_response.headers = {}
+mock_response.status_code = 200
+msg = '"content" attribute was accessed but it shouldn\'t have been'
+type(mock_response).content = PropertyMock(name='mock content
attribute',
+ side_effect=Exception(msg))
+mock_response.iter_content.return_value = StringIO('a' * 1000)
+
+self.driver.connection.connection.getresponse = Mock()
+self.driver.connection.connection.getresponse.return_value =
mock_response
+
+container = Container(name='foo_bar_container', extra={},
+ driver=self.driver)
+obj = Object(name='foo_bar_object_NO_BUFFER', size=1000, hash=None,
extra={},
+ container=container, meta_data=None,
+ driver=self.driver)
+destination_path = os.path.abspath(__file__) + '.temp'
+result = self.driver.download_object(obj=obj,
+ destination_path=destination_path,
+ overwrite_existing=False,
+ delete_on_failure=True)
+self.assertTrue(result)
+
def test_upload_object_success(self):
def upload_file(self, object_name=None, content_type=None,
request_path=None, request_method=None,
@@ -1184,5 +1218,18 @@ class CloudFilesMockHttp(MockHttp, unittest.TestCase):
headers,
httplib.responses[httplib.OK])
+def _v1_MossoCloudFS_foo_bar_container_foo_bar_object_NO_BUFFER(
+self, method, url, body, headers):
+# test_download_object_data_is_not_buffered_in_memory
+headers = {}
+headers.update(self.base_headers)
+headers['etag'] = '577ef1154f3240ad5b9b413aa7346a1e'
+body = generate_random_data(1000)
+return (httplib.OK,
+body,
+headers,
+httplib.responses[httplib.OK])
+
+
if __name__ == '__main__':
sys.exit(unittest.main())
[libcloud] 01/05: Merge branch 'fix/LIBCLOUD-1039-fix-download-object-as-stream-cloudfiles' of https://github.com/mattseymour/libcloud into mattseymour-fix/LIBCLOUD-1039-fix-download-object-as-stream-
This is an automated email from the ASF dual-hosted git repository. tomaz pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/libcloud.git commit 49b2206729ff4bacfef6803a1424e2ea6fa9045b Merge: 2adabe4 494cac5 Author: Tomaz Muraus AuthorDate: Tue May 28 14:58:42 2019 +0200 Merge branch 'fix/LIBCLOUD-1039-fix-download-object-as-stream-cloudfiles' of https://github.com/mattseymour/libcloud into mattseymour-fix/LIBCLOUD-1039-fix-download-object-as-stream-cloudfiles libcloud/storage/drivers/cloudfiles.py | 8 ++-- 1 file changed, 6 insertions(+), 2 deletions(-)
[libcloud] 04/05: Fix and improve tests which were broken (not testing the correct thing) since the migration to requests.
This is an automated email from the ASF dual-hosted git repository.
tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git
commit 54dbe437b87956f67da7acfe429a20546e4067d0
Author: Tomaz Muraus
AuthorDate: Tue May 28 15:20:31 2019 +0200
Fix and improve tests which were broken (not testing the correct thing)
since the migration to requests.
---
libcloud/test/storage/test_cloudfiles.py | 28 ++--
libcloud/test/storage/test_s3.py | 29 -
2 files changed, 42 insertions(+), 15 deletions(-)
diff --git a/libcloud/test/storage/test_cloudfiles.py
b/libcloud/test/storage/test_cloudfiles.py
index a157408..e3c6ff4 100644
--- a/libcloud/test/storage/test_cloudfiles.py
+++ b/libcloud/test/storage/test_cloudfiles.py
@@ -33,6 +33,7 @@ from libcloud.utils.py3 import b
from libcloud.utils.py3 import httplib
from libcloud.utils.py3 import urlquote
from libcloud.utils.py3 import StringIO
+from libcloud.utils.files import exhaust_iterator
from libcloud.common.types import MalformedResponseError
from libcloud.storage.base import CHUNK_SIZE, Container, Object
@@ -365,34 +366,33 @@ class CloudFilesTests(unittest.TestCase):
obj=obj, chunk_size=None)
self.assertTrue(hasattr(stream, '__iter__'))
-def test_download_object_data_is_not_buffered_in_memory(self):
-# Test case which verifies that response.body attribute is not accessed
+def test_download_object_as_stream_data_is_not_buffered_in_memory(self):
+# Test case which verifies that response.response attribute is not
accessed
# and as such, whole body response is not buffered into RAM
-# If content is consumed and response.content attribute accessed
execption
+# If content is consumed and response.content attribute accessed
exception
# will be thrown and test will fail
mock_response = Mock(name='mock response')
mock_response.headers = {}
-mock_response.status_code = 200
-msg = '"content" attribute was accessed but it shouldn\'t have been'
+mock_response.status = 200
+msg1 = '"response" attribute was accessed but it shouldn\'t have been'
+msg2 = '"content" attribute was accessed but it shouldn\'t have been'
+type(mock_response).response = PropertyMock(name='mock response
attribute',
+
side_effect=Exception(msg1))
type(mock_response).content = PropertyMock(name='mock content
attribute',
- side_effect=Exception(msg))
+ side_effect=Exception(msg2))
mock_response.iter_content.return_value = StringIO('a' * 1000)
-self.driver.connection.connection.getresponse = Mock()
-self.driver.connection.connection.getresponse.return_value =
mock_response
+self.driver.connection.request = Mock()
+self.driver.connection.request.return_value = mock_response
container = Container(name='foo_bar_container', extra={},
driver=self.driver)
obj = Object(name='foo_bar_object_NO_BUFFER', size=1000, hash=None,
extra={},
container=container, meta_data=None,
driver=self.driver)
-destination_path = os.path.abspath(__file__) + '.temp'
-result = self.driver.download_object(obj=obj,
- destination_path=destination_path,
- overwrite_existing=False,
- delete_on_failure=True)
-self.assertTrue(result)
+result = self.driver.download_object_as_stream(obj=obj)
+self.assertEqual(exhaust_iterator(result), 'a' * 1000)
def test_upload_object_success(self):
def upload_file(self, object_name=None, content_type=None,
diff --git a/libcloud/test/storage/test_s3.py b/libcloud/test/storage/test_s3.py
index b214b51..df31920 100644
--- a/libcloud/test/storage/test_s3.py
+++ b/libcloud/test/storage/test_s3.py
@@ -31,6 +31,7 @@ from libcloud.utils.py3 import httplib
from libcloud.utils.py3 import urlparse
from libcloud.utils.py3 import parse_qs
from libcloud.utils.py3 import StringIO
+from libcloud.utils.files import exhaust_iterator
from libcloud.common.types import InvalidCredsError
from libcloud.common.types import LibcloudError, MalformedResponseError
@@ -641,7 +642,6 @@ class S3Tests(unittest.TestCase):
# If content is consumed and response.content attribute accessed
execption
# will be thrown and test will fail
-
mock_response = Mock(name='mock response')
mock_response.headers = {}
mock_response.status_code = 200
@@ -665,6 +665,33 @@ class S3Tests(unittest.TestCase):
[libcloud] 03/05: Add changelog entry for it.
This is an automated email from the ASF dual-hosted git repository. tomaz pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/libcloud.git commit a6346353d74b6a18bdba929f1bf23257d06c728e Author: Tomaz Muraus AuthorDate: Tue May 28 15:07:38 2019 +0200 Add changelog entry for it. --- CHANGES.rst | 5 + 1 file changed, 5 insertions(+) diff --git a/CHANGES.rst b/CHANGES.rst index 3fa02c6..5d5d3dc 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -146,6 +146,11 @@ Storage support seek, entire iterator content will be buffered in memory. (LIBCLOUD-1043, GITHUB-1287) [Clemens Wolff] +- [CloudFiles] Fix ``download_object_as_stream`` method in the CloudFiles + driver. This regression / bug was inadvertently introduced when migrating + code to ``requests``. + (LIBCLOUD-1039, GITHUB-1283) + [Matt Seymour] Container ~
[libcloud] 05/05: Fix Python 3 compatibility.
This is an automated email from the ASF dual-hosted git repository.
tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git
commit f40b2c4d2e6531cc4b52972b028f56b17f90cc27
Author: Tomaz Muraus
AuthorDate: Tue May 28 15:34:51 2019 +0200
Fix Python 3 compatibility.
---
libcloud/test/storage/test_cloudfiles.py | 8 +++-
libcloud/test/storage/test_s3.py | 13 +++--
2 files changed, 18 insertions(+), 3 deletions(-)
diff --git a/libcloud/test/storage/test_cloudfiles.py
b/libcloud/test/storage/test_cloudfiles.py
index e3c6ff4..650cc53 100644
--- a/libcloud/test/storage/test_cloudfiles.py
+++ b/libcloud/test/storage/test_cloudfiles.py
@@ -33,6 +33,7 @@ from libcloud.utils.py3 import b
from libcloud.utils.py3 import httplib
from libcloud.utils.py3 import urlquote
from libcloud.utils.py3 import StringIO
+from libcloud.utils.py3 import PY3
from libcloud.utils.files import exhaust_iterator
from libcloud.common.types import MalformedResponseError
@@ -392,7 +393,12 @@ class CloudFilesTests(unittest.TestCase):
container=container, meta_data=None,
driver=self.driver)
result = self.driver.download_object_as_stream(obj=obj)
-self.assertEqual(exhaust_iterator(result), 'a' * 1000)
+result = exhaust_iterator(result)
+
+if PY3:
+result = result.decode('utf-8')
+
+self.assertEqual(result, 'a' * 1000)
def test_upload_object_success(self):
def upload_file(self, object_name=None, content_type=None,
diff --git a/libcloud/test/storage/test_s3.py b/libcloud/test/storage/test_s3.py
index df31920..cc756fb 100644
--- a/libcloud/test/storage/test_s3.py
+++ b/libcloud/test/storage/test_s3.py
@@ -31,6 +31,7 @@ from libcloud.utils.py3 import httplib
from libcloud.utils.py3 import urlparse
from libcloud.utils.py3 import parse_qs
from libcloud.utils.py3 import StringIO
+from libcloud.utils.py3 import PY3
from libcloud.utils.files import exhaust_iterator
from libcloud.common.types import InvalidCredsError
@@ -666,7 +667,10 @@ class S3Tests(unittest.TestCase):
self.assertTrue(result)
def test_download_object_as_stream_data_is_not_buffered_in_memory(self):
-# If content is consumed and response.content attribute accessed
execption
+# Test case which verifies that response.response attribute is not
accessed
+# and as such, whole body response is not buffered into RAM
+
+# If content is consumed and response.content attribute accessed
exception
# will be thrown and test will fail
mock_response = Mock(name='mock response')
mock_response.headers = {}
@@ -690,7 +694,12 @@ class S3Tests(unittest.TestCase):
driver=self.driver_type)
destination_path = self._file_path
result = self.driver.download_object_as_stream(obj=obj)
-self.assertEqual(exhaust_iterator(result), 'a' * 1000)
+result = exhaust_iterator(result)
+
+if PY3:
+result = result.decode('utf-8')
+
+self.assertEqual(result, 'a' * 1000)
def test_download_object_invalid_file_size(self):
self.mock_response_klass.type = 'INVALID_SIZE'
[libcloud] 02/02: Wrap lines in CHANGES.rst at 80 characters.
This is an automated email from the ASF dual-hosted git repository. tomaz pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/libcloud.git commit 2adabe4489ebd4841e5fed2108dce7a7ff2de1af Author: Tomaz Muraus AuthorDate: Tue May 28 14:54:34 2019 +0200 Wrap lines in CHANGES.rst at 80 characters. --- CHANGES.rst | 172 +++- 1 file changed, 111 insertions(+), 61 deletions(-) diff --git a/CHANGES.rst b/CHANGES.rst index 26265ba..3fa02c6 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -7,16 +7,19 @@ Changes in Apache Libcloud in development General ~~~ -- [NTT CIS] Add loadbalancer and compute drivers for NTT-CIS, rename dimensiondata modules to NTT-CIS (GITHUB-1250) +- [NTT CIS] Add loadbalancer and compute drivers for NTT-CIS, rename + dimensiondata modules to NTT-CIS. (GITHUB-1250) [Mitch Raful] -- [NTT CIS] Fix loadbalancer docs (GITHUB-1270) +- [NTT CIS] Fix loadbalancer docs. (GITHUB-1270) [Mitch Raful] - Use assertIsNone instead of assertEqual with None in tests (GITHUB-1264) [Ken Dreyer] -- Updating command line arguments to current version in Azure examples (GITHUB-1273) [mitar] +- Updating command line arguments to current version in Azure examples. + (GITHUB-1273) + [mitar] - [GCE, SoftLayer] Update GCE and Softlayer drivers to utilize crypto primitives from the ``cryptography`` library instead of deprecated and @@ -31,7 +34,8 @@ Common - [OpenStack] Handle missing user enabled attribute (GITHUB-1261) [Ken Dreyer] -- [Google Cloud Storage] Handle Interoperability access keys of more than 20 characters (GITHUB-1272) +- [Google Cloud Storage] Handle Interoperability access keys of more than 20 + characters. (GITHUB-1272) [Yoan Tournade] Compute @@ -43,75 +47,95 @@ Compute - [OpenStack] Pagination in various OpenStack_2_NodeDriver methods (GITHUB-1263) [Rick van de Loo] -- [OpenStack] Implement OpenStack_2_NodeDriver ex_create_subnet (LIBCLOUD-874, GITHUB-1242) +- [OpenStack] Implement OpenStack_2_NodeDriver ex_create_subnet (LIBCLOUD-874, + GITHUB-1242) [Miguel Caballer] -- [OpenStack] Implement OpenStack_2_NodeDriver ex_delete_subnet (LIBCLOUD-874, GITHUB-1242) +- [OpenStack] Implement OpenStack_2_NodeDriver ex_delete_subnet (LIBCLOUD-874, + GITHUB-1242) [Miguel Caballer] -- [OpenStack] Implement OpenStack_2_NodeDriver list_volumes (LIBCLOUD-874, GITHUB-1242) +- [OpenStack] Implement OpenStack_2_NodeDriver list_volumes (LIBCLOUD-874, + GITHUB-1242) [Miguel Caballer] -- [OpenStack] Implement OpenStack_2_NodeDriver ex_get_volume (LIBCLOUD-874, GITHUB-1242) +- [OpenStack] Implement OpenStack_2_NodeDriver ex_get_volume (LIBCLOUD-874, + GITHUB-1242) [Miguel Caballer] -- [OpenStack] Implement OpenStack_2_NodeDriver create_volume (LIBCLOUD-874, GITHUB-1242) +- [OpenStack] Implement OpenStack_2_NodeDriver create_volume (LIBCLOUD-874, + GITHUB-1242) [Miguel Caballer] -- [OpenStack] Implement OpenStack_2_NodeDriver destroy_volume (LIBCLOUD-874, GITHUB-1242) +- [OpenStack] Implement OpenStack_2_NodeDriver destroy_volume (LIBCLOUD-874, + GITHUB-1242) [Miguel Caballer] -- [OpenStack] Implement OpenStack_2_NodeDriver ex_list_snapshots (LIBCLOUD-874, GITHUB-1242) +- [OpenStack] Implement OpenStack_2_NodeDriver ex_list_snapshots (LIBCLOUD-874, + GITHUB-1242) [Miguel Caballer] -- [OpenStack] Implement OpenStack_2_NodeDriver create_volume_snapshot (LIBCLOUD-874, GITHUB-1242) +- [OpenStack] Implement OpenStack_2_NodeDriver create_volume_snapshot + (LIBCLOUD-874, GITHUB-1242) [Miguel Caballer] -- [OpenStack] Implement OpenStack_2_NodeDriver destroy_volume_snapshot (LIBCLOUD-874, GITHUB-1242) +- [OpenStack] Implement OpenStack_2_NodeDriver destroy_volume_snapshot + (LIBCLOUD-874, GITHUB-1242) [Miguel Caballer] -- [OpenStack] Implement OpenStack_2_NodeDriver ex_list_security_groups (LIBCLOUD-874, GITHUB-1242) +- [OpenStack] Implement OpenStack_2_NodeDriver ex_list_security_groups + (LIBCLOUD-874, GITHUB-1242) [Miguel Caballer] -- [OpenStack] Implement OpenStack_2_NodeDriver ex_create_security_group (LIBCLOUD-874, GITHUB-1242) +- [OpenStack] Implement OpenStack_2_NodeDriver ex_create_security_group + (LIBCLOUD-874, GITHUB-1242) [Miguel Caballer] -- [OpenStack] Implement OpenStack_2_NodeDriver ex_delete_security_group (LIBCLOUD-874, GITHUB-1242) +- [OpenStack] Implement OpenStack_2_NodeDriver ex_delete_security_group + (LIBCLOUD-874, GITHUB-1242) [Miguel Caballer] -- [OpenStack] Implement OpenStack_2_NodeDriver ex_create_security_group_rule (LIBCLOUD-874, GITHUB-1242) +- [OpenStack] Implement OpenStack_2_NodeDriver ex_create_security_group_rule + (LIBCLOUD-874, GITHUB-1242) [Miguel Caballer] -- [OpenStack] Implement OpenStack_2_NodeDriver ex_delete_security_group_rule (LIBCLOUD-874, GITHUB-1242) +- [OpenStack] Implement OpenStack_2_NodeDriver ex_delete_security_group_rule +
[libcloud] 01/02: Add changelog entry for #1287.
This is an automated email from the ASF dual-hosted git repository. tomaz pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/libcloud.git commit f3f45e16414c2be417aa800cbb6c3136a4bd461f Author: Tomaz Muraus AuthorDate: Tue May 28 14:47:28 2019 +0200 Add changelog entry for #1287. --- CHANGES.rst | 9 + 1 file changed, 9 insertions(+) diff --git a/CHANGES.rst b/CHANGES.rst index 0f8ea33..26265ba 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -114,6 +114,15 @@ Compute (LIBCLOUD-1040) [aki-k, Tomaz Muraus] +Storage +~~~ + +- [Azure] Fix ``upload_object_via_stream`` method so it also works with + iterators which don't implement ``seek()`` method. If the iterator doesn't + support seek, entire iterator content will be buffered in memory. + (LIBCLOUD-1043, GITHUB-1287) + [Clemens Wolff] + Container ~
[libcloud] branch trunk updated (f086aa6 -> 2adabe4)
This is an automated email from the ASF dual-hosted git repository. tomaz pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/libcloud.git. from f086aa6 Merge pull request #1287 from CatalystCode/1043_fix_azure_upload_object_via_stream_with_iterator new f3f45e1 Add changelog entry for #1287. new 2adabe4 Wrap lines in CHANGES.rst at 80 characters. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.rst | 181 1 file changed, 120 insertions(+), 61 deletions(-)
[jira] [Commented] (LIBCLOUD-1043) Azure Storage driver crashes when passed a non file-like iterator
[
https://issues.apache.org/jira/browse/LIBCLOUD-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849684#comment-16849684
]
ASF subversion and git services commented on LIBCLOUD-1043:
---
Commit f086aa68dfa543bc6b0cc348cb6a0bc1f43d6621 in libcloud's branch
refs/heads/trunk from Tomaz Muraus
[ https://gitbox.apache.org/repos/asf?p=libcloud.git;h=f086aa6 ]
Merge pull request #1287 from
CatalystCode/1043_fix_azure_upload_object_via_stream_with_iterator
[LIBCLOUD-1043] Fix Azure upload_object_via_stream used with iter
> Azure Storage driver crashes when passed a non file-like iterator
> -
>
> Key: LIBCLOUD-1043
> URL: https://issues.apache.org/jira/browse/LIBCLOUD-1043
> Project: Libcloud
> Issue Type: Bug
>Reporter: Clemens Wolff
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The azure_blobs implementation of {{upload_object_via_stream(iterator, ...)}}
> currently assumes that the passed-in iterator always is a file-like object
> that implements {{seek(...)}}
> ([source|https://github.com/apache/libcloud/blob/a1f9897ce107eb1b01674262ca1b9ff1f91d46f0/libcloud/storage/drivers/azure_blobs.py#L828-L830]).
> This means that function crashes when used with arbitrary iterators as
> returned by {{iter(...)}}.
> Notably, the libcloud integration for django-storages exercises this
> functionality
> ([source|https://github.com/jschneier/django-storages/blob/b441b74a17a46eb87ee4b10f60774b9a080c0fe1/storages/backends/apache_libcloud.py#L158])
> which makes django-storages + libcloud + Azure Blobs currently unusable. See
> detailed stacktrace below:
> {code:python}
> Traceback (most recent call last):
> File ".../app/manage.py", line 15, in
> execute_from_command_line(sys.argv)
> File "...\lib\site-packages\django\core\management\__init__.py", line 381,
> in execute_from_command_line
> utility.execute()
> File "...\lib\site-packages\django\core\management\__init__.py", line 375,
> in execute
> self.fetch_command(subcommand).run_from_argv(self.argv)
> File "...\lib\site-packages\django\core\management\base.py", line 316, in
> run_from_argv
> self.execute(*args, **cmd_options)
> File "...\lib\site-packages\django\core\management\base.py", line 353, in
> execute
> output = self.handle(*args, **options)
> File
> "...\lib\site-packages\django\contrib\staticfiles\management\commands\collectstatic.py",
> line 188, in handle
> collected = self.collect()
> File
> "...\lib\site-packages\django\contrib\staticfiles\management\commands\collectstatic.py",
> line 114, in collect
> handler(path, prefixed_path, storage)
> File
> "...\lib\site-packages\django\contrib\staticfiles\management\commands\collectstatic.py",
> line 353, in copy_file
> self.storage.save(prefixed_path, source_file)
> File "...\lib\site-packages\django\core\files\storage.py", line 49, in save
> return self._save(name, content)
> File "...\lib\site-packages\storages\backends\apache_libcloud.py", line
> 157, in _save
> self.driver.upload_object_via_stream(iter(file), self._get_bucket(), name)
> File "...\lib\site-packages\libcloud\storage\drivers\azure_blobs.py", line
> 828, in upload_object_via_stream
> iterator.seek(0, os.SEEK_END)
> AttributeError: 'generator' object has no attribute 'seek'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [libcloud] Kami commented on issue #1287: [LIBCLOUD-1043] Fix Azure upload_object_via_stream used with iter
Kami commented on issue #1287: [LIBCLOUD-1043] Fix Azure upload_object_via_stream used with iter URL: https://github.com/apache/libcloud/pull/1287#issuecomment-496500912 Merged, thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [libcloud] Kami merged pull request #1287: [LIBCLOUD-1043] Fix Azure upload_object_via_stream used with iter
Kami merged pull request #1287: [LIBCLOUD-1043] Fix Azure upload_object_via_stream used with iter URL: https://github.com/apache/libcloud/pull/1287 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[libcloud] branch trunk updated: Fix Azure upload_object_via_stream used with iter
This is an automated email from the ASF dual-hosted git repository.
tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git
The following commit(s) were added to refs/heads/trunk by this push:
new d949d76 Fix Azure upload_object_via_stream used with iter
new f086aa6 Merge pull request #1287 from
CatalystCode/1043_fix_azure_upload_object_via_stream_with_iterator
d949d76 is described below
commit d949d76b9bb24ca7b8f1839c45090db64689453a
Author: Clemens Wolff
AuthorDate: Fri May 17 14:05:55 2019 -0400
Fix Azure upload_object_via_stream used with iter
---
libcloud/storage/drivers/azure_blobs.py | 11 ++-
libcloud/test/storage/test_azure_blobs.py | 18 ++
2 files changed, 28 insertions(+), 1 deletion(-)
diff --git a/libcloud/storage/drivers/azure_blobs.py
b/libcloud/storage/drivers/azure_blobs.py
index ca63d3a..06c3184 100644
--- a/libcloud/storage/drivers/azure_blobs.py
+++ b/libcloud/storage/drivers/azure_blobs.py
@@ -18,6 +18,7 @@ from __future__ import with_statement
import base64
import os
import binascii
+from io import BytesIO
from libcloud.utils.py3 import ET
from libcloud.utils.py3 import httplib
@@ -799,6 +800,9 @@ class AzureBlobsStorageDriver(StorageDriver):
"""
@inherits: :class:`StorageDriver.upload_object_via_stream`
+Note that if ``iterator`` does not support ``seek``, the
+entire generator will be buffered in memory.
+
:param ex_blob_type: Storage class
:type ex_blob_type: ``str``
@@ -825,7 +829,12 @@ class AzureBlobsStorageDriver(StorageDriver):
"""
self._check_values(ex_blob_type, ex_page_blob_size)
if ex_blob_type == "BlockBlob":
-iterator.seek(0, os.SEEK_END)
+try:
+iterator.seek(0, os.SEEK_END)
+except AttributeError:
+buffer = BytesIO()
+buffer.writelines(iterator)
+iterator = buffer
blob_size = iterator.tell()
iterator.seek(0)
else:
diff --git a/libcloud/test/storage/test_azure_blobs.py
b/libcloud/test/storage/test_azure_blobs.py
index aa09c66..cbd71e1 100644
--- a/libcloud/test/storage/test_azure_blobs.py
+++ b/libcloud/test/storage/test_azure_blobs.py
@@ -874,6 +874,24 @@ class AzureBlobsTests(unittest.TestCase):
self.assertEqual(obj.size, 3)
self.mock_response_klass.use_param = None
+def test_upload_blob_object_via_stream_from_iterable(self):
+self.mock_response_klass.use_param = 'comp'
+container = Container(name='foo_bar_container', extra={},
+ driver=self.driver)
+
+object_name = 'foo_test_upload'
+iterator = iter([b('34'), b('5')])
+extra = {'content_type': 'text/plain'}
+obj = self.driver.upload_object_via_stream(container=container,
+ object_name=object_name,
+ iterator=iterator,
+ extra=extra,
+ ex_blob_type='BlockBlob')
+
+self.assertEqual(obj.name, object_name)
+self.assertEqual(obj.size, 3)
+self.mock_response_klass.use_param = None
+
def test_upload_blob_object_via_stream_with_lease(self):
self.mock_response_klass.use_param = 'comp'
container = Container(name='foo_bar_container', extra={},
[GitHub] [libcloud] codecov-io edited a comment on issue #1281: Add router management functions in OpenStack
codecov-io edited a comment on issue #1281: Add router management functions in OpenStack URL: https://github.com/apache/libcloud/pull/1281#issuecomment-475203814 # [Codecov](https://codecov.io/gh/apache/libcloud/pull/1281?src=pr=h1) Report > Merging [#1281](https://codecov.io/gh/apache/libcloud/pull/1281?src=pr=desc) into [trunk](https://codecov.io/gh/apache/libcloud/commit/064ad6cd4cf145ddabd0fc15cb00d965ef7688e4?src=pr=desc) will **decrease** coverage by `0.01%`. > The diff coverage is `79.02%`. [](https://codecov.io/gh/apache/libcloud/pull/1281?src=pr=tree) ```diff @@Coverage Diff @@ ##trunk#1281 +/- ## == - Coverage 85.98% 85.97% -0.02% == Files 359 359 Lines 7380973947 +138 Branches 6698 6719 +21 == + Hits6346663573 +107 - Misses 7666 7681 +15 - Partials 2677 2693 +16 ``` | [Impacted Files](https://codecov.io/gh/apache/libcloud/pull/1281?src=pr=tree) | Coverage Δ | | |---|---|---| | [libcloud/compute/drivers/openstack.py](https://codecov.io/gh/apache/libcloud/pull/1281/diff?src=pr=tree#diff-bGliY2xvdWQvY29tcHV0ZS9kcml2ZXJzL29wZW5zdGFjay5weQ==) | `83.97% <74.11%> (-1.28%)` | :arrow_down: | | [libcloud/test/compute/test\_openstack.py](https://codecov.io/gh/apache/libcloud/pull/1281/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3Rfb3BlbnN0YWNrLnB5) | `94.7% <86.2%> (-0.19%)` | :arrow_down: | | [libcloud/test/compute/test\_ec2.py](https://codecov.io/gh/apache/libcloud/pull/1281/diff?src=pr=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3RfZWMyLnB5) | `97.9% <0%> (+0.16%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/libcloud/pull/1281?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/libcloud/pull/1281?src=pr=footer). Last update [064ad6c...ae9d81e](https://codecov.io/gh/apache/libcloud/pull/1281?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
