[
https://issues.apache.org/jira/browse/LIBCLOUD-986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408315#comment-16408315
]
Joshua Hawkinson edited comment on LIBCLOUD-986 at 3/21/18 5:51 PM:
--------------------------------------------------------------------
Okay, done... Here are the diffs for your review:
{code:java}
diff --git a/libcloud/storage/drivers/azure_blobs.py
b/libcloud/storage/drivers/azure_blobs.py
index 82ee03a..2475793 100644
--- a/libcloud/storage/drivers/azure_blobs.py
+++ b/libcloud/storage/drivers/azure_blobs.py
@@ -381,11 +381,12 @@ class AzureBlobsStorageDriver(StorageDriver):
if not params['marker']:
break
- def iterate_container_objects(self, container):
+ def iterate_container_objects(self, container, ex_prefix=None):
"""
@inherits: :class:`StorageDriver.iterate_container_objects`
"""
- params = {'restype': 'container',
+ params = {'prefix': ex_prefix,
+ 'restype': 'container',
'comp': 'list',
'maxresults': RESPONSES_PER_REQUEST,
'include': 'metadata'}
@@ -416,6 +417,22 @@ class AzureBlobsStorageDriver(StorageDriver):
if not params['marker']:
break
+ def list_container_objects(self, container, ex_prefix=None):
+ """
+ Return a list of objects for the given container.
+
+ :param container: Container instance.
+ :type container: :class:`Container`
+
+ :param ex_prefix: Only return objects starting with ex_prefix
+ :type ex_prefix: ``str``
+
+ :return: A list of Object instances.
+ :rtype: ``list`` of :class:`Object`
+ """
+ return list(self.iterate_container_objects(container,
+ ex_prefix=ex_prefix))
+
def get_container(self, container_name):
"""
@inherits: :class:`StorageDriver.get_container`
diff --git a/libcloud/test/storage/test_azure_blobs.py
b/libcloud/test/storage/test_azure_blobs.py
index 95ab3f1..69bdebb 100644
--- a/libcloud/test/storage/test_azure_blobs.py
+++ b/libcloud/test/storage/test_azure_blobs.py
@@ -447,6 +447,28 @@ class AzureBlobsTests(unittest.TestCase):
self.assertTrue('content_encoding' in obj.extra)
self.assertTrue('content_language' in obj.extra)
+ def test_list_container_objects_with_prefix(self):
+ self.mock_response_klass.type = None
+ AzureBlobsStorageDriver.RESPONSES_PER_REQUEST = 2
+
+ container = Container(name='test_container', extra={},
+ driver=self.driver)
+ objects = self.driver.list_container_objects(container=container,
+ ex_prefix='test_prefix')
+ self.assertEqual(len(objects), 4)
+
+ obj = objects[1]
+ self.assertEqual(obj.name, 'object2.txt')
+ self.assertEqual(obj.hash, '0x8CFB90F1BA8CD8F')
+ self.assertEqual(obj.size, 1048576)
+ self.assertEqual(obj.container.name, 'test_container')
+ self.assertTrue('meta1' in obj.meta_data)
+ self.assertTrue('meta2' in obj.meta_data)
+ self.assertTrue('last_modified' in obj.extra)
+ self.assertTrue('content_type' in obj.extra)
+ self.assertTrue('content_encoding' in obj.extra)
+ self.assertTrue('content_language' in obj.extra)
+
{code}
And the before and after test results:
{code:java}
[root@localhost libcloud]# python3 -m pytest -k azure_blobs
=================================================================== test
session starts
====================================================================
platform linux -- Python 3.4.3, pytest-3.4.2, py-1.5.2, pluggy-0.6.0
rootdir: /usr/libcloud, inifile: pytest.ini
collected 9116 items
libcloud/test/storage/test_azure_blobs.py ....................F................
[100%]
=========================================================================
FAILURES
=========================================================================
_________________________________________________
AzureBlobsTests.test_list_container_objects_with_prefix
__________________________________________________
self = <libcloud.test.storage.test_azure_blobs.AzureBlobsTests
testMethod=test_list_container_objects_with_prefix>
def test_list_container_objects_with_prefix(self):
self.mock_response_klass.type = None
AzureBlobsStorageDriver.RESPONSES_PER_REQUEST = 2
container = Container(name='test_container', extra={},
driver=self.driver)
objects = self.driver.list_container_objects(container=container,
> ex_prefix='test_prefix')
E TypeError: list_container_objects() got an unexpected keyword argument
'ex_prefix'
libcloud/test/storage/test_azure_blobs.py:457: TypeError
================================================================== 9079 tests
deselected ===================================================================
=================================================== 1 failed, 36 passed, 9079
deselected in 3.39 seconds ===================================================
[root@localhost libcloud]# python3 -m pytest -k azure_blobs
=================================================================== test
session starts
====================================================================
platform linux -- Python 3.4.3, pytest-3.4.2, py-1.5.2, pluggy-0.6.0
rootdir: /usr/libcloud, inifile: pytest.ini
collected 9116 items
libcloud/test/storage/test_azure_blobs.py .....................................
[100%]
================================================================== 9079 tests
deselected ===================================================================
======================================================== 37 passed, 9079
deselected in 3.08 seconds
========================================================{code}
Some notes about the unit test for this feature... Ideally you'd have one
container that the Mock HTTP server would be able to return different results
for based on the prefix, I didn't see a way to simply accomplish this without
modifying some of the base mock server code. This test method was largely
pulled from the other driver's tests, but it seems to meet your base
requirements. Another note, I'm not sure why this Jira instance always strips
spaces, even in noformat, or code blocks (Sorry about the formatting).
As a bonus, Here are my personal unit test results using my Azure account
{code:java}
LOG.info("Testing basic object store operations...")
LOG.info("Creating unit test bucket %s" % UNITTEST_BUCKET)
obj_stores.create_container(UNITTEST_BUCKET)
LOG.info("Getting bucket....")
bucket = obj_stores.get_container(UNITTEST_BUCKET)
LOG.info("Creating a couple of files")
_ = obj_stores.upload_object(__file__, container=bucket,
object_name=UNITTEST_NAME1)
_ = obj_stores.upload_object(__file__, container=bucket,
object_name=UNITTEST_NAME2)
LOG.info("Validating list directory like behavior")
obj_list = list(obj_stores.iterate_container_objects(bucket,
ex_prefix=UNITTEST_DIR1))
obj_list2 = obj_stores.list_container_objects(bucket, ex_prefix=UNITTEST_DIR2)
{code}
output
{code:java}
2018-03-15 16:42:29 CAELUM_UNIT_TEST INFO 139630231111488 Starting unit test
for object store abstraction...
2018-03-15 16:42:29 CAELUM_UNIT_TEST INFO 139630231111488 ###### TEST VALIDATES
CONFIG FILE AND AWS CONNECTION
2018-03-15 16:42:29 CAELUM_UNIT_TEST INFO 139630231111488 Forcing config file
read...
2018-03-15 16:42:29 CAELUM_UNIT_TEST INFO 139630231111488 Connecting to BUILDER
AWS account
2018-03-15 16:42:29 CAELUM_UNIT_TEST INFO 139630231111488 Testing basic object
store operations...
2018-03-15 16:42:29 CAELUM_UNIT_TEST INFO 139630231111488 Creating unit test
bucket peaxy-caelum-unit-test
2018-03-15 16:42:30 CAELUM_UNIT_TEST INFO 139630231111488 Getting bucket....
2018-03-15 16:42:30 CAELUM_UNIT_TEST INFO 139630231111488 Creating a couple of
files
2018-03-15 16:42:31 CAELUM_UNIT_TEST INFO 139630231111488 Validating list
directory like behavior
2018-03-15 18:00:31 CAELUM_UNIT_TEST INFO 139630231111488 Attempting to delete
bucket
2018-03-15 18:00:35 CAELUM_UNIT_TEST INFO 139630231111488 ###### TEST PASSED
2018-03-15 18:00:35 CAELUM_UNIT_TEST INFO 139630231111488 ###### TEST VALIDATES
CONFIG VARIABLES AND AZURE CONNECTION
2018-03-15 18:00:35 CAELUM_UNIT_TEST INFO 139630231111488 Instantiating object
and then manually setting the config variables
2018-03-15 18:00:35 CAELUM_UNIT_TEST INFO 139630231111488 Connecting to MSDN
Azure account
2018-03-15 18:00:35 CAELUM_UNIT_TEST INFO 139630231111488 Testing basic object
store operations...
2018-03-15 18:00:35 CAELUM_UNIT_TEST INFO 139630231111488 Creating unit test
bucket peaxy-caelum-unit-test
2018-03-15 18:00:35 CAELUM_UNIT_TEST INFO 139630231111488 Getting bucket....
2018-03-15 18:00:35 CAELUM_UNIT_TEST INFO 139630231111488 Creating a couple of
files
2018-03-15 18:00:35 CAELUM_UNIT_TEST INFO 139630231111488 Validating list
directory like behavior
>>> obj_list
[<Object: name=testing1/caelum_test1, size=5709, hash=0x8D58AD95464B16D,
provider=Microsoft Azure (blobs) ...>]
>>> obj_list2
[<Object: name=testing2/caelum_test2, size=5709, hash=0x8D58AD954709A90,
provider=Microsoft Azure (blobs) ...>]{code}
was (Author: sehafoc):
Okay, done... Here are the diffs for your review:
{code:java}
diff --git a/libcloud/storage/drivers/azure_blobs.py
b/libcloud/storage/drivers/azure_blobs.py
index 82ee03a..2475793 100644
--- a/libcloud/storage/drivers/azure_blobs.py
+++ b/libcloud/storage/drivers/azure_blobs.py
@@ -381,11 +381,12 @@ class AzureBlobsStorageDriver(StorageDriver):
if not params['marker']:
break
- def iterate_container_objects(self, container):
+ def iterate_container_objects(self, container, ex_prefix=None):
"""
@inherits: :class:`StorageDriver.iterate_container_objects`
"""
- params = {'restype': 'container',
+ params = {'prefix': ex_prefix,
+ 'restype': 'container',
'comp': 'list',
'maxresults': RESPONSES_PER_REQUEST,
'include': 'metadata'}
@@ -416,6 +417,22 @@ class AzureBlobsStorageDriver(StorageDriver):
if not params['marker']:
break
+ def list_container_objects(self, container, ex_prefix=None):
+ """
+ Return a list of objects for the given container.
+
+ :param container: Container instance.
+ :type container: :class:`Container`
+
+ :param ex_prefix: Only return objects starting with ex_prefix
+ :type ex_prefix: ``str``
+
+ :return: A list of Object instances.
+ :rtype: ``list`` of :class:`Object`
+ """
+ return list(self.iterate_container_objects(container,
+ ex_prefix=ex_prefix))
+
def get_container(self, container_name):
"""
@inherits: :class:`StorageDriver.get_container`
diff --git a/libcloud/test/storage/test_azure_blobs.py
b/libcloud/test/storage/test_azure_blobs.py
index 95ab3f1..69bdebb 100644
--- a/libcloud/test/storage/test_azure_blobs.py
+++ b/libcloud/test/storage/test_azure_blobs.py
@@ -447,6 +447,28 @@ class AzureBlobsTests(unittest.TestCase):
self.assertTrue('content_encoding' in obj.extra)
self.assertTrue('content_language' in obj.extra)
+ def test_list_container_objects_with_prefix(self):
+ self.mock_response_klass.type = None
+ AzureBlobsStorageDriver.RESPONSES_PER_REQUEST = 2
+
+ container = Container(name='test_container', extra={},
+ driver=self.driver)
+ objects = self.driver.list_container_objects(container=container,
+ ex_prefix='test_prefix')
+ self.assertEqual(len(objects), 4)
+
+ obj = objects[1]
+ self.assertEqual(obj.name, 'object2.txt')
+ self.assertEqual(obj.hash, '0x8CFB90F1BA8CD8F')
+ self.assertEqual(obj.size, 1048576)
+ self.assertEqual(obj.container.name, 'test_container')
+ self.assertTrue('meta1' in obj.meta_data)
+ self.assertTrue('meta2' in obj.meta_data)
+ self.assertTrue('last_modified' in obj.extra)
+ self.assertTrue('content_type' in obj.extra)
+ self.assertTrue('content_encoding' in obj.extra)
+ self.assertTrue('content_language' in obj.extra)
+
{code}
And the before and after test results:
{code:java}
[root@localhost libcloud]# python3 -m pytest -k azure_blobs
=================================================================== test
session starts
====================================================================
platform linux -- Python 3.4.3, pytest-3.4.2, py-1.5.2, pluggy-0.6.0
rootdir: /usr/libcloud, inifile: pytest.ini
collected 9116 items
libcloud/test/storage/test_azure_blobs.py ....................F................
[100%]
=========================================================================
FAILURES
=========================================================================
_________________________________________________
AzureBlobsTests.test_list_container_objects_with_prefix
__________________________________________________
self = <libcloud.test.storage.test_azure_blobs.AzureBlobsTests
testMethod=test_list_container_objects_with_prefix>
def test_list_container_objects_with_prefix(self):
self.mock_response_klass.type = None
AzureBlobsStorageDriver.RESPONSES_PER_REQUEST = 2
container = Container(name='test_container', extra={},
driver=self.driver)
objects = self.driver.list_container_objects(container=container,
> ex_prefix='test_prefix')
E TypeError: list_container_objects() got an unexpected keyword argument
'ex_prefix'
libcloud/test/storage/test_azure_blobs.py:457: TypeError
================================================================== 9079 tests
deselected ===================================================================
=================================================== 1 failed, 36 passed, 9079
deselected in 3.39 seconds ===================================================
[root@localhost libcloud]# python3 -m pytest -k azure_blobs
=================================================================== test
session starts
====================================================================
platform linux -- Python 3.4.3, pytest-3.4.2, py-1.5.2, pluggy-0.6.0
rootdir: /usr/libcloud, inifile: pytest.ini
collected 9116 items
libcloud/test/storage/test_azure_blobs.py .....................................
[100%]
================================================================== 9079 tests
deselected ===================================================================
======================================================== 37 passed, 9079
deselected in 3.08 seconds
========================================================{code}
Some notes about the unit test for this feature... Ideally you'd have one
container that the Mock HTTP server would be able to return different results
for based on the prefix, I didn't see a way to simply accomplish this without
modifying some of the base mock server code. This test method was largely
pulled from the other driver's tests, but it seems to meet your base
requirements. Another note, I'm not sure why this Jira instance always strips
spaces, even in noformat, or code blocks (Sorry about the formatting).
> List / Iterate container with prefix for Azure Blob service (This support
> exists in the other drivers already)
> --------------------------------------------------------------------------------------------------------------
>
> Key: LIBCLOUD-986
> URL: https://issues.apache.org/jira/browse/LIBCLOUD-986
> Project: Libcloud
> Issue Type: Improvement
> Components: Storage
> Reporter: Joshua Hawkinson
> Priority: Major
>
> While testing out the libcloud storage abstraction I've come across a curious
> omission in driver support. It seems that for many drivers there is
> additional support for the {{list_container_objects()}} and
> {{iterate_container_objects()}} to support a prefix (or {{ex_prefix)}}.
> This functionality is not present in the Azure driver for some reason. I've
> gone and checked the list_blobs API and it also supports this functionality
> as seen here:
> [https://docs.microsoft.com/en-us/rest/api/storageservices/list-blobs]
>
> I've attempted a quick and dirty patch that seems to work as well.
> {code}
> azure_blobs.py Line 384
> - def iterate_container_objects(self, container):
> + def iterate_container_objects(self, container, ex_prefix=None):
> """
> @inherits: :class:`StorageDriver.iterate_container_objects`
> """
> params = {'restype': 'container',
> 'comp': 'list',
> + 'prefix': ex_prefix,
> 'maxresults': RESPONSES_PER_REQUEST,
> 'include': 'metadata'}
> Line 419 (Add new function)
> def list_container_objects(self, container, ex_prefix=None):
> """
> Return a list of objects for the given container.
>
> :param container: Container instance.
> :type container: :class:`Container`
>
> :param ex_prefix: Only return objects starting with ex_prefix
> :type ex_prefix: ``str``
>
> :return: A list of Object instances.
> :rtype: ``list`` of :class:`Object`
> """
> return list(self.iterate_container_objects(container,
> ex_prefix=ex_prefix))
>
> {code}
>
> I've just worked around this at a higher level for now. There isn't any
> urgency on my end, but it would be nice if the abstraction was more
> consistent so I could remove the workaround code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
