[jira] [Commented] (LIBCLOUD-986) List / Iterate container with prefix for Azure Blob service (This support exists in the other drivers already)

2018-03-22 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/LIBCLOUD-986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410830#comment-16410830
 ] 

ASF GitHub Bot commented on LIBCLOUD-986:
-

Github user asfgit closed the pull request at:

https://github.com/apache/libcloud/pull/1193


> List / Iterate container with prefix for Azure Blob service (This support 
> exists in the other drivers already)
> --
>
> Key: LIBCLOUD-986
> URL: https://issues.apache.org/jira/browse/LIBCLOUD-986
> Project: Libcloud
>  Issue Type: Improvement
>  Components: Storage
>Reporter: Joshua Hawkinson
>Priority: Major
>
> While testing out the libcloud storage abstraction I've come across a curious 
> omission in driver support. It seems that for many drivers there is 
> additional support for the {{list_container_objects()}} and 
> {{iterate_container_objects()}} to support a prefix (or {{ex_prefix)}}. 
> This functionality is not present in the Azure driver for some reason.   I've 
> gone and checked the list_blobs API and it also supports this functionality 
> as seen here:
> [https://docs.microsoft.com/en-us/rest/api/storageservices/list-blobs]
>  
> I've attempted a quick and dirty patch that seems to work as well.
> {code}
> azure_blobs.py Line 384
> -def iterate_container_objects(self, container):
> +def iterate_container_objects(self, container, ex_prefix=None):
> """
> @inherits: :class:`StorageDriver.iterate_container_objects`
> """
> params = {'restype': 'container',
>   'comp': 'list',
> +  'prefix': ex_prefix,
>   'maxresults': RESPONSES_PER_REQUEST,
>   'include': 'metadata'}
> Line 419 (Add new function)
> def list_container_objects(self, container, ex_prefix=None):
> """
> Return a list of objects for the given container.
> 
> :param container: Container instance.
> :type container: :class:`Container`
> 
> :param ex_prefix: Only return objects starting with ex_prefix
> :type ex_prefix: ``str``
> 
> :return: A list of Object instances.
> :rtype: ``list`` of :class:`Object`
> """
> return list(self.iterate_container_objects(container,
>ex_prefix=ex_prefix))
> 
>  {code}
>  
> I've just worked around this at a higher level for now. There isn't any 
> urgency on my end,  but it would be nice if the abstraction was more 
> consistent so I could remove the workaround code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[2/2] libcloud git commit: Add changes for #1193

2018-03-22 Thread quentinp 
Add changes for #1193

Closes #1193


Project: http://git-wip-us.apache.org/repos/asf/libcloud/repo
Commit: http://git-wip-us.apache.org/repos/asf/libcloud/commit/f9520305
Tree: http://git-wip-us.apache.org/repos/asf/libcloud/tree/f9520305
Diff: http://git-wip-us.apache.org/repos/asf/libcloud/diff/f9520305

Branch: refs/heads/trunk
Commit: f9520305279253ec854a0d58aef4c2b722da5f37
Parents: 1af5caf
Author: Quentin Pradet 
Authored: Fri Mar 23 09:33:11 2018 +0400
Committer: Quentin Pradet 
Committed: Fri Mar 23 09:33:11 2018 +0400

--
 CHANGES.rst | 6 ++
 1 file changed, 6 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/libcloud/blob/f9520305/CHANGES.rst
--
diff --git a/CHANGES.rst b/CHANGES.rst
index 5193ba4..c98b208 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -51,6 +51,12 @@ Compute
 - [OpenStack v2] Fix image members methods (GITHUB-1190)
   [Rick van de Loo]
 
+Storage
+~~~
+
+- [Azure Blob storage] Allow filtering lists by prefix (LIBCLOUD-986, 
GITHUB-1193)
+  [Joshua Hawkinson]
+
 Changes in Apache Libcloud 2.3.0

[GitHub] libcloud pull request #1193: LIBCLOUD-986: Add prefix option to azure blob l...

2018-03-22 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/libcloud/pull/1193


---

[jira] [Commented] (LIBCLOUD-986) List / Iterate container with prefix for Azure Blob service (This support exists in the other drivers already)

2018-03-22 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/LIBCLOUD-986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410829#comment-16410829
 ] 

ASF subversion and git services commented on LIBCLOUD-986:
--

Commit 1af5caf92a97092f7f11221ca881127e90807cb5 in libcloud's branch 
refs/heads/trunk from jhawk
[ https://git-wip-us.apache.org/repos/asf?p=libcloud.git;h=1af5caf ]

LIBCLOUD-986: Add prefix option to azure blob list and iterate container methods

Signed-off-by: Quentin Pradet 


> List / Iterate container with prefix for Azure Blob service (This support 
> exists in the other drivers already)
> --
>
> Key: LIBCLOUD-986
> URL: https://issues.apache.org/jira/browse/LIBCLOUD-986
> Project: Libcloud
>  Issue Type: Improvement
>  Components: Storage
>Reporter: Joshua Hawkinson
>Priority: Major
>
> While testing out the libcloud storage abstraction I've come across a curious 
> omission in driver support. It seems that for many drivers there is 
> additional support for the {{list_container_objects()}} and 
> {{iterate_container_objects()}} to support a prefix (or {{ex_prefix)}}. 
> This functionality is not present in the Azure driver for some reason.   I've 
> gone and checked the list_blobs API and it also supports this functionality 
> as seen here:
> [https://docs.microsoft.com/en-us/rest/api/storageservices/list-blobs]
>  
> I've attempted a quick and dirty patch that seems to work as well.
> {code}
> azure_blobs.py Line 384
> -def iterate_container_objects(self, container):
> +def iterate_container_objects(self, container, ex_prefix=None):
> """
> @inherits: :class:`StorageDriver.iterate_container_objects`
> """
> params = {'restype': 'container',
>   'comp': 'list',
> +  'prefix': ex_prefix,
>   'maxresults': RESPONSES_PER_REQUEST,
>   'include': 'metadata'}
> Line 419 (Add new function)
> def list_container_objects(self, container, ex_prefix=None):
> """
> Return a list of objects for the given container.
> 
> :param container: Container instance.
> :type container: :class:`Container`
> 
> :param ex_prefix: Only return objects starting with ex_prefix
> :type ex_prefix: ``str``
> 
> :return: A list of Object instances.
> :rtype: ``list`` of :class:`Object`
> """
> return list(self.iterate_container_objects(container,
>ex_prefix=ex_prefix))
> 
>  {code}
>  
> I've just worked around this at a higher level for now. There isn't any 
> urgency on my end,  but it would be nice if the abstraction was more 
> consistent so I could remove the workaround code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[1/2] libcloud git commit: LIBCLOUD-986: Add prefix option to azure blob list and iterate container methods

2018-03-22 Thread quentinp 
Repository: libcloud
Updated Branches:
  refs/heads/trunk f8634a8cd -> f95203052


LIBCLOUD-986: Add prefix option to azure blob list and iterate container methods

Signed-off-by: Quentin Pradet 


Project: http://git-wip-us.apache.org/repos/asf/libcloud/repo
Commit: http://git-wip-us.apache.org/repos/asf/libcloud/commit/1af5caf9
Tree: http://git-wip-us.apache.org/repos/asf/libcloud/tree/1af5caf9
Diff: http://git-wip-us.apache.org/repos/asf/libcloud/diff/1af5caf9

Branch: refs/heads/trunk
Commit: 1af5caf92a97092f7f11221ca881127e90807cb5
Parents: f8634a8
Author: jhawk 
Authored: Wed Mar 21 12:08:02 2018 -0700
Committer: Quentin Pradet 
Committed: Fri Mar 23 09:27:28 2018 +0400

--
 libcloud/storage/drivers/azure_blobs.py   | 21 +++--
 libcloud/test/storage/test_azure_blobs.py | 22 ++
 2 files changed, 41 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/libcloud/blob/1af5caf9/libcloud/storage/drivers/azure_blobs.py
--
diff --git a/libcloud/storage/drivers/azure_blobs.py 
b/libcloud/storage/drivers/azure_blobs.py
index 82ee03a..2475793 100644
--- a/libcloud/storage/drivers/azure_blobs.py
+++ b/libcloud/storage/drivers/azure_blobs.py
@@ -381,11 +381,12 @@ class AzureBlobsStorageDriver(StorageDriver):
 if not params['marker']:
 break
 
-def iterate_container_objects(self, container):
+def iterate_container_objects(self, container, ex_prefix=None):
 """
 @inherits: :class:`StorageDriver.iterate_container_objects`
 """
-params = {'restype': 'container',
+params = {'prefix': ex_prefix,
+  'restype': 'container',
   'comp': 'list',
   'maxresults': RESPONSES_PER_REQUEST,
   'include': 'metadata'}
@@ -416,6 +417,22 @@ class AzureBlobsStorageDriver(StorageDriver):
 if not params['marker']:
 break
 
+def list_container_objects(self, container, ex_prefix=None):
+"""
+Return a list of objects for the given container.
+
+:param container: Container instance.
+:type container: :class:`Container`
+
+:param ex_prefix: Only return objects starting with ex_prefix
+:type ex_prefix: ``str``
+
+:return: A list of Object instances.
+:rtype: ``list`` of :class:`Object`
+"""
+return list(self.iterate_container_objects(container,
+   ex_prefix=ex_prefix))
+
 def get_container(self, container_name):
 """
 @inherits: :class:`StorageDriver.get_container`

http://git-wip-us.apache.org/repos/asf/libcloud/blob/1af5caf9/libcloud/test/storage/test_azure_blobs.py
--
diff --git a/libcloud/test/storage/test_azure_blobs.py 
b/libcloud/test/storage/test_azure_blobs.py
index 95ab3f1..69bdebb 100644
--- a/libcloud/test/storage/test_azure_blobs.py
+++ b/libcloud/test/storage/test_azure_blobs.py
@@ -447,6 +447,28 @@ class AzureBlobsTests(unittest.TestCase):
 self.assertTrue('content_encoding' in obj.extra)
 self.assertTrue('content_language' in obj.extra)
 
+def test_list_container_objects_with_prefix(self):
+self.mock_response_klass.type = None
+AzureBlobsStorageDriver.RESPONSES_PER_REQUEST = 2
+
+container = Container(name='test_container', extra={},
+  driver=self.driver)
+objects = self.driver.list_container_objects(container=container,
+ ex_prefix='test_prefix')
+self.assertEqual(len(objects), 4)
+
+obj = objects[1]
+self.assertEqual(obj.name, 'object2.txt')
+self.assertEqual(obj.hash, '0x8CFB90F1BA8CD8F')
+self.assertEqual(obj.size, 1048576)
+self.assertEqual(obj.container.name, 'test_container')
+self.assertTrue('meta1' in obj.meta_data)
+self.assertTrue('meta2' in obj.meta_data)
+self.assertTrue('last_modified' in obj.extra)
+self.assertTrue('content_type' in obj.extra)
+self.assertTrue('content_encoding' in obj.extra)
+self.assertTrue('content_language' in obj.extra)
+
 def test_get_container_doesnt_exist(self):
 self.mock_response_klass.type = None
 try: