[
https://issues.apache.org/jira/browse/LIBCLOUD-275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13546710#comment-13546710
]
Mahendra M commented on LIBCLOUD-275:
-------------------------------------
I was thinking on the points that you mentioned. Very valid. I think we can
tackle this problem in one of the following ways
1) Have the function catch all exceptions and return a tuple as follows
( [list, of, successful, files], [list, of, failed, files or exceptions] )
Good - it is easy to implement and handle
Bad
* The caller does not have any control. The caller will have to wait for the
entire operation to succeed.
* In case of large directories or files, the lists can become pretty big. (more
memory)
2) Pass an argument - fail_on_error, which will cause the function to return on
the first error
3) Let the function return a generator, which will make the function behave as
follows
for (file_name, status, exp) in driver.sync_from_local( .... ):
if not status:
...
Good
* Easy to implement.
* Lets the developer have control. He can chose to continue or abort based on
business rules.
* Can be used for showing progress information. (I was planning on something
similar for file uploads/downloads also)
* No need of returning big lists. Lesser memory used.
Bad:
* Developer has to always call the function in loop or call list( function() )
etc.
> Utility APIs for containers
> ---------------------------
>
> Key: LIBCLOUD-275
> URL: https://issues.apache.org/jira/browse/LIBCLOUD-275
> Project: Libcloud
> Issue Type: New Feature
> Components: Storage
> Reporter: Mahendra M
> Assignee: Mahendra M
> Priority: Minor
>
> While working with libcloud, I thought it will be good to have some utility
> APIs to containers, which will help in easier use. Some APIs will have a base
> class implementation, which can be easily over-written in each driver based
> on the storage provider.
> Some APIs that I have in mind are:
> 1. container.copy_object(obj, target_container)
> Copies an object from one container to another. Base class implementation can
> be
>
> target_container.upload_object_via_stream(container.download_object_as_stream(obj),
> obj.name)
> In each driver, this can be over-written to use specific methods to do this
> (AWS, Azure etc. provide APIs for doing this).
> 2. container.move_object(obj, target_container)
> Base class implementation can be:
> container.copy_object(obj, target_container)
> obj.delete()
> Again, driver specific extensions can be done.
> 3. container.sync_to_local(local_path, only_missing=True, only_newer=False)
> 4. container.sync_from_local(local_path, followlinks=False, prefix='',
> only_missing=True, only_newer=False)
> The above two functions are good for syncing local folder repos with cloud.
> The functions are ridiculously easy to implement and will provide good
> mechanism for sync. (For eg: syncing a folder which contains 500 HLS chunks
> of a video)
> The parameters used are:
> * only_missing - sync only files that missing in the target (container or
> folder)
> * only_newer - sync only files that have last-modified (or m_time) more than
> the target
> * prefix - A prefix to be prefixed to each object name while syncing from a
> local folder.
> I am just opening a ticket so that we can discuss this and finalize it. Once
> everyone agree, I can submit/check-in a patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
