[
https://issues.apache.org/jira/browse/HADOOP-17015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106524#comment-17106524
]
Sneha Vijayarajan commented on HADOOP-17015:
--------------------------------------------
Thanks [[email protected]] for the pointers. S3Guard seems to be providing a
higher cached service for object store operations.
In case of ABFS, for PUT/POST operations like create, mkdirs, append, flush,
setowner, setpermission, setacl, the retry will re-execute the operation and
hence idempotency is maintained by store REST API level. Rename and Delete seem
to be the only 2 operations which will result in user error on retry.
I checked on the possibilities of using etag which does get set at server end.
But any checks on etag would lead to additional GetFileStatus call, which is
better to avoid for the not so common retry case.
For rename i have added a LastModifiedTime check to identify if the destination
file was recently modified to qualify for the rename operation to be success.
Delete I have followed your pointer of assuming to be idempotent by default.
Both Rename and delete operations do have the some corner cases if parallel
operations are happening over the file through another interface. But as the
chances are quite low to hit these corner cases, going ahead with this logic.
> ABFS: Make PUT and POST operations idempotent
> ---------------------------------------------
>
> Key: HADOOP-17015
> URL: https://issues.apache.org/jira/browse/HADOOP-17015
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.2.1
> Reporter: Sneha Vijayarajan
> Assignee: Sneha Vijayarajan
> Priority: Major
> Fix For: 3.4.0
>
>
> Currently when a PUT or POST operation timeouts and the server has already
> successfully executed the operation, there is no check in driver to see if
> the operation did succeed or not and just retries the same operation again.
> This can cause driver to through invalid user errors.
>
> Sample scenario:
> # Rename request times out. Though server has successfully executed the
> operation.
> # Driver retries rename and get source not found error.
> In the scenario, driver needs to check if rename is being retried and success
> if source if not found, but destination is present.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
