Thomas Marqardt reopened HADOOP-17015:

We should revisit PR 2021 and try to find a better solution for rename.  Users 
expect Rename to be atomic.  The service implementation is atomic, but we have 
this client-side idempotency issue.  This fix relies on time and assumes that 
if the destination was recently updated while we are executing a retry policy, 
that we succeeded.  This may not be the case.  For example, users may rely on 
rename (with overwrite = false) of a file to synchronize or act like a 
distributed lock, so who ever renames successfully acquires the lock.  With the 
fix in PR 2021, more than one caller could acquire this lock at the same time.

Instead, I think we could allow the client to provide a UUID for the rename 
operation and persist this UUID in the metadata of the destination blob upon 
successful completion of a rename, then if we get into this idempotency issue 
and the client gets a 404 source does not exist, we can check the destination 
blob's metadata to see if the UUID is a match.

> ABFS: Make PUT and POST operations idempotent
> ---------------------------------------------
>                 Key: HADOOP-17015
>                 URL: https://issues.apache.org/jira/browse/HADOOP-17015
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure
>    Affects Versions: 3.2.1
>            Reporter: Sneha Vijayarajan
>            Assignee: Sneha Vijayarajan
>            Priority: Major
>             Fix For: 3.4.0
> Currently when a PUT or POST operation timeouts and the server has already 
> successfully executed the operation, there is no check in driver to see if 
> the operation did succeed or not and just retries the same operation again. 
> This can cause driver to through invalid user errors.
> Sample scenario:
>  # Rename request times out. Though server has successfully executed the 
> operation.
>  # Driver retries rename and get source not found error.
> In the scenario, driver needs to check if rename is being retried and success 
> if source if not found, but destination is present.

This message was sent by Atlassian Jira

To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Reply via email to