Thomas Marqardt reopened HADOOP-17054:

We should revisit this and try to find a better solution for rename.  App 
developers expect our Rename to be atomic.  The service implementation is 
atomic, but we have this client-side idempotency issue.  Your fix relies on 
time and assumes that if the destination was recently updated while we are 
executing a retry policy, that we succeeded.  This may not be the case.  For 
example, app developers may rely on rename (with overwrite = false) of a file 
to synchronize or act like a distributed lock, so who ever renames successfully 
acquires the lock.  With your fix, multiple callers could acquire this lock at 
the same time.

Instead, I think we could allow the client to provide a UUID for the rename 
operation and persist this UUID in the metadata of the destination blob upon 
successful completion of a rename, then if we get into this idempotency issue 
and the client gets a 404 source does not exist, we can check the destination 
blob's metadata to see if the UUID is a match.

> ABFS: Fix idempotency test failures when SharedKey is set as AuthType
> ---------------------------------------------------------------------
>                 Key: HADOOP-17054
>                 URL: https://issues.apache.org/jira/browse/HADOOP-17054
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure
>    Affects Versions: 3.2.1
>            Reporter: Sneha Vijayarajan
>            Assignee: Sneha Vijayarajan
>            Priority: Major
>             Fix For: 3.4.0
> Idempotency related tests added as part of 
> https://issues.apache.org/jira/browse/HADOOP-17015
> create a test AbfsClient instance. This mock instance wrongly accepts valid 
> sharedKey and oauth token provider instance. This leads to test failures with 
> exceptions:
> [ERROR] 
> testRenameRetryFailureAsHTTP404(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRename)
>   Time elapsed: 9.133 s  <<< ERROR!
>  Invalid auth type: SharedKey is being used, expecting OAuth
>          at 
> org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getTokenProvider(AbfsConfiguration.java:643)
> This Jira is to fix these tests.

This message was sent by Atlassian Jira

To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Reply via email to