[ https://issues.apache.org/jira/browse/HADOOP-17015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Marqardt reopened HADOOP-17015: -------------------------------------- We should revisit PR 2021 and try to find a better solution for rename. Users expect Rename to be atomic. The service implementation is atomic, but we have this client-side idempotency issue. This fix relies on time and assumes that if the destination was recently updated while we are executing a retry policy, that we succeeded. This may not be the case. For example, users may rely on rename (with overwrite = false) of a file to synchronize or act like a distributed lock, so who ever renames successfully acquires the lock. With the fix in PR 2021, more than one caller could acquire this lock at the same time. Instead, I think we could allow the client to provide a UUID for the rename operation and persist this UUID in the metadata of the destination blob upon successful completion of a rename, then if we get into this idempotency issue and the client gets a 404 source does not exist, we can check the destination blob's metadata to see if the UUID is a match. > ABFS: Make PUT and POST operations idempotent > --------------------------------------------- > > Key: HADOOP-17015 > URL: https://issues.apache.org/jira/browse/HADOOP-17015 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure > Affects Versions: 3.2.1 > Reporter: Sneha Vijayarajan > Assignee: Sneha Vijayarajan > Priority: Major > Fix For: 3.4.0 > > > Currently when a PUT or POST operation timeouts and the server has already > successfully executed the operation, there is no check in driver to see if > the operation did succeed or not and just retries the same operation again. > This can cause driver to through invalid user errors. > > Sample scenario: > # Rename request times out. Though server has successfully executed the > operation. > # Driver retries rename and get source not found error. > In the scenario, driver needs to check if rename is being retried and success > if source if not found, but destination is present. > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org