[
https://issues.apache.org/jira/browse/HADOOP-17015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt resolved HADOOP-17015.
--------------------------------------
Resolution: Fixed
Sneha and I discussed this. The common Hadoop scenario is a case where you
have one or more tasks, each operating on different source files, all
attempting to rename to a common destination. In this scenario, the fix in PR
2021 is correct. There are scenarios where PR 2021 will lead to incorrect
results, but they seem to be very contrived and unlikely in Hadoop. A work
item will be opened to investigate the need to improve this on the server-side,
for example by allowing an operation-id to be passed to the rename operation
and persisted in the destination metadata, but for now we have this fix to the
driver on the client-side.
> ABFS: Make PUT and POST operations idempotent
> ---------------------------------------------
>
> Key: HADOOP-17015
> URL: https://issues.apache.org/jira/browse/HADOOP-17015
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.2.1
> Reporter: Sneha Vijayarajan
> Assignee: Sneha Vijayarajan
> Priority: Major
> Fix For: 3.4.0
>
>
> Currently when a PUT or POST operation timeouts and the server has already
> successfully executed the operation, there is no check in driver to see if
> the operation did succeed or not and just retries the same operation again.
> This can cause driver to through invalid user errors.
>
> Sample scenario:
> # Rename request times out. Though server has successfully executed the
> operation.
> # Driver retries rename and get source not found error.
> In the scenario, driver needs to check if rename is being retried and success
> if source if not found, but destination is present.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
