Thomas Marqardt resolved HADOOP-17015.
    Resolution: Fixed

Sneha and I discussed this.  The common Hadoop scenario is a case where you 
have one or more tasks, each operating on different source files, all 
attempting to rename to a common destination. In this scenario, the fix in PR 
2021 is correct.  There are scenarios where PR 2021 will lead to incorrect 
results, but they seem to be very contrived and unlikely in Hadoop.  A work 
item will be opened to investigate the need to improve this on the server-side, 
for example by allowing an operation-id to be passed to the rename operation 
and persisted in the destination metadata, but for now we have this fix to the 
driver on the client-side. 

> ABFS: Make PUT and POST operations idempotent
> ---------------------------------------------
>                 Key: HADOOP-17015
>                 URL: https://issues.apache.org/jira/browse/HADOOP-17015
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure
>    Affects Versions: 3.2.1
>            Reporter: Sneha Vijayarajan
>            Assignee: Sneha Vijayarajan
>            Priority: Major
>             Fix For: 3.4.0
> Currently when a PUT or POST operation timeouts and the server has already 
> successfully executed the operation, there is no check in driver to see if 
> the operation did succeed or not and just retries the same operation again. 
> This can cause driver to through invalid user errors.
> Sample scenario:
>  # Rename request times out. Though server has successfully executed the 
> operation.
>  # Driver retries rename and get source not found error.
> In the scenario, driver needs to check if rename is being retried and success 
> if source if not found, but destination is present.

This message was sent by Atlassian Jira

To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Reply via email to