ThomasMarquardt commented on pull request #2246:
URL: https://github.com/apache/hadoop/pull/2246#issuecomment-695146807
commit e31a636e922a8fdbe0aa7cca53f6de7175e97254
Author: Sneha Vijayarajan <sneha.vijayara...@gmail.com>
Date: Wed Aug 26 00:31:35 2020 +0530
HADOOP-17215: Support for conditional overwrite.
Contributed by Sneha Vijayarajan
DETAILS:
This change adds config key
"fs.azure.enable.conditional.create.overwrite" with
a default of true. When enabled, if create(path, overwrite: true)
is invoked
and the file exists, the ABFS driver will first obtain its etag and
then attempt
to overwrite the file on the condition that the etag matches. The
purpose of this
is to mitigate the non-idempotency of this method. Specifically, in
the event of
a network error or similar, the client will retry and this can
result in the file
being created more than once which may result in data loss. In
essense this is
like a poor man's file handle, and will be addressed more thoroughly
in the future
when support for lease is added to ABFS.
TEST RESULTS:
namespace.enabled=true
auth.type=SharedKey
-------------------
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean
verify
Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
Tests run: 457, Failures: 0, Errors: 0, Skipped: 42
Tests run: 207, Failures: 0, Errors: 0, Skipped: 24
namespace.enabled=true
auth.type=OAuth
-------------------
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean
verify
Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
Tests run: 457, Failures: 0, Errors: 0, Skipped: 74
Tests run: 207, Failures: 0, Errors: 0, Skipped: 140
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
