pranavsaxena-microsoft opened a new pull request, #5109: URL: https://github.com/apache/hadoop/pull/5109
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
JIRA: https://issues.apache.org/jira/browse/HADOOP-18501
Error Description:
For partial read (due to account backend throttling), the ABFS driver retry
but doesn't add up in the throttling metrics.
In case of partial read with connection-reset exception, ABFS driver retry
for the full request and doesn't add up in throttling metrics.
Mitigation:
In case of partial read, ABFS Driver should retry for the remaining bytes,
and it should be added in throttling metrics.
### How was this patch tested?
Ran integeration and unit tests on the following accounts:
1. pranavsaxenahns: EAST_US: HNS account
2. pranavsaxenanonhns: EAST_US: Non-HNS account
Test results:
NonHNS-SharedKey
========================
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 107, Failures: 1, Errors: 0, Skipped: 2
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:269->Assert.assertTrue:42->Assert.fail:89
There should not be any network I/O
(elapsedTimeMs=24).
[ERROR]
ITestPartialRead.testPartialReadWithConnectionReset:189->Assert.assertEquals:633->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89
expected:<4> but was:<8>
[ERROR]
ITestPartialRead.testRecoverPartialRead:112->Assert.assertEquals:633->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89
expected:<4> but was:<0>
[ERROR] Errors:
[ERROR]
ITestAzureBlobFileSystemLease.testAcquireRetry:344->lambda$testAcquireRetry$6:345
» TestTimedOut
[INFO]
[ERROR] Tests run: 568, Failures: 3,
Errors: 1, Skipped: 276
[INFO] Results:
[INFO]
[ERRO
R] Failures:
[ERROR]
ITestAbfsTerasort.test_110_teragen:244->executeStage:211->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89
teragen(1000,
abfs://testcontai...@pranavsaxenanonhns.dfs.core.windows.net/ITestAbfsTerasort/sortin)
failed expected:<0> but was:<1>
[ERROR] ITestReadBufferManager.testPurgeBufferManagerForParallelStreams:85
[After closing all streams free list contents should match with [0, 1, 2, 3, 4,
5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]]
Expected size:<16> but was:<9> in:
<[9, 12, 10, 13, 11, 8, 7, 15, 14]>
[ERROR] Errors:
[ERROR]
ITestAbfsJobThroughManifestCommitter.test_0420_validateJob » OutputValidation
...
[ERROR]
ITestAbfsManifestCommitProtocol.testCommitLifecycle » OutputValidation
`abfs:/...
[ERROR]
ITestAbfsManifestCommitProtocol.testCommitterWithDuplicatedCommit »
OutputValidation
[ERROR] ITestAbfsManifestCommitProtocol.testConcurrentCommitTaskWithSubDir »
OutputValidation
[ERROR] ITestAbfsManifestCommitProtocol.testMapFileOutputCommitter »
OutputValidation ...
[ERROR] ITestAbfsManifestCommitProtocol.testOutputFormatIntegration »
OutputValidation
[ERROR]
ITestAbfsManifestCommitProtocol.testParallelJobsToAdjacentPaths »
OutputValidation
[ERROR] ITestAbfsManifestCommitProtocol.testTwoTaskAttemptsCommit »
OutputValidation `...
[INFO]
[ERROR] Tests run: 333, Failures: 2, Errors: 8, Skipped: 46
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
