[
https://issues.apache.org/jira/browse/HADOOP-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070387#comment-16070387
]
Santhosh G Nayak commented on HADOOP-14443:
-------------------------------------------
[~liuml07] v7 Patch looks good. Tested against {{Azure Storage South India}}
endpoint. Following is the test report,
{code}
PS D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure> mvn test -q
-------------------------------------------------------
T E S T S
-------------------------------------------------------
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend
Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 0.451 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate
Tests run: 11, Failures: 0, Errors: 0, Skipped: 11, Time elapsed: 0.47 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete
Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 0.432 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
Tests run: 4, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 0.42 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.524 sec -
in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir
Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 0.378 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen
Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.448 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename
Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.385 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek
Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.589 sec -
in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek
Running org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.732 sec - in
org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
Running org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.039 sec - in
org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
Running
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.403 sec - in
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
Running org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.263 sec - in
org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.731 sec - in
org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.885 sec - in
org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode
Running org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.62 sec - in
org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions
Running org.apache.hadoop.fs.azure.TestBlobDataValidation
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.685 sec - in
org.apache.hadoop.fs.azure.TestBlobDataValidation
Running org.apache.hadoop.fs.azure.TestBlobMetadata
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.24 sec - in
org.apache.hadoop.fs.azure.TestBlobMetadata
Running org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.167 sec - in
org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference
Running org.apache.hadoop.fs.azure.TestContainerChecks
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.588 sec - in
org.apache.hadoop.fs.azure.TestContainerChecks
Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.563 sec -
in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionMessage
Tests run: 47, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 378.382 sec -
in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionMessage
Running
org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.937 sec -
in
org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Running org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 425.129 sec -
in org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 130.504 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.671 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization
Tests run: 21, Failures: 0, Errors: 0, Skipped: 21, Time elapsed: 7.166 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization
Running
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner
Tests run: 24, Failures: 0, Errors: 0, Skipped: 24, Time elapsed: 5.773 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.291 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.518 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.771 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrencyLive
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.408 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrencyLive
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator
Tests run: 43, Failures: 0, Errors: 0, Skipped: 43, Time elapsed: 0.717 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 149.625 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 2.365 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 166.31 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.187 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 452.95 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.95 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
Tests run: 50, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 2.394 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic
Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 0.085 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic
Running org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 428.76 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive
Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.566 sec - in
org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations
Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.849 sec - in
org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive
Running org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 173.356 sec -
in org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite
Running org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.385 sec - in
org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
Running org.apache.hadoop.fs.azure.TestWasbFsck
Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 1.066 sec - in
org.apache.hadoop.fs.azure.TestWasbFsck
Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 2.896 sec -
in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
Running org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration
Tests run: 18, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 14.022 sec -
in org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration
Results :
Tests run: 707, Failures: 0, Errors: 0, Skipped: 203
{code}
> Azure: Add retry and client side failover for authorization, SASKey and
> delegation token generation
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-14443
> URL: https://issues.apache.org/jira/browse/HADOOP-14443
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
> Affects Versions: 2.9.0
> Reporter: Santhosh G Nayak
> Assignee: Santhosh G Nayak
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HADOOP-14443.1.patch, HADOOP-14443.2.patch,
> HADOOP-14443.3.patch, HADOOP-14443.4.patch, HADOOP-14443.5.patch,
> HADOOP-14443.6.patch, HADOOP-14443.7.patch
>
>
> Currently, {{WasRemoteCallHelper}} can be configured to talk to only one URL
> for authorization, SASKey generation and delegation token generation. If for
> some reason the service is down, all the requests will fail.
> So proposal is to,
> - Add support to configure multiple URLs, so that if communication to one URL
> fails, client can retry on another instance of the service running on
> different node for authorization, SASKey generation and delegation token
> generation.
> - Rename the configurations {{fs.azure.authorization.remote.service.url}} to
> {{fs.azure.authorization.remote.service.urls}} and
> {{fs.azure.cred.service.url}} to {{fs.azure.cred.service.urls}} to support
> the comma separated list of URLs.
> - Introduce a new configuration {{fs.azure.delegation.token.service.urls}} to
> configure the comma separated list of service URLs to get the delegation
> token.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
