[
https://issues.apache.org/jira/browse/HADOOP-14640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085909#comment-16085909
]
Santhosh G Nayak edited comment on HADOOP-14640 at 7/13/17 4:12 PM:
--------------------------------------------------------------------
Thanks [~jnp] for reviewing the patch.
Yes. {{AuthenticatedURL.Token}} does not provide the expiry time of the token.
So, introduced {{SpnegoToken}} class to represent {{SPNEGO}} token, which
provides information about the validity of the token along with the token
itself.
Token used in {{hadoop.auth}} cookie follows SPNEGO standard and will have
expiry time (represented as {{e}}) in it. So, I think it will be better, if we
can reuse that information, instead of setting different expiry time on our end
and treating it opaque.
Regarding {{expiryTime > System.currentTimeMillis() + 1000 * 60 * 5L}}, adding
5 minutes buffer to proactively get new token to save additional network calls
of requests which can definitely fail because of the token expiry and to
eliminate any issues which can arise because of time difference between
machines (if any).
I have attached another patch addressing the checkstyle related comments and a
bug fix where impersonation context is getting lost in
{{SecureWasbRemoteCallHelper.getHttpRequest()}} as it is being invoked inside
{{ugi.doAs()}}.
Tested with {{Azure South India}} Storage endpoint.
{code}
PS D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure> mvn clean package
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Hadoop Azure support 3.0.0-beta1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hadoop-azure ---
[INFO] Deleting D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target
[INFO] Deleting D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure (includes =
[dependency-reduced-pom.xml], excludes = [])
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-azure ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\test-dir
[INFO] Executed tasks
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hadoop-azure
---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hadoop-azure ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hadoop-azure
---
[INFO] Compiling 57 source files to
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\classes
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SecureStorageInterfaceImpl.java:
Some input files use or override a deprecated API.
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SecureStorageInterfaceImpl.java:
Recompile with -Xlint:deprecation for details.
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java:
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\src\main\java\org\apache\hadoop\fs\azure\security\TokenUtils.java
uses unchecked or unsafe operations.
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java:
Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-dependency-plugin:2.10:list (deplist) @ hadoop-azure ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
hadoop-azure ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 4 resources
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hadoop-azure ---
[INFO] Compiling 58 source files to
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\test-classes
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java:
Some input files use or override a deprecated API.
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java:
Recompile with -Xlint:deprecation for details.
[INFO]
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-azure ---
[INFO] Surefire report directory:
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\surefire-reports
-------------------------------------------------------
T E S T S
-------------------------------------------------------
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend
Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 0.525 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate
Tests run: 11, Failures: 0, Errors: 0, Skipped: 11, Time elapsed: 0.467 sec -
in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete
Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 0.386 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
Tests run: 4, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 0.386 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.455 sec -
in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir
Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 0.386 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen
Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.401 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename
Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.35 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek
Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.544 sec -
in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek
Running org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.093 sec - in
org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
Running org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.539 sec - in
org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
Running
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.031 sec - in
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
Running org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.268 sec - in
org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.328 sec - in
org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.871 sec - in
org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode
Running org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.568 sec - in
org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions
Running org.apache.hadoop.fs.azure.TestBlobDataValidation
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.077 sec - in
org.apache.hadoop.fs.azure.TestBlobDataValidation
Running org.apache.hadoop.fs.azure.TestBlobMetadata
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.183 sec - in
org.apache.hadoop.fs.azure.TestBlobMetadata
Running org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.243 sec - in
org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference
Running org.apache.hadoop.fs.azure.TestContainerChecks
Tests run: 4, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 4.482 sec - in
org.apache.hadoop.fs.azure.TestContainerChecks
Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 60.004 sec -
in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionMessage
Tests run: 47, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 398.648 sec -
in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionMessage
Running
org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.549 sec -
in
org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Running org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 450.974 sec -
in org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 111.067 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.359 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization
Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 44.088 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization
Running
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.393 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.202 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.578 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.219 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrencyLive
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.898 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrencyLive
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator
Tests run: 43, Failures: 0, Errors: 0, Skipped: 43, Time elapsed: 0.817 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 131.049 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 2.17 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 149.315 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.143 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 408.38 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.866 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
Tests run: 50, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 2.472 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic
Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 0.147 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic
Running org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 422.47 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive
Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.373 sec - in
org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations
Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.348 sec - in
org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive
Running org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 127.899 sec -
in org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite
Running org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.369 sec - in
org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
Running org.apache.hadoop.fs.azure.TestWasbFsck
Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 1.102 sec - in
org.apache.hadoop.fs.azure.TestWasbFsck
Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 162.873 sec -
in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
Running org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration
Tests run: 18, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 7.954 sec - in
org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration
Results :
Tests run: 707, Failures: 0, Errors: 0, Skipped: 152
[INFO]
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-azure ---
[INFO] Building jar:
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\hadoop-azure-3.0.0-beta1-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @
hadoop-azure ---
[INFO]
[INFO] --- maven-jar-plugin:2.5:test-jar (default) @ hadoop-azure ---
[INFO] Building jar:
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\hadoop-azure-3.0.0-beta1-SNAPSHOT-tests.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 46:44.589s
[INFO] Finished at: Thu Jul 13 20:47:17 IST 2017
[INFO] Final Memory: 35M/514M
[INFO] ------------------------------------------------------------------------
{code}
was (Author: snayak):
Thanks [~jnp] for reviewing the patch.
Yes. {{AuthenticatedURL.Token}} does not provide the expiry time of the token.
So, introduced {{SpnegoToken}} class to represent {{SPNEGO}} token, which
provides information about the validity of the token along with the token
itself.
Token used in {{hadoop.auth}} cookie follows SPNEGO standard and will have
expiry time (represented as {{e}}) in it. So, I think it will be better, if we
can reuse that information, instead of setting different expiry time on our end
and treating it opaque.
Regarding {{expiryTime > System.currentTimeMillis() + 1000 * 60 * 5L}}, adding
5 minutes buffer to proactively get new token to save additional network calls
of requests which can definitely fail because of the token expiry and to
eliminate any issues which can arise because of time difference between
machines (if any).
I have attached another patch addressing the checkstyle related comments and a
bug fix where impersonation context is getting lost in
{{SecureWasbRemoteCallHelper.getHttpRequest()}} as it is being invoked inside
{{ugi.doAs()}}.
Tested against {{Azure South India}} Storage endpoint.
{code}
PS D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure> mvn clean package
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Hadoop Azure support 3.0.0-beta1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hadoop-azure ---
[INFO] Deleting D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target
[INFO] Deleting D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure (includes =
[dependency-reduced-pom.xml], excludes = [])
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-azure ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\test-dir
[INFO] Executed tasks
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hadoop-azure
---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hadoop-azure ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hadoop-azure
---
[INFO] Compiling 57 source files to
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\classes
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SecureStorageInterfaceImpl.java:
Some input files use or override a deprecated API.
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SecureStorageInterfaceImpl.java:
Recompile with -Xlint:deprecation for details.
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java:
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\src\main\java\org\apache\hadoop\fs\azure\security\TokenUtils.java
uses unchecked or unsafe operations.
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java:
Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-dependency-plugin:2.10:list (deplist) @ hadoop-azure ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
hadoop-azure ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 4 resources
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hadoop-azure ---
[INFO] Compiling 58 source files to
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\test-classes
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java:
Some input files use or override a deprecated API.
[WARNING]
/D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java:
Recompile with -Xlint:deprecation for details.
[INFO]
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-azure ---
[INFO] Surefire report directory:
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\surefire-reports
-------------------------------------------------------
T E S T S
-------------------------------------------------------
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend
Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 0.525 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate
Tests run: 11, Failures: 0, Errors: 0, Skipped: 11, Time elapsed: 0.467 sec -
in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete
Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 0.386 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
Tests run: 4, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 0.386 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.455 sec -
in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir
Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 0.386 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen
Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.401 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename
Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.35 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek
Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.544 sec -
in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek
Running org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.093 sec - in
org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
Running org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.539 sec - in
org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
Running
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.031 sec - in
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
Running org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.268 sec - in
org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.328 sec - in
org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.871 sec - in
org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode
Running org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.568 sec - in
org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions
Running org.apache.hadoop.fs.azure.TestBlobDataValidation
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.077 sec - in
org.apache.hadoop.fs.azure.TestBlobDataValidation
Running org.apache.hadoop.fs.azure.TestBlobMetadata
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.183 sec - in
org.apache.hadoop.fs.azure.TestBlobMetadata
Running org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.243 sec - in
org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference
Running org.apache.hadoop.fs.azure.TestContainerChecks
Tests run: 4, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 4.482 sec - in
org.apache.hadoop.fs.azure.TestContainerChecks
Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 60.004 sec -
in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionMessage
Tests run: 47, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 398.648 sec -
in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionMessage
Running
org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.549 sec -
in
org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Running org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 450.974 sec -
in org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 111.067 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.359 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization
Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 44.088 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization
Running
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.393 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.202 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.578 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.219 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrencyLive
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.898 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrencyLive
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator
Tests run: 43, Failures: 0, Errors: 0, Skipped: 43, Time elapsed: 0.817 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 131.049 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 2.17 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 149.315 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.143 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 408.38 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.866 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
Tests run: 50, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 2.472 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic
Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 0.147 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic
Running org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 422.47 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive
Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.373 sec - in
org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations
Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.348 sec - in
org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive
Running org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 127.899 sec -
in org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite
Running org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.369 sec - in
org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
Running org.apache.hadoop.fs.azure.TestWasbFsck
Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 1.102 sec - in
org.apache.hadoop.fs.azure.TestWasbFsck
Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 162.873 sec -
in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
Running org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration
Tests run: 18, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 7.954 sec - in
org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration
Results :
Tests run: 707, Failures: 0, Errors: 0, Skipped: 152
[INFO]
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-azure ---
[INFO] Building jar:
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\hadoop-azure-3.0.0-beta1-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @
hadoop-azure ---
[INFO]
[INFO] --- maven-jar-plugin:2.5:test-jar (default) @ hadoop-azure ---
[INFO] Building jar:
D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\hadoop-azure-3.0.0-beta1-SNAPSHOT-tests.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 46:44.589s
[INFO] Finished at: Thu Jul 13 20:47:17 IST 2017
[INFO] Final Memory: 35M/514M
[INFO] ------------------------------------------------------------------------
{code}
> Azure: Support affinity for service running on localhost and reuse SPNEGO
> hadoop.auth cookie for authorization, SASKey and delegation token generation
> ------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-14640
> URL: https://issues.apache.org/jira/browse/HADOOP-14640
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
> Affects Versions: 2.9.0
> Reporter: Santhosh G Nayak
> Assignee: Santhosh G Nayak
> Labels: security
> Attachments: HADOOP-14640.1.patch, HADOOP-14640.2.patch
>
>
> Currently, {{WasbRemoteCallHelper}} can be configured to talk to comma
> separated list of URLs for authorization, SASKey generation and delegation
> token generation.
> To improve the performance, if service runs on the local machine, give it
> first preference over the other configured list of URLs.
> Currently, {{WasbRemoteCallHelper}} generates {{hadoop.auth}} cookie for
> every request by talking to the remote service, before making actual rest
> requests.
> The proposal is to reuse the {{hadoop.auth}} cookie for subsequent requests
> from same {{WasbRemoteCallHelper}} object until its expiry time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
