[
https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949782#comment-16949782
]
Jeetesh Mangwani commented on HADOOP-16612:
-------------------------------------------
I have fixed the checkstyle issues. Please take a look.
When I run tests on my machine, I see 4 failing tests on my 'trunk' branch.
These tests fail on my feature branch too.
1. ITestGetNameSpaceEnabled.testNonXNSAccount: fails because the HTTP response
status is not 400, but is 404
2. ITestAzureBlobFileSystemCLI.testMkdirRootNonExistentContainer: fails in the
setup phase because the HTTP response status is not 400, but is 404
3. ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek:
times out, probably because this is a scale test and my machine is slow
4. ITestAzureBlobFileSystemE2EScale.testWriteHeavyBytesToFileAcrossThreads:
times out, probably because there are lot of heavy writes and my machine is slow
Here are the details:
trunk, non-xns
[INFO] Tests run: 42, Failures: 0, Errors: 0, Skipped: 0
[ERROR] Failures:
[ERROR]
ITestGetNameSpaceEnabled.testNonXNSAccount:59->Assert.assertFalse:64->Assert.assertTrue:41->Assert.fail:88
Expecting getIsNamespaceEnabled() return false
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemCLI>AbstractAbfsIntegrationTest.setup:137 »
AbfsRestOperation
[INFO]
[ERROR] Tests run: 395, Failures: 1, Errors: 1, Skipped: 21
[ERROR] Errors:
[ERROR]
ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek:60->testReadWriteAndSeek:75
» TestTimedOut
[ERROR]
ITestAzureBlobFileSystemE2EScale.testWriteHeavyBytesToFileAcrossThreads:77 »
TestTimedOut
[INFO]
[ERROR] Tests run: 192, Failures: 0, Errors: 2, Skipped: 24
===================================
feature branch, non-xns
[INFO] Tests run: 53, Failures: 0, Errors: 0, Skipped: 0
[ERROR] Failures:
[ERROR]
ITestGetNameSpaceEnabled.testNonXNSAccount:59->Assert.assertFalse:64->Assert.assertTrue:41->Assert.fail:88
Expecting getIsNamespaceEnabled() return false
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemCLI>AbstractAbfsIntegrationTest.setup:137 »
AbfsRestOperation
[INFO]
[ERROR] Tests run: 395, Failures: 1, Errors: 1, Skipped: 21
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]
ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek:60->testReadWriteAndSeek:75
» TestTimedOut
[ERROR]
ITestAzureBlobFileSystemE2EScale.testWriteHeavyBytesToFileAcrossThreads:77 »
TestTimedOut
[INFO]
[ERROR] Tests run: 192, Failures: 0, Errors: 2, Skipped: 24
================================
trunk, xns
[INFO] Tests run: 42, Failures: 0, Errors: 0, Skipped: 0
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemCLI>AbstractAbfsIntegrationTest.setup:137 »
AbfsRestOperation
[INFO]
[ERROR] Tests run: 395, Failures: 0, Errors: 1, Skipped: 21
[ERROR] Errors:
[ERROR]
ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek:60->testReadWriteAndSeek:75
» TestTimedOut
[ERROR]
ITestAzureBlobFileSystemE2EScale.testWriteHeavyBytesToFileAcrossThreads:77 »
TestTimedOut
[INFO]
[ERROR] Tests run: 192, Failures: 0, Errors: 2, Skipped: 24
=================================
feature branch, xns
[INFO] Tests run: 53, Failures: 0, Errors: 0, Skipped: 0
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemCLI>AbstractAbfsIntegrationTest.setup:137 »
AbfsRestOperation
[INFO]
[ERROR] Tests run: 395, Failures: 0, Errors: 1, Skipped: 21
[ERROR] Errors:
[ERROR]
ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek:60->testReadWriteAndSeek:75
» TestTimedOut
[ERROR]
ITestAzureBlobFileSystemE2EScale.testWriteHeavyBytesToFileAcrossThreads:77 »
TestTimedOut
[INFO]
[ERROR] Tests run: 192, Failures: 0, Errors: 2, Skipped: 24
Error stack traces:
[ERROR]
testMkdirRootNonExistentContainer(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCLI)
Time elapsed: 18.27 s <<< ERROR!
Operation failed: "The specified filesystem does not exist.", 404, HEAD,
https://abfstest02.dfs.core.windows.net/abfs-testcontainer-aa478873-647e-455a-9a71-4cb6da30a088//?upn=false&action=getAccessControl&timeout=90
at
org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:143)
at
org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:523)
at
org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:506)
at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getIsNamespaceEnabled(AzureBlobFileSystemStore.java:224)
at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getIsNamespaceEnabled(AzureBlobFileSystem.java:1108)
at
org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.setup(AbstractAbfsIntegrationTest.java:137)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
[ERROR]
testNonXNSAccount(org.apache.hadoop.fs.azurebfs.ITestGetNameSpaceEnabled) Time
elapsed: 0.808 s <<< FAILURE!
java.lang.AssertionError: Expecting getIsNamespaceEnabled() return false
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertFalse(Assert.java:64)
at
org.apache.hadoop.fs.azurebfs.ITestGetNameSpaceEnabled.testNonXNSAccount(ITestGetNameSpaceEnabled.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
[ERROR]
testReadAndWriteWithDifferentBufferSizesAndSeek[Size=104,857,600](org.apache.hadoop.fs.azurebfs.ITestAbfsReadWriteAndSeek)
Time elapsed: 1,800.017 s <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 1800000
milliseconds
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at
org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.flushWrittenBytesToService(AbfsOutputStream.java:317)
at
org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.flushInternal(AbfsOutputStream.java:265)
at
org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.close(AbfsOutputStream.java:248)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at
org.apache.hadoop.fs.azurebfs.ITestAbfsReadWriteAndSeek.testReadWriteAndSeek(ITestAbfsReadWriteAndSeek.java:75)
at
org.apache.hadoop.fs.azurebfs.ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek(ITestAbfsReadWriteAndSeek.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
[ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
1,970.475 s <<< FAILURE! - in
org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2EScale
[ERROR]
testWriteHeavyBytesToFileAcrossThreads(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2EScale)
Time elapsed: 1,800.018 s <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 1800000
milliseconds
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at
org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2EScale.testWriteHeavyBytesToFileAcrossThreads(ITestAzureBlobFileSystemE2EScale.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
> Track Azure Blob File System client-perceived latency
> -----------------------------------------------------
>
> Key: HADOOP-16612
> URL: https://issues.apache.org/jira/browse/HADOOP-16612
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, hdfs-client
> Reporter: Jeetesh Mangwani
> Assignee: Jeetesh Mangwani
> Priority: Major
>
> Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring
> latencies in the Hadoop ABFS driver.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
