[
https://issues.apache.org/jira/browse/HADOOP-15370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16696674#comment-16696674
]
Gabor Bota commented on HADOOP-15370:
-------------------------------------
mvn verify ran against eu-west-1. s3guard: dynamo, non-auth and auth after that.
Results are not looking good. For non-authoritative:
{noformat}
[ERROR] Failures:
[ERROR]
ITestDynamoDBMetadataStore>MetadataStoreTestBase.testDeleteSubtree:321->MetadataStoreTestBase.deleteSubtreeHelper:346->MetadataStoreTestBase.assertEmptyDirectory:927->MetadataStoreTestBase.assertDirectorySize:882->Assert.assertNotNull:712->Assert.assertTrue:41->Assert.fail:88
Directory /ADirectory2 is null in cache
[ERROR]
ITestDynamoDBMetadataStore>MetadataStoreTestBase.testDeleteSubtreeHostPath:326->MetadataStoreTestBase.deleteSubtreeHelper:346->MetadataStoreTestBase.assertEmptyDirectory:927->MetadataStoreTestBase.assertDirectorySize:882->Assert.assertNotNull:712->Assert.assertTrue:41->Assert.fail:88
Directory s3a://cloudera-dev-gabor-ireland/ADirectory2 is null in cache
[ERROR]
ITestDynamoDBMetadataStore.testProvisionTable:594->Assert.assertEquals:631->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88
expected:<20> but was:<10>
[ERROR]
ITestDynamoDBMetadataStore>MetadataStoreTestBase.testPutNew:243->MetadataStoreTestBase.assertEmptyDirs:932->MetadataStoreTestBase.assertEmptyDirectory:927->MetadataStoreTestBase.assertDirectorySize:882->Assert.assertNotNull:712->Assert.assertTrue:41->Assert.fail:88
Directory /da2 is null in cache
[ERROR] Errors:
[ERROR] ITestDynamoDBMetadataStore.testBatchWrite:318->doTestBatchWrite:365
NullPointer
[ERROR]
ITestDynamoDBMetadataStore>MetadataStoreTestBase.testPruneUnsetsAuthoritative:737
? NullPointer
[ERROR]
ITestS3GuardToolDynamoDB>AbstractS3GuardToolTestBase.testDestroyNoBucket:336->AbstractS3GuardToolTestBase.run:115
? IllegalArgument
[ERROR] ITestS3AMiniYarnCluster.setup:68 ? NoClassDefFound
org/bouncycastle/jce/provid...
{noformat}
I've seen {{ITestS3AMiniYarnCluster#testWithMiniCluster}} and
{{ITestS3GuardToolDynamoDB#testDestroyNoBucket}} but not the others.
These failures/errors are not related to the patch (obviously). I did a
regression testing with clean trunk, and got the same results.
Failures/errors with non-authoritative:
{noformat}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 19.912
s <<< FAILURE! - in org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
[ERROR]
testWithMiniCluster(org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster)
Time elapsed: 19.697 s <<< ERROR!
java.lang.NoClassDefFoundError:
org/bouncycastle/jce/provider/BouncyCastleProvider
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:840)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1270)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:328)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.yarn.server.MiniYARNCluster.initResourceManager(MiniYARNCluster.java:348)
at
org.apache.hadoop.yarn.server.MiniYARNCluster.access$200(MiniYARNCluster.java:128)
at
org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceInit(MiniYARNCluster.java:497)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at
org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:316)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster.setup(ITestS3AMiniYarnCluster.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException:
org.bouncycastle.jce.provider.BouncyCastleProvider
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 27 more
{noformat}
{noformat}
ERROR] Tests run: 40, Failures: 4, Errors: 2, Skipped: 0, Time elapsed: 177.303
s <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore
[ERROR]
testBatchWrite(org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore)
Time elapsed: 1.262 s <<< ERROR!
java.lang.NullPointerException
at
org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore.doTestBatchWrite(ITestDynamoDBMetadataStore.java:365)
(...)
[ERROR]
testProvisionTable(org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore)
Time elapsed: 11.394 s <<< FAILURE!
java.lang.AssertionError: expected:<20> but was:<10> (...)
org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore.testProvisionTable(ITestDynamoDBMetadataStore.java:594)
(...)
[ERROR]
testPruneUnsetsAuthoritative(org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore)
Time elapsed: 2.323 s <<< ERROR!
java.lang.NullPointerException
at
org.apache.hadoop.fs.s3a.s3guard.MetadataStoreTestBase.testPruneUnsetsAuthoritative(MetadataStoreTestBase.java:737)
(...)
[ERROR]
testDeleteSubtree(org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore)
Time elapsed: 2.377 s <<< FAILURE!
java.lang.AssertionError: Directory /ADirectory2 is null in cache (...)
[ERROR] testPutNew(org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore)
Time elapsed: 1.466 s <<< FAILURE!
java.lang.AssertionError: Directory /da2 is null in cache (...)
[ERROR]
testDeleteSubtreeHostPath(org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore)
Time elapsed: 2.378 s <<< FAILURE!
java.lang.AssertionError: Directory
s3a://cloudera-dev-gabor-ireland/ADirectory2 is null in cache (...)
{noformat}
{noformat}
[ERROR]
testDestroyNoBucket(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
Time elapsed: 1.921 s <<< ERROR!
java.lang.IllegalArgumentException: No DynamoDB table name configured (...)
{noformat}
I've created a gist with the results for authoritative and non-authoritative:
https://gist.github.com/bgaborg/e0d8092659cf009fcd2985ca95c574be
> S3A log message on rm s3a://bucket/ not intuitive
> -------------------------------------------------
>
> Key: HADOOP-15370
> URL: https://issues.apache.org/jira/browse/HADOOP-15370
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.1.0
> Reporter: Steve Loughran
> Assignee: Gabor Bota
> Priority: Trivial
> Attachments: HADOOP-15370.001.patch
>
>
> when you try to delete the root of a bucket from command line, e.g. {{hadoop
> fs -rm -r -skipTrash s3a://hwdev-steve-new/}}, the output isn't that useful
> {code}
> 2018-04-06 16:35:23,048 [main] INFO s3a.S3AFileSystem
> (S3AFileSystem.java:rejectRootDirectoryDelete(1837)) - s3a delete the
> hwdev-steve-new root directory of true
> rm: `s3a://hwdev-steve-new/': Input/output error
> 2018-04-06 16:35:23,050 [pool-2-thread-1] DEBUG s3a.S3AFileSystem
> {code}
> the single log message doesn't parse, and the error message raised is lost by
> the FS -rm CLI command (why?)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
