[
https://issues.apache.org/jira/browse/HADOOP-13650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826630#comment-15826630
]
Chris Nauroth commented on HADOOP-13650:
----------------------------------------
I'm getting failures in {{TestS3GuardTool}}. Is anyone else seeing this?
{code}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.427 sec <<<
FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.TestS3GuardTool
testInitDynamoDBMetadataStore(org.apache.hadoop.fs.s3a.s3guard.TestS3GuardTool)
Time elapsed: 1.377 sec <<< FAILURE!
java.lang.AssertionError: expected:<0> but was:<-1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.fs.s3a.s3guard.TestS3GuardTool.testInitDynamoDBMetadataStore(TestS3GuardTool.java:112)
{code}
{code}
Must specify -e ENDPOINT or fs.s3a.s3guard.ddb.endpoint in conf.
{code}
I had specified {{fs.s3a.endpoint}}, but not {{fs.s3a.s3guard.ddb.endpoint}},
so I tried adding {{fs.s3a.s3guard.ddb.endpoint}}, and then I got a different
error:
{code}
2017-01-17 10:59:59,696 [main] INFO s3guard.S3GuardTool
(S3GuardTool.java:initMetadataStore(148)) - create metadata store:
dynamodb://s3guard_test_init_ddb_table scheme: dynamodb
2017-01-17 11:00:00,092 [main] INFO json.JsonContent
(JsonContent.java:parseJsonContent(74)) - Unable to parse HTTP response content
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code
60)): expected a valid value (number, String, array, object, 'true', 'false' or
'null')
at [Source: [B@2b5825fa; line: 1, column: 2]
at
com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1586)
at
com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:521)
at
com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:450)
at
com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2631)
at
com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:854)
at
com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:748)
at
com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3847)
at
com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3792)
at
com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2355)
at
com.amazonaws.protocol.json.JsonContent.parseJsonContent(JsonContent.java:72)
at com.amazonaws.protocol.json.JsonContent.<init>(JsonContent.java:64)
at
com.amazonaws.protocol.json.JsonContent.createJsonContent(JsonContent.java:54)
at
com.amazonaws.http.JsonErrorResponseHandler.handle(JsonErrorResponseHandler.java:61)
at
com.amazonaws.http.JsonErrorResponseHandler.handle(JsonErrorResponseHandler.java:33)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1495)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.describeTable(AmazonDynamoDBClient.java:1096)
at
com.amazonaws.services.dynamodbv2.document.Table.describe(Table.java:130)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initTable(DynamoDBMetadataStore.java:572)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:270)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.initMetadataStore(S3GuardTool.java:171)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$InitMetadata.run(S3GuardTool.java:257)
at
org.apache.hadoop.fs.s3a.s3guard.TestS3GuardTool.testInitDynamoDBMetadataStore(TestS3GuardTool.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:254)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:149)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
{code}
Is there a certain configuration assumed that I need to set up in order to be
able to run these tests?
Also, it appears these tests can connect to AWS services, and if you don't have
an auth-keys.xml file, then the tests get skipped via the
{{AssumptionViolatedException}} thrown from
{{S3ATestUtils#createTestFileSystem}}. If this is intentional, then we likely
need to rename {{TestS3GuardTool}} to {{ITestS3GuardTool}}, because the unit
test phase is intended for tests that can run without integrating with AWS
services (including during Jenkins pre-commit).
> S3Guard: Provide command line tools to manipulate metadata store.
> -----------------------------------------------------------------
>
> Key: HADOOP-13650
> URL: https://issues.apache.org/jira/browse/HADOOP-13650
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: HADOOP-13345
> Reporter: Lei (Eddy) Xu
> Assignee: Lei (Eddy) Xu
> Fix For: HADOOP-13345
>
> Attachments: HADOOP-13650-HADOOP-13345.000.patch,
> HADOOP-13650-HADOOP-13345.001.patch, HADOOP-13650-HADOOP-13345.002.patch,
> HADOOP-13650-HADOOP-13345.003.patch, HADOOP-13650-HADOOP-13345.004.patch,
> HADOOP-13650-HADOOP-13345.005.patch, HADOOP-13650-HADOOP-13345.006.patch,
> HADOOP-13650-HADOOP-13345.007.patch, HADOOP-13650-HADOOP-13345.008.patch,
> HADOOP-13650-HADOOP-13345.009.patch
>
>
> Similar systems like EMRFS has the CLI tools to manipulate the metadata
> store, i.e., create or delete metadata store, or {{import}}, {{sync}} the
> file metadata between metadata store and S3.
> http://docs.aws.amazon.com//ElasticMapReduce/latest/ReleaseGuide/emrfs-cli-reference.html
> S3Guard should offer similar functionality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
