[
https://issues.apache.org/jira/browse/HADOOP-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15830156#comment-15830156
]
Steve Loughran commented on HADOOP-13985:
-----------------------------------------
This patch is going to force everyone running the tests to delete their tables
via the s3 CLI or simply the AWS console. Otherwise: stack traces on fs init.
Which is how I've verified that when you run with {{-Ds3guard -Ddynamo}} then
ddb is used and the version checking kicks in.
{code}
test_100_renameHugeFile(org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesDiskBlocks)
Time elapsed: 0.386 sec <<< ERROR!
java.io.IOException: S3Guard table lacks version marker. Table:
hwdev-steve-frankfurt-new
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.verifyVersionCompatibility(DynamoDBMetadataStore.java:618)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initTable(DynamoDBMetadataStore.java:583)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:246)
at
org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:92)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:258)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3246)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3295)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3263)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:476)
at
org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72)
at
org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:177)
at
org.apache.hadoop.fs.s3a.scale.S3AScaleTestBase.setup(S3AScaleTestBase.java:90)
at
org.apache.hadoop.fs.s3a.scale.AbstractSTestS3AHugeFiles.setup(AbstractSTestS3AHugeFiles.java:75)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
> s3guard: add a version marker to every table
> --------------------------------------------
>
> Key: HADOOP-13985
> URL: https://issues.apache.org/jira/browse/HADOOP-13985
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: HADOOP-13345
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-13985-HADOOP-13345-001.patch
>
>
> This is something else we need before any preview: a way to identify a table
> version, so that if future versions change the table structure:
> * older clients can recognise that it's a newer format, and fail
> * the future version can identify that it's an older format, and fail until
> some fsck-upgrade operation has taken place
> I think something like a row on a path which is impossible in a real
> filesystem, such as "../VERSION" would allow a version marker to go in; the
> length field could be abused for the version number.
> This field would be something that'd be checked in init(), so be the simple
> test for table existence we need for faster init
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
