[jira] [Comment Edited] (HADOOP-14918) Remove the Local Dynamo DB test option
[ https://issues.apache.org/jira/browse/HADOOP-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046740#comment-17046740 ] Gabor Bota edited comment on HADOOP-14918 at 2/27/20 3:34 PM: -- Thanks for [~jhung] for working on this. Could you create a PR for your patch? We prefer using PRs for merging. And reviews. was (Author: gabor.bota): Thanks for [~jhung] for working on this. Could you create a PR for your patch? We prefer using PRs for merging. > Remove the Local Dynamo DB test option > -- > > Key: HADOOP-14918 > URL: https://issues.apache.org/jira/browse/HADOOP-14918 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-14918-001.patch, HADOOP-14918-002.patch, > HADOOP-14918-003.patch, HADOOP-14918-004.patch, > HADOOP-14918-branch-2.10.001.patch, HADOOP-14918.005.patch, > HADOOP-14918.006.patch > > > I'm going to propose cutting out the localdynamo test option for s3guard > * the local DDB JAR is unmaintained/lags the SDK We work with...eventually > there'll be differences in API. > * as the local dynamo DB is unshaded. it complicates classpath setup for the > build. Remove it and there's no need to worry about versions of anything > other than the shaded AWS > * it complicates test runs. Now we need to test for both localdynamo *and* > real dynamo > * but we can't ignore real dynamo, because that's the one which matters > While the local option promises to reduce test costs, really, it's just > adding complexity. If you are testing with s3guard, you need to have a real > table to test against., And with the exception of those people testing s3a > against non-AWS, consistent endpoints, everyone should be testing with > S3Guard. > -Straightforward to remove.- -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14918) Remove the Local Dynamo DB test option
[ https://issues.apache.org/jira/browse/HADOOP-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513492#comment-16513492 ] Gabor Bota edited comment on HADOOP-14918 at 6/15/18 7:55 AM: -- Thanks for the review [~mackrorysd]! We need a separate fs.s3a.s3guard.ddb.test.table because fs.s3a.s3guard.ddb.table will be used by other tests, and we want to avoid clearing or destroying that table. That was really an intent in my latest patch - to avoid touching the main table by adding a testing table variable. That way we can run these tests parallelly with others. Maybe this need to be documented better? I'll fix the other two issues, and submit another patch soon. was (Author: gabor.bota): Thanks for the review [~mackrorysd]! We need a separate fs.s3a.s3guard.ddb.test.table because fs.s3a.s3guard.ddb.table will be used by other tests, and we want to avoid clearing or destroying that table. That was really an intent in my latest patch - to avoid touching the main table by adding a testing table variable. That way we can run these tests parallelly with others. Maybe this need to be documented better? I'll fix the other three issues, and submit another patch soon. > Remove the Local Dynamo DB test option > -- > > Key: HADOOP-14918 > URL: https://issues.apache.org/jira/browse/HADOOP-14918 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > Attachments: HADOOP-14918-001.patch, HADOOP-14918-002.patch, > HADOOP-14918-003.patch, HADOOP-14918-004.patch, HADOOP-14918.005.patch, > HADOOP-14918.006.patch > > > I'm going to propose cutting out the localdynamo test option for s3guard > * the local DDB JAR is unmaintained/lags the SDK We work with...eventually > there'll be differences in API. > * as the local dynamo DB is unshaded. it complicates classpath setup for the > build. Remove it and there's no need to worry about versions of anything > other than the shaded AWS > * it complicates test runs. Now we need to test for both localdynamo *and* > real dynamo > * but we can't ignore real dynamo, because that's the one which matters > While the local option promises to reduce test costs, really, it's just > adding complexity. If you are testing with s3guard, you need to have a real > table to test against., And with the exception of those people testing s3a > against non-AWS, consistent endpoints, everyone should be testing with > S3Guard. > -Straightforward to remove.- -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14918) Remove the Local Dynamo DB test option
[ https://issues.apache.org/jira/browse/HADOOP-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513492#comment-16513492 ] Gabor Bota edited comment on HADOOP-14918 at 6/15/18 7:47 AM: -- Thanks for the review [~mackrorysd]! We need a separate fs.s3a.s3guard.ddb.test.table because fs.s3a.s3guard.ddb.table will be used by other tests, and we want to avoid clearing or destroying that table. That was really an intent in my latest patch - to avoid touching the main table by adding a testing table variable. That way we can run these tests parallelly with others. Maybe this need to be documented better? I'll fix the other three issues, and submit another patch soon. was (Author: gabor.bota): Thanks for the review [~mackrorysd]! We need a separate fs.s3a.s3guard.ddb.test.table because fs.s3a.s3guard.ddb.table will be used by other tests, and we want to avoid clearing or destroying that table. That was really an intent in my latest patch - to avoid touching the main table by adding a testing table variable. That way we can run these tests parallelly with others. I'll fix the other three issues, and submit another patch soon. > Remove the Local Dynamo DB test option > -- > > Key: HADOOP-14918 > URL: https://issues.apache.org/jira/browse/HADOOP-14918 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > Attachments: HADOOP-14918-001.patch, HADOOP-14918-002.patch, > HADOOP-14918-003.patch, HADOOP-14918-004.patch, HADOOP-14918.005.patch > > > I'm going to propose cutting out the localdynamo test option for s3guard > * the local DDB JAR is unmaintained/lags the SDK We work with...eventually > there'll be differences in API. > * as the local dynamo DB is unshaded. it complicates classpath setup for the > build. Remove it and there's no need to worry about versions of anything > other than the shaded AWS > * it complicates test runs. Now we need to test for both localdynamo *and* > real dynamo > * but we can't ignore real dynamo, because that's the one which matters > While the local option promises to reduce test costs, really, it's just > adding complexity. If you are testing with s3guard, you need to have a real > table to test against., And with the exception of those people testing s3a > against non-AWS, consistent endpoints, everyone should be testing with > S3Guard. > -Straightforward to remove.- -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14918) Remove the Local Dynamo DB test option
[
https://issues.apache.org/jira/browse/HADOOP-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511603#comment-16511603
]
Gabor Bota edited comment on HADOOP-14918 at 6/13/18 9:33 PM:
--
In my latest (v005) I've managed to solve the destructive test issue.
All tests pass on eu-west-1 with {{mvn -Dparallel-tests -DtestsThreadCount=8
clean verify -Ds3guard -Ddynamo}}
(testDestroyNoBucket will fail because of HADOOP-14927 is not fixed yet)
* I've added a new config param {{fs.s3a.s3guard.ddb.table}} which should be
set in order to run the test in the config. This is the name of the table that
will be shared and used by the metadatastore object created in {{@BeforeClass}}
and destroyed in {{@AfterClass}} and shared between the tests. It's not
documented yet in this patch, so I should add further documentation if this
solution will be accepted (markdown).
* The contents of this table will be cleared out in {{@After}} of each test,
but there will be no ms teardown until {{@AfterClass}}
* Some of the tests still need to create a new ms (so a new table), but the
tables used will be named after the name of those tests.
* Sometimes a test could take more time than what is defined in
{{HadoopTestBase}} by default, so I had to increase it in the hadoop-aws
pom.xml.
was (Author: gabor.bota):
In my latest (v005) I've managed to solve the destructive test issue.
All tests pass on eu-west-1 with {{mvn -Dparallel-tests -DtestsThreadCount=8
clean verify -Ds3guard -Ddynamo}}
* I've added a new config param {{fs.s3a.s3guard.ddb.table}} which should be
set in order to run the test in the config. This is the name of the table that
will be shared and used by the metadatastore object created in {{@BeforeClass}}
and destroyed in {{@AfterClass}} and shared between the tests. It's not
documented yet in this patch, so I should add further documentation if this
solution will be accepted (markdown).
* The contents of this table will be cleared out in {{@After}} of each test,
but there will be no ms teardown until {{@AfterClass}}
* Some of the tests still need to create a new ms (so a new table), but the
tables used will be named after the name of those tests.
* Sometimes a test could take more time than what is defined in
{{HadoopTestBase}} by default, so I had to increase it in the hadoop-aws
pom.xml.
> Remove the Local Dynamo DB test option
> --
>
> Key: HADOOP-14918
> URL: https://issues.apache.org/jira/browse/HADOOP-14918
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.9.0, 3.0.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
> Attachments: HADOOP-14918-001.patch, HADOOP-14918-002.patch,
> HADOOP-14918-003.patch, HADOOP-14918-004.patch, HADOOP-14918.005.patch
>
>
> I'm going to propose cutting out the localdynamo test option for s3guard
> * the local DDB JAR is unmaintained/lags the SDK We work with...eventually
> there'll be differences in API.
> * as the local dynamo DB is unshaded. it complicates classpath setup for the
> build. Remove it and there's no need to worry about versions of anything
> other than the shaded AWS
> * it complicates test runs. Now we need to test for both localdynamo *and*
> real dynamo
> * but we can't ignore real dynamo, because that's the one which matters
> While the local option promises to reduce test costs, really, it's just
> adding complexity. If you are testing with s3guard, you need to have a real
> table to test against., And with the exception of those people testing s3a
> against non-AWS, consistent endpoints, everyone should be testing with
> S3Guard.
> -Straightforward to remove.-
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14918) remove the Local Dynamo DB test option
[ https://issues.apache.org/jira/browse/HADOOP-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489164#comment-16489164 ] Gabor Bota edited comment on HADOOP-14918 at 5/24/18 3:05 PM: -- Thanks for testing it [~ste...@apache.org], but right now it won't work in parallel with other tests, as I said in the earlier comment. I've added it to the sequential-integration-tests in maven. It will delete and re-create the table for each test method. If this behavior is not good - so we need to keep the table - I have to work with this patch a little more. was (Author: gabor.bota): Thanks for testing it [~ste...@apache.org], but right now it won't work in parallel with other tests, as I said in the earlier comment. I've added it to the sequential-integration-tests in maven. > remove the Local Dynamo DB test option > -- > > Key: HADOOP-14918 > URL: https://issues.apache.org/jira/browse/HADOOP-14918 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > Attachments: HADOOP-14918-001.patch, HADOOP-14918-002.patch, > HADOOP-14918-003.patch, HADOOP-14918-004.patch > > > I'm going to propose cutting out the localdynamo test option for s3guard > * the local DDB JAR is unmaintained/lags the SDK We work with...eventually > there'll be differences in API. > * as the local dynamo DB is unshaded. it complicates classpath setup for the > build. Remove it and there's no need to worry about versions of anything > other than the shaded AWS > * it complicates test runs. Now we need to test for both localdynamo *and* > real dynamo > * but we can't ignore real dynamo, because that's the one which matters > While the local option promises to reduce test costs, really, it's just > adding complexity. If you are testing with s3guard, you need to have a real > table to test against., And with the exception of those people testing s3a > against non-AWS, consistent endpoints, everyone should be testing with > S3Guard. > -Straightforward to remove.- -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14918) remove the Local Dynamo DB test option
[ https://issues.apache.org/jira/browse/HADOOP-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489164#comment-16489164 ] Gabor Bota edited comment on HADOOP-14918 at 5/24/18 3:05 PM: -- Thanks for testing it [~ste...@apache.org], but right now it won't work in parallel with other tests, as I said in the earlier comment. I've added it to the sequential-integration-tests in maven. It will delete and re-create the table for each test method. If this behavior is not good - so we need to keep the table - I have to work on this patch a little more. was (Author: gabor.bota): Thanks for testing it [~ste...@apache.org], but right now it won't work in parallel with other tests, as I said in the earlier comment. I've added it to the sequential-integration-tests in maven. It will delete and re-create the table for each test method. If this behavior is not good - so we need to keep the table - I have to work with this patch a little more. > remove the Local Dynamo DB test option > -- > > Key: HADOOP-14918 > URL: https://issues.apache.org/jira/browse/HADOOP-14918 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > Attachments: HADOOP-14918-001.patch, HADOOP-14918-002.patch, > HADOOP-14918-003.patch, HADOOP-14918-004.patch > > > I'm going to propose cutting out the localdynamo test option for s3guard > * the local DDB JAR is unmaintained/lags the SDK We work with...eventually > there'll be differences in API. > * as the local dynamo DB is unshaded. it complicates classpath setup for the > build. Remove it and there's no need to worry about versions of anything > other than the shaded AWS > * it complicates test runs. Now we need to test for both localdynamo *and* > real dynamo > * but we can't ignore real dynamo, because that's the one which matters > While the local option promises to reduce test costs, really, it's just > adding complexity. If you are testing with s3guard, you need to have a real > table to test against., And with the exception of those people testing s3a > against non-AWS, consistent endpoints, everyone should be testing with > S3Guard. > -Straightforward to remove.- -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
