Re: s3a hadoop 2.8.0 tests vs 2.7.3

2017-05-31 Thread Vasu Kulkarni 
Thanks Steve

On Wed, May 31, 2017 at 4:18 AM, Steve Loughran 
wrote:

> see also
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-
> tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
>
>
> On 26 May 2017, at 23:22, Vasu Kulkarni  wrote:
>
> Thanks Liu
>
> On Fri, May 26, 2017 at 2:42 PM, Mingliang Liu  wrote:
>
> Hi,
>
> Many tests of S3A have been moved to “integration tests”, whose names start
> with “ITestS3A”. Moreover, the phase of Maven for those tests are “verify”
> instead of “test” now.
>
> So, you can specify the "mvn -Dit.test=‘ ITestS3A*’ verify" for integration
> tests (and unit tests). “mvn test” will run unit tests only. Make sure the
> credentials are provided.
>
> By the way, upgrading from 2.7 to 2.8 is a smart choice from S3A point of
> view.
>
> https://github.com/apache/hadoop/blob/branch-2.8.1/
> hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-
> aws/index.md#running-the-tests
>
> L
>
> On May 26, 2017, at 9:00 AM, Vasu Kulkarni  wrote:
>
> Sorry resending because i had problems with group subscribe
>
> Hi,
>
> I am trying to run hadoop s3a unit tests on 2.8.0 release(using ceph
> radosgw),  I notice that many tests that ran in 2.7.3 have been
> skipped in 2.8.0 hadoop release,
> I am following the configuration options from here that worked here
> for 2.7.3:
> https://github.com/apache/hadoop/blob/trunk/hadoop-
> tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
>
> Has any of the configuration options changed for 2.8.0 which are not
> yet documented or the test structure has changed ? Thanks
>
> on 2.8.0:
>
> Tests: (mvn test -Dtest=S3a*,TestS3A* )
> Running org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.124
> sec - in org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation
> Running org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
> Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.385
> sec - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
> Running org.apache.hadoop.fs.s3a.TestS3AInputPolicies
> Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.146
> sec - in org.apache.hadoop.fs.s3a.TestS3AInputPolicies
> Running org.apache.hadoop.fs.s3a.TestS3AGetFileStatus
>
>
> Tests run: 40, Failures: 0, Errors: 0, Skipped: 0
>
>
>
> on 2.7.3:
>
> Tests: ( mvn test -Dtest=S3a*,TestS3A*)
>
> Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
> Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
> Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
> .
> .
> Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
>
> Tests run: 88, Failures: 0, Errors: 0, Skipped: 48
>
> Thanks
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>
>
>

Re: s3a hadoop 2.8.0 tests vs 2.7.3

2017-05-31 Thread Steve Loughran 
see also

https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md


On 26 May 2017, at 23:22, Vasu Kulkarni 
> wrote:

Thanks Liu

On Fri, May 26, 2017 at 2:42 PM, Mingliang Liu 
> wrote:
Hi,

Many tests of S3A have been moved to “integration tests”, whose names start
with “ITestS3A”. Moreover, the phase of Maven for those tests are “verify”
instead of “test” now.

So, you can specify the "mvn -Dit.test=‘ ITestS3A*’ verify" for integration
tests (and unit tests). “mvn test” will run unit tests only. Make sure the
credentials are provided.

By the way, upgrading from 2.7 to 2.8 is a smart choice from S3A point of
view.

https://github.com/apache/hadoop/blob/branch-2.8.1/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md#running-the-tests

L

On May 26, 2017, at 9:00 AM, Vasu Kulkarni  wrote:

Sorry resending because i had problems with group subscribe

Hi,

I am trying to run hadoop s3a unit tests on 2.8.0 release(using ceph
radosgw),  I notice that many tests that ran in 2.7.3 have been
skipped in 2.8.0 hadoop release,
I am following the configuration options from here that worked here
for 2.7.3:
https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md

Has any of the configuration options changed for 2.8.0 which are not
yet documented or the test structure has changed ? Thanks

on 2.8.0:

Tests: (mvn test -Dtest=S3a*,TestS3A* )
Running org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.124
sec - in org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation
Running org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.385
sec - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
Running org.apache.hadoop.fs.s3a.TestS3AInputPolicies
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.146
sec - in org.apache.hadoop.fs.s3a.TestS3AInputPolicies
Running org.apache.hadoop.fs.s3a.TestS3AGetFileStatus


Tests run: 40, Failures: 0, Errors: 0, Skipped: 0



on 2.7.3:

Tests: ( mvn test -Dtest=S3a*,TestS3A*)

Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
.
.
Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles

Tests run: 88, Failures: 0, Errors: 0, Skipped: 48

Thanks

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org



-
To unsubscribe, e-mail: 
common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: 
common-dev-h...@hadoop.apache.org

Re: s3a hadoop 2.8.0 tests vs 2.7.3

2017-05-26 Thread Vasu Kulkarni 
Thanks Liu

On Fri, May 26, 2017 at 2:42 PM, Mingliang Liu  wrote:
> Hi,
>
> Many tests of S3A have been moved to “integration tests”, whose names start
> with “ITestS3A”. Moreover, the phase of Maven for those tests are “verify”
> instead of “test” now.
>
> So, you can specify the "mvn -Dit.test=‘ ITestS3A*’ verify" for integration
> tests (and unit tests). “mvn test” will run unit tests only. Make sure the
> credentials are provided.
>
> By the way, upgrading from 2.7 to 2.8 is a smart choice from S3A point of
> view.
>
> https://github.com/apache/hadoop/blob/branch-2.8.1/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md#running-the-tests
>
> L
>
> On May 26, 2017, at 9:00 AM, Vasu Kulkarni  wrote:
>
> Sorry resending because i had problems with group subscribe
>
> Hi,
>
> I am trying to run hadoop s3a unit tests on 2.8.0 release(using ceph
> radosgw),  I notice that many tests that ran in 2.7.3 have been
> skipped in 2.8.0 hadoop release,
> I am following the configuration options from here that worked here
> for 2.7.3:
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
>
> Has any of the configuration options changed for 2.8.0 which are not
> yet documented or the test structure has changed ? Thanks
>
> on 2.8.0:
>
> Tests: (mvn test -Dtest=S3a*,TestS3A* )
> Running org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.124
> sec - in org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation
> Running org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
> Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.385
> sec - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
> Running org.apache.hadoop.fs.s3a.TestS3AInputPolicies
> Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.146
> sec - in org.apache.hadoop.fs.s3a.TestS3AInputPolicies
> Running org.apache.hadoop.fs.s3a.TestS3AGetFileStatus
>
>
> Tests run: 40, Failures: 0, Errors: 0, Skipped: 0
>
>
>
> on 2.7.3:
>
> Tests: ( mvn test -Dtest=S3a*,TestS3A*)
>
> Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
> Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
> Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
> .
> .
> Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
>
> Tests run: 88, Failures: 0, Errors: 0, Skipped: 48
>
> Thanks
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: s3a hadoop 2.8.0 tests vs 2.7.3

2017-05-26 Thread Mingliang Liu 
Hi,

Many tests of S3A have been moved to “integration tests”, whose names start 
with “ITestS3A”. Moreover, the phase of Maven for those tests are “verify” 
instead of “test” now.

So, you can specify the "mvn -Dit.test=‘ITestS3A*’ verify" for integration 
tests (and unit tests). “mvn test” will run unit tests only. Make sure the 
credentials are provided.

By the way, upgrading from 2.7 to 2.8 is a smart choice from S3A point of view.

https://github.com/apache/hadoop/blob/branch-2.8.1/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md#running-the-tests
 


L

> On May 26, 2017, at 9:00 AM, Vasu Kulkarni  wrote:
> 
> Sorry resending because i had problems with group subscribe
> 
> Hi,
> 
> I am trying to run hadoop s3a unit tests on 2.8.0 release(using ceph
> radosgw),  I notice that many tests that ran in 2.7.3 have been
> skipped in 2.8.0 hadoop release,
> I am following the configuration options from here that worked here
> for 2.7.3: 
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
> 
> Has any of the configuration options changed for 2.8.0 which are not
> yet documented or the test structure has changed ? Thanks
> 
> on 2.8.0:
> 
> Tests: (mvn test -Dtest=S3a*,TestS3A* )
> Running org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.124
> sec - in org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation
> Running org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
> Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.385
> sec - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
> Running org.apache.hadoop.fs.s3a.TestS3AInputPolicies
> Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.146
> sec - in org.apache.hadoop.fs.s3a.TestS3AInputPolicies
> Running org.apache.hadoop.fs.s3a.TestS3AGetFileStatus
> 
> 
> Tests run: 40, Failures: 0, Errors: 0, Skipped: 0
> 
> 
> 
> on 2.7.3:
> 
> Tests: ( mvn test -Dtest=S3a*,TestS3A*)
> 
> Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
> Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
> Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
> .
> .
> Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
> 
> Tests run: 88, Failures: 0, Errors: 0, Skipped: 48
> 
> Thanks
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>

Fwd: s3a hadoop 2.8.0 tests vs 2.7.3

2017-05-26 Thread Vasu Kulkarni 
Sorry resending because i had problems with group subscribe

Hi,

I am trying to run hadoop s3a unit tests on 2.8.0 release(using ceph
radosgw),  I notice that many tests that ran in 2.7.3 have been
skipped in 2.8.0 hadoop release,
I am following the configuration options from here that worked here
for 2.7.3: 
https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md

Has any of the configuration options changed for 2.8.0 which are not
yet documented or the test structure has changed ? Thanks

on 2.8.0:

Tests: (mvn test -Dtest=S3a*,TestS3A* )
Running org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.124
sec - in org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation
Running org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.385
sec - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
Running org.apache.hadoop.fs.s3a.TestS3AInputPolicies
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.146
sec - in org.apache.hadoop.fs.s3a.TestS3AInputPolicies
Running org.apache.hadoop.fs.s3a.TestS3AGetFileStatus


Tests run: 40, Failures: 0, Errors: 0, Skipped: 0



on 2.7.3:

Tests: ( mvn test -Dtest=S3a*,TestS3A*)

Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
.
.
Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles

Tests run: 88, Failures: 0, Errors: 0, Skipped: 48

Thanks

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org