Repository: sqoop Updated Branches: refs/heads/trunk d58e5f106 -> bb9c2dd85
SQOOP-3404: Categorize all tests in the project (Szabolcs Vasas via Fero Szabo) Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/bb9c2dd8 Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/bb9c2dd8 Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/bb9c2dd8 Branch: refs/heads/trunk Commit: bb9c2dd85b0cac84503e69906a05c76d6a0413e1 Parents: d58e5f1 Author: Fero Szabo <[email protected]> Authored: Mon Nov 19 10:44:35 2018 +0100 Committer: Fero Szabo <[email protected]> Committed: Mon Nov 19 10:44:35 2018 +0100 ---------------------------------------------------------------------- COMPILING.txt | 251 ++++++++++++++++--- build.gradle | 81 +++--- .../sqoop/importjob/SplitByImportTest.java | 5 + .../sqoop/manager/TestMainframeManager.java | 1 + .../TestOraOopDBInputSplitGetDebugDetails.java | 3 + .../TestNetezzaExternalTableExportMapper.java | 2 + .../TestNetezzaExternalTableImportMapper.java | 3 + .../TestMainframeDatasetBinaryRecord.java | 3 + .../TestMainframeFTPFileGdgEntryParser.java | 4 + .../org/apache/sqoop/s3/TestS3AvroImport.java | 3 + .../sqoop/s3/TestS3ExternalHiveTableImport.java | 5 + .../s3/TestS3ImportWithHadoopCredProvider.java | 3 + .../s3/TestS3IncrementalAppendAvroImport.java | 3 + .../TestS3IncrementalAppendParquetImport.java | 3 + ...stS3IncrementalAppendSequenceFileImport.java | 3 + .../s3/TestS3IncrementalAppendTextImport.java | 3 + .../s3/TestS3IncrementalMergeParquetImport.java | 3 + .../s3/TestS3IncrementalMergeTextImport.java | 3 + .../apache/sqoop/s3/TestS3ParquetImport.java | 3 + .../sqoop/s3/TestS3SequenceFileImport.java | 3 + .../org/apache/sqoop/s3/TestS3TextImport.java | 3 + .../testcategories/thirdpartytest/S3Test.java | 10 + ...estS3IncrementalImportOptionValidations.java | 5 + 23 files changed, 328 insertions(+), 78 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/COMPILING.txt ---------------------------------------------------------------------- diff --git a/COMPILING.txt b/COMPILING.txt index 835ba33..0383707 100644 --- a/COMPILING.txt +++ b/COMPILING.txt @@ -1,7 +1,7 @@ = Compiling -This document explains how to compile Sqoop. +This document explains how to compile and test Sqoop. //// Licensed to the Apache Software Foundation (ASF) under one @@ -27,7 +27,7 @@ under the License. Compiling Sqoop requires the following tools: -* Apache ant (1.9.7) or Gradle (3.5.1) +* Apache ant (1.9.7) or Gradle (4.9) * Java JDK 1.8 Additionally, building the documentation requires these tools: @@ -47,21 +47,22 @@ Furthermore, Sqoop's build can be instrumented with the following: == The Basics -Sqoop is compiled with ant. Type +ant -p+ to see the list of available targets. +Sqoop can be compiled and tested with both Ant and Gradle. Type +ant -p+ or +./gradlew tasks --all+ to see the list of available targets/tasks. -Type +ant+ to compile all java sources. You can then run Sqoop with +bin/sqoop+. +Type +ant+ or +./gradlew jar+ to compile all java sources. You can then run Sqoop with +bin/sqoop+. If you want to build everything (including the documentation), type +ant package+ or +./gradlew package+. This will appear in the +build/sqoop-(version)/+ directory. -This version of Sqoop is built against Hadoop 0.23 available from Apache -maven repository by default. These dependencies are obtained via IVY which -downloads the necessary binaries. - == Testing Sqoop -Sqoop has three main test categories: unit, third party and Amazon S3 tests. +Sqoop supports both Ant and Gradle but the test tasks are a bit different in each build tools. + +=== Testing with Ant +The Ant build defines two main test categories: unit and third party tests. + +Classes with the +Test+ prefix are unit tests and classes with the +Test+ postfix are third party tests. Sqoop unit tests can be run with +ant test+. This command will run all the "basic" checks against an in-memory database, HSQLDB. @@ -70,9 +71,84 @@ Sqoop's third party tests are compatibility tests that check its ability to work several third-party databases. To enable these tests, you will need to install and configure the databases or run them in Docker containers, and download the JDBC drivers for each one. -Sqoop's Amazon S3 test suite tests the "RDBMS to Amazon S3" use case with an in-memory database, HSQLDB. +For more information about how to run this suite see the 'Third party tests' section. + +Note that the unit test suite contains several Amazon S3 test classes too which test the "RDBMS to Amazon S3" use case with an in-memory database, HSQLDB. To enable these tests, you will need to have either permanent or temporary Amazon S3 security credentials. +For more information about how to run this suite see the 'Amazon S3 tests' section. + +=== Testing with Gradle + +The Gradle build supports JUnit's +@Category+ annotation so we have access to much more fine grained test categories here. +The test categories are defined as marker interfaces under +org.apache.sqoop.testcategories+ package. + +The following table shows the currently supported test categories and their hierarchy: + +.Available test categories +[width="40%",frame="topbot",options="header"] +|====================== +|Category |Subcategory +.3+|+SqoopTest+ |+UnitTest+ + |+IntegrationTest+ + |+ManualTest+ +.9+|+ThirdPartyTest+ |+CubridTest+ + |+Db2Test+ + |+MainFrameTest+ + |+MysqlTest+ + |+NetezzaTest+ + |+OracleTest+ + |+PostgresqlTest+ + |+SqlServerTest+ + |+S3Test+ +|+KerberizedTest+ | +|====================== + +==== SqoopTest +A general category including UnitTest, IntegrationTest and ManualTest. + +==== UnitTest +A unit test shall test one class at a time having it's dependencies mocked. +A unit test shall not start a mini cluster nor an embedded database and it shall not use a JDBC driver. + +==== IntegrationTest +An integration test shall test if independently developed classes work together correctly. +An integration test checks a whole scenario and thus may start mini clusters or embedded databases and may connect to +external resources like RDBMS instances. + +==== ManualTest +Deprecated category, shall not be used nor extended. + +==== ThirdPartyTest +A third party test shall test a scenario where a third party side is required such as a JDBC driver or an external RDBMS instance. +The subcategories define what kind of third party dependency is needed by the test. + +==== KerberizedTest +A kerberized test shall run in kerberized environment thus it starts mini KDC server. + +==== Categorizing tests +Note that if you add a new test you have to make sure that it is categorized otherwise Gradle will not execute it. + +The categorizing steps are the following: + +* Decide if the test is a unit or an integration test, mark the test class with the +@Category+ annotation and add the +corresponding marker interface to it. +* If the test needs a JDBC driver or an external service then add +ThirdPartyTest+ or one of its subinterfaces to the ++@Category+ annotation. Try to use the most specific interface. +* If the test starts a Mini KDC then add the +KerberizedTest+ interface to the +@Category+ annotation. + +==== Available test targets + +* +unitTest+: Runs unit tests which do not need proprietary JDBC driver. +* +integrationTest+: Runs integration tests which do not need a docker container or an external database/service. +* +kerberizedTest+: Runs kerberized tests. +* +thirdPartyTest+: Runs third-party tests. For more information see the 'Third party tests' section. +* +test+: Runs tests that do not need external JDBC driver and/or a docker container. +This the same as running unitTest, integrationTest and kerberizedTest. +* +s3Test+: Runs S3 tests. For more information see the 'Amazon S3 tests' section. +* +allTest+: Runs all Sqoop tests. + + === Third party tests ==== Installing the necessary databases @@ -291,10 +367,72 @@ You can stop and remove the Docker containers using the following command: ===== Running the third party tests using docker containers -You can execute the third party tests against the DBs running in Docker containers using the following command (replace <path_to_thirdparty_lib_directory> with the path you have the necessary JDBC drivers): +You can execute the third party tests against the DBs running in Docker containers using the following command (replace <path_to_thirdparty_lib_directory> with the path you have the necessary JDBC drivers, +<your-bucket-url> and <your-credential-generator-command> with the values described in the 'Amazon S3 tests' section): + +---- +ant clean test -Dthirdparty=true -Dsqoop.thirdparty.lib.dir=<path_to_thirdparty_lib_directory> \ + -Dsqoop.test.mysql.connectstring.host_url=jdbc:mysql://127.0.0.1:3306/ \ + -Dsqoop.test.mysql.databasename=sqoop \ + -Dsqoop.test.mysql.password=Sqoop12345 \ + -Dsqoop.test.mysql.username=sqoop \ + -Dsqoop.test.oracle.connectstring=jdbc:oracle:thin:@//localhost:1521/sqoop \ + -Dsqoop.test.oracle.username=SYSTEM \ + -Dsqoop.test.oracle.password=Sqoop12345 \ + -Dsqoop.test.postgresql.connectstring.host_url=jdbc:postgresql://localhost/ \ + -Dsqoop.test.postgresql.database=sqoop \ + -Dsqoop.test.postgresql.username=sqoop \ + -Dsqoop.test.postgresql.password=Sqoop12345 \ + -Dsqoop.test.cubrid.connectstring.host_url=jdbc:cubrid:localhost:33000 \ + -Dsqoop.test.cubrid.connectstring.username=sqoop \ + -Dsqoop.test.cubrid.connectstring.database=sqoop \ + -Dsqoop.test.cubrid.connectstring.password=Sqoop12345 \ + -Dmapred.child.java.opts="\-Djava.security.egd=file:/dev/../dev/urandom" \ + -Dtest.timeout=10000000 \ + -Dsqoop.test.sqlserver.connectstring.host_url=jdbc:sqlserver://localhost:1433 \ + -Dsqoop.test.sqlserver.database=master \ + -Dms.sqlserver.username=sa \ + -Dms.sqlserver.password=Sqoop12345 \ + -Dsqoop.test.db2.connectstring.host_url=jdbc:db2://localhost:50000 \ + -Dsqoop.test.db2.connectstring.database=SQOOP \ + -Dsqoop.test.db2.connectstring.username=DB2INST1 \ + -Dsqoop.test.db2.connectstring.password=Sqoop12345 \ + -Ds3.bucket.url=<your-bucket-url> \ + -Ds3.generator.command=<your-credential-generator-command> +---- + +or ---- -ant clean test -Dthirdparty=true -Dsqoop.thirdparty.lib.dir=<path_to_thirdparty_lib_directory> -Dsqoop.test.mysql.connectstring.host_url=jdbc:mysql://127.0.0.1:3306/ -Dsqoop.test.mysql.databasename=sqoop -Dsqoop.test.mysql.password=Sqoop12345 -Dsqoop.test.mysql.username=sqoop -Dsqoop.test.oracle.connectstring=jdbc:oracle:thin:@//localhost:1521/sqoop -Dsqoop.test.oracle.username=SYSTEM -Dsqoop.test.oracle.password=Sqoop12345 -Dsqoop.test.postgresql.connectstring.host_url=jdbc:postgresql://localhost/ -Dsqoop.test.postgresql.database=sqoop -Dsqoop.test.postgresql.username=sqoop -Dsqoop.test.postgresql.password=Sqoop12345 -Dsqoop.test.cubrid.connectstring.host_url=jdbc:cubrid:localhost:33000 -Dsqoop.test.cubrid.connectstring.username=sqoop -Dsqoop.test.cubrid.connectstring.database=sqoop -Dsqoop.test.cubrid.connectstring.password=Sqoop12345 -Dmapred.child.java.opts="\-Djava.security.egd=file:/dev/../dev/urandom" -Dtest.timeout=10000000 -Dsqoop.test.sqlserver.connectstring.host_url=jdbc: sqlserver://localhost:1433 -Dsqoop.test.sqlserver.database=master -Dms.sqlserver.username=sa -Dms.sqlserver.password=Sqoop12345 -Dsqoop.test.db2.connectstring.host_url=jdbc:db2://localhost:50000 -Dsqoop.test.db2.connectstring.database=SQOOP -Dsqoop.test.db2.connectstring.username=DB2INST1 -Dsqoop.test.db2.connectstring.password=Sqoop12345 +./gradlew -Dsqoop.thirdparty.lib.dir=<path_to_thirdparty_lib_directory> \ + -Dsqoop.test.mysql.connectstring.host_url=jdbc:mysql://127.0.0.1:3306/ \ + -Dsqoop.test.mysql.databasename=sqoop \ + -Dsqoop.test.mysql.password=Sqoop12345 \ + -Dsqoop.test.mysql.username=sqoop \ + -Dsqoop.test.oracle.connectstring=jdbc:oracle:thin:@//localhost:1521/sqoop \ + -Dsqoop.test.oracle.username=SYSTEM \ + -Dsqoop.test.oracle.password=Sqoop12345 \ + -Dsqoop.test.postgresql.connectstring.host_url=jdbc:postgresql://localhost/ \ + -Dsqoop.test.postgresql.database=sqoop \ + -Dsqoop.test.postgresql.username=sqoop \ + -Dsqoop.test.postgresql.password=Sqoop12345 \ + -Dsqoop.test.cubrid.connectstring.host_url=jdbc:cubrid:localhost:33000 \ + -Dsqoop.test.cubrid.connectstring.username=sqoop \ + -Dsqoop.test.cubrid.connectstring.database=sqoop \ + -Dsqoop.test.cubrid.connectstring.password=Sqoop12345 \ + -Dmapred.child.java.opts="\-Djava.security.egd=file:/dev/../dev/urandom" \ + -Dtest.timeout=10000000 \ + -Dsqoop.test.sqlserver.connectstring.host_url=jdbc:sqlserver://localhost:1433 \ + -Dsqoop.test.sqlserver.database=master \ + -Dms.sqlserver.username=sa \ + -Dms.sqlserver.password=Sqoop12345 \ + -Dsqoop.test.db2.connectstring.host_url=jdbc:db2://localhost:50000 \ + -Dsqoop.test.db2.connectstring.database=SQOOP \ + -Dsqoop.test.db2.connectstring.username=DB2INST1 \ + -Dsqoop.test.db2.connectstring.password=Sqoop12345 \ + -Ds3.bucket.url=<your-bucket-url> \ + -Ds3.generator.command=<your-credential-generator-command> \ + thirdPartyTest ---- Please note that even if you do not need to install RDBMSs to run Sqoop third party tests against the Docker containers you still need to install the following tools: @@ -312,11 +450,15 @@ of temporary credentials) having them separated by spaces. You can then pass the bucket URL and the generator command to the tests via system properties as follows: -++++ +---- ant clean test -Ds3.bucket.url=<your-bucket-url> -Ds3.generator.command=<your-credential-generator-command> +---- + or -./gradlew test -Ds3.bucket.url=<your-bucket-url> -Ds3.generator.command=<your-credential-generator-command> -++++ + +---- +./gradlew s3Test -Ds3.bucket.url=<your-bucket-url> -Ds3.generator.command=<your-credential-generator-command> +---- == Code Quality Analysis @@ -331,11 +473,15 @@ trigger additional warnings in Findbugs. Install findbugs (1.3.9) according to its instructions. To use it, run: -++++ +---- ant findbugs -Dfindbugs.home=/path/to/findbugs/ +---- + or + +---- ./gradlew findbugsMain -++++ +---- A report will be generated in +build/findbugs/+ @@ -347,25 +493,30 @@ all possible paths. Install Cobertura according to its instructions. Then run a test with: -++++ +---- ant clean ant cobertura -Dcobertura.home=/path/to/cobertura ant cobertura -Dcobertura.home=/path/to/cobertura \ -Dthirdparty=true -Dsqoop.thirdparty.lib.dir=/path/to/thirdparty +---- For Gradle we run Jacoco for code coverage checks. You can create single reports or composite reports, where you can check the combined coverage of unit and thirdparty tests. +---- ./gradlew clean ./gradlew test ./gradlew jacocoTestReport ./gradlew -Dsqoop.thirdparty.lib.dir=<path_to_thirdparty_lib_directory> thirdPartyTest ./gradlew jacocoThirdPartyReport +---- or generate the composite report after running test and thirdPartyTest + +---- ./gradlew jacocoCompositeReport -++++ +---- (You'll need to run the cobertura target twice; once against the regular test targets, and once against the thirdparty targets.) @@ -388,10 +539,15 @@ Ivy when necessary. To run checkstyle, execute: -++++ +---- ant checkstyle +---- + +or + +---- ./gradlew checkStyleMain -++++ +---- A report will be generated as +build/checkstyle-errors.html+ @@ -403,22 +559,32 @@ dependency management system through Maven. To install Sqoop in your local +.m2+ cache, run: -++++ +---- ant mvn-install +---- + +or + +---- ./gradlew publishToMavenLocal -++++ +---- This will install a pom and the Sqoop jar. To deploy Sqoop to a public repository, use: -++++ +---- ant mvn-deploy +---- + +or + +---- ./gradlew publishSqoopPublicationToCloudera.snapshot.repoRepository ./gradlew -DmvnRepo=x publishSqoopPublicationToCloudera.x.repoRepository -++++ +---- -By default, this deploys to repository.cloudera.com. You can choose +By default, this deploys to +repository.cloudera.com+. You can choose the complete URL to deploy to with the +mvn.deploy.url+ property. By default, this deploys to the "snapshots" repository. To deploy to "staging" or "releases" on repository.cloudera.com, set the @@ -444,28 +610,33 @@ the +mvn-install+ or +mvn-deploy+ targets as well. Running +ant eclipse+ will generate +.project+ and +.classpath+ files that will allow you to edit Sqoop sources in Eclipse with all the library dependencies correctly resolved. To compile the jars, you should still -use ant. +use Ant or Gradle. == Building the documentation Building the documentation requires that you have toxml installed. Also, one needs to set the XML_CATALOG_FILES environment variable. -++++ +---- export XML_CATALOG_FILES=/usr/local/etc/xml/catalog ant docs -++++ +---- + +or + +---- +export XML_CATALOG_FILES=/usr/local/etc/xml/catalog +./gradlew docs +---- == Other important Gradle commands -+ Generate gradle wrapper (to ensure you are using the correct, compatible version of gradle) by running +./gradlew wrapper+ -+ ./gradlew tasks to list all top-level gradle tasks or run ./gradlew tasks --all to show all tasks and subtasks -+ ./gradlew compileJava to compile the main Java source -+ ./gradlew -Dsqoop.thirdparty.lib.dir=<relative_path_to_thirdparty_lib_directory> -Dsqoop.test.mysql.connectstring.host_url=jdbc:mysql://127.0.0.1:3306/ -Dsqoop.test.mysql.databasename=sqoop -Dsqoop.test.mysql.password=Sqoop12345 -Dsqoop.test.mysql.username=sqoop -Dsqoop.test.oracle.connectstring=jdbc:oracle:thin:@//localhost:1521/sqoop -Dsqoop.test.oracle.username=SYSTEM -Dsqoop.test.oracle.password=Sqoop12345 -Dsqoop.test.postgresql.connectstring.host_url=jdbc:postgresql://localhost/ -Dsqoop.test.postgresql.database=sqoop -Dsqoop.test.postgresql.username=sqoop -Dsqoop.test.postgresql.password=Sqoop12345 -Dsqoop.test.cubrid.connectstring.host_url=jdbc:cubrid:localhost:33000 -Dsqoop.test.cubrid.connectstring.username=sqoop -Dsqoop.test.cubrid.connectstring.database=sqoop -Dsqoop.test.cubrid.connectstring.password=Sqoop12345 -Dmapred.child.java.opts="\-Djava.security.egd=file:/dev/../dev/urandom" -Dtest.timeout=10000000 -Dsqoop.test.sqlserver.connectstring.host_url=jdbc:sqlserver:// localhost:1433 -Dsqoop.test.sqlserver.database=master -Dms.sqlserver.username=sa -Dms.sqlserver.password=Sqoop12345 -Dsqoop.test.db2.connectstring.host_url=jdbc:db2://localhost:50000 -Dsqoop.test.db2.connectstring.database=SQOOP -Dsqoop.test.db2.connectstring.username=DB2INST1 -Dsqoop.test.db2.connectstring.password=Sqoop12345 thirdPartyTest -+ ./gradlew test --debug-jvm : to run remote debug on port 5005 -+ ./gradlew -Dtest.single=ClassName*Test test or ./gradlew tests --tests -+ To refresh dependencies for a build ./gradlew build --refresh-dependencies -+ For skipping a single test or a set of tests ./gradle build -x test -+ To run a single test class use --tests ClassName*Test or -DtestType.single=ClassName*Test (use with test, thirdPartyTest or manualTest) -+ To get a dependency tree: ./gradlew dependencyInsight --configuration optionalConfiguration --dependency searchedForDependency -+ For a list of the dependencies of the selected project, broken down by configuration run: ./gradlew dependencies \ No newline at end of file +* +./gradlew wrapper+ to generate gradle wrapper (to ensure you are using the correct, compatible version of gradle) +* +./gradlew tasks+ to list all top-level gradle tasks or run +./gradlew tasks --all+ to show all tasks and subtasks +* +./gradlew compileJava+ to compile the main Java source +* +./gradlew test --debug-jvm+ to run remote debug on port 5005 +* +./gradlew test --tests TestSqoopOptions+, +./gradlew thirdPartyTest --tests TestS3*+ to run one or more tests matching a pattern (use with test or thirdPartyTest) +* +./gradlew build --refresh-dependencies+ to refresh dependencies for a build +* +./gradle build -x test+ for skipping a single test or a set of tests +* +./gradlew dependencyInsight --configuration optionalConfiguration --dependency searchedForDependency+ to get a dependency tree +* +./gradlew dependencies+ to get the list of the dependencies of the selected project, broken down by configuration \ No newline at end of file http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/build.gradle ---------------------------------------------------------------------- diff --git a/build.gradle b/build.gradle index cb9eeca..954935d 100644 --- a/build.gradle +++ b/build.gradle @@ -154,21 +154,37 @@ dependencies { } task unitTest (type: Test) { - description 'Run unit tests' + description 'Run unit tests which do not need proprietary JDBC driver' useJUnit { includeCategories 'org.apache.sqoop.testcategories.sqooptest.UnitTest' - excludeCategories 'org.apache.sqoop.testcategories.KerberizedTest' excludeCategories 'org.apache.sqoop.testcategories.thirdpartytest.ThirdPartyTest' } } -task allIntegrationTest (type: Test){ - description 'Run all integration tests' +task integrationTest(type: Test) { + description 'Run integration tests which do not need a docker container or an external database/service' useJUnit { includeCategories 'org.apache.sqoop.testcategories.sqooptest.IntegrationTest' + excludeCategories 'org.apache.sqoop.testcategories.sqooptest.ManualTest' + excludeCategories 'org.apache.sqoop.testcategories.KerberizedTest' + excludeCategories 'org.apache.sqoop.testcategories.thirdpartytest.ThirdPartyTest' } } +task kerberizedTest (type: Test){ + description 'Run Kerberized Test' + // A Gradle test task with forkEvery 1 and includeCategories performs poorly because it starts a new JVM even for the filtered tests. + // To work around this performance problem we need to add every kerberized test in a separate include field here. + include '**/TestKerberosAuthenticator*' + include '**/HBaseKerberizedConnectivityTest*' + include '**/TestHiveMiniCluster*' + include '**/TestHiveServer2TextImport*' + useJUnit { + includeCategories 'org.apache.sqoop.testcategories.KerberizedTest' + } + forkEvery 1 +} + task thirdPartyTest (type: Test) { description 'Run Third-party Tests - you need to specify -Dsqoop.thirdparty.lib.dir where the Third party driver ' + 'jars reside (relative to the project directory)' @@ -183,49 +199,37 @@ task thirdPartyTest (type: Test) { systemProperty "ms.datatype.test.hdfsprefix" ,"file:///" } -task manualTest(type: Test) { - description 'Run manual tests' - useJUnit { - includeCategories 'org.apache.sqoop.testcategories.sqooptest.ManualTest' - } -} - -task integrationPlainTest(type: Test) { - description 'Run integration tests which do not need a docker container or an external database' +test { + description 'Run tests that do not need external JDBC driver and/or a docker container. ' + + 'This is the same as running unitTest, integrationTest and kerberizedTest.' useJUnit { + includeCategories 'org.apache.sqoop.testcategories.sqooptest.UnitTest' includeCategories 'org.apache.sqoop.testcategories.sqooptest.IntegrationTest' - excludeCategories 'org.apache.sqoop.testcategories.sqooptest.ManualTest' excludeCategories 'org.apache.sqoop.testcategories.KerberizedTest' excludeCategories 'org.apache.sqoop.testcategories.thirdpartytest.ThirdPartyTest' + excludeCategories 'org.apache.sqoop.testcategories.sqooptest.ManualTest' } } +test.finalizedBy(kerberizedTest) -task kerberizedTest (type: Test){ - description 'Run Kerberized Test' - // A Gradle test task with forkEvery 1 and includeCategories performs poorly because it starts a new JVM even for the filtered tests. - // To work around this performance problem we need to add every kerberized test in a separate include field here. - include '**/TestKerberosAuthenticator*' - include '**/HBaseKerberizedConnectivityTest*' - include '**/TestHiveMiniCluster*' - include '**/TestHiveServer2TextImport*' +task s3Test(type: Test) { + description 'Run S3 tests' useJUnit { - includeCategories 'org.apache.sqoop.testcategories.KerberizedTest' + includeCategories 'org.apache.sqoop.testcategories.thirdpartytest.S3Test' } - forkEvery 1 -} - -test { - description 'Run tests that do not need exteral JDBC driver and/or a docker container. ' + - 'This includes integration plain tests and unit tests' - include '' } -test.finalizedBy(integrationPlainTest, unitTest, kerberizedTest) -task sqoopTest (type: Test){ +task allTest (type: Test){ description 'Run all Sqoop tests' - include '' + useJUnit { + includeCategories 'org.apache.sqoop.testcategories.sqooptest.UnitTest' + includeCategories 'org.apache.sqoop.testcategories.sqooptest.IntegrationTest' + includeCategories 'org.apache.sqoop.testcategories.thirdpartytest.ThirdPartyTest' + excludeCategories 'org.apache.sqoop.testcategories.KerberizedTest' + excludeCategories 'org.apache.sqoop.testcategories.sqooptest.ManualTest' + } } -sqoopTest.finalizedBy(test, kerberizedTest, thirdPartyTest) +allTest.finalizedBy(kerberizedTest) def testBuildDir = "$buildDir/test/" def testBuildDirData ="$testBuildDir/data/" @@ -236,7 +240,7 @@ task buildFolder { tasks.withType(Test) { testLogging { - events 'started', 'passed', 'skipped' + events 'passed', 'skipped' } workingDir = testBuildDirData project.mkdir(workingDir) @@ -253,8 +257,8 @@ tasks.withType(Test) { systemProperty "sqoop.throwOnError", "" minHeapSize = "512m" - maxHeapSize = "5120m" - jvmArgs '-Xmx5012m', "-da:org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge" + maxHeapSize = "8G" + jvmArgs "-da:org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge" dependsOn buildFolder @@ -291,6 +295,7 @@ checkstyle { jacocoTestReport { description "Generates a coverage report of the unit test results under $buildDir/jacocoHtml/test/index.html" + executionData test, kerberizedTest reports { xml.enabled true csv.enabled false @@ -312,7 +317,7 @@ task jacocoThirdPartyReport(type: JacocoReport){ task jacocoCompositeReport(type: JacocoReport){ description "Generates a composite coverage report of test and thirdparty test results under $buildDir/jacocoHtml/index.html" sourceSets sourceSets.main - executionData test, thirdPartyTest + executionData test, kerberizedTest, thirdPartyTest reports { xml.enabled true csv.enabled false http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/src/test/org/apache/sqoop/importjob/SplitByImportTest.java ---------------------------------------------------------------------- diff --git a/src/test/org/apache/sqoop/importjob/SplitByImportTest.java b/src/test/org/apache/sqoop/importjob/SplitByImportTest.java index c6fe4f2..90b7cbb 100644 --- a/src/test/org/apache/sqoop/importjob/SplitByImportTest.java +++ b/src/test/org/apache/sqoop/importjob/SplitByImportTest.java @@ -24,6 +24,7 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.sqoop.SqoopOptions; import org.apache.sqoop.importjob.configuration.GenericImportJobSplitByTestConfiguration; +import org.apache.sqoop.testcategories.thirdpartytest.ThirdPartyTest; import org.apache.sqoop.importjob.configuration.ImportJobTestConfiguration; import org.apache.sqoop.importjob.configuration.ParquetTestConfiguration; import org.apache.sqoop.testutil.ArgumentArrayBuilder; @@ -33,11 +34,13 @@ import org.apache.sqoop.testutil.adapter.MSSQLServerDatabaseAdapter; import org.apache.sqoop.testutil.adapter.MySqlDatabaseAdapter; import org.apache.sqoop.testutil.adapter.OracleDatabaseAdapter; import org.apache.sqoop.testutil.adapter.PostgresDatabaseAdapter; +import org.apache.sqoop.util.BlockJUnit4ClassRunnerWithParametersFactory; import org.apache.sqoop.util.ParquetReader; import org.junit.After; import org.junit.Before; import org.junit.Rule; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.junit.rules.ExpectedException; import org.junit.runner.RunWith; import org.junit.runners.Parameterized; @@ -51,6 +54,8 @@ import static java.util.Arrays.asList; import static org.junit.Assert.assertEquals; @RunWith(Parameterized.class) +@Category(ThirdPartyTest.class) [email protected](BlockJUnit4ClassRunnerWithParametersFactory.class) public class SplitByImportTest extends ImportJobTestCase { public static final Log LOG = LogFactory.getLog(SplitByImportTest.class.getName()); http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/src/test/org/apache/sqoop/manager/TestMainframeManager.java ---------------------------------------------------------------------- diff --git a/src/test/org/apache/sqoop/manager/TestMainframeManager.java b/src/test/org/apache/sqoop/manager/TestMainframeManager.java index c84f05f..654e20c 100644 --- a/src/test/org/apache/sqoop/manager/TestMainframeManager.java +++ b/src/test/org/apache/sqoop/manager/TestMainframeManager.java @@ -119,6 +119,7 @@ public class TestMainframeManager extends BaseSqoopTestCase { } catch (IOException e) { fail("No IOException should be thrown!"); } finally { + HBaseUtil.setAlwaysNoHBaseJarMode(false); opts.setHBaseTable(null); } } http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/src/test/org/apache/sqoop/manager/oracle/TestOraOopDBInputSplitGetDebugDetails.java ---------------------------------------------------------------------- diff --git a/src/test/org/apache/sqoop/manager/oracle/TestOraOopDBInputSplitGetDebugDetails.java b/src/test/org/apache/sqoop/manager/oracle/TestOraOopDBInputSplitGetDebugDetails.java index 6f33ad3..081ce47 100644 --- a/src/test/org/apache/sqoop/manager/oracle/TestOraOopDBInputSplitGetDebugDetails.java +++ b/src/test/org/apache/sqoop/manager/oracle/TestOraOopDBInputSplitGetDebugDetails.java @@ -18,8 +18,10 @@ package org.apache.sqoop.manager.oracle; +import org.apache.sqoop.testcategories.sqooptest.UnitTest; import org.junit.Before; import org.junit.Test; +import org.junit.experimental.categories.Category; import java.util.ArrayList; import java.util.Arrays; @@ -27,6 +29,7 @@ import java.util.List; import static org.junit.Assert.assertEquals; +@Category(UnitTest.class) public class TestOraOopDBInputSplitGetDebugDetails { private OraOopDBInputSplit firstSplit; private OraOopDBInputSplit secondSplit; http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableExportMapper.java ---------------------------------------------------------------------- diff --git a/src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableExportMapper.java b/src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableExportMapper.java index 5e55871..b93c646 100644 --- a/src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableExportMapper.java +++ b/src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableExportMapper.java @@ -26,9 +26,11 @@ import org.apache.hadoop.mapreduce.JobID; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.TaskAttemptID; import org.apache.sqoop.mapreduce.db.DBConfiguration; +import org.apache.sqoop.testcategories.sqooptest.UnitTest; import org.junit.Before; import org.junit.Rule; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.junit.rules.ExpectedException; import org.junit.rules.RuleChain; import org.junit.rules.TemporaryFolder; http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableImportMapper.java ---------------------------------------------------------------------- diff --git a/src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableImportMapper.java b/src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableImportMapper.java index 1a69437..bb908db 100644 --- a/src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableImportMapper.java +++ b/src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableImportMapper.java @@ -25,9 +25,11 @@ import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.TaskAttemptID; import org.apache.sqoop.mapreduce.db.DBConfiguration; +import org.apache.sqoop.testcategories.sqooptest.UnitTest; import org.junit.Before; import org.junit.Rule; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.junit.rules.ExpectedException; import org.junit.rules.TemporaryFolder; @@ -44,6 +46,7 @@ import static org.mockito.Matchers.anyString; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.when; +@Category(UnitTest.class) public class TestNetezzaExternalTableImportMapper { @Rule http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetBinaryRecord.java ---------------------------------------------------------------------- diff --git a/src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetBinaryRecord.java b/src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetBinaryRecord.java index b4cba28..f5add92 100644 --- a/src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetBinaryRecord.java +++ b/src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetBinaryRecord.java @@ -22,9 +22,11 @@ import org.apache.commons.logging.LogFactory; import org.apache.commons.net.ftp.FTPClient; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.mapreduce.TaskAttemptContext; +import org.apache.sqoop.testcategories.sqooptest.UnitTest; import org.junit.Assert; import org.junit.Before; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.mockito.invocation.InvocationOnMock; import org.mockito.stubbing.Answer; @@ -38,6 +40,7 @@ import static org.mockito.Mockito.when; import static org.mockito.Mockito.spy; import static org.mockito.Mockito.doReturn; +@Category(UnitTest.class) public class TestMainframeDatasetBinaryRecord { private MainframeDatasetFTPRecordReader ftpRecordReader; http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeFTPFileGdgEntryParser.java ---------------------------------------------------------------------- diff --git a/src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeFTPFileGdgEntryParser.java b/src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeFTPFileGdgEntryParser.java index 521a042..bd08ee8 100644 --- a/src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeFTPFileGdgEntryParser.java +++ b/src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeFTPFileGdgEntryParser.java @@ -23,11 +23,15 @@ import java.util.List; import java.util.Objects; import org.apache.commons.net.ftp.FTPFile; +import org.apache.sqoop.testcategories.sqooptest.UnitTest; import org.junit.Before; import org.junit.Test; +import org.junit.experimental.categories.Category; + import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; +@Category(UnitTest.class) public class TestMainframeFTPFileGdgEntryParser { /* Sample FTP listing Volume Unit Referred Ext Used Recfm Lrecl BlkSz Dsorg Dsname http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/src/test/org/apache/sqoop/s3/TestS3AvroImport.java ---------------------------------------------------------------------- diff --git a/src/test/org/apache/sqoop/s3/TestS3AvroImport.java b/src/test/org/apache/sqoop/s3/TestS3AvroImport.java index 7f5f5d6..9d37187 100644 --- a/src/test/org/apache/sqoop/s3/TestS3AvroImport.java +++ b/src/test/org/apache/sqoop/s3/TestS3AvroImport.java @@ -21,6 +21,7 @@ package org.apache.sqoop.s3; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.fs.FileSystem; +import org.apache.sqoop.testcategories.thirdpartytest.S3Test; import org.apache.sqoop.testutil.ArgumentArrayBuilder; import org.apache.sqoop.testutil.AvroTestUtils; import org.apache.sqoop.testutil.DefaultS3CredentialGenerator; @@ -32,10 +33,12 @@ import org.junit.Before; import org.junit.BeforeClass; import org.junit.Rule; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.junit.rules.ExpectedException; import java.io.IOException; +@Category(S3Test.class) public class TestS3AvroImport extends ImportJobTestCase { public static final Log LOG = LogFactory.getLog( http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/src/test/org/apache/sqoop/s3/TestS3ExternalHiveTableImport.java ---------------------------------------------------------------------- diff --git a/src/test/org/apache/sqoop/s3/TestS3ExternalHiveTableImport.java b/src/test/org/apache/sqoop/s3/TestS3ExternalHiveTableImport.java index 0c3161e..1f01669 100644 --- a/src/test/org/apache/sqoop/s3/TestS3ExternalHiveTableImport.java +++ b/src/test/org/apache/sqoop/s3/TestS3ExternalHiveTableImport.java @@ -22,16 +22,19 @@ import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.fs.FileSystem; import org.apache.sqoop.hive.minicluster.HiveMiniCluster; +import org.apache.sqoop.testcategories.thirdpartytest.S3Test; import org.apache.sqoop.testutil.ArgumentArrayBuilder; import org.apache.sqoop.testutil.DefaultS3CredentialGenerator; import org.apache.sqoop.testutil.HiveServer2TestUtil; import org.apache.sqoop.testutil.ImportJobTestCase; import org.apache.sqoop.testutil.S3CredentialGenerator; import org.apache.sqoop.testutil.S3TestUtils; +import org.apache.sqoop.util.BlockJUnit4ClassRunnerWithParametersFactory; import org.junit.After; import org.junit.Before; import org.junit.BeforeClass; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.junit.runner.RunWith; import org.junit.runners.Parameterized; @@ -43,7 +46,9 @@ import static org.apache.sqoop.tool.BaseSqoopTool.FMT_PARQUETFILE_ARG; import static org.apache.sqoop.tool.BaseSqoopTool.FMT_TEXTFILE_ARG; import static org.junit.Assert.assertEquals; +@Category(S3Test.class) @RunWith(Parameterized.class) [email protected](BlockJUnit4ClassRunnerWithParametersFactory.class) public class TestS3ExternalHiveTableImport extends ImportJobTestCase { @Parameterized.Parameters(name = "fileFormatArg = {0}, expectedResult = {1}") http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/src/test/org/apache/sqoop/s3/TestS3ImportWithHadoopCredProvider.java ---------------------------------------------------------------------- diff --git a/src/test/org/apache/sqoop/s3/TestS3ImportWithHadoopCredProvider.java b/src/test/org/apache/sqoop/s3/TestS3ImportWithHadoopCredProvider.java index 3a0d636..e1d7cbd 100644 --- a/src/test/org/apache/sqoop/s3/TestS3ImportWithHadoopCredProvider.java +++ b/src/test/org/apache/sqoop/s3/TestS3ImportWithHadoopCredProvider.java @@ -25,6 +25,7 @@ import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.s3a.Constants; import org.apache.hadoop.security.alias.CredentialShell; import org.apache.hadoop.util.ToolRunner; +import org.apache.sqoop.testcategories.thirdpartytest.S3Test; import org.apache.sqoop.testutil.ArgumentArrayBuilder; import org.apache.sqoop.testutil.DefaultS3CredentialGenerator; import org.apache.sqoop.testutil.ImportJobTestCase; @@ -40,6 +41,7 @@ import org.junit.ClassRule; import org.junit.Rule; import org.junit.Test; import org.junit.contrib.java.lang.system.EnvironmentVariables; +import org.junit.experimental.categories.Category; import org.junit.rules.ExpectedException; import java.io.File; @@ -48,6 +50,7 @@ import java.nio.file.Files; import static junit.framework.TestCase.fail; +@Category(S3Test.class) public class TestS3ImportWithHadoopCredProvider extends ImportJobTestCase { public static final Log LOG = LogFactory.getLog( TestS3ImportWithHadoopCredProvider.class.getName()); http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/src/test/org/apache/sqoop/s3/TestS3IncrementalAppendAvroImport.java ---------------------------------------------------------------------- diff --git a/src/test/org/apache/sqoop/s3/TestS3IncrementalAppendAvroImport.java b/src/test/org/apache/sqoop/s3/TestS3IncrementalAppendAvroImport.java index 5faf59e..69b6e66 100644 --- a/src/test/org/apache/sqoop/s3/TestS3IncrementalAppendAvroImport.java +++ b/src/test/org/apache/sqoop/s3/TestS3IncrementalAppendAvroImport.java @@ -21,6 +21,7 @@ package org.apache.sqoop.s3; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.fs.FileSystem; +import org.apache.sqoop.testcategories.thirdpartytest.S3Test; import org.apache.sqoop.testutil.ArgumentArrayBuilder; import org.apache.sqoop.testutil.AvroTestUtils; import org.apache.sqoop.testutil.DefaultS3CredentialGenerator; @@ -32,12 +33,14 @@ import org.junit.Before; import org.junit.BeforeClass; import org.junit.Rule; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.junit.rules.ExpectedException; import java.io.IOException; import static org.apache.sqoop.util.AppendUtils.MAPREDUCE_OUTPUT_BASENAME_PROPERTY; +@Category(S3Test.class) public class TestS3IncrementalAppendAvroImport extends ImportJobTestCase { public static final Log LOG = LogFactory.getLog( http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/src/test/org/apache/sqoop/s3/TestS3IncrementalAppendParquetImport.java ---------------------------------------------------------------------- diff --git a/src/test/org/apache/sqoop/s3/TestS3IncrementalAppendParquetImport.java b/src/test/org/apache/sqoop/s3/TestS3IncrementalAppendParquetImport.java index a4f9864..0abfb81 100644 --- a/src/test/org/apache/sqoop/s3/TestS3IncrementalAppendParquetImport.java +++ b/src/test/org/apache/sqoop/s3/TestS3IncrementalAppendParquetImport.java @@ -21,6 +21,7 @@ package org.apache.sqoop.s3; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.fs.FileSystem; +import org.apache.sqoop.testcategories.thirdpartytest.S3Test; import org.apache.sqoop.testutil.ArgumentArrayBuilder; import org.apache.sqoop.testutil.DefaultS3CredentialGenerator; import org.apache.sqoop.testutil.ImportJobTestCase; @@ -32,6 +33,7 @@ import org.junit.Before; import org.junit.BeforeClass; import org.junit.Rule; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.junit.rules.ExpectedException; import java.io.IOException; @@ -40,6 +42,7 @@ import java.util.List; import static org.apache.sqoop.util.AppendUtils.MAPREDUCE_OUTPUT_BASENAME_PROPERTY; import static org.junit.Assert.assertEquals; +@Category(S3Test.class) public class TestS3IncrementalAppendParquetImport extends ImportJobTestCase { public static final Log LOG = LogFactory.getLog( http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/src/test/org/apache/sqoop/s3/TestS3IncrementalAppendSequenceFileImport.java ---------------------------------------------------------------------- diff --git a/src/test/org/apache/sqoop/s3/TestS3IncrementalAppendSequenceFileImport.java b/src/test/org/apache/sqoop/s3/TestS3IncrementalAppendSequenceFileImport.java index d271588..7f00706 100644 --- a/src/test/org/apache/sqoop/s3/TestS3IncrementalAppendSequenceFileImport.java +++ b/src/test/org/apache/sqoop/s3/TestS3IncrementalAppendSequenceFileImport.java @@ -21,6 +21,7 @@ package org.apache.sqoop.s3; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.fs.FileSystem; +import org.apache.sqoop.testcategories.thirdpartytest.S3Test; import org.apache.sqoop.testutil.ArgumentArrayBuilder; import org.apache.sqoop.testutil.DefaultS3CredentialGenerator; import org.apache.sqoop.testutil.ImportJobTestCase; @@ -32,12 +33,14 @@ import org.junit.Before; import org.junit.BeforeClass; import org.junit.Rule; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.junit.rules.ExpectedException; import java.io.IOException; import static org.apache.sqoop.util.AppendUtils.MAPREDUCE_OUTPUT_BASENAME_PROPERTY; +@Category(S3Test.class) public class TestS3IncrementalAppendSequenceFileImport extends ImportJobTestCase { public static final Log LOG = LogFactory.getLog( http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/src/test/org/apache/sqoop/s3/TestS3IncrementalAppendTextImport.java ---------------------------------------------------------------------- diff --git a/src/test/org/apache/sqoop/s3/TestS3IncrementalAppendTextImport.java b/src/test/org/apache/sqoop/s3/TestS3IncrementalAppendTextImport.java index 52d89c7..eabdae8 100644 --- a/src/test/org/apache/sqoop/s3/TestS3IncrementalAppendTextImport.java +++ b/src/test/org/apache/sqoop/s3/TestS3IncrementalAppendTextImport.java @@ -21,6 +21,7 @@ package org.apache.sqoop.s3; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.fs.FileSystem; +import org.apache.sqoop.testcategories.thirdpartytest.S3Test; import org.apache.sqoop.testutil.ArgumentArrayBuilder; import org.apache.sqoop.testutil.DefaultS3CredentialGenerator; import org.apache.sqoop.testutil.ImportJobTestCase; @@ -32,12 +33,14 @@ import org.junit.Before; import org.junit.BeforeClass; import org.junit.Rule; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.junit.rules.ExpectedException; import java.io.IOException; import static org.apache.sqoop.util.AppendUtils.MAPREDUCE_OUTPUT_BASENAME_PROPERTY; +@Category(S3Test.class) public class TestS3IncrementalAppendTextImport extends ImportJobTestCase { public static final Log LOG = LogFactory.getLog( http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/src/test/org/apache/sqoop/s3/TestS3IncrementalMergeParquetImport.java ---------------------------------------------------------------------- diff --git a/src/test/org/apache/sqoop/s3/TestS3IncrementalMergeParquetImport.java b/src/test/org/apache/sqoop/s3/TestS3IncrementalMergeParquetImport.java index 39238c5..738e6f8 100644 --- a/src/test/org/apache/sqoop/s3/TestS3IncrementalMergeParquetImport.java +++ b/src/test/org/apache/sqoop/s3/TestS3IncrementalMergeParquetImport.java @@ -21,6 +21,7 @@ package org.apache.sqoop.s3; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.fs.FileSystem; +import org.apache.sqoop.testcategories.thirdpartytest.S3Test; import org.apache.sqoop.testutil.ArgumentArrayBuilder; import org.apache.sqoop.testutil.DefaultS3CredentialGenerator; import org.apache.sqoop.testutil.ImportJobTestCase; @@ -32,6 +33,7 @@ import org.junit.Before; import org.junit.BeforeClass; import org.junit.Rule; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.junit.rules.ExpectedException; import java.io.IOException; @@ -40,6 +42,7 @@ import java.util.List; import static org.apache.sqoop.util.AppendUtils.MAPREDUCE_OUTPUT_BASENAME_PROPERTY; import static org.junit.Assert.assertEquals; +@Category(S3Test.class) public class TestS3IncrementalMergeParquetImport extends ImportJobTestCase { public static final Log LOG = LogFactory.getLog( http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/src/test/org/apache/sqoop/s3/TestS3IncrementalMergeTextImport.java ---------------------------------------------------------------------- diff --git a/src/test/org/apache/sqoop/s3/TestS3IncrementalMergeTextImport.java b/src/test/org/apache/sqoop/s3/TestS3IncrementalMergeTextImport.java index 597e3de..7645054 100644 --- a/src/test/org/apache/sqoop/s3/TestS3IncrementalMergeTextImport.java +++ b/src/test/org/apache/sqoop/s3/TestS3IncrementalMergeTextImport.java @@ -21,6 +21,7 @@ package org.apache.sqoop.s3; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.fs.FileSystem; +import org.apache.sqoop.testcategories.thirdpartytest.S3Test; import org.apache.sqoop.testutil.ArgumentArrayBuilder; import org.apache.sqoop.testutil.DefaultS3CredentialGenerator; import org.apache.sqoop.testutil.ImportJobTestCase; @@ -32,12 +33,14 @@ import org.junit.Before; import org.junit.BeforeClass; import org.junit.Rule; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.junit.rules.ExpectedException; import java.io.IOException; import static org.apache.sqoop.util.AppendUtils.MAPREDUCE_OUTPUT_BASENAME_PROPERTY; +@Category(S3Test.class) public class TestS3IncrementalMergeTextImport extends ImportJobTestCase { public static final Log LOG = LogFactory.getLog( http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/src/test/org/apache/sqoop/s3/TestS3ParquetImport.java ---------------------------------------------------------------------- diff --git a/src/test/org/apache/sqoop/s3/TestS3ParquetImport.java b/src/test/org/apache/sqoop/s3/TestS3ParquetImport.java index c9785d8..58309de 100644 --- a/src/test/org/apache/sqoop/s3/TestS3ParquetImport.java +++ b/src/test/org/apache/sqoop/s3/TestS3ParquetImport.java @@ -21,6 +21,7 @@ package org.apache.sqoop.s3; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.fs.FileSystem; +import org.apache.sqoop.testcategories.thirdpartytest.S3Test; import org.apache.sqoop.testutil.ArgumentArrayBuilder; import org.apache.sqoop.testutil.DefaultS3CredentialGenerator; import org.apache.sqoop.testutil.ImportJobTestCase; @@ -32,6 +33,7 @@ import org.junit.Before; import org.junit.BeforeClass; import org.junit.Rule; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.junit.rules.ExpectedException; import java.io.IOException; @@ -39,6 +41,7 @@ import java.util.List; import static org.junit.Assert.assertEquals; +@Category(S3Test.class) public class TestS3ParquetImport extends ImportJobTestCase { public static final Log LOG = LogFactory.getLog( http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/src/test/org/apache/sqoop/s3/TestS3SequenceFileImport.java ---------------------------------------------------------------------- diff --git a/src/test/org/apache/sqoop/s3/TestS3SequenceFileImport.java b/src/test/org/apache/sqoop/s3/TestS3SequenceFileImport.java index bba8b74..5bc747b 100644 --- a/src/test/org/apache/sqoop/s3/TestS3SequenceFileImport.java +++ b/src/test/org/apache/sqoop/s3/TestS3SequenceFileImport.java @@ -21,6 +21,7 @@ package org.apache.sqoop.s3; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.fs.FileSystem; +import org.apache.sqoop.testcategories.thirdpartytest.S3Test; import org.apache.sqoop.testutil.ArgumentArrayBuilder; import org.apache.sqoop.testutil.DefaultS3CredentialGenerator; import org.apache.sqoop.testutil.ImportJobTestCase; @@ -32,10 +33,12 @@ import org.junit.Before; import org.junit.BeforeClass; import org.junit.Rule; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.junit.rules.ExpectedException; import java.io.IOException; +@Category(S3Test.class) public class TestS3SequenceFileImport extends ImportJobTestCase { public static final Log LOG = LogFactory.getLog( http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/src/test/org/apache/sqoop/s3/TestS3TextImport.java ---------------------------------------------------------------------- diff --git a/src/test/org/apache/sqoop/s3/TestS3TextImport.java b/src/test/org/apache/sqoop/s3/TestS3TextImport.java index 114f97c..f1c3fa4 100644 --- a/src/test/org/apache/sqoop/s3/TestS3TextImport.java +++ b/src/test/org/apache/sqoop/s3/TestS3TextImport.java @@ -21,6 +21,7 @@ package org.apache.sqoop.s3; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.fs.FileSystem; +import org.apache.sqoop.testcategories.thirdpartytest.S3Test; import org.apache.sqoop.testutil.ArgumentArrayBuilder; import org.apache.sqoop.testutil.DefaultS3CredentialGenerator; import org.apache.sqoop.testutil.ImportJobTestCase; @@ -32,10 +33,12 @@ import org.junit.Before; import org.junit.BeforeClass; import org.junit.Rule; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.junit.rules.ExpectedException; import java.io.IOException; +@Category(S3Test.class) public class TestS3TextImport extends ImportJobTestCase { public static final Log LOG = LogFactory.getLog( http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/src/test/org/apache/sqoop/testcategories/thirdpartytest/S3Test.java ---------------------------------------------------------------------- diff --git a/src/test/org/apache/sqoop/testcategories/thirdpartytest/S3Test.java b/src/test/org/apache/sqoop/testcategories/thirdpartytest/S3Test.java new file mode 100644 index 0000000..f2ebb45 --- /dev/null +++ b/src/test/org/apache/sqoop/testcategories/thirdpartytest/S3Test.java @@ -0,0 +1,10 @@ +package org.apache.sqoop.testcategories.thirdpartytest; + +/** + * An S3 test shall test the integration with the Amazon S3 cloud service. + * These tests also require AWS credentials to access S3 and they run only if these + * credentials are provided via the -Ds3.generator.command=<credential-generator-command> property + * as well as the target S3 location via the -Ds3.bucket.url=<bucket-url> property. + */ +public interface S3Test extends ThirdPartyTest { +} http://git-wip-us.apache.org/repos/asf/sqoop/blob/bb9c2dd8/src/test/org/apache/sqoop/tool/TestS3IncrementalImportOptionValidations.java ---------------------------------------------------------------------- diff --git a/src/test/org/apache/sqoop/tool/TestS3IncrementalImportOptionValidations.java b/src/test/org/apache/sqoop/tool/TestS3IncrementalImportOptionValidations.java index 7745f1b..d6aca99 100644 --- a/src/test/org/apache/sqoop/tool/TestS3IncrementalImportOptionValidations.java +++ b/src/test/org/apache/sqoop/tool/TestS3IncrementalImportOptionValidations.java @@ -19,9 +19,12 @@ package org.apache.sqoop.tool; import org.apache.sqoop.SqoopOptions; +import org.apache.sqoop.testcategories.sqooptest.UnitTest; +import org.apache.sqoop.util.BlockJUnit4ClassRunnerWithParametersFactory; import org.junit.Before; import org.junit.Rule; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.junit.rules.ExpectedException; import org.junit.runner.RunWith; import org.junit.runners.Parameterized; @@ -36,7 +39,9 @@ import static org.apache.sqoop.tool.BaseSqoopTool.TEMP_ROOTDIR_ARG; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.when; +@Category(UnitTest.class) @RunWith(Parameterized.class) [email protected](BlockJUnit4ClassRunnerWithParametersFactory.class) public class TestS3IncrementalImportOptionValidations { @Parameterized.Parameters(name = "incrementalMode = {0}")
