Repository: sqoop Updated Branches: refs/heads/trunk 6c6963abe -> b148d5400
SQOOP-3052: Introduce Gradle based build for Sqoop to make it more developer friendly / open (Anna Szonyi via Szabolcs Vasas) Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/b148d540 Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/b148d540 Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/b148d540 Branch: refs/heads/trunk Commit: b148d54000da8d718f95917f8bf805860a6cbfe6 Parents: 6c6963a Author: Szabolcs Vasas <[email protected]> Authored: Mon Jul 23 13:48:22 2018 +0200 Committer: Szabolcs Vasas <[email protected]> Committed: Mon Jul 23 13:48:22 2018 +0200 ---------------------------------------------------------------------- .gitignore | 5 + COMPILING.txt | 57 ++- build.gradle | 400 ++++++++++++++++++++++ config/checkstyle/checkstyle-java-header.txt | 17 + config/checkstyle/checkstyle-noframes.xsl | 198 +++++++++++ config/checkstyle/checkstyle.xml | 226 ++++++++++++ gradle.properties | 57 +++ gradle/customUnixStartScript.txt | 100 ++++++ gradle/customWindowsStartScript.txt | 26 ++ gradle/sqoop-package.gradle | 209 +++++++++++ gradle/sqoop-version-gen.gradle | 76 ++++ gradle/wrapper/gradle-wrapper.jar | Bin 0 -> 54333 bytes gradle/wrapper/gradle-wrapper.properties | 5 + gradlew | 172 ++++++++++ gradlew.bat | 84 +++++ settings.gradle | 18 + src/scripts/rat-violations.sh | 4 + testdata/hcatalog/conf/hive-site.xml | 2 +- 18 files changed, 1639 insertions(+), 17 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/sqoop/blob/b148d540/.gitignore ---------------------------------------------------------------------- diff --git a/.gitignore b/.gitignore index 68cbe28..36f53f9 100644 --- a/.gitignore +++ b/.gitignore @@ -36,3 +36,8 @@ target /conf/managers.d /conf/tools.d /tags +src/java/org/apache/sqoop/SqoopVersion.java +gradle/build +gradle/.gradle +.gradle +out http://git-wip-us.apache.org/repos/asf/sqoop/blob/b148d540/COMPILING.txt ---------------------------------------------------------------------- diff --git a/COMPILING.txt b/COMPILING.txt index 3b82250..9abf0b2 100644 --- a/COMPILING.txt +++ b/COMPILING.txt @@ -27,7 +27,7 @@ under the License. Compiling Sqoop requires the following tools: -* Apache ant (1.7.1) +* Apache ant (1.7.1) or Gradle (3.5.1) * Java JDK 1.6 Additionally, building the documentation requires these tools: @@ -52,7 +52,7 @@ Sqoop is compiled with ant. Type +ant -p+ to see the list of available targets. Type +ant+ to compile all java sources. You can then run Sqoop with +bin/sqoop+. If you want to build everything (including the documentation), type -+ant package+. This will appear in the ++ant package+ or +./gradlew package+. This will appear in the +build/sqoop-(version)/+ directory. This version of Sqoop is built against Hadoop 0.23 available from Apache @@ -61,7 +61,7 @@ downloads the necessary binaries. == Testing Sqoop -Sqoop has several unit tests which can be run with +ant test+. This command +Sqoop has several unit tests which can be run with +ant test+ or +./gradlew test+. This command will run all the "basic" checks against an in-memory database, HSQLDB. Sqoop also has compatibility tests that check its ability to work with @@ -203,6 +203,7 @@ After the third-party databases are installed and configured, run: ++++ ant test -Dthirdparty=true -Dsqoop.thirdparty.lib.dir=/path/to/jdbc/drivers/ +./gradlew -Dsqoop.thirdparty.lib.dir=/relative/path/to/jdbc/drivers/ thirdPartyTest ++++ This command will run all thirdparty tests except some DB2 tests. @@ -211,6 +212,7 @@ as follows: ++++ ant test -Dmanual=true -Dsqoop.thirdparty.lib.dir=/path/to/jdbc/drivers/ +./gradlew -Dsqoop.thirdparty.lib.dir=/relative/path/to/jdbc/drivers/ manualTest ++++ Note that +sqoop.thirdparty.lib.dir+ can also be specified in @@ -308,13 +310,15 @@ run: ++++ ant findbugs -Dfindbugs.home=/path/to/findbugs/ +or +./gradlew findbugsMain ++++ A report will be generated in +build/findbugs/+ -=== Cobertura +=== Code Coverage reports -Cobertura runs code coverage checks. It instruments the build and +For ant Cobertura runs code coverage checks. It instruments the build and checks that each line and conditional expression is evaluated along all possible paths. @@ -325,6 +329,19 @@ ant clean ant cobertura -Dcobertura.home=/path/to/cobertura ant cobertura -Dcobertura.home=/path/to/cobertura \ -Dthirdparty=true -Dsqoop.thirdparty.lib.dir=/path/to/thirdparty + +For Gradle we run Jacoco for code coverage checks. You can create single reports or composite reports, +where you can check the combined coverage of unit and thirdparty tests. + +./gradlew clean +./gradlew test +./gradlew jacocoTestReport + +./gradlew -Dsqoop.thirdparty.lib.dir=<path_to_thirdparty_lib_directory> thirdPartyTest +./gradlew jacocoThirdPartyReport + +or generate the composite report after running test and thirdPartyTest +./gradlew jacocoCompositeReport ++++ (You'll need to run the cobertura target twice; once against the regular @@ -350,6 +367,7 @@ To run checkstyle, execute: ++++ ant checkstyle +./gradlew checkStyleMain ++++ A report will be generated as +build/checkstyle-errors.html+ @@ -364,6 +382,7 @@ To install Sqoop in your local +.m2+ cache, run: ++++ ant mvn-install +./gradlew publishToMavenLocal ++++ This will install a pom and the Sqoop jar. @@ -372,6 +391,8 @@ To deploy Sqoop to a public repository, use: ++++ ant mvn-deploy +./gradlew publishSqoopPublicationToCloudera.snapshot.repoRepository +./gradlew -DmvnRepo=x publishSqoopPublicationToCloudera.x.repoRepository ++++ By default, this deploys to repository.cloudera.com. You can choose @@ -387,7 +408,7 @@ This will build a binary release tarball and the web-based documentation as well as run a release audit which flags any source files which may be missing license headers. -(The release audit can be run standalone with the +ant releaseaudit+ +(The release audit can be run standalone with the +ant releaseaudit+ (+./gradlew rat+) target.) You must set the +version+ property explicitly; you cannot release a @@ -402,16 +423,6 @@ will allow you to edit Sqoop sources in Eclipse with all the library dependencies correctly resolved. To compile the jars, you should still use ant. - -== Using a specific version of Hadoop - -Now Sqoop defaults to use Hadoop 0.23 available from Apache maven repository. -To switch back to the previous version of Hadoop 0.20, for example, run: - -++++ -ant test -Dhadoopversion=20 -++++ - == Building the documentation Building the documentation requires that you have toxml installed. @@ -421,3 +432,17 @@ Also, one needs to set the XML_CATALOG_FILES environment variable. export XML_CATALOG_FILES=/usr/local/etc/xml/catalog ant docs ++++ + +== Other important Gradle commands + ++ Generate gradle wrapper (to ensure you are using the correct, compatible version of gradle) by running +./gradlew wrapper+ ++ ./gradlew tasks to list all top-level gradle tasks or run ./gradlew tasks --all to show all tasks and subtasks ++ ./gradlew compileJava to compile the main Java source ++ ./gradlew -Dsqoop.thirdparty.lib.dir=<relative_path_to_thirdparty_lib_directory> -Dsqoop.test.mysql.connectstring.host_url=jdbc:mysql://127.0.0.1:3306/ -Dsqoop.test.mysql.databasename=sqoop -Dsqoop.test.mysql.password=Sqoop12345 -Dsqoop.test.mysql.username=sqoop -Dsqoop.test.oracle.connectstring=jdbc:oracle:thin:@//localhost:1521/sqoop -Dsqoop.test.oracle.username=SYSTEM -Dsqoop.test.oracle.password=Sqoop12345 -Dsqoop.test.postgresql.connectstring.host_url=jdbc:postgresql://localhost/ -Dsqoop.test.postgresql.database=sqoop -Dsqoop.test.postgresql.username=sqoop -Dsqoop.test.postgresql.password=Sqoop12345 -Dsqoop.test.cubrid.connectstring.host_url=jdbc:cubrid:localhost:33000 -Dsqoop.test.cubrid.connectstring.username=sqoop -Dsqoop.test.cubrid.connectstring.database=sqoop -Dsqoop.test.cubrid.connectstring.password=Sqoop12345 -Dmapred.child.java.opts="\-Djava.security.egd=file:/dev/../dev/urandom" -Dtest.timeout=10000000 -Dsqoop.test.sqlserver.connectstring.host_url=jdbc:sqlserver:// localhost:1433 -Dsqoop.test.sqlserver.database=master -Dms.sqlserver.username=sa -Dms.sqlserver.password=Sqoop12345 -Dsqoop.test.db2.connectstring.host_url=jdbc:db2://localhost:50000 -Dsqoop.test.db2.connectstring.database=SQOOP -Dsqoop.test.db2.connectstring.username=DB2INST1 -Dsqoop.test.db2.connectstring.password=Sqoop12345 thirdPartyTest ++ ./gradlew test --debug-jvm : to run remote debug on port 5005 ++ ./gradlew -Dtest.single=ClassName*Test test or ./gradlew tests --tests ++ To refresh dependencies for a build ./gradlew build --refresh-dependencies ++ For skipping a single test or a set of tests ./gradle build -x test ++ To run a single test class use --tests ClassName*Test or -DtestType.single=ClassName*Test (use with test, thirdPartyTest or manualTest) ++ To get a dependency tree: ./gradlew dependencyInsight --configuration optionalConfiguration --dependency searchedForDependency ++ For a list of the dependencies of the selected project, broken down by configuration run: ./gradlew dependencies \ No newline at end of file http://git-wip-us.apache.org/repos/asf/sqoop/blob/b148d540/build.gradle ---------------------------------------------------------------------- diff --git a/build.gradle b/build.gradle new file mode 100644 index 0000000..8b8b555 --- /dev/null +++ b/build.gradle @@ -0,0 +1,400 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +plugins { + id "org.nosphere.apache.rat" version "0.3.0" +} +apply plugin: 'java' +apply plugin: 'eclipse' +apply plugin: 'idea' +apply plugin: 'checkstyle' +apply plugin: 'jacoco' +apply plugin: 'findbugs' +apply plugin: 'maven-publish' + +apply from: 'gradle/sqoop-version-gen.gradle' +apply from: 'gradle/sqoop-package.gradle' + +group = 'org.apache.sqoopgradle' + +sourceCompatibility = javaSourceCompatibilityVersion + +repositories { + mavenCentral() + maven { + url 'http://conjars.org/repo/' + } +} + +sourceSets { + main { + java { + srcDirs = ['src/java'] + } + resources { + srcDirs = ['src/java'] + } + } + test { + java { + srcDirs = ['src/test'] + exclude 'aop/' + } + resources { + srcDirs = ['src/test', 'testdata/hcatalog/conf', 'conf'] + exclude 'aop/' + } + } + aop { + java { + srcDirs = ['src/test/aop'] + } + resources { + srcDirs = ['src/test/aop'] + } + } +} + +configurations { + common.extendsFrom(redist) + compile.extendsFrom(common) +} + +compileJava.dependsOn(SqoopVersionFileGen) + +configurations.all { + resolutionStrategy { + force group: 'org.apache.avro', name: 'avro', version: avroVersion + force group: 'org.apache.avro', name: 'avro-mapred', version: avroVersion + } + exclude group: 'org.apache.hadoop', module: 'avro' +} + +def sqoopThirdPartyLib = System.getProperty("sqoop.thirdparty.lib.dir") + +dependencies { + if (sqoopThirdPartyLib != null) runtime fileTree(dir: sqoopThirdPartyLib, include: '*.jar') + compile group: 'org.apache.hadoop', name: 'hadoop-common', version: hadoopVersion + compile group: 'org.apache.hadoop', name: 'hadoop-hdfs', version: hadoopVersion + compile group: 'org.apache.hadoop', name: 'hadoop-mapreduce-client-common', version: hadoopVersion + compile group: 'org.apache.hadoop', name: 'hadoop-mapreduce-client-core', version: hadoopVersion + compile group: 'org.apache.hbase', name: 'hbase-hadoop-compat', version: hbaseVersion + + aopCompile group: 'org.aspectj', name: 'aspectjtools', version: aspectjVersion + aopCompile group: 'org.aspectj', name: 'aspectjrt', version: aspectjVersion + aopCompile group: 'org.apache.hadoop', name: 'hadoop-common', version: hadoopVersion + + common group: 'org.apache.accumulo', name: 'accumulo-core', version: accumuloVersion + common group: 'org.apache.accumulo', name: 'accumulo-minicluster', version: accumuloVersion + common group: 'org.apache.hbase', name: 'hbase-hadoop2-compat', version: hbaseVersion + common group: 'org.apache.hbase', name: 'hbase-server', version: hbaseVersion + common group: 'org.apache.hbase', name: 'hbase-client', version: hbaseVersion + common group: 'org.apache.hbase', name: 'hbase-common', version: hbaseVersion + common (group: 'org.apache.hive.hcatalog', name: 'hive-hcatalog-core', version: hcatalogVersion) { + exclude group: 'org.apache.avro', module: 'avro' + } + common (group: 'org.apache.hive', name: 'hive-jdbc', version: hcatalogVersion) { + exclude group: 'org.apache.avro', module: 'avro' + } + common group: 'commons-cli', name: 'commons-cli', version: commonscliVersion + common group: 'commons-logging', name: 'commons-logging', version: commonsloggingVersion + common group: 'commons-net', name: 'commons-net', version: commonsnetVersion + common group: 'log4j', name: 'log4j', version: log4jVersion + common group: 'org.postgresql', name: 'postgresql', version: postgresqlVersion + + testCompile group: 'com.h2database', name: 'h2', version: h2Version + testCompile group: 'org.apache.hbase', name: 'hbase-server', version: hbaseVersion, classifier: 'tests' + testCompile group: 'org.apache.hbase', name: 'hbase-hadoop2-compat', version: hbaseVersion, classifier: 'tests' + testCompile group: 'org.apache.hbase', name: 'hbase-hadoop-compat', version: hbaseVersion, classifier: 'tests' + testCompile( group: 'org.apache.hadoop', name: 'hadoop-minikdc', version: hadoopVersion) { + exclude group: 'org.apache.directory.api', module: 'api-ldap-schema-data' + } + testCompile group: 'junit', name: 'junit', version: junitVersion + testCompile group: 'org.assertj', name: 'assertj-core', version: assertjVersion + testCompile group: 'org.mockito', name: 'mockito-core', version: mockitoallVersion + testCompile group: 'org.apache.zookeeper', name: 'zookeeper', version: zookeeperVersion, ext: 'jar' +} + +test { + include '**/Test*.*' +} + +task thirdPartyTest(type: Test) { + description 'Run ThirdParty tests - you need to specify -Dsqoop.thirdparty.lib.dir where the Third party driver jars reside (relative to the project directory)' + exclude '**/*ManualTest.*' + exclude '**/Test*.*' + include '**/*Test*.*' + systemProperty "ms.datatype.test.data.file.export" ,"DatatypeTestData-export-lite.txt" + systemProperty "ms.datatype.test.data.file.import" ,"DatatypeTestData-import-lite.txt" + systemProperty "ms.datatype.test.data.file.delim" ,"," + systemProperty "ms.datatype.test.hdfsprefix" ,"file:///" +} + +task manualTest(type: Test) { + description 'Run manual tests' + include '**/*ManualTest.*' +} + +def testBuildDir = "$buildDir/test/" +def testBuildDirData ="$testBuildDir/data/" + +task buildFolder { + project.mkdir(testBuildDirData) +} + +tasks.withType(Test) { + testLogging { + events 'started', 'passed', 'skipped' + } + workingDir = testBuildDirData + project.mkdir(workingDir) + systemProperties(System.getProperties()) + systemProperty "test.data.dir", "$projectDir/testdata" + systemProperty 'test.build.data', "$testBuildDir/data" + systemProperty "hadoop.root.logger", "DEBUG,console" + systemProperty "user.dir", workingDir + systemProperty "sqoop.src.dir", "$testBuildDir/data" + systemProperty "hadoop.tmp.dir", "$testBuildDir/hadoop" + systemProperty "fs.default.name", "" + systemProperty "hadoop.log.dir", "$testBuildDir/logs" + systemProperty "hive.home", "$projectDir/testdata/hive" + systemProperty "sqoop.throwOnError", "" + + minHeapSize = "512m" + maxHeapSize = "5120m" + jvmArgs '-Xmx5012m', '-XX:PermSize=256m', '-XX:MaxPermSize=512m', "-da:org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge" + forkEvery 1 + + dependsOn buildFolder + + jacoco{ + excludes = ["**/SqoopVersion*"] + } +} + +tasks.withType(Checkstyle) { + reports { + xml.enabled false + html.enabled true + html.stylesheet resources.text.fromFile('config/checkstyle/checkstyle-noframes.xsl') + } + ignoreFailures = true +} + +tasks.withType(FindBugs) { + reports { + xml.enabled false + html.enabled true + } + ignoreFailures = false +} + +findbugs { + sourceSets = [] +} + +checkstyle { + sourceSets = [] + toolVersion checkstyleVersion +} + +jacocoTestReport { + description "Generates a coverage report of the unit test results under $buildDir/jacocoHtml/test/index.html" + reports { + xml.enabled true + csv.enabled false + html.destination "$buildDir/jacocoHtml/test" + } +} + +task jacocoThirdPartyReport(type: JacocoReport){ + description "Generates a coverage report of the thirdparty test results under $buildDir/jacocoHtml/3rd/index.html" + sourceSets sourceSets.main + executionData thirdPartyTest + reports { + xml.enabled true + csv.enabled false + html.destination "$buildDir/jacocoHtml/3rd" + } +} + +task jacocoCompositeReport(type: JacocoReport){ + description "Generates a composite coverage report of test and thirdparty test results under $buildDir/jacocoHtml/index.html" + sourceSets sourceSets.main + executionData test, thirdPartyTest + reports { + xml.enabled true + csv.enabled false + html.destination "$buildDir/jacocoHtml/" + } +} + +javadoc { + source = sourceSets.main.allJava + include 'com/cloudera/sqoop/lib/**' + include 'org/apache/sqoop/lib/**' + destinationDir = reporting.file("${project.docsDir}/api") +} + +task faultInjectionJar(type: Jar) { + description 'Create the test jar' + appendix = 'test' + from sourceSets.aop.output +} + +def archiveName = jar.baseName + "-" + jar.version +def binArtifactName = archiveName + ".bin__hadoop-" + hadoopVersion +def distDir = "$buildDir/$binArtifactName" + +rat { + inputDir = distDir + reportDir = project.file(buildDir) + excludes = ['**/docs/**', '**/testdata/**', '**/.idea/**', '**/test-dir/**', '**/.settings/**', '**/.gradle/**','**/gradle/**' ,'**/gradle/build/**', '*.iml', '*.iws', '*.ipr', 'derby.log', '**/gradlew**' ] + failOnError = true + xmlOutput = false + htmlOutput = true + plainOutput = false +} +rat.dependsOn('packageDist') + +def scripts = ['export', 'list-databases', 'metastore', 'create-hive-table', 'help', + 'import-mainframe', 'list-tables', 'version', + 'eval', 'import', 'job', 'merge', + 'import-all-tables', 'codegen'] + + +task runSqoopHelp(type: JavaExec, dependsOn: 'classes') { + main = 'org.apache.sqoop.Sqoop' + jvmArgs = ['-Dhadoop.security.log.file=./build/security-audit.log'] + classpath = sourceSets.main.runtimeClasspath + configurations.runtime + args = ['help'] + standardOutput = new ByteArrayOutputStream() + ext.helpOut = { + standardOutput.toString() + } + +} + +task createAllStartScripts(dependsOn: 'runSqoopHelp' ) doLast{ + // Placeholder task with validation + // We call Sqoop help to verify that all tasks that we create scripts for are present in Sqoop help and vice-versa + // We currently depend on the format of the help output to parse the relevant information out + def matcher = (tasks.runSqoopHelp.helpOut() =~/(?m)^\s\s([a-z-]+)\s+\w+/) + def matches = new String[matcher.getCount()] + for (def i=0; i < matcher.getCount(); i++) { + matches[i] = matcher[i][1] + } + if (matches.sort() != scripts.sort()) + throw new GradleException('Error: script list in gradle build script and result of Sqoop help are not the same, but should be') +} + +scripts.each() { scriptName -> + def t = tasks.create(name: scriptName + 'StartScript', type: CreateStartScripts) { + unixStartScriptGenerator.template = resources.text.fromFile('gradle/customUnixStartScript.txt') + windowsStartScriptGenerator.template = resources.text.fromFile('gradle/customWindowsStartScript.txt') + outputDir = file("$buildDir/bin") + applicationName = scriptName + mainClassName = 'com.cloudera.sqoop.Sqoop' + classpath = project.configurations.runtime + doLast { + file("$buildDir/bin/"+ scriptName).renameTo(file("$buildDir/bin/"+"sqoop-"+scriptName)) + file("$buildDir/bin/"+ scriptName +".bat").renameTo(file("$buildDir/bin/"+"sqoop-"+scriptName+".bat")) + } + } + t.dependsOn('compileJava') + createAllStartScripts.dependsOn(t) +} + +task checkVersion doLast { + if (version.contains('SNAPSHOT')) + throw new GradleException('Error: cannot release a snapshot. Set -Pversion') +} + +task release(dependsOn: ['checkVersion', 'tar', 'rat']) { + doLast { + println 'Release complete' + println "Binary tar: $buildDir/$binArtifactName-${version}.tar.gz" + println "Documentation: $buildDir/docs" + println "Release notes: $buildDir/docs/sqoop-$version" + ".releasenotes.html" + println "Release audit report: $buildDir/index.html" + } +} + +task relnotes(type: Exec) { + workingDir 'src/scripts' + if (!version.contains('SNAPSHOT')) { + commandLine "python", "relnotes.py", "$buildDir/docs", "$projectDir", "$oldHash\\..HEAD", "$version", "$oldVersion" + } else { + commandLine "true" //noop + println "Will not run releasenotes for SNAPSHOT version." + } +} + +task realDocs(type: Exec, dependsOn: ['relnotes', 'javadoc']) { + workingDir projectDir + commandLine "make", "-C", "$projectDir/src/docs", "BUILDROOT=$buildDir", "VERSION=$version" +} + +task docs(dependsOn: ['realDocs', 'relnotes', 'javadoc']) { + //placeholder +} +def $mvnRepo = System.getProperty("mvnRepo", "snapshots") + +publishing { + repositories { + maven { + name "cloudera." + $mvnRepo + ".repo" + url "https://repository.cloudera.com/content/repositories/"+$mvnRepo + credentials { + username 'username' + password 'password' + } + } + } +} +publishing { + publications { + sqoop(MavenPublication) { + groupId group + version version + from components.java + artifact sourceJar { + classifier "sources" + } + } + sqoopTest(MavenPublication) { + groupId group + version version + artifactId 'sqoop-test' + artifact testJar + artifact sourceTestJar + pom.withXml { + def dependencies = asNode().appendNode('dependencies') + configurations.testRuntime.getResolvedConfiguration().getFirstLevelModuleDependencies().each { + def dependency = dependencies.appendNode('dependency') + dependency.appendNode('groupId', it.moduleGroup) + dependency.appendNode('artifactId', it.moduleName) + dependency.appendNode('version', it.moduleVersion) + } + } + } + } +} + +task wrapper(type: Wrapper) { + gradleVersion = '3.5.1' +} http://git-wip-us.apache.org/repos/asf/sqoop/blob/b148d540/config/checkstyle/checkstyle-java-header.txt ---------------------------------------------------------------------- diff --git a/config/checkstyle/checkstyle-java-header.txt b/config/checkstyle/checkstyle-java-header.txt new file mode 100644 index 0000000..5d5f1e3 --- /dev/null +++ b/config/checkstyle/checkstyle-java-header.txt @@ -0,0 +1,17 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ http://git-wip-us.apache.org/repos/asf/sqoop/blob/b148d540/config/checkstyle/checkstyle-noframes.xsl ---------------------------------------------------------------------- diff --git a/config/checkstyle/checkstyle-noframes.xsl b/config/checkstyle/checkstyle-noframes.xsl new file mode 100644 index 0000000..60175c6 --- /dev/null +++ b/config/checkstyle/checkstyle-noframes.xsl @@ -0,0 +1,198 @@ +<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"; version="1.0"> +<!-- + + Copyright 2011 The Apache Software Foundation + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + + +<xsl:output method="html" indent="yes"/> +<xsl:decimal-format decimal-separator="." grouping-separator="," /> + +<xsl:key name="files" match="file" use="@name" /> + +<!-- Checkstyle XML Style Sheet by Stephane Bailliez <[email protected]> --> +<!-- Part of the Checkstyle distribution found at http://checkstyle.sourceforge.net --> +<!-- Usage (generates checkstyle_report.html): --> +<!-- <checkstyle failonviolation="false" config="${check.config}"> --> +<!-- <fileset dir="${src.dir}" includes="**/*.java"/> --> +<!-- <formatter type="xml" toFile="${doc.dir}/checkstyle_report.xml"/> --> +<!-- </checkstyle> --> +<!-- <style basedir="${doc.dir}" destdir="${doc.dir}" --> +<!-- includes="checkstyle_report.xml" --> +<!-- style="${doc.dir}/checkstyle-noframes-sorted.xsl"/> --> + +<xsl:template match="checkstyle"> + <html> + <head> + <style type="text/css"> + .bannercell { + border: 0px; + padding: 0px; + } + body { + margin-left: 10; + margin-right: 10; + font:normal 80% arial,helvetica,sanserif; + background-color:#FFFFFF; + color:#000000; + } + .a td { + background: #efefef; + } + .b td { + background: #fff; + } + th, td { + text-align: left; + vertical-align: top; + } + th { + font-weight:bold; + background: #ccc; + color: black; + } + table, th, td { + font-size:100%; + border: none + } + table.log tr td, tr th { + + } + h2 { + font-weight:bold; + font-size:140%; + margin-bottom: 5; + } + h3 { + font-size:100%; + font-weight:bold; + background: #525D76; + color: white; + text-decoration: none; + padding: 5px; + margin-right: 2px; + margin-left: 2px; + margin-bottom: 0; + } + </style> + </head> + <body> + <a name="top"></a> + <!-- jakarta logo --> + <table border="0" cellpadding="0" cellspacing="0" width="100%"> + <tr> + <td class="bannercell" rowspan="2"> + <!--a href="http://jakarta.apache.org/";> + <img src="http://jakarta.apache.org/images/jakarta-logo.gif"; alt="http://jakarta.apache.org"; align="left" border="0"/> + </a--> + </td> + <td class="text-align:right"><h2>CheckStyle Audit</h2></td> + </tr> + <tr> + <td class="text-align:right">Designed for use with <a href='http://checkstyle.sourceforge.net/'>CheckStyle</a> and <a href='http://jakarta.apache.org'>Ant</a>.</td> + </tr> + </table> + <hr size="1"/> + + <!-- Summary part --> + <xsl:apply-templates select="." mode="summary"/> + <hr size="1" width="100%" align="left"/> + + <!-- Package List part --> + <xsl:apply-templates select="." mode="filelist"/> + <hr size="1" width="100%" align="left"/> + + <!-- For each package create its part --> + <xsl:apply-templates select="file[@name and generate-id(.) = generate-id(key('files', @name))]" /> + + <hr size="1" width="100%" align="left"/> + + + </body> + </html> +</xsl:template> + + + + <xsl:template match="checkstyle" mode="filelist"> + <h3>Files</h3> + <table class="log" border="0" cellpadding="5" cellspacing="2" width="100%"> + <tr> + <th>Name</th> + <th>Errors</th> + </tr> + <xsl:for-each select="file[@name and generate-id(.) = generate-id(key('files', @name))]"> + <xsl:sort data-type="number" order="descending" select="count(key('files', @name)/error)"/> + <xsl:variable name="errorCount" select="count(error)"/> + <tr> + <xsl:call-template name="alternated-row"/> + <td><a href="#f-{@name}"><xsl:value-of select="@name"/></a></td> + <td><xsl:value-of select="$errorCount"/></td> + </tr> + </xsl:for-each> + </table> + </xsl:template> + + + <xsl:template match="file"> + <a name="f-{@name}"></a> + <h3>File <xsl:value-of select="@name"/></h3> + + <table class="log" border="0" cellpadding="5" cellspacing="2" width="100%"> + <tr> + <th>Error Description</th> + <th>Line</th> + </tr> + <xsl:for-each select="key('files', @name)/error"> + <xsl:sort data-type="number" order="ascending" select="@line"/> + <tr> + <xsl:call-template name="alternated-row"/> + <td><xsl:value-of select="@message"/></td> + <td><xsl:value-of select="@line"/></td> + </tr> + </xsl:for-each> + </table> + <a href="#top">Back to top</a> + </xsl:template> + + + <xsl:template match="checkstyle" mode="summary"> + <h3>Summary</h3> + <xsl:variable name="fileCount" select="count(file[@name and generate-id(.) = generate-id(key('files', @name))])"/> + <xsl:variable name="errorCount" select="count(file/error)"/> + <table class="log" border="0" cellpadding="5" cellspacing="2" width="100%"> + <tr> + <th>Files</th> + <th>Errors</th> + </tr> + <tr> + <xsl:call-template name="alternated-row"/> + <td><xsl:value-of select="$fileCount"/></td> + <td><xsl:value-of select="$errorCount"/></td> + </tr> + </table> + </xsl:template> + + <xsl:template name="alternated-row"> + <xsl:attribute name="class"> + <xsl:if test="position() mod 2 = 1">a</xsl:if> + <xsl:if test="position() mod 2 = 0">b</xsl:if> + </xsl:attribute> + </xsl:template> +</xsl:stylesheet> http://git-wip-us.apache.org/repos/asf/sqoop/blob/b148d540/config/checkstyle/checkstyle.xml ---------------------------------------------------------------------- diff --git a/config/checkstyle/checkstyle.xml b/config/checkstyle/checkstyle.xml new file mode 100644 index 0000000..b7e5143 --- /dev/null +++ b/config/checkstyle/checkstyle.xml @@ -0,0 +1,226 @@ +<?xml version="1.0"?> +<!-- + Copyright 2011 The Apache Software Foundation + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +<!DOCTYPE module PUBLIC + "-//Puppy Crawl//DTD Check Configuration 1.2//EN" + "http://www.puppycrawl.com/dtds/configuration_1_2.dtd";> + +<!-- + + Checkstyle configuration for Sqoop that is based on the sun_checks.xml file + that is bundled with Checkstyle and includes checks for: + + - the Java Language Specification at + http://java.sun.com/docs/books/jls/second_edition/html/index.html + + - the Sun Code Conventions at http://java.sun.com/docs/codeconv/ + + - the Javadoc guidelines at + http://java.sun.com/j2se/javadoc/writingdoccomments/index.html + + - the JDK Api documentation http://java.sun.com/j2se/docs/api/index.html + + - some best practices + + Checkstyle is very configurable. Be sure to read the documentation at + http://checkstyle.sf.net (or in your downloaded distribution). + + Most Checks are configurable, be sure to consult the documentation. + + To completely disable a check, just comment it out or delete it from the file. + + Finally, it is worth reading the documentation. + +--> + +<module name="Checker"> + + <!-- Checks that a package.html file exists for each package. --> + <!-- See http://checkstyle.sf.net/config_javadoc.html#PackageHtml --> + <!-- module name="PackageHtml"/ --> + + <!-- Checks whether files end with a new line. --> + <!-- See http://checkstyle.sf.net/config_misc.html#NewlineAtEndOfFile --> + <module name="NewlineAtEndOfFile"/> + + <!-- Checks for Headers --> + <!-- See http://checkstyle.sf.net/config_header.html --> + <module name="Header"> + <property name="headerFile" + value="config/checkstyle/checkstyle-java-header.txt" /> + </module> + + <module name="FileLength"/> + <module name="FileTabCharacter"/> + + <module name="TreeWalker"> + <!-- Checks for Javadoc comments. --> + <!-- See http://checkstyle.sf.net/config_javadoc.html --> + <module name="JavadocType"> + <property name="scope" value="public"/> + <property name="allowMissingParamTags" value="true"/> + </module> + <module name="JavadocStyle"/> + + <module name="SuperClone"/> + <module name="SuperFinalize"/> + + <!-- Checks for Naming Conventions. --> + <!-- See http://checkstyle.sf.net/config_naming.html --> + <module name="ConstantName"/> + <module name="ClassTypeParameterName"> + <property name="format" value="^[A-Z]+$"/> + </module> + <module name="LocalFinalVariableName"> + <property name="format" value="^[A-Z][_A-Z0-9]*$" /> + </module> + <module name="LocalVariableName"/> + <module name="MemberName"/> + <module name="MethodName"/> + <module name="MethodTypeParameterName"> + <property name="format" value="^[A-Z]+$"/> + </module> + <module name="PackageName"/> + <module name="ParameterName"/> + <module name="StaticVariableName"/> + <module name="TypeName"/> + + <!-- Checks for imports --> + <!-- See http://checkstyle.sf.net/config_import.html --> + <module name="IllegalImport"/> <!-- defaults to sun.* packages --> + <module name="RedundantImport"/> + <module name="UnusedImports"/> + + + <!-- Checks for Size Violations. --> + <!-- See http://checkstyle.sf.net/config_sizes.html --> + <module name="LineLength"/> + <module name="MethodLength"/> + <module name="ParameterNumber"/> + <module name="OuterTypeNumber"/> + + <!-- Checks for whitespace --> + <!-- See http://checkstyle.sf.net/config_whitespace.html --> + <module name="GenericWhitespace"/> + <module name="EmptyForIteratorPad"/> + <module name="MethodParamPad"/> + <module name="WhitespaceAround"> + <property name="tokens" value="LITERAL_IF" /> + </module> + <module name="NoWhitespaceAfter"> + <property name="tokens" + value="BNOT, DEC, DOT, INC, LNOT, UNARY_MINUS, UNARY_PLUS" /> + </module> + <module name="NoWhitespaceBefore"/> + <module name="OperatorWrap"/> + <module name="ParenPad"/> + <module name="TypecastParenPad"/> + <module name="WhitespaceAfter"> + <property name="tokens" value="COMMA, SEMI"/> + </module> + + <module name="Regexp"> + <property name="format" value="[ \t]+$"/> + <property name="illegalPattern" value="true"/> + <property name="message" value="Trailing whitespace"/> + </module> + + <!-- Modifier Checks --> + <!-- See http://checkstyle.sf.net/config_modifiers.html --> + <module name="ModifierOrder"/> + <module name="RedundantModifier"/> + + + <!-- Checks for blocks. You know, those {}'s --> + <!-- See http://checkstyle.sf.net/config_blocks.html --> + <module name="AvoidNestedBlocks"/> + <module name="EmptyBlock"> + <!-- catch blocks need a statement or a comment. --> + <property name="option" value="text" /> + <property name="tokens" value="LITERAL_CATCH" /> + </module> + <module name="EmptyBlock"> + <!-- all other blocks need a real statement. --> + <property name="option" value="stmt" /> + <property name="tokens" value="LITERAL_DO, LITERAL_ELSE, LITERAL_FINALLY, + LITERAL_IF, LITERAL_FOR, LITERAL_TRY, LITERAL_WHILE, INSTANCE_INIT, + STATIC_INIT" /> + </module> + <module name="LeftCurly"/> + <module name="NeedBraces"/> + <module name="RightCurly"/> + + + <!-- Checks for common coding problems --> + <!-- See http://checkstyle.sf.net/config_coding.html --> + <!-- module name="AvoidInlineConditionals"/--> + <module name="DoubleCheckedLocking"/> + <module name="EmptyStatement"/> + <module name="EqualsHashCode"/> + <module name="StringLiteralEquality" /> + <module name="HiddenField"> + <property name="ignoreConstructorParameter" value="true"/> + </module> + <module name="IllegalInstantiation"/> + <module name="InnerAssignment"/> + <module name="MissingSwitchDefault"/> + <module name="RedundantThrows"/> + <module name="SimplifyBooleanExpression"/> + <module name="SimplifyBooleanReturn"/> + <module name="DefaultComesLast" /> + + <!-- Checks for class design --> + <!-- See http://checkstyle.sf.net/config_design.html --> + <module name="FinalClass"/> + <module name="HideUtilityClassConstructor"/> + <module name="InterfaceIsType"/> + <module name="VisibilityModifier"> + <property name="protectedAllowed" value="true" /> + </module> + <module name="MissingOverride" /> + + + <!-- Miscellaneous other checks. --> + <!-- See http://checkstyle.sf.net/config_misc.html --> + <module name="ArrayTypeStyle"/> + <module name="ArrayTrailingComma"/> + <!-- + This generates too many false-positives on wrapped 'throws' clauses + to be really useful. Disabled for now. + + Sqoop style is: + * Spaces, not tabs. + * Indent by two spaces. + * Indent by four spaces when wrapping a line. + <module name="Indentation"> + <property name="basicOffset" value="2" /> + <property name="caseIndent" value="0" /> + </module> + --> + <!-- module name="TodoComment"/ --> + <module name="UpperEll"/> + + <module name="FileContentsHolder" /> + </module> + + <!-- allow warnings to be suppressed --> + <module name="SuppressionCommentFilter" /> +</module> http://git-wip-us.apache.org/repos/asf/sqoop/blob/b148d540/gradle.properties ---------------------------------------------------------------------- diff --git a/gradle.properties b/gradle.properties new file mode 100644 index 0000000..a571e6c --- /dev/null +++ b/gradle.properties @@ -0,0 +1,57 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# This properties file lists the versions of the various artifacts we use in gradle. + +javaSourceCompatibilityVersion=1.7 + +avroVersion=1.8.1 +kitedataVersion=1.1.0 +hadoopVersion=2.8.0 +aspectjVersion=1.7.4 +zookeeperVersion=3.4.6 +hbaseVersion=1.2.4 +hcatalogVersion=1.2.1 + +accumuloVersion=1.6.2 + +h2Version=1.3.170 +hsqldbVersion=1.8.0.10 + +commonscliVersion=1.2 +commonscollectionsVersion=3.2.2 +commonsioVersion=1.4 +commonslangVersion=2.4 +commonslang3Version=3.4 +commonsloggingVersion=1.0.4 +commonsnetVersion=3.1 + +log4jVersion=1.2.16 +junitVersion=4.12 +mockitoallVersion=1.9.5 +assertjVersion=2.8.0 + +checkstyleVersion=5.5 + +version=1.5.0-SNAPSHOT + +postgresqlVersion=9.2-1003-jdbc4 + +oldHash=b0f391e75154be86f95378ab141f6dd1b3b59475 +oldVersion=1.4.7 + +org.gradle.daemon=true http://git-wip-us.apache.org/repos/asf/sqoop/blob/b148d540/gradle/customUnixStartScript.txt ---------------------------------------------------------------------- diff --git a/gradle/customUnixStartScript.txt b/gradle/customUnixStartScript.txt new file mode 100644 index 0000000..702ff69 --- /dev/null +++ b/gradle/customUnixStartScript.txt @@ -0,0 +1,100 @@ +#!/bin/bash +# +# Copyright 2011 The Apache Software Foundation +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +follow_one() { + # Resolve symlinks and relative path components along a path. This requires + # its argument to be an absolute path. This does not recursively re-resolve + # symlinks; if that is required, use the 'follow' method. + + target=\$1 + OIFS=\$IFS + IFS='/' + + # Taking each dir component along the way, build up a new target directory, + # resolving '.', '..', and symlinks. + newtarget='' + for part in \${target}; do + if [ -z "\${part}" ]; then + continue # Empty dir part. 'foo//bar' + elif [ "." == "\${part}" ]; then + continue # Nothing special to do for '.' + elif [ ".." == "\${part}" ]; then + IFS=\$OIFS + newtarget=`dirname \${newtarget}` # pop a component. + elif [ -h "\${newtarget}/\${part}" ]; then + IFS=\$OIFS + link=`readlink \${newtarget}/\${part}` + # links can be relative or absolute. Relative ones get appended to + # newtarget; absolute ones replace it. + if [ "\${link:0:1}" != "/" ]; then + newtarget="\${newtarget}/\${link}" # relative + else + newtarget="\${link}" # absolute + fi + else # Regular file component. + newtarget="\${newtarget}/\${part}" + fi + IFS='/' + done + + IFS=\$OIFS + echo \$newtarget +} + +follow() { + # Portable 'readlink -f' function to follow a file's links to the final + # target. Calls follow_one recursively til we're finished tracing symlinks. + + target=\$1 + depth=\$2 + + if [ -z "\$depth" ]; then + depth=0 + elif [ "\$depth" == "1000" ]; then + # Don't recurse indefinitely; we've probably hit a symlink cycle. + # Just bail out here. + echo \$target + return 1 + fi + + # Canonicalize the target to be an absolute path. + targetdir=`dirname \${target}` + targetdir=`cd \${targetdir} && pwd` + target=\${targetdir}/`basename \${target}` + + # Use follow_one to resolve links. Test that we get the same result twice, + # to terminate iteration. + first=`follow_one \${target}` + second=`follow_one \${first}` + if [ "\${first}" == "\${second}" ]; then + # We're done. + echo "\${second}" + else + # Need to continue resolving links. + echo `follow \${second} \$(( \$depth + 1 ))` + fi +} + +prgm=`follow \$0` +bin=`dirname \${prgm}` +bin=`cd \${bin} && pwd` + +exec \${bin}/sqoop ${applicationName} "\$@" http://git-wip-us.apache.org/repos/asf/sqoop/blob/b148d540/gradle/customWindowsStartScript.txt ---------------------------------------------------------------------- diff --git a/gradle/customWindowsStartScript.txt b/gradle/customWindowsStartScript.txt new file mode 100644 index 0000000..57a23f7 --- /dev/null +++ b/gradle/customWindowsStartScript.txt @@ -0,0 +1,26 @@ +@echo off +:: Licensed to the Apache Software Foundation (ASF) under one or more +:: contributor license agreements. See the NOTICE file distributed with +:: this work for additional information regarding copyright ownership. +:: The ASF licenses this file to You under the Apache License, Version 2.0 +:: (the "License"); you may not use this file except in compliance with +:: the License. You may obtain a copy of the License at +:: +:: +:: http://www.apache.org/licenses/LICENSE-2.0 +:: +:: Unless required by applicable law or agreed to in writing, software +:: distributed under the License is distributed on an "AS IS" BASIS, +:: WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +:: See the License for the specific language governing permissions and +:: limitations under the License. +setlocal + +set prgm=%~f0 +set bin=%~dp0 + +if "%bin:~-1%" == "\" ( + set bin=%bin:~0,-1% +) + +call "%bin%\\sqoop.cmd" SUBCOMMANDMARKER %* http://git-wip-us.apache.org/repos/asf/sqoop/blob/b148d540/gradle/sqoop-package.gradle ---------------------------------------------------------------------- diff --git a/gradle/sqoop-package.gradle b/gradle/sqoop-package.gradle new file mode 100644 index 0000000..076fed8 --- /dev/null +++ b/gradle/sqoop-package.gradle @@ -0,0 +1,209 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +configurations { + redist +} + +dependencies { + redist group: 'commons-io', name: 'commons-io', version: commonsioVersion + redist(group: 'org.apache.avro', name: 'avro', version: avroVersion) { + exclude group: 'org.slf4j', module: 'slf4j-api' + exclude group: 'org.mortbay.jetty', module: 'jetty' + exclude group: 'org.mortbay.jetty', module: 'jetty-util' + exclude group: 'org.mortbay.jetty', module: 'servlet-api' + exclude module: 'netty' + exclude group: 'org.apache.velocity', module: 'velocity' + } + redist(group: 'org.apache.avro', name: 'avro-mapred', version: avroVersion, classifier: 'hadoop2') { + exclude group: 'org.slf4j', module: 'slf4j-api' + exclude group: 'org.mortbay.jetty', module: 'jetty' + exclude group: 'org.mortbay.jetty', module: 'jetty-util' + exclude group: 'org.mortbay.jetty', module: 'servlet-api' + exclude module: 'netty' + exclude group: 'org.apache.velocity', module: 'velocity' + } + redist group: 'hsqldb', name: 'hsqldb', version: hsqldbVersion + redist group: 'org.apache.commons', name: 'commons-lang3', version: commonslang3Version + redist group: 'org.kitesdk', name: 'kite-data-mapreduce', version: kitedataVersion + redist group: 'org.kitesdk', name: 'kite-data-hive', version: kitedataVersion +} + +//Jar tasks +jar { + baseName = rootProject.name.toLowerCase() + destinationDir = buildDir +} + +task testJar(type: Jar) { + baseName = rootProject.name.toLowerCase() + destinationDir = buildDir + description 'Create the test jar' + appendix = 'test' + from sourceSets.test.output +} + +task jarAll(dependsOn: ['testJar', 'jar']) + +def buildSrcJarDir = "$buildDir/srcjars" + +task sourceJar(type: Jar) { + description 'Create source jars' + classifier = 'sources' + from sourceSets.main.allSource + into buildSrcJarDir +} + +task sourceTestJar(type: Jar) { + description 'Create source jars' + classifier = 'sources' + appendix = 'test' + from sourceSets.test.allSource + into buildSrcJarDir +} + +task sourceJars(dependsOn: ['jarAll', 'sourceJar', 'sourceTestJar']) + +//tar tasks +def srcArchiveName =jar.baseName.toLowerCase() + "-" + jar.version +def srcDirstDir = "$buildDir/$srcArchiveName" +def binArtifactName = srcArchiveName + ".bin__hadoop-" + hadoopVersion +def distDir = "$buildDir/$binArtifactName" + +task copyToSrcDistDir(type: Copy) { + from(projectDir) { + include "**/*" + exclude "build/**", ".git/**", "tags", ".project", ".classpath", "conf/managers.d/**", "conf/tools.d/**", ".gradle/**" + } + into srcDirstDir + includeEmptyDirs = false +} + +task filepermissionForSrcDist(type: Exec) { + commandLine 'chmod', 'ugo+x', "$srcDirstDir/bin" + commandLine 'chmod', 'ugo+x', "$srcDirstDir/testdata/hive/bin" + fileTree("$srcDirstDir").matching { include "**/*.sh" }.each { aFile -> + exec { + commandLine 'chmod' + args 'ugo+x', aFile.absolutePath + } + } +} + +task srcPackageDist(dependsOn: ['copyToSrcDistDir', 'filepermissionForSrcDist']) + +task srctar(type: Tar, dependsOn: srcPackageDist) { + from("$srcDirstDir") { + include "**" + exclude "**/*.sh", "testdata/hive/bin/*", "bin/*" + fileMode 0644 + } + from("$srcDirstDir") { + include "**/*.sh", "testdata/hive/bin/*", "bin/*", "testdata/hcatalog/conf/*" + fileMode 0755 + } + baseName = srcArchiveName + destinationDir = file("$buildDir") + extension = 'tar.gz' + compression = Compression.GZIP +} + +task copyToDistDir(type: Copy, dependsOn: ['jarAll', 'docs', 'createAllStartScripts']) { + from jar + from testJar + from(projectDir) { + include "**/*" + exclude "build/**", "lib/**", ".git/**", "tags", ".project", ".classpath", "conf/managers.d/**", "conf/tools.d/**", ".gradle/**" + } + into distDir + includeEmptyDirs = false +} + +task copyToDistLibDir(type: Copy, dependsOn: [configurations.redist, copyToDistDir]) { + from configurations.redist + from("$projectDir/lib") { + include "**/*" + exclude "ivy*" + } + into "$distDir/lib" + includeEmptyDirs = false +} + +task copyToDistDirDocs(type: Copy, dependsOn: ['docs', 'copyToDistLibDir']) { + from("$buildDir/docs") { + include "**/*.html", "**/*.css", "images/**" + } + into "$distDir/docs" + includeEmptyDirs = false +} + +task copyToDistDirDocsMan(type: Copy, dependsOn: ['docs', 'copyToDistDirDocs']) { + from("$buildDir/docs") { + include "**/*.gz" + } + into "$distDir/docs/man" + includeEmptyDirs = false +} + +task copyToDistBinDir(type: Copy, dependsOn: ['copyToDistDirDocsMan']) { + from("$buildDir/bin") { + include "*" + } + into "$distDir/bin" + includeEmptyDirs = false +} + +task filepermissionForDist(type: Exec, dependsOn: ['copyToDistBinDir']) { + commandLine 'chmod', 'ugo+x', "$distDir/bin" + commandLine 'chmod', 'ugo+x', "$distDir/testdata/hive/bin" + fileTree("$distDir").matching { include "**/*.sh" }.each { aFile -> + exec { + commandLine 'chmod' + args 'ugo+x', aFile.absolutePath + } + } +} + +def contentSpec = copySpec { + from("$distDir/conf") { + include "sqoop-site-template.xml" + } + rename('sqoop-site-template.xml', 'sqoop-site.xml') +} + +task copyAndOverwriteSqoopSiteXML(type: Copy, dependsOn: filepermissionForDist) { + into "$distDir/conf" + with contentSpec +} + +task packageDist(dependsOn: ['copyToDistDir', 'copyToDistLibDir', 'copyToDistDirDocs', 'copyToDistDirDocsMan', 'copyToDistBinDir', 'filepermissionForDist', 'copyAndOverwriteSqoopSiteXML']) + +task tar(type: Tar, dependsOn: packageDist) { + from("$buildDir") { + include "$binArtifactName/**" + exclude "$binArtifactName/**/*.sh", "$binArtifactName/testdata/hive/bin/*", "$binArtifactName/bin/*" + fileMode 0644 + } + from("$buildDir") { + include "$binArtifactName/**/*.sh", "$binArtifactName/testdata/hive/bin/*", "$binArtifactName/bin/*", "$binArtifactName/testdata/hcatalog/conf/*" + fileMode 0755 + } + baseName = binArtifactName + destinationDir = file("$buildDir") + extension = 'tar.gz' + compression = Compression.GZIP +} http://git-wip-us.apache.org/repos/asf/sqoop/blob/b148d540/gradle/sqoop-version-gen.gradle ---------------------------------------------------------------------- diff --git a/gradle/sqoop-version-gen.gradle b/gradle/sqoop-version-gen.gradle new file mode 100644 index 0000000..2afcea4 --- /dev/null +++ b/gradle/sqoop-version-gen.gradle @@ -0,0 +1,76 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import java.text.SimpleDateFormat + +def getGitHash = { -> + def stdout = new ByteArrayOutputStream() + exec { + commandLine 'git', 'log', '-1', '--pretty=format:%H' + standardOutput = stdout + } + return stdout.toString().trim() +} +def signature = getGitHash + +def user = System.getProperty("user.name") + +task SqoopVersionFileGen() { + doLast { + def file = new File("$projectDir/src/java/org/apache/sqoop/SqoopVersion.java") + file.createNewFile() + file.text = """ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +// generated by gradle task +package org.apache.sqoop; + +public class SqoopVersion { + public SqoopVersion() { + } + + public static final String VERSION="$version"; + public static final String GIT_HASH="$signature"; + public static final String COMPILE_USER="$user"; + public static final String COMPILE_DATE="${new SimpleDateFormat("dd-MM-yyyy HH:mm:ss").format(new Date())}"; + + @Override + public String toString() { + return "Sqoop " + VERSION + "\\n" + + "git commit id " + GIT_HASH + "\\n" + + "Compiled by " + COMPILE_USER + + " on " + COMPILE_DATE + "\\n"; + } + } +""" + } +} http://git-wip-us.apache.org/repos/asf/sqoop/blob/b148d540/gradle/wrapper/gradle-wrapper.jar ---------------------------------------------------------------------- diff --git a/gradle/wrapper/gradle-wrapper.jar b/gradle/wrapper/gradle-wrapper.jar new file mode 100644 index 0000000..99340b4 Binary files /dev/null and b/gradle/wrapper/gradle-wrapper.jar differ http://git-wip-us.apache.org/repos/asf/sqoop/blob/b148d540/gradle/wrapper/gradle-wrapper.properties ---------------------------------------------------------------------- diff --git a/gradle/wrapper/gradle-wrapper.properties b/gradle/wrapper/gradle-wrapper.properties new file mode 100644 index 0000000..90a06ce --- /dev/null +++ b/gradle/wrapper/gradle-wrapper.properties @@ -0,0 +1,5 @@ +distributionBase=GRADLE_USER_HOME +distributionPath=wrapper/dists +zipStoreBase=GRADLE_USER_HOME +zipStorePath=wrapper/dists +distributionUrl=https\://services.gradle.org/distributions/gradle-3.5.1-bin.zip http://git-wip-us.apache.org/repos/asf/sqoop/blob/b148d540/gradlew ---------------------------------------------------------------------- diff --git a/gradlew b/gradlew new file mode 100755 index 0000000..cccdd3d --- /dev/null +++ b/gradlew @@ -0,0 +1,172 @@ +#!/usr/bin/env sh + +############################################################################## +## +## Gradle start up script for UN*X +## +############################################################################## + +# Attempt to set APP_HOME +# Resolve links: $0 may be a link +PRG="$0" +# Need this for relative symlinks. +while [ -h "$PRG" ] ; do + ls=`ls -ld "$PRG"` + link=`expr "$ls" : '.*-> \(.*\)$'` + if expr "$link" : '/.*' > /dev/null; then + PRG="$link" + else + PRG=`dirname "$PRG"`"/$link" + fi +done +SAVED="`pwd`" +cd "`dirname \"$PRG\"`/" >/dev/null +APP_HOME="`pwd -P`" +cd "$SAVED" >/dev/null + +APP_NAME="Gradle" +APP_BASE_NAME=`basename "$0"` + +# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. +DEFAULT_JVM_OPTS="" + +# Use the maximum available, or set MAX_FD != -1 to use that value. +MAX_FD="maximum" + +warn () { + echo "$*" +} + +die () { + echo + echo "$*" + echo + exit 1 +} + +# OS specific support (must be 'true' or 'false'). +cygwin=false +msys=false +darwin=false +nonstop=false +case "`uname`" in + CYGWIN* ) + cygwin=true + ;; + Darwin* ) + darwin=true + ;; + MINGW* ) + msys=true + ;; + NONSTOP* ) + nonstop=true + ;; +esac + +CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar + +# Determine the Java command to use to start the JVM. +if [ -n "$JAVA_HOME" ] ; then + if [ -x "$JAVA_HOME/jre/sh/java" ] ; then + # IBM's JDK on AIX uses strange locations for the executables + JAVACMD="$JAVA_HOME/jre/sh/java" + else + JAVACMD="$JAVA_HOME/bin/java" + fi + if [ ! -x "$JAVACMD" ] ; then + die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME + +Please set the JAVA_HOME variable in your environment to match the +location of your Java installation." + fi +else + JAVACMD="java" + which java >/dev/null 2>&1 || die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. + +Please set the JAVA_HOME variable in your environment to match the +location of your Java installation." +fi + +# Increase the maximum file descriptors if we can. +if [ "$cygwin" = "false" -a "$darwin" = "false" -a "$nonstop" = "false" ] ; then + MAX_FD_LIMIT=`ulimit -H -n` + if [ $? -eq 0 ] ; then + if [ "$MAX_FD" = "maximum" -o "$MAX_FD" = "max" ] ; then + MAX_FD="$MAX_FD_LIMIT" + fi + ulimit -n $MAX_FD + if [ $? -ne 0 ] ; then + warn "Could not set maximum file descriptor limit: $MAX_FD" + fi + else + warn "Could not query maximum file descriptor limit: $MAX_FD_LIMIT" + fi +fi + +# For Darwin, add options to specify how the application appears in the dock +if $darwin; then + GRADLE_OPTS="$GRADLE_OPTS \"-Xdock:name=$APP_NAME\" \"-Xdock:icon=$APP_HOME/media/gradle.icns\"" +fi + +# For Cygwin, switch paths to Windows format before running java +if $cygwin ; then + APP_HOME=`cygpath --path --mixed "$APP_HOME"` + CLASSPATH=`cygpath --path --mixed "$CLASSPATH"` + JAVACMD=`cygpath --unix "$JAVACMD"` + + # We build the pattern for arguments to be converted via cygpath + ROOTDIRSRAW=`find -L / -maxdepth 1 -mindepth 1 -type d 2>/dev/null` + SEP="" + for dir in $ROOTDIRSRAW ; do + ROOTDIRS="$ROOTDIRS$SEP$dir" + SEP="|" + done + OURCYGPATTERN="(^($ROOTDIRS))" + # Add a user-defined pattern to the cygpath arguments + if [ "$GRADLE_CYGPATTERN" != "" ] ; then + OURCYGPATTERN="$OURCYGPATTERN|($GRADLE_CYGPATTERN)" + fi + # Now convert the arguments - kludge to limit ourselves to /bin/sh + i=0 + for arg in "$@" ; do + CHECK=`echo "$arg"|egrep -c "$OURCYGPATTERN" -` + CHECK2=`echo "$arg"|egrep -c "^-"` ### Determine if an option + + if [ $CHECK -ne 0 ] && [ $CHECK2 -eq 0 ] ; then ### Added a condition + eval `echo args$i`=`cygpath --path --ignore --mixed "$arg"` + else + eval `echo args$i`="\"$arg\"" + fi + i=$((i+1)) + done + case $i in + (0) set -- ;; + (1) set -- "$args0" ;; + (2) set -- "$args0" "$args1" ;; + (3) set -- "$args0" "$args1" "$args2" ;; + (4) set -- "$args0" "$args1" "$args2" "$args3" ;; + (5) set -- "$args0" "$args1" "$args2" "$args3" "$args4" ;; + (6) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" ;; + (7) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" ;; + (8) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" ;; + (9) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" "$args8" ;; + esac +fi + +# Escape application args +save () { + for i do printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/" ; done + echo " " +} +APP_ARGS=$(save "$@") + +# Collect all arguments for the java command, following the shell quoting and substitution rules +eval set -- $DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS "\"-Dorg.gradle.appname=$APP_BASE_NAME\"" -classpath "\"$CLASSPATH\"" org.gradle.wrapper.GradleWrapperMain "$APP_ARGS" + +# by default we should be in the correct project dir, but when run from Finder on Mac, the cwd is wrong +if [ "$(uname)" = "Darwin" ] && [ "$HOME" = "$PWD" ]; then + cd "$(dirname "$0")" +fi + +exec "$JAVACMD" "$@" http://git-wip-us.apache.org/repos/asf/sqoop/blob/b148d540/gradlew.bat ---------------------------------------------------------------------- diff --git a/gradlew.bat b/gradlew.bat new file mode 100644 index 0000000..f955316 --- /dev/null +++ b/gradlew.bat @@ -0,0 +1,84 @@ +@if "%DEBUG%" == "" @echo off +@rem ########################################################################## +@rem +@rem Gradle startup script for Windows +@rem +@rem ########################################################################## + +@rem Set local scope for the variables with windows NT shell +if "%OS%"=="Windows_NT" setlocal + +set DIRNAME=%~dp0 +if "%DIRNAME%" == "" set DIRNAME=. +set APP_BASE_NAME=%~n0 +set APP_HOME=%DIRNAME% + +@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. +set DEFAULT_JVM_OPTS= + +@rem Find java.exe +if defined JAVA_HOME goto findJavaFromJavaHome + +set JAVA_EXE=java.exe +%JAVA_EXE% -version >NUL 2>&1 +if "%ERRORLEVEL%" == "0" goto init + +echo. +echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. +echo. +echo Please set the JAVA_HOME variable in your environment to match the +echo location of your Java installation. + +goto fail + +:findJavaFromJavaHome +set JAVA_HOME=%JAVA_HOME:"=% +set JAVA_EXE=%JAVA_HOME%/bin/java.exe + +if exist "%JAVA_EXE%" goto init + +echo. +echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% +echo. +echo Please set the JAVA_HOME variable in your environment to match the +echo location of your Java installation. + +goto fail + +:init +@rem Get command-line arguments, handling Windows variants + +if not "%OS%" == "Windows_NT" goto win9xME_args + +:win9xME_args +@rem Slurp the command line arguments. +set CMD_LINE_ARGS= +set _SKIP=2 + +:win9xME_args_slurp +if "x%~1" == "x" goto execute + +set CMD_LINE_ARGS=%* + +:execute +@rem Setup the command line + +set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar + +@rem Execute Gradle +"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %CMD_LINE_ARGS% + +:end +@rem End local scope for the variables with windows NT shell +if "%ERRORLEVEL%"=="0" goto mainEnd + +:fail +rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of +rem the _cmd.exe /c_ return code! +if not "" == "%GRADLE_EXIT_CONSOLE%" exit 1 +exit /b 1 + +:mainEnd +if "%OS%"=="Windows_NT" endlocal + +:omega http://git-wip-us.apache.org/repos/asf/sqoop/blob/b148d540/settings.gradle ---------------------------------------------------------------------- diff --git a/settings.gradle b/settings.gradle new file mode 100644 index 0000000..7d64af5 --- /dev/null +++ b/settings.gradle @@ -0,0 +1,18 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +rootProject.name ='Sqoop' http://git-wip-us.apache.org/repos/asf/sqoop/blob/b148d540/src/scripts/rat-violations.sh ---------------------------------------------------------------------- diff --git a/src/scripts/rat-violations.sh b/src/scripts/rat-violations.sh index 1cfbc15..9911321 100755 --- a/src/scripts/rat-violations.sh +++ b/src/scripts/rat-violations.sh @@ -47,7 +47,11 @@ sed -i -e "s|${basedir}||" ${auditlog} # Anything in /testdata is a file that is supposed to represent exact output. grep '!?????' ${auditlog} \ | grep -v ' \/docs\/' \ + | grep -v ' \/target\/' \ | grep -v ' \/testdata\/' \ + | grep -v ' \/gradle\/' \ + | grep -v ' \/gradlew' \ + | grep -v ' \/gradlew.bat' \ > ${filtered} # Check: did we find any violations after filtering? http://git-wip-us.apache.org/repos/asf/sqoop/blob/b148d540/testdata/hcatalog/conf/hive-site.xml ---------------------------------------------------------------------- diff --git a/testdata/hcatalog/conf/hive-site.xml b/testdata/hcatalog/conf/hive-site.xml index edac7aa..4922abd 100644 --- a/testdata/hcatalog/conf/hive-site.xml +++ b/testdata/hcatalog/conf/hive-site.xml @@ -34,7 +34,7 @@ </property> <property> <name>javax.jdo.option.ConnectionURL</name> - <value>jdbc:derby:;databaseName=${test.build.data}/sqoop/metastore_db;create=true</value> + <value>jdbc:derby:memory:metastore_db;create=true</value> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name>
![http://jakarta.apache.org/images/jakarta-logo.gif"](http://jakarta.apache.org/images/jakarta-logo.gif"){kind=link}