This is an automated email from the ASF dual-hosted git repository.
abukor pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new e456bc775 [build] KUDU-3551 Upgrade gradle to 7.6.4
e456bc775 is described below
commit e456bc775805d4555d99ce12d04e3ca0b8950760
Author: Zoltan Chovan <[email protected]>
AuthorDate: Mon Feb 5 11:06:01 2024 +0100
[build] KUDU-3551 Upgrade gradle to 7.6.4
The current version of gradle is 3+ years old, and the highest Java
version supported by it is Java 13, which is ~5years old. Upgrading
Kudu's gradle version will make it possible to move from Java8 to a more
recently released stable Java version, that would futureproof the Kudu
Java client. Gradle 7.6 makes it possible to use up to Java19.
Additionally Gradle has fixed multiple security vulnerabilities and
issues since 6.8.3 [1]
* Gradle version has been upgraded to 7.6.4 from 6.8.3, both in the
dependencies declaration and the actual wrapper code itself
(/java/gradlew)
* Removed jcenter from list of repositories [2]
* Replaced compile and testCompile configurations with implementation
and testImplementation respectively [3]
* Replaced extension methods with archiveExtension [4]
* Removed scopes.gradle and the propdeps plugin as optional and
provided scopes are no longer supported/working in Gradle7, these
have been replaced by either compileOnly
* Upgraded scalafmt version [5]
* Pinned the scalafmt config to version 1.5.1 to avoid the new
formatting errors that popped up with the new scalafmt version
* Upgraded Spark3 version to 3.2.4 to solve Guava related compilation
errors (some classes have been deprecated and removed, which led to
compilation failure)
* Updated the shadow plugin version from 6.1.0 to 7.1.2
* Added LogCaptor to resolve issues with CapturingLogAppender, which
stopped working in the Scala tests, fixing was taking too much time
so this was implemented as a workaround, it only affected 4 tests
in TestKuduBackup.scala, if the CLA errors are fixed later, the
dependency can be removed and the test changes reverted.
* As the legacy maven plugin was removed, the signing and publishing
has been rewritten using the new maven-publish plugin. I tested the
signing and publishing with a locally hosted Maven repository,
however this needs further testing and confirmation from a commiter
to ensure it works as expected. The new plugin uses a different
task to publish the artifacts (publish instead of uploadArchives)
an alias was created to keep backwards compatibility with existing
docs/scripts.
* The permissions for assign-location.py script were updated, as
previously the permissions of it's symlink were copied, but that
changed in the new gradle version and the original (non executable)
permissions led to test failures.
* Two tasks were updated in the shadow.gradle file as well, this
needs another round of verification after the jars are built
correctly
* Dependency scopes were only modified when necessary (compilation
failure)
Additional manual testing:
* Attila Bukor has tested and confirmed that the java artifact signing
and publishing works.
[1]
https://docs.gradle.org/7.6.3/release-notes.html?_gl=1*n7ayww*_ga*MjQ4NTk5ODI5LjE2ODc0MDc0OTA.*_ga_7W7NC6YNPT*MTcxNzc2MzIzOS44NC4xLjE3MTc3NjQyMTUuMjYuMC4w
[2]
https://docs.gradle.org/6.9.2/userguide/upgrading_version_6.html?_gl=1*1xt1kk*_ga*MjQ4NTk5ODI5LjE2ODc0MDc0OTA.*_ga_7W7NC6YNPT*MTcwNzEyMzc1Ni4zLjEuMTcwNzEyODU2MS41NC4wLjA.#jcenter_deprecation
[3]
https://docs.gradle.org/6.9.2/userguide/upgrading_version_5.html?_gl=1*1ldn5ls*_ga*MjQ4NTk5ODI5LjE2ODc0MDc0OTA.*_ga_7W7NC6YNPT*MTcwNzEyMzc1Ni4zLjEuMTcwNzEyODY1My40NS4wLjA.#dependencies_should_no_longer_be_declared_using_the_compile_and_runtime_configurations
[4]
https://docs.gradle.org/6.9.2/dsl/org.gradle.api.tasks.bundling.AbstractArchiveTask.html?_gl=1*1ldn5ls*_ga*MjQ4NTk5ODI5LjE2ODc0MDc0OTA.*_ga_7W7NC6YNPT*MTcwNzEyMzc1Ni4zLjEuMTcwNzEyODY1My40NS4wLjA.#org.gradle.api.tasks.bundling.AbstractArchiveTask:extension
[5] https://github.com/alenkacz/gradle-scalafmt/issues/39
Change-Id: I915dab011aba633d55a79c72ea6f459d703d7f47
Reviewed-on: http://gerrit.cloudera.org:8080/21030
Tested-by: Kudu Jenkins
Reviewed-by: Attila Bukor <[email protected]>
---
RELEASING.adoc | 8 +-
java/.scalafmt.conf | 1 +
java/build.gradle | 23 +-
java/buildSrc/build.gradle | 28 ++-
.../org/apache/kudu/gradle/DistTestTask.java | 4 +
java/gradle/artifacts.gradle | 16 +-
java/gradle/dependencies.gradle | 6 +-
java/gradle/publishing.gradle | 80 ++++---
java/gradle/quality.gradle | 2 +-
java/gradle/scopes.gradle | 23 --
java/gradle/shadow.gradle | 6 +-
java/gradle/wrapper/gradle-wrapper.properties | 2 +-
java/gradlew | 257 ++++++++++++---------
java/kudu-backup-common/build.gradle | 30 +--
java/kudu-backup-tools/build.gradle | 29 ++-
java/kudu-backup/build.gradle | 72 ++++--
.../org/apache/kudu/backup/TestKuduBackup.scala | 40 ++--
java/kudu-client/build.gradle | 39 ++--
java/kudu-hive/build.gradle | 67 +++++-
java/kudu-jepsen/build.gradle | 16 +-
java/kudu-proto/build.gradle | 6 +-
java/kudu-spark-tools/build.gradle | 67 ++++--
java/kudu-spark/build.gradle | 64 +++--
java/kudu-subprocess/build.gradle | 44 ++--
java/kudu-test-utils/build.gradle | 57 +++--
src/kudu/hms/CMakeLists.txt | 2 +-
src/kudu/scripts/assign-location.py | 0
27 files changed, 614 insertions(+), 375 deletions(-)
diff --git a/RELEASING.adoc b/RELEASING.adoc
index 649b3f552..0a26ffebd 100644
--- a/RELEASING.adoc
+++ b/RELEASING.adoc
@@ -210,7 +210,7 @@ Example doc: https://s.apache.org/kudu1.16rn
# Take a note of the identifier of the key you want to use for signing.
gpg --list-secret-keys --keyid-format=short
cd java
- ./gradlew clean install -PforceSigning
-Psigning.gnupg.keyName=<8-character-pgp-key-id>
+ ./gradlew clean assemble -PforceSigning
-Psigning.gnupg.keyName=<8-character-pgp-key-id>
----
. Create a new version update commit which removes the -SNAPSHOT suffix (same
@@ -252,11 +252,11 @@ Example doc: https://s.apache.org/kudu1.16rn
# Run a gpg-agent if you don't normally
gpg-agent --daemon
cd java
- ./gradlew clean assemble
+ ./gradlew clean assemble -PforceSigning
-Psigning.gnupg.keyName=<8-character-pgp-key-id>
# Turn off bash history: this is to avoid exposing the credentials
# via .bash_history file.
set +o history
- ./gradlew --no-parallel uploadArchives \
+ ./gradlew --no-parallel publish \
-Psigning.gnupg.keyName=<8-character-pgp-key-id> \
-PmavenUsername='<APACHE-LDAP-USERNAME>' \
-PmavenPassword='<APACHE-LDAP-PASSWORD>'
@@ -264,7 +264,7 @@ Example doc: https://s.apache.org/kudu1.16rn
set -o history
----
+
-NOTE: If `uploadArchives` is executed without `--no-parallel`, uploading a
+NOTE: If `publish` is executed without `--no-parallel`, uploading a
number of artifacts fails with "peer not authenticated" errors.
. Build and deploy new binary test JARs for the RC on macOS and Linux. Build
diff --git a/java/.scalafmt.conf b/java/.scalafmt.conf
index dc8ad3976..ca8a0cc59 100644
--- a/java/.scalafmt.conf
+++ b/java/.scalafmt.conf
@@ -1,3 +1,4 @@
+version = 1.5.1 // TODO: remove this and fix the formatting errors
style = defaultWithAlign
align = false
diff --git a/java/build.gradle b/java/build.gradle
index 4fa917e7d..ea8f68930 100755
--- a/java/build.gradle
+++ b/java/build.gradle
@@ -48,7 +48,6 @@ subprojects {
// These are common to all subprojects. However, subprojects may
// include their own plugins and scripts as well.
apply plugin: "java"
- apply from: "$rootDir/gradle/scopes.gradle"
apply from: "$rootDir/gradle/compile.gradle"
apply from: "$rootDir/gradle/tests.gradle"
apply from: "$rootDir/gradle/quality.gradle"
@@ -58,19 +57,17 @@ subprojects {
// Ignore the transitive annotations libraries that are
// not marked as optional in Guava version 22.0+.
// See https://github.com/google/guava/issues/2824
- configurations.compile {
+ configurations.implementation {
exclude group: "com.google.errorprone", module: "error_prone_annotations"
- exclude group: "com.google.code.findbugs", module: "jsr305"
exclude group: "com.google.j2objc", module: "j2objc-annotations"
exclude group: "org.checkerframework", module: "checker-compat-qual"
exclude group: "org.codehaus.mojo", module: "animal-sniffer-annotations"
}
- sourceSets {
- all {
- configurations.all { conf ->
- attributes.attribute(Attribute.of("org.gradle.jvm.environment",
String), "standard-jvm")
- }
+ // Force gradle to use the same (declared in dependencies) guava version
everywhere
+ configurations.all {
+ resolutionStrategy {
+ force libs.guava
}
}
}
@@ -96,10 +93,7 @@ task javadocAggregate(type: Javadoc, group: "Documentation")
{
task copyDistTestJars(type: Copy) {
into "$buildDir/jars/"
from subprojects.collect {
- it.configurations.testRuntime
- }
- from subprojects.collect {
- it.configurations.provided
+ it.configurations.testRuntimeClasspath
}
}
@@ -112,3 +106,8 @@ task distTest(type: DistTestTask, dependsOn:
copyDistTestJars) {
}
}
}
+
+// Task alias to keep compatibility with previous Gradle version's publishing
plugin
+task uploadArchives () {
+ dependsOn 'publish'
+}
diff --git a/java/buildSrc/build.gradle b/java/buildSrc/build.gradle
index 28f03f1ec..f603ab10f 100644
--- a/java/buildSrc/build.gradle
+++ b/java/buildSrc/build.gradle
@@ -19,7 +19,6 @@
repositories {
mavenCentral()
- jcenter()
maven { url "https://clojars.org/repo"; } // Only used for the clojure plugin
below.
maven { url "https://plugins.gradle.org/m2/"; }
}
@@ -27,20 +26,19 @@ repositories {
// Manage plugin dependencies since the plugin block can't be used in included
build scripts yet.
// For more details see:
https://docs.gradle.org/current/userguide/plugins.html#plugins_dsl_limitations
dependencies {
- compile "com.github.ben-manes:gradle-versions-plugin:0.41.0"
- compile "com.github.jengelman.gradle.plugins:shadow:6.1.0"
- compile "gradle.plugin.org.barfuin.gradle.jacocolog:gradle-jacoco-log:1.2.4"
- compile "gradle.plugin.com.google.gradle:osdetector-gradle-plugin:1.7.0"
- compile "com.google.protobuf:protobuf-gradle-plugin:0.8.18"
- compile "com.netflix.nebula:nebula-clojure-plugin:10.1.1"
- compile "gradle.plugin.com.github.spotbugs.snom:spotbugs-gradle-plugin:4.7.0"
- compile "gradle.plugin.cn.bestwu.gradle:propdeps-plugin:0.0.10"
- compile "net.ltgt.gradle:gradle-errorprone-plugin:2.0.2"
- compile "ru.vyarus:gradle-animalsniffer-plugin:1.5.4"
- compile "com.google.code.gson:gson:2.8.9"
- compile "cz.alenkacz:gradle-scalafmt:1.14.0"
- compile "com.google.guava:guava:31.0.1-jre"
- compile "me.champeau.gradle:jmh-gradle-plugin:0.5.3"
+ implementation "com.github.ben-manes:gradle-versions-plugin:0.41.0"
+ implementation "gradle.plugin.com.github.johnrengelman:shadow:7.1.2"
+ implementation
"gradle.plugin.org.barfuin.gradle.jacocolog:gradle-jacoco-log:1.2.4"
+ implementation
"gradle.plugin.com.google.gradle:osdetector-gradle-plugin:1.7.0"
+ implementation "com.google.protobuf:protobuf-gradle-plugin:0.8.18"
+ implementation "com.netflix.nebula:nebula-clojure-plugin:10.1.1"
+ implementation
"gradle.plugin.com.github.spotbugs.snom:spotbugs-gradle-plugin:4.7.0"
+ implementation "net.ltgt.gradle:gradle-errorprone-plugin:2.0.2"
+ implementation "ru.vyarus:gradle-animalsniffer-plugin:1.5.4"
+ implementation "com.google.code.gson:gson:2.8.9"
+ implementation "gradle.plugin.cz.alenkacz:gradle-scalafmt:1.16.2"
+ implementation "com.google.guava:guava:31.0.1-jre"
+ implementation "me.champeau.gradle:jmh-gradle-plugin:0.5.3"
}
// Compiler configuration
diff --git
a/java/buildSrc/src/main/groovy/org/apache/kudu/gradle/DistTestTask.java
b/java/buildSrc/src/main/groovy/org/apache/kudu/gradle/DistTestTask.java
index 8b7065fdc..10dc1ed71 100644
--- a/java/buildSrc/src/main/groovy/org/apache/kudu/gradle/DistTestTask.java
+++ b/java/buildSrc/src/main/groovy/org/apache/kudu/gradle/DistTestTask.java
@@ -116,6 +116,10 @@ public class DistTestTask extends DefaultTask {
return fc;
}
+ public File getOutputDir() {
+ return this.outputDir;
+ }
+
@TaskAction
public void doStuff() throws IOException {
getProject().delete(outputDir);
diff --git a/java/gradle/artifacts.gradle b/java/gradle/artifacts.gradle
index 844fdf838..b3ebd99d3 100755
--- a/java/gradle/artifacts.gradle
+++ b/java/gradle/artifacts.gradle
@@ -23,29 +23,29 @@ configurations.create("test")
task testJar(type: Jar, dependsOn: testClasses, group: "Build") {
description = "Assembles a jar archive containing the test classes."
from sourceSets.test.output
- classifier = "tests"
- extension "jar"
+ archiveClassifier = "tests"
+ archiveExtension = "jar"
}
task sourcesJar(type: Jar, dependsOn: classes, group: "Build") {
description = "Assembles a jar archive containing the main source."
from sourceSets.main.allSource
- classifier "sources"
- extension "jar"
+ archiveClassifier = "sources"
+ archiveExtension = "jar"
}
task testSourcesJar(type: Jar, dependsOn: testJar, group: "Build") {
description = "Assembles a jar archive containing the test source."
from sourceSets.test.allSource
- classifier "test-sources"
- extension "jar"
+ archiveClassifier = "test-sources"
+ archiveExtension = "jar"
}
task javadocJar(type: Jar, dependsOn: javadoc, group: "Build") {
description = "Assembles a jar archive containing the javadoc."
from javadoc.destinationDir
- classifier "javadoc"
- extension "jar"
+ archiveClassifier = "javadoc"
+ archiveExtension = "jar"
}
tasks.withType(Jar) {
diff --git a/java/gradle/dependencies.gradle b/java/gradle/dependencies.gradle
index e205649b3..812ac2541 100755
--- a/java/gradle/dependencies.gradle
+++ b/java/gradle/dependencies.gradle
@@ -32,7 +32,7 @@ versions += [
commonsIo : "2.11.0",
errorProne : "2.3.3",
errorProneJavac: "9+181-r4173-1",
- gradle : "6.9.2",
+ gradle : "7.6.4",
guava : "32.1.1-jre",
hadoop : "3.3.1",
hamcrest : "2.2",
@@ -46,6 +46,7 @@ versions += [
jsr305 : "3.0.2",
junit : "4.13.2",
log4j : "2.17.1",
+ logCaptor : "2.9.2",
micrometer : "1.8.2",
mockito : "4.2.0",
murmur : "1.0.0",
@@ -59,7 +60,7 @@ versions += [
scopt : "4.0.1",
slf4j : "1.7.33",
spark2 : "2.4.8",
- spark : "3.1.2",
+ spark : "3.2.4",
spotBugs : "4.1.1",
yetus : "0.13.0"
]
@@ -112,6 +113,7 @@ libs += [
log4jCompat :
"org.apache.logging.log4j:log4j-1.2-api:$versions.log4j",
log4jCore :
"org.apache.logging.log4j:log4j-core:$versions.log4j",
log4jSlf4jImpl :
"org.apache.logging.log4j:log4j-slf4j-impl:$versions.log4j",
+ logCaptor : "io.github.hakky54:logcaptor:$versions.logCaptor",
micrometerCore :
"io.micrometer:micrometer-core:$versions.micrometer",
mockitoCore : "org.mockito:mockito-core:$versions.mockito",
murmur : "com.sangupta:murmur:$versions.murmur",
diff --git a/java/gradle/publishing.gradle b/java/gradle/publishing.gradle
index 9ed2d52d3..56206c0db 100644
--- a/java/gradle/publishing.gradle
+++ b/java/gradle/publishing.gradle
@@ -17,7 +17,7 @@
// This file contains common tasks and configuration for artifact publishing.
-apply plugin: "maven"
+apply plugin: "maven-publish"
apply plugin: "signing"
ext {
@@ -40,51 +40,57 @@ ext {
mavenPassword = propertyWithDefault("mavenPassword", "")
}
-uploadArchives {
- repositories {
- signing {
- required { shouldSign }
- // Check if we are going to sign, because CI environments may not have
- // gpg on their path and useGpgCmd evaluates eagerly.
- if (shouldSign) {
- useGpgCmd() // Use gpg-agent to sign
+publishing {
+ publications {
+ maven(MavenPublication) {
+ from components.java
+
+ groupId "org.apache.kudu"
+ version "${project.version}"
+
+ artifact(sourcesJar) {
+ classifier = "sources"
}
- sign configurations.archives
- mavenDeployer {
- beforeDeployment { MavenDeployment deployment ->
signing.signPom(deployment) }
- repository(url: "${mavenPublishUrl}") {
- authentication(userName: "${mavenUsername}", password:
"${mavenPassword}")
- }
- afterEvaluate {
- pom.project {
- url = "$url"
- licenses {
- license {
- name = "The Apache Software License, Version 2.0"
- url = "http://www.apache.org/licenses/LICENSE-2.0.txt";
- distribution = "repo"
- }
- }
+ artifact(testSourcesJar) {
+ classifier = "testSources"
+ }
+ artifact(javadocJar) {
+ classifier = "javadoc"
+ }
+
+ pom {
+ url = "$url"
+ licenses {
+ license {
+ name = "The Apache License, Version 2.0"
+ url = "https://www.apache.org/licenses/LICENSE-2.0.txt";
+ distribution = "repo"
}
}
}
}
}
-}
-
-// Add the install task to the "Upload" group so it's visible in the tasks
output.
-install.group = "Upload"
-
-// Sort the generated maven dependencies to make pom comparisons easier.
-tasks.withType(Upload) {
- def installer = install.repositories.mavenInstaller
- def deployer = uploadArchives.repositories.mavenDeployer
- [installer, deployer]*.pom*.whenConfigured { pom ->
- pom.dependencies = pom.dependencies.sort { dep ->
- "$dep.scope:$dep.optional:$dep.groupId:$dep.artifactId"
+ repositories {
+ maven {
+ url "${mavenPublishUrl}"
+ credentials {
+ username "${mavenUsername}"
+ password "${mavenPassword}"
+ }
}
}
}
+signing {
+ required { shouldSign }
+ if (shouldSign) {
+ useGpgCmd() // Use gpg-agent to sign
+ }
+
+ sign configurations.archives
+}
+
+// Add the install task to the "Upload" group so it's visible in the tasks
output.
+publish.group = "Upload"
diff --git a/java/gradle/quality.gradle b/java/gradle/quality.gradle
index de58c8283..72ba51757 100644
--- a/java/gradle/quality.gradle
+++ b/java/gradle/quality.gradle
@@ -40,7 +40,7 @@ if (!propertyExists("skipCodeStyleChecks")) {
checkstyle {
toolVersion = versions.checkstyle
- configDir = file("$rootProject.projectDir/config/checkstyle")
+ configDirectory = file("$rootProject.projectDir/config/checkstyle")
ignoreFailures = ignoreCheckFailures
maxWarnings = 0
showViolations = true
diff --git a/java/gradle/scopes.gradle b/java/gradle/scopes.gradle
deleted file mode 100644
index 84a94a3d8..000000000
--- a/java/gradle/scopes.gradle
+++ /dev/null
@@ -1,23 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-// This file adds provided scope and optional maven support.
-
-apply plugin: "cn.bestwu.propdeps"
-apply plugin: "cn.bestwu.propdeps-eclipse"
-apply plugin: "cn.bestwu.propdeps-idea"
-apply plugin: "cn.bestwu.propdeps-maven"
diff --git a/java/gradle/shadow.gradle b/java/gradle/shadow.gradle
index 9eb6cfd11..1f1786331 100644
--- a/java/gradle/shadow.gradle
+++ b/java/gradle/shadow.gradle
@@ -82,13 +82,13 @@ shadowJar {
// By default shadow assumes all dependencies are shaded.
configurations.create("compileUnshaded")
configurations.shadow.extendsFrom(configurations.compileUnshaded)
-configurations.compile.extendsFrom(configurations.compileUnshaded)
+configurations.implementation.extendsFrom(configurations.compileUnshaded)
// We use afterEvaluate to add additional configuration once all the
definitions
// in the projects build script have been applied
afterEvaluate {
- // Ensure compileUnshaded dependencies are included in the pom.
- [install, uploadArchives].each { task ->
+// Ensure compileUnshaded dependencies are included in the pom.
+ tasks.withType(Upload) { task ->
task.repositories.each {
configure(it.pom.scopeMappings) {
// The priority value is arbitrary.
diff --git a/java/gradle/wrapper/gradle-wrapper.properties
b/java/gradle/wrapper/gradle-wrapper.properties
index 8cf6eb5ad..763217c50 100644
--- a/java/gradle/wrapper/gradle-wrapper.properties
+++ b/java/gradle/wrapper/gradle-wrapper.properties
@@ -1,5 +1,5 @@
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-6.8.3-all.zip
+distributionUrl=https\://services.gradle.org/distributions/gradle-7.6.4-all.zip
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
diff --git a/java/gradlew b/java/gradlew
index 6e24fce06..717594ad8 100755
--- a/java/gradlew
+++ b/java/gradlew
@@ -1,7 +1,7 @@
-#!/usr/bin/env sh
+#!/bin/sh
#
-# Copyright 2015 the original author or authors.
+# Copyright © 2015-2021 the original authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -17,67 +17,101 @@
#
##############################################################################
-##
-## Gradle start up script for UN*X
-##
+#
+# Gradle start up script for POSIX generated by Gradle.
+#
+# Important for running:
+#
+# (1) You need a POSIX-compliant shell to run this script. If your /bin/sh is
+# noncompliant, but you have some other compliant shell such as ksh or
+# bash, then to run this script, type that shell name before the whole
+# command line, like:
+#
+# ksh Gradle
+#
+# Busybox and similar reduced shells will NOT work, because this script
+# requires all of these POSIX shell features:
+# * functions;
+# * expansions «$var», «${var}», «${var:-default}», «${var+SET}»,
+# «${var#prefix}», «${var%suffix}», and «$( cmd )»;
+# * compound commands having a testable exit status, especially «case»;
+# * various built-in commands including «command», «set», and «ulimit».
+#
+# Important for patching:
+#
+# (2) This script targets any POSIX shell, so it avoids extensions provided
+# by Bash, Ksh, etc; in particular arrays are avoided.
+#
+# The "traditional" practice of packing multiple parameters into a
+# space-separated string is a well documented source of bugs and security
+# problems, so this is (mostly) avoided, by progressively accumulating
+# options in "$@", and eventually passing that to Java.
+#
+# Where the inherited environment variables (DEFAULT_JVM_OPTS, JAVA_OPTS,
+# and GRADLE_OPTS) rely on word-splitting, this is performed explicitly;
+# see the in-line comments for details.
+#
+# There are tweaks for specific operating systems such as AIX, CygWin,
+# Darwin, MinGW, and NonStop.
+#
+# (3) This script is generated from the Groovy template
+#
https://github.com/gradle/gradle/blob/master/subprojects/plugins/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
+# within the Gradle project.
+#
+# You can find Gradle at https://github.com/gradle/gradle/.
+#
##############################################################################
# Attempt to set APP_HOME
+
# Resolve links: $0 may be a link
-PRG="$0"
-# Need this for relative symlinks.
-while [ -h "$PRG" ] ; do
- ls=`ls -ld "$PRG"`
- link=`expr "$ls" : '.*-> \(.*\)$'`
- if expr "$link" : '/.*' > /dev/null; then
- PRG="$link"
- else
- PRG=`dirname "$PRG"`"/$link"
- fi
+app_path=$0
+
+# Need this for daisy-chained symlinks.
+while
+ APP_HOME=${app_path%"${app_path##*/}"} # leaves a trailing /; empty if no
leading path
+ [ -h "$app_path" ]
+do
+ ls=$( ls -ld "$app_path" )
+ link=${ls#*' -> '}
+ case $link in #(
+ /*) app_path=$link ;; #(
+ *) app_path=$APP_HOME$link ;;
+ esac
done
-SAVED="`pwd`"
-cd "`dirname \"$PRG\"`/" >/dev/null
-APP_HOME="`pwd -P`"
-cd "$SAVED" >/dev/null
+
+APP_HOME=$( cd "${APP_HOME:-./}" && pwd -P ) || exit
APP_NAME="Gradle"
-APP_BASE_NAME=`basename "$0"`
+APP_BASE_NAME=${0##*/}
# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to
pass JVM options to this script.
DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
# Use the maximum available, or set MAX_FD != -1 to use that value.
-MAX_FD="maximum"
+MAX_FD=maximum
warn () {
echo "$*"
-}
+} >&2
die () {
echo
echo "$*"
echo
exit 1
-}
+} >&2
# OS specific support (must be 'true' or 'false').
cygwin=false
msys=false
darwin=false
nonstop=false
-case "`uname`" in
- CYGWIN* )
- cygwin=true
- ;;
- Darwin* )
- darwin=true
- ;;
- MINGW* )
- msys=true
- ;;
- NONSTOP* )
- nonstop=true
- ;;
+case "$( uname )" in #(
+ CYGWIN* ) cygwin=true ;; #(
+ Darwin* ) darwin=true ;; #(
+ MSYS* | MINGW* ) msys=true ;; #(
+ NONSTOP* ) nonstop=true ;;
esac
@@ -100,9 +134,9 @@ CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar
if [ -n "$JAVA_HOME" ] ; then
if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
# IBM's JDK on AIX uses strange locations for the executables
- JAVACMD="$JAVA_HOME/jre/sh/java"
+ JAVACMD=$JAVA_HOME/jre/sh/java
else
- JAVACMD="$JAVA_HOME/bin/java"
+ JAVACMD=$JAVA_HOME/bin/java
fi
if [ ! -x "$JAVACMD" ] ; then
die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
@@ -111,7 +145,7 @@ Please set the JAVA_HOME variable in your environment to
match the
location of your Java installation."
fi
else
- JAVACMD="java"
+ JAVACMD=java
which java >/dev/null 2>&1 || die "ERROR: JAVA_HOME is not set and no
'java' command could be found in your PATH.
Please set the JAVA_HOME variable in your environment to match the
@@ -119,80 +153,95 @@ location of your Java installation."
fi
# Increase the maximum file descriptors if we can.
-if [ "$cygwin" = "false" -a "$darwin" = "false" -a "$nonstop" = "false" ] ;
then
- MAX_FD_LIMIT=`ulimit -H -n`
- if [ $? -eq 0 ] ; then
- if [ "$MAX_FD" = "maximum" -o "$MAX_FD" = "max" ] ; then
- MAX_FD="$MAX_FD_LIMIT"
- fi
- ulimit -n $MAX_FD
- if [ $? -ne 0 ] ; then
- warn "Could not set maximum file descriptor limit: $MAX_FD"
- fi
- else
- warn "Could not query maximum file descriptor limit: $MAX_FD_LIMIT"
- fi
+if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then
+ case $MAX_FD in #(
+ max*)
+ MAX_FD=$( ulimit -H -n ) ||
+ warn "Could not query maximum file descriptor limit"
+ esac
+ case $MAX_FD in #(
+ '' | soft) :;; #(
+ *)
+ ulimit -n "$MAX_FD" ||
+ warn "Could not set maximum file descriptor limit to $MAX_FD"
+ esac
fi
-# For Darwin, add options to specify how the application appears in the dock
-if $darwin; then
- GRADLE_OPTS="$GRADLE_OPTS \"-Xdock:name=$APP_NAME\"
\"-Xdock:icon=$APP_HOME/media/gradle.icns\""
-fi
+# Collect all arguments for the java command, stacking in reverse order:
+# * args from the command line
+# * the main class name
+# * -classpath
+# * -D...appname settings
+# * --module-path (only if needed)
+# * DEFAULT_JVM_OPTS, JAVA_OPTS, and GRADLE_OPTS environment variables.
# For Cygwin or MSYS, switch paths to Windows format before running java
-if [ "$cygwin" = "true" -o "$msys" = "true" ] ; then
- APP_HOME=`cygpath --path --mixed "$APP_HOME"`
- CLASSPATH=`cygpath --path --mixed "$CLASSPATH"`
-
- JAVACMD=`cygpath --unix "$JAVACMD"`
-
- # We build the pattern for arguments to be converted via cygpath
- ROOTDIRSRAW=`find -L / -maxdepth 1 -mindepth 1 -type d 2>/dev/null`
- SEP=""
- for dir in $ROOTDIRSRAW ; do
- ROOTDIRS="$ROOTDIRS$SEP$dir"
- SEP="|"
- done
- OURCYGPATTERN="(^($ROOTDIRS))"
- # Add a user-defined pattern to the cygpath arguments
- if [ "$GRADLE_CYGPATTERN" != "" ] ; then
- OURCYGPATTERN="$OURCYGPATTERN|($GRADLE_CYGPATTERN)"
- fi
+if "$cygwin" || "$msys" ; then
+ APP_HOME=$( cygpath --path --mixed "$APP_HOME" )
+ CLASSPATH=$( cygpath --path --mixed "$CLASSPATH" )
+
+ JAVACMD=$( cygpath --unix "$JAVACMD" )
+
# Now convert the arguments - kludge to limit ourselves to /bin/sh
- i=0
- for arg in "$@" ; do
- CHECK=`echo "$arg"|egrep -c "$OURCYGPATTERN" -`
- CHECK2=`echo "$arg"|egrep -c "^-"` ###
Determine if an option
-
- if [ $CHECK -ne 0 ] && [ $CHECK2 -eq 0 ] ; then ###
Added a condition
- eval `echo args$i`=`cygpath --path --ignore --mixed "$arg"`
- else
- eval `echo args$i`="\"$arg\""
+ for arg do
+ if
+ case $arg in #(
+ -*) false ;; # don't mess with
options #(
+ /?*) t=${arg#/} t=/${t%%/*} # looks like a POSIX
filepath
+ [ -e "$t" ] ;; #(
+ *) false ;;
+ esac
+ then
+ arg=$( cygpath --path --ignore --mixed "$arg" )
fi
- i=`expr $i + 1`
+ # Roll the args list around exactly as many times as the number of
+ # args, so each arg winds up back in the position where it started, but
+ # possibly modified.
+ #
+ # NB: a `for` loop captures its iteration list before it begins, so
+ # changing the positional parameters here affects neither the number of
+ # iterations, nor the values presented in `arg`.
+ shift # remove old arg
+ set -- "$@" "$arg" # push replacement arg
done
- case $i in
- 0) set -- ;;
- 1) set -- "$args0" ;;
- 2) set -- "$args0" "$args1" ;;
- 3) set -- "$args0" "$args1" "$args2" ;;
- 4) set -- "$args0" "$args1" "$args2" "$args3" ;;
- 5) set -- "$args0" "$args1" "$args2" "$args3" "$args4" ;;
- 6) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" ;;
- 7) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5"
"$args6" ;;
- 8) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5"
"$args6" "$args7" ;;
- 9) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5"
"$args6" "$args7" "$args8" ;;
- esac
fi
-# Escape application args
-save () {
- for i do printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/" ;
done
- echo " "
-}
-APP_ARGS=`save "$@"`
+# Collect all arguments for the java command;
+# * $DEFAULT_JVM_OPTS, $JAVA_OPTS, and $GRADLE_OPTS can contain fragments of
+# shell script including quotes and variable substitutions, so put them in
+# double quotes to make sure that they get re-expanded; and
+# * put everything else in single quotes, so that it's not re-expanded.
+
+set -- \
+ "-Dorg.gradle.appname=$APP_BASE_NAME" \
+ -classpath "$CLASSPATH" \
+ org.gradle.wrapper.GradleWrapperMain \
+ "$@"
+
+# Use "xargs" to parse quoted args.
+#
+# With -n1 it outputs one arg per line, with the quotes and backslashes
removed.
+#
+# In Bash we could simply go:
+#
+# readarray ARGS < <( xargs -n1 <<<"$var" ) &&
+# set -- "${ARGS[@]}" "$@"
+#
+# but POSIX shell has neither arrays nor command substitution, so instead we
+# post-process each arg (as a line of input to sed) to backslash-escape any
+# character that might be a shell metacharacter, then use eval to reverse
+# that process (while maintaining the separation between arguments), and wrap
+# the whole thing up as a single "set" statement.
+#
+# This will of course break if any of these variables contains a newline or
+# an unmatched quote.
+#
-# Collect all arguments for the java command, following the shell quoting and
substitution rules
-eval set -- $DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS
"\"-Dorg.gradle.appname=$APP_BASE_NAME\"" -classpath "\"$CLASSPATH\""
org.gradle.wrapper.GradleWrapperMain "$APP_ARGS"
+eval "set -- $(
+ printf '%s\n' "$DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS" |
+ xargs -n1 |
+ sed ' s~[^-[:alnum:]+,./:=@_]~\\&~g; ' |
+ tr '\n' ' '
+ )" '"$@"'
exec "$JAVACMD" "$@"
diff --git a/java/kudu-backup-common/build.gradle
b/java/kudu-backup-common/build.gradle
index 56d8ad8ac..6a6292315 100644
--- a/java/kudu-backup-common/build.gradle
+++ b/java/kudu-backup-common/build.gradle
@@ -19,25 +19,25 @@ apply plugin: "scala"
apply from: "$rootDir/gradle/protobuf.gradle"
dependencies {
- compile project(path: ":kudu-client", configuration: "shadow")
- compile libs.protobufJava
- compile (libs.protobufJavaUtil) {
+ implementation project(path: ":kudu-client", configuration: "shadow")
+ implementation libs.protobufJava
+ implementation (libs.protobufJavaUtil) {
// Make sure wrong Guava version is not pulled in.
exclude group: "com.google.guava", module: "guava"
}
- compile libs.slf4jApi
+ implementation libs.slf4jApi
- provided libs.hadoopCommon
- provided libs.scalaLibrary
+ implementation libs.hadoopCommon
+ implementation libs.scalaLibrary
- optional libs.yetusAnnotations
+ implementation libs.yetusAnnotations
- testCompile project(path: ":kudu-test-utils", configuration: "shadow")
- testCompile libs.junit
- testCompile libs.log4jApi
- testCompile libs.log4jCore
- testCompile libs.log4jSlf4jImpl
- testCompile libs.scalatest
+ testImplementation project(path: ":kudu-test-utils", configuration: "shadow")
+ testImplementation libs.junit
+ testImplementation libs.log4jApi
+ testImplementation libs.log4jCore
+ testImplementation libs.log4jSlf4jImpl
+ testImplementation libs.scalatest
}
// Add protobuf files to the proto source set.
@@ -56,5 +56,5 @@ javadoc {
// Skip publishing kudu-backup-common artifact because it will always be
shaded into
// kudu-backup and kudu-backup-tools.
-uploadArchives.enabled = false
-install.enabled = false
\ No newline at end of file
+publish.enabled = false
+publishToMavenLocal.enabled = false
diff --git a/java/kudu-backup-tools/build.gradle
b/java/kudu-backup-tools/build.gradle
index 2f5bd0cb8..ff5c842a3 100644
--- a/java/kudu-backup-tools/build.gradle
+++ b/java/kudu-backup-tools/build.gradle
@@ -19,23 +19,30 @@ apply plugin: "scala"
apply from: "$rootDir/gradle/shadow.gradle"
dependencies {
- compile project(path: ":kudu-backup-common")
- compile project(path: ":kudu-client", configuration: "shadow")
- compile (libs.scopt) {
+ implementation project(path: ":kudu-backup-common")
+ implementation project(path: ":kudu-client", configuration: "shadow")
+ implementation libs.protobufJava
+ implementation (libs.scopt) {
// Make sure wrong Scala version is not pulled in.
exclude group: "org.scala-lang", module: "scala-library"
}
- compile libs.scalaLibrary
- compile libs.slf4jApi
+ implementation libs.scalaLibrary
+ implementation libs.slf4jApi
- provided libs.hadoopClient
+ compileOnly libs.hadoopClient
- optional libs.yetusAnnotations
+ implementation libs.yetusAnnotations
- testCompile project(path: ":kudu-test-utils", configuration: "shadow")
- testCompile libs.hadoopMiniCluster
- testCompile libs.junit
- testCompile libs.scalatest
+ testImplementation project(path: ":kudu-test-utils", configuration: "shadow")
+ testImplementation libs.hadoopMiniCluster
+ testImplementation libs.junit
+ testImplementation libs.scalatest
+}
+
+shadowJar {
+ dependencies {
+ minimize()
+ }
}
// kudu-backup-tools has no public Javadoc.
diff --git a/java/kudu-backup/build.gradle b/java/kudu-backup/build.gradle
index 0a30de936..f0c5b710e 100644
--- a/java/kudu-backup/build.gradle
+++ b/java/kudu-backup/build.gradle
@@ -20,34 +20,68 @@ apply from: "$rootDir/gradle/shadow.gradle"
dependencies {
// Note: We don't use the shaded version, so we can control the dependencies.
- compile(project(path: ":kudu-backup-common")) {
+ implementation(project(path: ":kudu-backup-common")) {
// Ensure we use the hadoop-client provided by Spark to avoid any
compatibility issues.
exclude group: "org.apache.hadoop", module: "hadoop-common"
// Ensure we use the scala-library provided by Spark to avoid any
compatibility issues.
exclude group: "org.scala-lang", module: "scala-library"
}
- compile project(path: ":kudu-client", configuration: "shadow")
- compile project(path: ":kudu-spark", configuration: "shadow")
- compile (libs.scopt) {
+ implementation project(path: ":kudu-client", configuration: "shadow")
+ implementation project(path: ":kudu-spark", configuration: "shadow")
+ implementation libs.protobufJava
+ implementation (libs.scopt) {
// Make sure wrong Scala version is not pulled in.
exclude group: "org.scala-lang", module: "scala-library"
}
// TODO(KUDU-2500): Spark uses reflection which requires the annotations at
runtime.
- compile libs.yetusAnnotations
-
- provided libs.scalaLibrary
- provided libs.sparkAvro
- provided libs.sparkCore
- provided libs.sparkSql
- provided libs.slf4jApi
-
- testCompile project(path: ":kudu-test-utils", configuration: "shadow")
- testCompile project(path: ":kudu-spark", configuration: "test")
- testCompile libs.junit
- testCompile libs.log4jApi
- testCompile libs.log4jCore
- testCompile libs.log4jSlf4jImpl
- testCompile libs.scalatest
+ implementation libs.yetusAnnotations
+
+ implementation libs.scalaLibrary
+ implementation libs.sparkAvro
+ implementation libs.sparkCore
+ implementation libs.sparkSql
+ implementation libs.slf4jApi
+
+ testImplementation project(path: ":kudu-test-utils", configuration: "shadow")
+ testImplementation project(path: ":kudu-spark", configuration: "test")
+ testImplementation libs.junit
+ testImplementation libs.log4jApi
+ testImplementation libs.log4jCore
+ testImplementation libs.log4jSlf4jImpl
+ testImplementation libs.scalatest
+ testImplementation libs.logCaptor
+}
+
+shadowJar {
+ dependencies {
+ exclude(dependency(libs.jsr305))
+ exclude(dependency("log4j:log4j:.*"))
+ exclude(dependency("org.apache.arrow::.*"))
+ exclude(dependency("org.apache.avro::.*"))
+ exclude(dependency("org.apache.curator::.*"))
+ exclude(dependency("org.apache.hadoop::.*"))
+ exclude(dependency("org.apache.ivy:ivy::.*"))
+ exclude(dependency("org.apache.parquet:.*"))
+ exclude(dependency("org.apache.spark::.*"))
+ exclude(dependency("org.codehaus.janino::.*"))
+ exclude(dependency("org.glassfish.jersey.core::.*"))
+ exclude(dependency("org.scala-lang::.*"))
+ exclude(dependency("org.xerial.snappy::.*"))
+ }
+ exclude '**/*.adoc'
+ exclude '**/*.default'
+ exclude '**/*.dll'
+ exclude '**/*.dylib'
+ exclude '**/*.html'
+ exclude '**/*.md'
+ exclude 'META-INF/services/**'
+ exclude 'META-INF/versions/11/org/roaringbitmap/**'
+ exclude 'codegen/**'
+ exclude 'assets/**'
+ exclude 'javax/**'
+ exclude 'org/jetbrains/**'
+
+ minimize()
}
// Adjust the artifact name to match the maven build.
diff --git
a/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
b/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
index cc9bcd4de..ea13fcf62 100644
---
a/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
+++
b/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
@@ -49,6 +49,7 @@ import org.junit.Before
import org.junit.Test
import org.slf4j.Logger
import org.slf4j.LoggerFactory
+import nl.altindag.log.LogCaptor
import scala.collection.JavaConverters._
@@ -234,8 +235,11 @@ class TestKuduBackup extends KuduTestSuite {
// table that does exist, it should not throw an exception, and it should
return 1 to indicate
// some error. The logs should contain a message about the missing table.
val options = createBackupOptions(Seq("missingTable", tableName))
- val logs = captureLogs(() => assertFalse(KuduBackup.run(options, ss)))
- assertTrue(logs.contains("the table does not exist"))
+ // Setting up log capture for the class that generates the logs we're
+ // interested in and not for the test class.
+ val logCaptor = LogCaptor.forClass(KuduBackup.getClass)
+ assertFalse(KuduBackup.run(options, ss))
+ assertTrue(logCaptor.getLogs().toString().contains("the table does not
exist"))
// Restore the backup of the non-failed table and validate the end result.
restoreAndValidateTable(tableName, 100)
@@ -264,11 +268,13 @@ class TestKuduBackup extends KuduTestSuite {
insertRows(table, 100)
KuduBackup.run(createBackupOptions(Seq(tableName)), ss)
+ // setting up log capture for the class that generates the logs we're
+ // interested in and not for the test class
+ val logCaptor = LogCaptor.forClass(KuduRestore.getClass)
// There's no guarantee about the order restores run in, so it doesn't
work to test fail-fast
// and then the default no-fail-fast because the actual table may have
been restored.
- val logs = captureLogs(
- () => assertFalse(runRestore(createRestoreOptions(Seq("missingTable",
tableName)))))
- assertTrue(logs.contains("Failed to restore table"))
+ assertFalse(runRestore(createRestoreOptions(Seq("missingTable",
tableName))))
+ assertTrue(logCaptor.getLogs().toString().contains("Failed to restore
table"))
}
@Test
@@ -283,10 +289,16 @@ class TestKuduBackup extends KuduTestSuite {
// Force an incremental backup without a full backup.
// It will use a diff scan and won't check the existing dependency graph.
val options = createBackupOptions(Seq(tableName), fromMs = beforeMs)
- val logs = captureLogs { () =>
- assertTrue(runBackup(options))
- }
- assertTrue(logs.contains("Performing an incremental backup: fromMs was set
to"))
+
+ // setting up log capture for the class that generates the logs we're
+ // interested in and not for the test class
+ val logCaptor = LogCaptor.forClass(KuduBackup.getClass)
+ assertTrue(runBackup(options))
+ assertTrue(
+ logCaptor
+ .getLogs()
+ .toString()
+ .contains("Performing an incremental backup: fromMs was set to"))
validateBackup(options, 100, true)
}
@@ -299,10 +311,12 @@ class TestKuduBackup extends KuduTestSuite {
// Force a full backup. It should contain all the rows.
val options = createBackupOptions(Seq(tableName), forceFull = true)
- val logs = captureLogs { () =>
- assertTrue(runBackup(options))
- }
- assertTrue(logs.contains("Performing a full backup: forceFull was set to
true"))
+
+ // setting up log capture for the class that generates the logs we're
+ // interested in and not for the test class
+ val logCaptor = LogCaptor.forClass(KuduBackup.getClass)
+ assertTrue(runBackup(options))
+ assertTrue(logCaptor.getLogs().contains("Performing a full backup:
forceFull was set to true"))
validateBackup(options, 200, false)
}
diff --git a/java/kudu-client/build.gradle b/java/kudu-client/build.gradle
index 082b7d930..61abb96ba 100644
--- a/java/kudu-client/build.gradle
+++ b/java/kudu-client/build.gradle
@@ -19,27 +19,38 @@ apply from: "$rootDir/gradle/shadow.gradle"
apply from: "$rootDir/gradle/benchmarks.gradle"
dependencies {
- compile project(path: ":kudu-proto")
+ implementation project(path: ":kudu-proto")
+ implementation libs.protobufJava
+
// Not shaded in the client JAR because it's part of the public API.
- compileUnshaded(libs.async) {
+ compileUnshaded (libs.async) {
// async uses versions ranges for slf4j making builds non-deterministic.
// Remove this once the following is merged:
https://github.com/OpenTSDB/async/pull/8
exclude group: "org.slf4j", module: "slf4j-api"
}
compileUnshaded libs.slf4jApi
- compile libs.guava
- compile libs.micrometerCore
- compile libs.murmur
- compile libs.netty
+ implementation (libs.guava) {
+ exclude(group: "com.google.errorprone")
+ }
+ implementation libs.micrometerCore
+ implementation libs.murmur
+ implementation libs.netty
+
+ compileOnly libs.jsr305
+ compileOnly libs.yetusAnnotations
- optional libs.jsr305
- optional libs.yetusAnnotations
+ testImplementation project(":kudu-test-utils")
+ testImplementation libs.junit
+ testImplementation libs.log4jApi
+ testImplementation libs.log4jCore
+ testImplementation libs.log4jSlf4jImpl
+ testImplementation libs.mockitoCore
- testCompile project(":kudu-test-utils")
- testCompile libs.junit
- testCompile libs.log4jApi
- testCompile libs.log4jCore
- testCompile libs.log4jSlf4jImpl
- testCompile libs.mockitoCore
+ shadowJar {
+ dependencies {
+ exclude(dependency(libs.jsr305))
+ }
+ minimize()
+ }
}
diff --git a/java/kudu-hive/build.gradle b/java/kudu-hive/build.gradle
index b86d912f3..b81e0b2ae 100644
--- a/java/kudu-hive/build.gradle
+++ b/java/kudu-hive/build.gradle
@@ -17,21 +17,66 @@
apply from: "$rootDir/gradle/shadow.gradle"
dependencies {
- compile project(path: ":kudu-client", configuration: "shadow")
+ implementation project(path: ":kudu-client", configuration: "shadow")
- provided libs.hiveMetastore
- provided libs.slf4jApi
+ implementation libs.hiveMetastore
+ compileOnly libs.slf4jApi
// Transitively required through hive-metastore,
// which has marked it as an optional dependency.
- provided libs.hadoopCommon
- provided libs.hadoopMRClientCommon
+ compileOnly libs.hadoopCommon
+ testImplementation libs.hadoopMRClientCommon
- testCompile project(path: ":kudu-test-utils", configuration: "shadow")
- testCompile libs.hiveTestUtils
- testCompile libs.junit
- testCompile libs.log4jApi
- testCompile libs.log4jCore
- testCompile libs.log4jSlf4jImpl
+ testImplementation project(path: ":kudu-test-utils", configuration: "shadow")
+ testImplementation libs.hiveTestUtils
+ testImplementation libs.junit
+ testImplementation libs.log4jApi
+ testImplementation libs.log4jCore
+ testImplementation libs.log4jSlf4jImpl
+}
+
+shadowJar {
+ dependencies {
+ exclude(dependency("log4j::.*"))
+ exclude(dependency("org.apache.hadoop::.*"))
+ exclude(dependency("org.apache.hive::.*"))
+ exclude(dependency("org.apache.hbase::.*"))
+ exclude(dependency("org.apache.hbase.thirdparty::.*"))
+ exclude(dependency("junit::.*"))
+ exclude(dependency("javax.servlet.jsp::.*"))
+ exclude(dependency("javax.jdo::.*"))
+ exclude(dependency("com.fasterxml.woodstox:::.*"))
+ exclude(dependency("org.fusesource.leveldbjni::.*"))
+ exclude(dependency("org.apache.parquet:.*"))
+ exclude(dependency("com.fasterxml.jackson.core::.*"))
+ exclude(dependency("org.codehaus.jackson::.*"))
+ exclude(dependency("com.fasterxml.jackson.jaxrs::.*"))
+ exclude(dependency("com.fasterxml.jackson.module::.*"))
+ exclude(dependency("commons-dbcp::.*"))
+ exclude(dependency("org.apache.kerby:.*"))
+ exclude(dependency("org.apache.commons::.*"))
+ exclude(dependency("org.eclipse.jetty::.*"))
+ exclude(dependency("com.nimbusds::.*"))
+ exclude(dependency("org.apache.curator::.*"))
+ exclude(dependency("joda-time::.*"))
+ exclude(dependency("org.apache.logging.log4j::.*"))
+ exclude(dependency("org.apache.orc::.*"))
+ exclude(dependency("org.xerial.snappy::.*"))
+ exclude(dependency("org.datanucleus::.*"))
+ exclude(dependency("com.sun.jersey::.*"))
+ exclude(dependency("sqlline::.*"))
+ exclude(dependency("org.apache.thrift::.*"))
+ exclude(dependency("org.ehcache::.*"))
+ exclude(dependency("com.sun.xml.bind::.*"))
+ }
+ exclude 'META-INF/services/**'
+ exclude 'tables/**'
+ exclude 'codegen/**'
+ exclude 'javax/**'
+ exclude '**/*.dll'
+ exclude 'images/**'
+ exclude 'org/apache/tools/ant/**'
+
+ minimize()
}
// kudu-hive has no public Javadoc.
diff --git a/java/kudu-jepsen/build.gradle b/java/kudu-jepsen/build.gradle
index c23082a2f..3a496fa97 100644
--- a/java/kudu-jepsen/build.gradle
+++ b/java/kudu-jepsen/build.gradle
@@ -33,13 +33,13 @@ repositories {
}
dependencies {
- compile project(path: ":kudu-client", configuration: "shadow")
- compile project(path: ":kudu-test-utils", configuration: "shadow")
- compile libs.clojure
- compile libs.clojureToolsCli
- compile libs.jepsen
+ implementation project(path: ":kudu-client", configuration: "shadow")
+ implementation project(path: ":kudu-test-utils", configuration: "shadow")
+ implementation libs.clojure
+ implementation libs.clojureToolsCli
+ implementation libs.jepsen
- optional libs.yetusAnnotations
+ implementation libs.yetusAnnotations
}
compileClojure {
@@ -69,8 +69,8 @@ task runJepsen(type: JavaExec) {
}
// We don't publish kudu-jepsen
-uploadArchives.enabled = false
-install.enabled = false
+publish.enabled = false
+publishToMavenLocal.enabled = false
// SpotBugs doesn't work on Clojure.
spotbugsMain.enabled = false
diff --git a/java/kudu-proto/build.gradle b/java/kudu-proto/build.gradle
index c32cad421..42b9dac7b 100644
--- a/java/kudu-proto/build.gradle
+++ b/java/kudu-proto/build.gradle
@@ -18,7 +18,7 @@
apply from: "$rootDir/gradle/protobuf.gradle"
dependencies {
- compile libs.protobufJava
+ implementation libs.protobufJava
}
// Add protobuf files to the proto source set.
@@ -38,5 +38,5 @@ javadoc {
}
// Skip publishing kudu-proto artifact because it will always be shaded into
kudu-client.
-uploadArchives.enabled = false
-install.enabled = false
+publish.enabled = false
+publishToMavenLocal.enabled = false
diff --git a/java/kudu-spark-tools/build.gradle
b/java/kudu-spark-tools/build.gradle
index e754ec2c5..95daad590 100644
--- a/java/kudu-spark-tools/build.gradle
+++ b/java/kudu-spark-tools/build.gradle
@@ -19,28 +19,61 @@ apply plugin: "scala"
apply from: "$rootDir/gradle/shadow.gradle"
dependencies {
- compile project(path: ":kudu-client", configuration: "shadow")
- compile project(path: ":kudu-spark", configuration: "shadow")
+ implementation project(path: ":kudu-client", configuration: "shadow")
+ implementation project(':kudu-spark')
// TODO(KUDU-2500): Spark uses reflection which requires the annotations at
runtime.
- compile libs.yetusAnnotations
- compile (libs.scopt) {
+ implementation libs.yetusAnnotations
+ implementation (libs.scopt) {
// Make sure wrong Scala version is not pulled in.
exclude group: "org.scala-lang", module: "scala-library"
}
- provided libs.scalaLibrary
- provided libs.sparkAvro
- provided libs.sparkCore
- provided libs.sparkSql
- provided libs.slf4jApi
-
- testCompile project(path: ":kudu-test-utils", configuration: "shadow")
- testCompile project(path: ":kudu-spark", configuration: "test")
- testCompile libs.junit
- testCompile libs.log4jApi
- testCompile libs.log4jCore
- testCompile libs.log4jSlf4jImpl
- testCompile libs.scalatest
+ implementation libs.sparkCore
+ implementation libs.sparkSql
+
+ compileOnly libs.scalaLibrary
+ compileOnly libs.slf4jApi
+
+ testImplementation project(path: ":kudu-test-utils", configuration: "shadow")
+ testImplementation project(path: ":kudu-spark", configuration: "test")
+ testImplementation libs.junit
+ testImplementation libs.log4jApi
+ testImplementation libs.log4jCore
+ testImplementation libs.log4jSlf4jImpl
+ testImplementation libs.scalatest
+ testImplementation libs.sparkAvro
+}
+
+shadowJar {
+ dependencies {
+ exclude(dependency(libs.sparkCore))
+ exclude(dependency(libs.sparkSql))
+ exclude(dependency("log4j:log4j:.*"))
+ exclude(dependency("org.apache.avro::.*"))
+ exclude(dependency("org.apache.commons::.*"))
+ exclude(dependency("org.apache.curator::.*"))
+ exclude(dependency("org.apache.hadoop::.*"))
+ exclude(dependency("org.apache.ivy:ivy::.*"))
+ exclude(dependency("org.apache.parquet:.*"))
+ exclude(dependency("org.apache.spark::.*"))
+ exclude(dependency("org.codehaus.janino::.*"))
+ exclude(dependency("org.glassfish.jersey.core::.*"))
+ exclude(dependency("org.scala-lang::.*"))
+ exclude(dependency("org.xerial.snappy::.*"))
+ }
+ exclude '**/*.adoc'
+ exclude '**/*.default'
+ exclude '**/*.dll'
+ exclude '**/*.dylib'
+ exclude '**/*.html'
+ exclude '**/*.md'
+ exclude 'META-INF/services/**'
+ exclude 'META-INF/versions/11/org/roaringbitmap/**'
+ exclude 'codegen/**'
+ exclude 'javax/**'
+ exclude 'org/jetbrains/**'
+
+ minimize()
}
// Adjust the artifact name to match the maven build.
diff --git a/java/kudu-spark/build.gradle b/java/kudu-spark/build.gradle
index abd7a4be5..28d7adce2 100644
--- a/java/kudu-spark/build.gradle
+++ b/java/kudu-spark/build.gradle
@@ -19,22 +19,56 @@ apply plugin: "scala"
apply from: "$rootDir/gradle/shadow.gradle"
dependencies {
- compile project(path: ":kudu-client", configuration: "shadow")
- compile libs.hdrhistogram
+ implementation project(path: ":kudu-client", configuration: "shadow")
+ implementation libs.hdrhistogram
// TODO(KUDU-2500): Spark uses reflection which requires the annotations at
runtime.
- compile libs.yetusAnnotations
-
- provided libs.scalaLibrary
- provided libs.sparkCore
- provided libs.sparkSql
- provided libs.slf4jApi
-
- testCompile project(path: ":kudu-test-utils", configuration: "shadow")
- testCompile libs.junit
- testCompile libs.scalatest
- testCompile libs.log4jApi
- testCompile libs.log4jCore
- testCompile libs.log4jSlf4jImpl
+ implementation libs.yetusAnnotations
+
+ implementation libs.sparkCore
+ implementation libs.sparkSql
+
+ compileOnly libs.scalaLibrary
+ compileOnly libs.slf4jApi
+
+ testImplementation project(path: ":kudu-test-utils", configuration: "shadow")
+ testImplementation libs.junit
+ testImplementation libs.scalatest
+ testImplementation libs.log4jApi
+ testImplementation libs.log4jCore
+ testImplementation libs.log4jSlf4jImpl
+}
+
+shadowJar {
+ dependencies {
+ exclude(dependency("com.fasterxml.jackson.core::.*"))
+ exclude(dependency("com.github.luben::.*"))
+ exclude(dependency("javax.servlet.jsp::.*"))
+ exclude(dependency("log4j:log4j:.*"))
+ exclude(dependency("org.apache.avro::.*"))
+ exclude(dependency("org.apache.commons::.*"))
+ exclude(dependency("org.apache.curator::.*"))
+ exclude(dependency("org.apache.hadoop::.*"))
+ exclude(dependency("org.apache.ivy:ivy::.*"))
+ exclude(dependency("org.apache.parquet:.*"))
+ exclude(dependency("org.apache.spark::.*"))
+ exclude(dependency("org.codehaus.janino::.*"))
+ exclude(dependency("org.glassfish.jersey.core::.*"))
+ exclude(dependency("org.scala-lang::.*"))
+ exclude(dependency("org.xerial.snappy::.*"))
+ }
+ exclude '**/*.adoc'
+ exclude '**/*.default'
+ exclude '**/*.dll'
+ exclude '**/*.dylib'
+ exclude '**/*.html'
+ exclude '**/*.md'
+ exclude 'META-INF/services/**'
+ exclude 'META-INF/versions/11/org/roaringbitmap/**'
+ exclude 'codegen/**'
+ exclude 'javax/**'
+ exclude 'org/jetbrains/**'
+
+ minimize()
}
// Adjust the artifact name to include the spark and scala base versions.
diff --git a/java/kudu-subprocess/build.gradle
b/java/kudu-subprocess/build.gradle
index 1f8745ab4..6f213bb8e 100644
--- a/java/kudu-subprocess/build.gradle
+++ b/java/kudu-subprocess/build.gradle
@@ -21,8 +21,8 @@ apply from: "$rootDir/gradle/shadow.gradle"
shadowIncludeSlf4j = true
dependencies {
- compile project(path: ":kudu-proto")
- compile(libs.hadoopCommon) {
+ implementation project(path: ":kudu-proto")
+ implementation(libs.hadoopCommon) {
// Exclude io.netty to ensure we use our own versions.
// The dependency on io.netty is a transitive one through the chain
// hadoop-common --> hadoop-auth --> zookeeper --> io.netty.netty-xxx
@@ -33,9 +33,9 @@ dependencies {
exclude group: "log4j"
exclude group: "org.slf4j"
}
- compile libs.protobufJavaUtil
+ implementation libs.protobufJavaUtil
- compile(libs.rangerPlugin) {
+ implementation(libs.rangerPlugin) {
// rangerPlugin depends on kafka which includes different versions
// of jersey than rangerPlugin.
exclude group: "org.apache.kafka"
@@ -43,29 +43,29 @@ dependencies {
exclude group: "log4j"
exclude group: "org.slf4j"
}
- compile libs.slf4jApi
- compile libs.log4jApi
- compile libs.log4jCore
- compile libs.log4jCompat
- compile libs.log4jSlf4jImpl
+ implementation libs.slf4jApi
+ implementation libs.log4jApi
+ implementation libs.log4jCore
+ implementation libs.log4jCompat
+ implementation libs.log4jSlf4jImpl
// Compiling exact modules from the io.netty group for hadoop-common:
// for some reason, if compiling just the libs.netty umbrella package
// (i.e. io.netty:netty-all), hadoop-common still depends on its own
// version of io.netty:netty-transport-native-epoll.
- compile libs.nettyBuffer
- compile libs.nettyCommon
- compile libs.nettyHandler
- compile libs.nettyTransport
- compile libs.nettyTransportEpoll
- compile libs.nettyTransportUnix
+ implementation libs.nettyBuffer
+ implementation libs.nettyCommon
+ implementation libs.nettyHandler
+ implementation libs.nettyTransport
+ implementation libs.nettyTransportEpoll
+ implementation libs.nettyTransportUnix
- optional libs.jsr305
- optional libs.yetusAnnotations
+ compileOnly libs.jsr305
+ implementation libs.yetusAnnotations
- testCompile project(path: ":kudu-test-utils", configuration: "shadow")
- testCompile libs.junit
- testCompile libs.mockitoCore
+ testImplementation project(path: ":kudu-test-utils", configuration: "shadow")
+ testImplementation libs.junit
+ testImplementation libs.mockitoCore
}
// kudu-subprocess has no public Javadoc.
@@ -74,5 +74,5 @@ javadoc {
}
// Skip publishing kudu-subprocess artifact because it's not intended for
external use.
-uploadArchives.enabled = false
-install.enabled = false
+publish.enabled = false
+publishToMavenLocal.enabled = false
diff --git a/java/kudu-test-utils/build.gradle
b/java/kudu-test-utils/build.gradle
index a4a23e4b7..f4203789a 100644
--- a/java/kudu-test-utils/build.gradle
+++ b/java/kudu-test-utils/build.gradle
@@ -18,14 +18,25 @@
apply from: "$rootDir/gradle/shadow.gradle"
dependencies {
- compile project(path: ":kudu-client")
- compile libs.commonsIo
- compile libs.guava
- compile libs.httpClient
- compile libs.httpMime
- compile libs.osdetector
-
- compileUnshaded libs.hamcrest
+ implementation project(path: ":kudu-client")
+ implementation project(path: ":kudu-proto")
+ implementation libs.commonsIo
+ implementation(libs.guava) {
+ exclude(group: "com.google.errorprone")
+ }
+ implementation libs.httpClient
+ implementation libs.httpMime
+ implementation libs.osdetector
+ implementation libs.protobufJava
+
+ // Not shaded in the client JAR because it's part of the public API.
+ compileUnshaded(libs.async) {
+ // async uses versions ranges for slf4j making builds non-deterministic.
+ // Remove this once the following is merged:
https://github.com/OpenTSDB/async/pull/8
+ exclude group: "org.slf4j", module: "slf4j-api"
+ }
+
+ implementation libs.hamcrest
compileUnshaded(libs.junit) {
// Use the Kudu specified hamcrest.
exclude group: "org.hamcrest"
@@ -36,19 +47,33 @@ dependencies {
if (propertyExists("useBinJar")) {
apply plugin: "com.google.osdetector"
def jarVersion = propertyWithDefault("useBinJar", project.version)
- runtime "org.apache.kudu:kudu-binary:$jarVersion:${osdetector.classifier}"
+ runtimeOnly
"org.apache.kudu:kudu-binary:$jarVersion:${osdetector.classifier}"
}
// Needed for CapturingLogAppender. Optional otherwise.
- optional libs.log4jApi
- optional libs.log4jCore
- optional libs.log4jSlf4jImpl
+ implementation libs.log4jApi
+ implementation libs.log4jCore
+ implementation libs.log4jSlf4jImpl
- optional libs.jsr305
- optional libs.yetusAnnotations
+ compileOnly libs.jsr305
+ compileOnly libs.yetusAnnotations
+
+ testImplementation libs.jetty
+ testImplementation libs.jettyServlet
+}
+
+shadowJar {
+ dependencies {
+ exclude(dependency("log4j:log4j:.*"))
+ exclude(dependency(libs.jsr305))
+ exclude(dependency(libs.log4jApi))
+ exclude(dependency(libs.log4jCore))
+ exclude(dependency(libs.log4jSlf4jImpl))
+ exclude(dependency(libs.junit))
+ exclude(dependency("org.jacoco::.*"))
+ }
- testCompile libs.jetty
- testCompile libs.jettyServlet
+ minimize()
}
// kudu-test-utils has no public Javadoc.
diff --git a/src/kudu/hms/CMakeLists.txt b/src/kudu/hms/CMakeLists.txt
index f777e007d..cf84f7ca2 100644
--- a/src/kudu/hms/CMakeLists.txt
+++ b/src/kudu/hms/CMakeLists.txt
@@ -66,7 +66,7 @@ execute_process(COMMAND ln -nsf
set(HMS_PLUGIN_JAR ${EXECUTABLE_OUTPUT_PATH}/hms-plugin.jar)
add_custom_command(OUTPUT ${HMS_PLUGIN_JAR}
- COMMAND ./gradlew :kudu-hive:jar ${GRADLE_FLAGS}
+ COMMAND ./gradlew :kudu-hive:assemble ${GRADLE_FLAGS}
COMMAND cp -f
"${JAVA_DIR}/kudu-hive/build/libs/kudu-hive-${KUDU_VERSION_NUMBER}.jar"
"${HMS_PLUGIN_JAR}"
diff --git a/src/kudu/scripts/assign-location.py
b/src/kudu/scripts/assign-location.py
old mode 100644
new mode 100755
