This is an automated email from the ASF dual-hosted git repository.
rzo1 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/storm.git
The following commit(s) were added to refs/heads/master by this push:
new 250f8d773 Reduce binary distribution size: unbundle optional
Hadoop/Kafka deps + de-duplicate shared jars (#8819)
250f8d773 is described below
commit 250f8d773aef95b4cbbb99ef0f47a3f4452d58a2
Author: Richard Zowalla <[email protected]>
AuthorDate: Wed Jul 15 20:16:25 2026 +0200
Reduce binary distribution size: unbundle optional Hadoop/Kafka deps +
de-duplicate shared jars (#8819)
* build: stop bundling storm-autocreds in the binary distribution
storm-autocreds pulls in the full Hadoop/HBase client dependency tree
(~79 MB, 43 jars unique to it) but is only needed on secure (Kerberos)
clusters and is off by default. Ship only the README, consistent with
the other external/* connectors, and add bin/storm-autocreds-fetch to
retrieve the plugin and its runtime dependencies from Maven Central into
extlib-daemon on demand.
Also removes the now-unused storm-autocreds-bin assembly module.
* build: stop bundling storm-kafka-monitor in the binary distribution
The storm-kafka-monitor jars (and their Kafka client dependencies, ~38 MB)
are only needed to display Kafka spout lag in the UI or to run the
bin/storm-kafka-monitor command. Ship only the README, consistent with the
other external/* connectors, and add bin/storm-kafka-monitor-fetch to
retrieve the tool and its runtime dependencies from Maven Central into
lib-tools/storm-kafka-monitor on demand.
Guard the UI against the jars being absent: TopologySpoutLag now detects
whether storm-kafka-monitor is installed and, when it is not, surfaces an
actionable message (and logs it once) instead of failing the lag shell-out.
The bin/storm-kafka-monitor wrapper prints the same hint instead of a
ClassNotFound error.
Also removes the now-unused storm-kafka-monitor-bin assembly module.
* build: add lib-common to the daemon and worker classpaths
Prepares de-duplication of the jars shared by the daemon (lib) and worker
(lib-worker) classpaths into a single lib-common directory. storm.py now
includes lib-common on both classpaths; when the directory is absent (older
layouts) it contributes nothing, so the change is backward compatible.
* build: add dedup-libs script to share daemon/worker jars via lib-common
The worker classpath (lib-worker) is a byte-identical subset of the daemon
classpath (lib). dedup-libs.py moves the shared jars into a single
lib-common
directory and removes the duplicate copies from lib, reclaiming ~71 MB. It
only
de-duplicates byte-identical jars (same name and sha-256), so a version
mismatch is never silently merged; tool classpaths (lib-tools/*, lib-webapp)
are left untouched.
Paired with the lib-common classpath support in bin/storm.py. Wiring this
into
the binary assembly is a follow-up (it must be validated by a full -Pdist
distribution build).
* build: wire lib-common de-duplication into the binary assembly
final-package now stages the daemon jars (copy-dependencies -> staging/lib)
and
the shared jars (storm-client-bin's lib-worker -> staging/lib-common), runs
dedup-libs.py to remove the byte-identical copies from lib, and the assembly
packages the staged lib/ and lib-common/ directories. The storm-client-bin
tree
(which only carried lib-worker) is no longer copied directly.
Verified through the prepare-package phase: staging/lib = 48 jars, staging/
lib-common = 40 jars, zero overlap, full 88-jar daemon set preserved across
lib + lib-common, ~71 MB reclaimed. The final tar/zip packaging requires a
full
-Pdist (native) distribution build and must be validated in CI / on Linux.
* test: add lib-common to expected classpaths in storm CLI tests
The binary distribution now de-duplicates jars shared by the daemon and
worker classpaths into a lib-common directory, which bin/storm.py adds to
the classpath after the storm home wildcard. Update the expected classpath
assertions in test_storm_cli.py to include lib-common.
* build: add a lean "storm-lite" distribution alongside the full one
The full distribution keeps bundling storm-autocreds (Hadoop/HBase) and
storm-kafka-monitor (Kafka client) so existing and air-gapped deployments
are
unaffected. A second "storm-lite" artifact omits those jars and ships only
their
READMEs, for users who do not need them and want a much smaller download.
The shared assembly content moves into a common.xml component descriptor, so
binary.xml (full) and binary-lite.xml differ only in the optional plugin
jars.
The lite artifact is attached with the "lite" Maven classifier and is
signed by
the existing GPG plugin. Both distributions keep the lib-common
de-duplication.
Measured (3.0.0-SNAPSHOT, tar.gz): baseline 367 MB, full now 303 MB (-64 MB
via
lib-common), lite 196 MB (-171 MB, ~47%).
RELEASING.md documents the new artifact and the extra signing/checksum
steps.
* build: keep tooling on the full distribution now that storm-lite is built
The binary build produces two zips/tarballs, which broke consumers that
assumed
exactly one artifact:
- integration-test/run-it.sh failed its "expected exactly one zip file"
guard.
- integration-test/config/Vagrantfile raised "Expected one storm-binary".
Both now ignore apache-storm-*-lite.zip and run against the full
distribution.
The cluster Dockerfile ADDs the full tarball by name, so its .dockerignore
also
excludes the lite tarball to keep it out of the build context.
* build: fail dedup-libs on same-name/different-sha jars, add test coverage
Address review feedback: detect every same-name/different-sha-256 conflict
between the daemon (lib) and worker (lib-common/lib-worker) jars in a
read-only pass and fail the build before any file is moved or removed, so
a diverged jar can never silently shadow its counterpart on the daemon
classpath. The summary now also reports the number of worker-only jars
promoted to lib-common (expected 0) as a drift canary.
Add unit tests for dedup-libs.py, run on every PR/push via a workflow step
and during dist builds via a unix_py_tests profile mirroring storm-client.
* build: catch version drift in dedup-libs by comparing artifact base names
The jars are maven artifacts named <artifactId>-<version>.jar, so a version
bump changes the file name and the same-name/different-sha guard alone would
let e.g. lib/foo-1.0.jar and lib-worker/foo-1.1.jar pass as unrelated jars,
promoting the worker copy onto the daemon classpath next to the lib copy.
Compare jars by their version-stripped artifact base name (the part before
the first "-<digit>") as well and fail the build when the same artifact is
present with different versions; only artifacts absent from lib entirely are
counted as worker-only.
---
.github/workflows/maven.yaml | 2 +
RELEASING.md | 31 ++
bin/storm-autocreds-fetch | 134 +++++++++
bin/storm-kafka-monitor | 9 +-
bin/storm-kafka-monitor-fetch | 133 +++++++++
bin/storm.py | 11 +-
bin/test_storm.py | 13 +
dev-tools/cluster/Dockerfile.dockerignore | 3 +
external/storm-autocreds/README.md | 112 ++++++++
external/storm-kafka-monitor/README.md | 30 ++
integration-test/config/Vagrantfile | 4 +-
integration-test/run-it.sh | 6 +-
storm-client/test/py/test_storm_cli.py | 50 ++--
.../org/apache/storm/utils/TopologySpoutLag.java | 37 ++-
storm-dist/binary/final-package/pom.xml | 136 ++++++++-
.../src/main/assembly/binary-lite.xml | 63 ++++
.../final-package/src/main/assembly/binary.xml | 320 ++-------------------
.../src/main/assembly/{binary.xml => common.xml} | 57 ++--
.../final-package/src/main/scripts/dedup-libs.py | 211 ++++++++++++++
.../src/main/scripts/test_dedup_libs.py | 265 +++++++++++++++++
20 files changed, 1253 insertions(+), 374 deletions(-)
diff --git a/.github/workflows/maven.yaml b/.github/workflows/maven.yaml
index 1f407ccbb..d493796cf 100644
--- a/.github/workflows/maven.yaml
+++ b/.github/workflows/maven.yaml
@@ -43,6 +43,8 @@ jobs:
with:
distribution: temurin
java-version: 25
+ - name: Run distribution packaging script tests
+ run: python3 -B
storm-dist/binary/final-package/src/main/scripts/test_dedup_libs.py
- name: Ensure a clean state without storm artifacts
run: rm -rf ~/.m2/repository/org/apache/storm
- name: Build project (compile + install, skip tests)
diff --git a/RELEASING.md b/RELEASING.md
index 270437cdd..bcfe91bae 100644
--- a/RELEASING.md
+++ b/RELEASING.md
@@ -101,9 +101,17 @@ In this way, you create a new release line and then you
can create PATCH version
pushd storm-dist/binary/final-package/target
sha512sum apache-storm-2.8.1.zip > apache-storm-2.8.1.zip.sha512
sha512sum apache-storm-2.8.1.tar.gz > apache-storm-2.8.1.tar.gz.sha512
+ sha512sum apache-storm-2.8.1-lite.zip > apache-storm-2.8.1-lite.zip.sha512
+ sha512sum apache-storm-2.8.1-lite.tar.gz >
apache-storm-2.8.1-lite.tar.gz.sha512
popd
```
+ > **Note:** the binary build produces two distributions. The full one
+ > (`apache-storm-x.x.x.tar.gz` / `.zip`) bundles the optional
`storm-autocreds` and
+ > `storm-kafka-monitor` jars, while the lean one
(`apache-storm-x.x.x-lite.tar.gz` / `.zip`)
+ > omits them. Both must be signed, checksummed and staged for the release
candidate.
+ > See [Storm lite distribution](#storm-lite-distribution) below.
+
6. Create a directory in the dist svn repo for the release candidate:
https://dist.apache.org/repos/dist/dev/storm/apache-storm-x.x.x-rcx
7. Before generating the release notes, please double check if all merged pull
requests for the version being released are assigned the milestone in question.
They won't be placed in the release notes otherwise.
@@ -135,8 +143,31 @@ In this way, you create a new release line and then you
can create PATCH version
apache-storm-2.8.3-src.tar.gz apache-storm-2.8.3-src.zip
apache-storm-2.8.3.tar.gz apache-storm-2.8.3.zip
RELEASE_NOTES.html
apache-storm-2.8.3-src.tar.gz.asc apache-storm-2.8.3-src.zip.asc
apache-storm-2.8.3.tar.gz.asc apache-storm-2.8.3.zip.asc
RELEASE_NOTES.html.asc
apache-storm-2.8.3-src.tar.gz.sha512 apache-storm-2.8.3-src.zip.sha512
apache-storm-2.8.3.tar.gz.sha512 apache-storm-2.8.3.zip.sha512
RELEASE_NOTES.html.sha512
+ apache-storm-2.8.3-lite.tar.gz apache-storm-2.8.3-lite.zip
+ apache-storm-2.8.3-lite.tar.gz.asc apache-storm-2.8.3-lite.zip.asc
+ apache-storm-2.8.3-lite.tar.gz.sha512 apache-storm-2.8.3-lite.zip.sha512
```
+## Storm lite distribution
+
+Since 3.0.0 the binary build produces two distributions from
`storm-dist/binary`:
+
+| Artifact | Contents |
+|---|---|
+| `apache-storm-x.x.x.tar.gz` / `.zip` | Full distribution. Bundles the
`storm-autocreds` (Hadoop/HBase) and `storm-kafka-monitor` (Kafka client) jars.
|
+| `apache-storm-x.x.x-lite.tar.gz` / `.zip` | Lean distribution. Ships only
the READMEs for those two plugins; everything else is identical. |
+
+Both are produced by the same `mvn package` run in `storm-dist/binary`
(assembly
+executions `bin` and `lite`), and both are automatically signed by the GPG
plugin.
+The lite artifact is attached with the `lite` Maven classifier.
+
+Release managers must sign, checksum and stage **both** distributions, and list
+both in the release candidate VOTE mail so reviewers know which to verify.
+
+Users of the lite distribution can install the omitted plugins on demand with
+`bin/storm-autocreds-fetch` and `bin/storm-kafka-monitor-fetch`, which resolve
them
+from Maven Central (or an internal mirror configured in `settings.xml`).
+
10. Add and commit the files to SVN. This makes them available in the Apache
staging repo.
```bash
diff --git a/bin/storm-autocreds-fetch b/bin/storm-autocreds-fetch
new file mode 100755
index 000000000..aabb98adb
--- /dev/null
+++ b/bin/storm-autocreds-fetch
@@ -0,0 +1,134 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Fetch the storm-autocreds plugin and its (Hadoop/HBase) runtime dependencies
+# into the daemon classpath, so Nimbus/Supervisor can populate and renew HDFS
+# and HBase delegation tokens on a secure (Kerberos) cluster.
+#
+# These jars ship only in the full binary distribution; the lite distribution
+# (apache-storm-x.x.x-lite.tar.gz) omits them because only secure-Hadoop
+# deployments need them. See external/storm-autocreds/README.md for details.
+
+set -euo pipefail
+
+usage() {
+ cat <<'EOF'
+Usage: storm-autocreds-fetch [options] [-- <extra maven args>]
+
+Resolves org.apache.storm:storm-autocreds and its runtime dependencies from a
+Maven repository (Maven Central by default) and copies them into the Storm
+daemon classpath directory (extlib-daemon).
+
+Options:
+ --version <ver> Storm version to fetch (default: read from
$STORM_HOME/RELEASE)
+ --dest <dir> Target directory (default: $STORM_HOME/extlib-daemon)
+ -h, --help Show this help
+
+Any arguments after "--" are passed through to Maven, e.g. to use an internal
+mirror or an offline local repository:
+ storm-autocreds-fetch -- -s /path/settings.xml
+ storm-autocreds-fetch -- -Dmaven.repo.local=/path/to/offline-repo -o
+EOF
+}
+
+# Resolve symlinks so STORM_HOME is correct even when invoked via a link.
+PRG="${0}"
+while [ -h "${PRG}" ]; do
+ ls=$(ls -ld "${PRG}")
+ link=$(expr "${ls}" : '.*-> \(.*\)$')
+ if expr "${link}" : '/.*' > /dev/null; then
+ PRG="${link}"
+ else
+ PRG="$(dirname "${PRG}")/${link}"
+ fi
+done
+STORM_BIN_DIR=$(dirname "${PRG}")
+STORM_HOME=$(cd "${STORM_BIN_DIR}/.." && pwd)
+
+VERSION=""
+DEST=""
+MVN_ARGS=()
+while [ $# -gt 0 ]; do
+ case "${1}" in
+ --version) VERSION="${2}"; shift 2 ;;
+ --dest) DEST="${2}"; shift 2 ;;
+ -h|--help) usage; exit 0 ;;
+ --) shift; MVN_ARGS=("$@"); break ;;
+ *) echo "Unknown option: ${1}" >&2; usage; exit 1 ;;
+ esac
+done
+
+if [ -z "${VERSION}" ]; then
+ if [ -f "${STORM_HOME}/RELEASE" ]; then
+ VERSION=$(tr -d '[:space:]' < "${STORM_HOME}/RELEASE")
+ fi
+fi
+if [ -z "${VERSION}" ]; then
+ echo "Error: could not determine Storm version. Pass --version <ver>." >&2
+ exit 1
+fi
+
+if [ -z "${DEST}" ]; then
+ DEST="${STORM_HOME}/extlib-daemon"
+fi
+
+MVN="${MAVEN_HOME:+${MAVEN_HOME}/bin/}mvn"
+if ! command -v "${MVN}" > /dev/null 2>&1; then
+ echo "Error: '${MVN}' not found on PATH. Install Apache Maven or set
MAVEN_HOME." >&2
+ exit 1
+fi
+
+mkdir -p "${DEST}"
+
+# Use a throwaway POM that depends on storm-autocreds; copy-dependencies then
+# pulls the exact runtime closure (honoring the exclusions declared in the
+# published storm-autocreds POM). storm-client is 'provided' there and is
+# correctly skipped, since it already ships in lib/.
+TMP_DIR=$(mktemp -d)
+trap 'rm -rf "${TMP_DIR}"' EXIT
+cat > "${TMP_DIR}/pom.xml" <<EOF
+<project xmlns="http://maven.apache.org/POM/4.0.0";>
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>org.apache.storm.tools</groupId>
+ <artifactId>storm-autocreds-fetch</artifactId>
+ <version>${VERSION}</version>
+ <packaging>pom</packaging>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.storm</groupId>
+ <artifactId>storm-autocreds</artifactId>
+ <version>${VERSION}</version>
+ </dependency>
+ </dependencies>
+</project>
+EOF
+
+echo "Fetching org.apache.storm:storm-autocreds:${VERSION} (runtime closure)
into:"
+echo " ${DEST}"
+"${MVN}" -q -f "${TMP_DIR}/pom.xml" \
+ org.apache.maven.plugins:maven-dependency-plugin:copy-dependencies \
+ -DincludeScope=runtime \
+ -DoutputDirectory="${DEST}" \
+ ${MVN_ARGS[@]+"${MVN_ARGS[@]}"}
+
+echo "Done. ${DEST} now contains:"
+ls -1 "${DEST}" | sed 's/^/ /'
+echo
+echo "Restart the Storm daemons (Nimbus, Supervisor) to pick up the new
classpath,"
+echo "then configure the autocreds plugins in storm.yaml. See"
+echo "external/storm-autocreds/README.md for the required settings."
diff --git a/bin/storm-kafka-monitor b/bin/storm-kafka-monitor
index 9bd11054c..8f2d6b109 100755
--- a/bin/storm-kafka-monitor
+++ b/bin/storm-kafka-monitor
@@ -49,4 +49,11 @@ if [ -z "$JAVA_HOME" ]; then
else
JAVA="$JAVA_HOME/bin/java"
fi
-exec $JAVA $STORM_JAAS_CONF_PARAM $STORM_JAR_JVM_OPTS -cp
"$STORM_BASE_DIR/lib-tools/storm-kafka-monitor/*"
org.apache.storm.kafka.monitor.KafkaOffsetLagUtil "$@"
+# The storm-kafka-monitor jars ship only in the full distribution; on the lite
one they are fetched on demand.
+KAFKA_MONITOR_LIB="$STORM_BASE_DIR/lib-tools/storm-kafka-monitor"
+if ! ls "$KAFKA_MONITOR_LIB"/*.jar >/dev/null 2>&1; then
+ echo "storm-kafka-monitor is not installed (no jars in $KAFKA_MONITOR_LIB)."
>&2
+ echo "Run '$STORM_BIN_DIR/storm-kafka-monitor-fetch' to download it, then
retry." >&2
+ exit 1
+fi
+exec $JAVA $STORM_JAAS_CONF_PARAM $STORM_JAR_JVM_OPTS -cp
"$KAFKA_MONITOR_LIB/*" org.apache.storm.kafka.monitor.KafkaOffsetLagUtil "$@"
diff --git a/bin/storm-kafka-monitor-fetch b/bin/storm-kafka-monitor-fetch
new file mode 100755
index 000000000..43dd90221
--- /dev/null
+++ b/bin/storm-kafka-monitor-fetch
@@ -0,0 +1,133 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Fetch the storm-kafka-monitor tool and its (Kafka client) runtime
dependencies
+# into lib-tools/storm-kafka-monitor, enabling the "Kafka spout lag" display in
+# the Storm UI and the bin/storm-kafka-monitor command.
+#
+# These jars ship only in the full binary distribution; the lite distribution
+# (apache-storm-x.x.x-lite.tar.gz) omits them because they are only needed when
+# running Kafka spouts and wanting lag info. The UI degrades gracefully when
they
+# are absent. See external/storm-kafka-monitor/README.md for details.
+
+set -euo pipefail
+
+usage() {
+ cat <<'EOF'
+Usage: storm-kafka-monitor-fetch [options] [-- <extra maven args>]
+
+Resolves org.apache.storm:storm-kafka-monitor and its runtime dependencies from
+a Maven repository (Maven Central by default) and copies them into
+lib-tools/storm-kafka-monitor.
+
+Options:
+ --version <ver> Storm version to fetch (default: read from
$STORM_HOME/RELEASE)
+ --dest <dir> Target directory (default:
$STORM_HOME/lib-tools/storm-kafka-monitor)
+ -h, --help Show this help
+
+Any arguments after "--" are passed through to Maven, e.g. to use an internal
+mirror or an offline local repository:
+ storm-kafka-monitor-fetch -- -s /path/settings.xml
+ storm-kafka-monitor-fetch -- -Dmaven.repo.local=/path/to/offline-repo -o
+EOF
+}
+
+# Resolve symlinks so STORM_HOME is correct even when invoked via a link.
+PRG="${0}"
+while [ -h "${PRG}" ]; do
+ ls=$(ls -ld "${PRG}")
+ link=$(expr "${ls}" : '.*-> \(.*\)$')
+ if expr "${link}" : '/.*' > /dev/null; then
+ PRG="${link}"
+ else
+ PRG="$(dirname "${PRG}")/${link}"
+ fi
+done
+STORM_BIN_DIR=$(dirname "${PRG}")
+STORM_HOME=$(cd "${STORM_BIN_DIR}/.." && pwd)
+
+VERSION=""
+DEST=""
+MVN_ARGS=()
+while [ $# -gt 0 ]; do
+ case "${1}" in
+ --version) VERSION="${2}"; shift 2 ;;
+ --dest) DEST="${2}"; shift 2 ;;
+ -h|--help) usage; exit 0 ;;
+ --) shift; MVN_ARGS=("$@"); break ;;
+ *) echo "Unknown option: ${1}" >&2; usage; exit 1 ;;
+ esac
+done
+
+if [ -z "${VERSION}" ]; then
+ if [ -f "${STORM_HOME}/RELEASE" ]; then
+ VERSION=$(tr -d '[:space:]' < "${STORM_HOME}/RELEASE")
+ fi
+fi
+if [ -z "${VERSION}" ]; then
+ echo "Error: could not determine Storm version. Pass --version <ver>." >&2
+ exit 1
+fi
+
+if [ -z "${DEST}" ]; then
+ DEST="${STORM_HOME}/lib-tools/storm-kafka-monitor"
+fi
+
+MVN="${MAVEN_HOME:+${MAVEN_HOME}/bin/}mvn"
+if ! command -v "${MVN}" > /dev/null 2>&1; then
+ echo "Error: '${MVN}' not found on PATH. Install Apache Maven or set
MAVEN_HOME." >&2
+ exit 1
+fi
+
+mkdir -p "${DEST}"
+
+# Use a throwaway POM that depends on storm-kafka-monitor; copy-dependencies
then
+# pulls the exact runtime closure. The artifact itself is a direct dependency
and
+# is therefore copied too.
+TMP_DIR=$(mktemp -d)
+trap 'rm -rf "${TMP_DIR}"' EXIT
+cat > "${TMP_DIR}/pom.xml" <<EOF
+<project xmlns="http://maven.apache.org/POM/4.0.0";>
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>org.apache.storm.tools</groupId>
+ <artifactId>storm-kafka-monitor-fetch</artifactId>
+ <version>${VERSION}</version>
+ <packaging>pom</packaging>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.storm</groupId>
+ <artifactId>storm-kafka-monitor</artifactId>
+ <version>${VERSION}</version>
+ </dependency>
+ </dependencies>
+</project>
+EOF
+
+echo "Fetching org.apache.storm:storm-kafka-monitor:${VERSION} (runtime
closure) into:"
+echo " ${DEST}"
+"${MVN}" -q -f "${TMP_DIR}/pom.xml" \
+ org.apache.maven.plugins:maven-dependency-plugin:copy-dependencies \
+ -DincludeScope=runtime \
+ -DoutputDirectory="${DEST}" \
+ ${MVN_ARGS[@]+"${MVN_ARGS[@]}"}
+
+echo "Done. ${DEST} now contains:"
+ls -1 "${DEST}" | sed 's/^/ /'
+echo
+echo "Restart the Storm UI to enable Kafka spout lag display, or run"
+echo "bin/storm-kafka-monitor directly. See
external/storm-kafka-monitor/README.md."
diff --git a/bin/storm.py b/bin/storm.py
index 81d6e4e4d..eb1afa2f8 100755
--- a/bin/storm.py
+++ b/bin/storm.py
@@ -107,6 +107,7 @@ def confvalue(name, storm_config_opts, extrapaths,
overriding_conf_file=None, da
def get_classpath(extrajars, daemon=True, client=False):
ret = get_wildcard_dir(STORM_DIR)
+ ret.extend(get_wildcard_dir(STORM_COMMON_LIB_DIR))
if client:
ret.extend(get_wildcard_dir(STORM_WORKER_LIB_DIR))
else:
@@ -125,9 +126,9 @@ def get_classpath(extrajars, daemon=True, client=False):
def init_storm_env(within_unittest=False):
global NORMAL_CLASS_PATH, STORM_DIR, USER_CONF_DIR, STORM_CONF_DIR,
STORM_WORKER_LIB_DIR, STORM_LIB_DIR,\
- STORM_TOOLS_LIB_DIR, STORM_WEBAPP_LIB_DIR, STORM_BIN_DIR,
STORM_LOG4J2_CONF_DIR, STORM_SUPERVISOR_LOG_FILE,\
- CLUSTER_CONF_DIR, JAR_JVM_OPTS, JAVA_HOME, JAVA_CMD, CONF_FILE,
STORM_EXT_CLASSPATH, \
- STORM_EXT_CLASSPATH_DAEMON, LOCAL_TTL_DEFAULT
+ STORM_COMMON_LIB_DIR, STORM_TOOLS_LIB_DIR, STORM_WEBAPP_LIB_DIR,
STORM_BIN_DIR, STORM_LOG4J2_CONF_DIR,\
+ STORM_SUPERVISOR_LOG_FILE, CLUSTER_CONF_DIR, JAR_JVM_OPTS, JAVA_HOME,
JAVA_CMD, CONF_FILE, \
+ STORM_EXT_CLASSPATH, STORM_EXT_CLASSPATH_DAEMON, LOCAL_TTL_DEFAULT
NORMAL_CLASS_PATH = cygpath if sys.platform == 'cygwin' else identity
STORM_DIR = os.sep.join(os.path.realpath( __file__ ).split(os.sep)[:-2])
@@ -141,6 +142,10 @@ def init_storm_env(within_unittest=False):
STORM_WORKER_LIB_DIR = os.path.join(STORM_DIR, "lib-worker")
STORM_LIB_DIR = os.path.join(STORM_DIR, "lib")
+ # Jars shared by the daemon (lib) and worker (lib-worker) classpaths are
de-duplicated into
+ # lib-common to keep the distribution small. It is added to both
classpaths; absent in older
+ # layouts, in which case it contributes nothing.
+ STORM_COMMON_LIB_DIR = os.path.join(STORM_DIR, "lib-common")
STORM_TOOLS_LIB_DIR = os.path.join(STORM_DIR, "lib-tools")
STORM_WEBAPP_LIB_DIR = os.path.join(STORM_DIR, "lib-webapp")
diff --git a/bin/test_storm.py b/bin/test_storm.py
index 11c8057b0..a44669919 100644
--- a/bin/test_storm.py
+++ b/bin/test_storm.py
@@ -66,6 +66,19 @@ class Test(unittest.TestCase):
expected = ":".join(extrajars)
self.assertEqual(s[-len(expected):], expected)
+ def test_get_classpath_includes_lib_common(self):
+ extrajars = []
+ # When lib-common exists, it is included on both the daemon and the
client classpaths.
+ storm.STORM_COMMON_LIB_DIR = storm.STORM_BIN_DIR
+ expected = os.path.join(storm.STORM_BIN_DIR, "*")
+ for client in (True, False):
+ cp = storm.get_classpath(extrajars, daemon=True, client=client)
+ self.assertIn(expected, cp.split(os.pathsep))
+ # When it does not exist, it contributes nothing (backward compatible
with older layouts).
+ storm.STORM_COMMON_LIB_DIR = os.path.join(storm.STORM_DIR,
"no-such-lib-common")
+ cp = storm.get_classpath(extrajars, daemon=True, client=False)
+ self.assertNotIn(os.path.join(storm.STORM_COMMON_LIB_DIR, "*"),
cp.split(os.pathsep))
+
def test_resolve_dependencies(self):
artifacts = "org.apache.commons.commons-api"
artifact_repositories = "maven-central"
diff --git a/dev-tools/cluster/Dockerfile.dockerignore
b/dev-tools/cluster/Dockerfile.dockerignore
index a12edc515..0780dbbc1 100644
--- a/dev-tools/cluster/Dockerfile.dockerignore
+++ b/dev-tools/cluster/Dockerfile.dockerignore
@@ -20,3 +20,6 @@
# (which also holds the .zip and the extracted distribution).
*
!apache-storm-*.tar.gz
+# The build ADDs the full distribution; keep the lean "storm-lite" tarball out
of
+# the context so it does not add ~200 MB for nothing.
+apache-storm-*-lite.tar.gz
diff --git a/external/storm-autocreds/README.md
b/external/storm-autocreds/README.md
new file mode 100644
index 000000000..459972434
--- /dev/null
+++ b/external/storm-autocreds/README.md
@@ -0,0 +1,112 @@
+# Storm Auto Credentials (HDFS / HBase)
+
+`storm-autocreds` lets Storm automatically acquire, distribute and renew
+**Hadoop delegation tokens** so that topologies can talk to a secure (Kerberos)
+HDFS or HBase cluster without distributing keytabs to every worker host.
+
+* On topology submission, **Nimbus** obtains delegation tokens on behalf of the
+ submitting user and ships them with the topology.
+* **Workers** unpack the tokens into their `Subject` / `UserGroupInformation`.
+* **Nimbus** periodically renews the tokens for long-running topologies.
+
+See `docs/SECURITY.md` ("Automatic Credentials Push and Renewal") for the full
+design.
+
+## Which distribution has the jars?
+
+These plugins run on the **daemon** classpath (Nimbus/Supervisor) and pull in
the
+full Hadoop and HBase client dependency trees, which is a large amount of
weight
+that only secure-Hadoop deployments need. Therefore:
+
+| Distribution | storm-autocreds jars |
+|---|---|
+| `apache-storm-x.x.x.tar.gz` (full) | bundled under
`external/storm-autocreds` |
+| `apache-storm-x.x.x-lite.tar.gz` (lite) | not bundled, README only |
+
+If you use the **lite** distribution, install the jars with the helper script
+below (or just use the full distribution).
+
+## Installing
+
+The plugins must be present on the **daemon** classpath, i.e. in
+`$STORM_HOME/extlib-daemon` on Nimbus and the Supervisors. With the full
+distribution, copy them from `external/storm-autocreds`:
+
+```bash
+cp $STORM_HOME/external/storm-autocreds/*.jar $STORM_HOME/extlib-daemon/
+```
+
+### Option 1 — use the helper script (recommended for the lite distribution)
+
+The distribution ships a helper that resolves `storm-autocreds` and its runtime
+dependencies from Maven Central and copies them into `extlib-daemon`:
+
+```bash
+$STORM_HOME/bin/storm-autocreds-fetch
+```
+
+It detects the Storm version from `$STORM_HOME/RELEASE`. Useful options:
+
+```bash
+# explicit version / target directory
+bin/storm-autocreds-fetch --version 3.0.0 --dest /opt/storm/extlib-daemon
+
+# pass extra arguments through to Maven (internal mirror / offline repo)
+bin/storm-autocreds-fetch -- -s /etc/maven/settings.xml
+bin/storm-autocreds-fetch -- -Dmaven.repo.local=/srv/offline-repo -o
+```
+
+Maven must be available on the host running the script (it does not have to be
+installed on the cluster nodes — you can run it once and copy the resulting
jars
+to every daemon host).
+
+### Option 2 — build from source
+
+```bash
+mvn -pl external/storm-autocreds -am package
+cp external/storm-autocreds/target/storm-autocreds-*.jar \
+ $(find ~/.m2 -name 'hadoop-auth-*.jar' -o -name 'hbase-client-*.jar') \
+ $STORM_HOME/extlib-daemon/
+```
+
+(Prefer Option 1 — it resolves the complete, correct dependency closure for
you.)
+
+Restart Nimbus and the Supervisors after adding the jars so the new classpath
+takes effect.
+
+## Configuring
+
+Add the following to `storm.yaml`. The `*Nimbus` classes run on Nimbus (acquire
+and renew tokens); the non-`Nimbus` classes run in the worker (unpack tokens).
+
+```yaml
+# Worker side: unpack the tokens into the worker Subject.
+topology.auto-credentials:
+ - org.apache.storm.hdfs.security.AutoHDFS
+ - org.apache.storm.hbase.security.AutoHBase
+
+# Nimbus side: obtain the tokens on behalf of the submitter.
+nimbus.autocredential.plugins.classes:
+ - org.apache.storm.hdfs.security.AutoHDFSNimbus
+ - org.apache.storm.hbase.security.AutoHBaseNimbus
+
+# Nimbus side: renew the tokens for long-running topologies.
+nimbus.credential.renewers.classes:
+ - org.apache.storm.hdfs.security.AutoHDFSNimbus
+ - org.apache.storm.hbase.security.AutoHBaseNimbus
+```
+
+Relevant credential settings:
+
+| Setting | Purpose |
+|---|---|
+| `hdfs.keytab.file` / `hdfs.kerberos.principal` | Nimbus principal used to
fetch HDFS tokens |
+| `hdfs.kerberos.principal` | HDFS service principal |
+| `hbase.keytab.file` / `hbase.kerberos.principal` | Nimbus principal used to
fetch HBase tokens |
+| `topology.hdfs.uri` | NameNode URI (defaults to the cluster `fs.defaultFS`) |
+| `hdfsCredentialsConfigKeys` / `hbaseCredentialsConfigKeys` | optional list
of per-cluster config keys when talking to multiple clusters |
+
+Use only the HDFS or only the HBase entries if you need just one of them.
+
+For the full secure-cluster setup (Kerberos, impersonation, ACLs) see
+`docs/SECURITY.md`.
diff --git a/external/storm-kafka-monitor/README.md
b/external/storm-kafka-monitor/README.md
index a483f4bde..24da7a9b8 100644
--- a/external/storm-kafka-monitor/README.md
+++ b/external/storm-kafka-monitor/README.md
@@ -2,6 +2,36 @@
Tool to query kafka spout lags and show in Storm UI
+## Installation
+
+The storm-kafka-monitor jars (and their Kafka client dependencies) are only
+needed to display Kafka spout lag in the UI or to run the `storm-kafka-monitor`
+command, so they are bundled only in the full distribution:
+
+| Distribution | storm-kafka-monitor jars |
+|---|---|
+| `apache-storm-x.x.x.tar.gz` (full) | bundled under
`lib-tools/storm-kafka-monitor`, works out of the box |
+| `apache-storm-x.x.x-lite.tar.gz` (lite) | not bundled, README only |
+
+The Storm UI degrades gracefully when they are absent (no lag is shown and a
+hint is logged once).
+
+With the **lite** distribution, install the jars on the UI host with the helper
+script, then restart the UI:
+
+```bash
+$STORM_HOME/bin/storm-kafka-monitor-fetch
+```
+
+It resolves `org.apache.storm:storm-kafka-monitor` and its runtime dependencies
+from Maven Central into `lib-tools/storm-kafka-monitor`. Useful options:
+
+```bash
+bin/storm-kafka-monitor-fetch --version 3.0.0
+# pass extra arguments through to Maven (internal mirror / offline repo)
+bin/storm-kafka-monitor-fetch -- -s /etc/maven/settings.xml
+```
+
## Usage
This tool provides a way to query kafka offsets that the spout has consumed
successfully and the latest
offsets in kafka. It provides an easy way to see how the topology is
performing. It is a command line
diff --git a/integration-test/config/Vagrantfile
b/integration-test/config/Vagrantfile
index ab6e77489..f09721190 100644
--- a/integration-test/config/Vagrantfile
+++ b/integration-test/config/Vagrantfile
@@ -20,7 +20,9 @@ require 'uri'
# Vagrantfile API/syntax version. Don't touch unless you know what you're
doing!
VAGRANTFILE_API_VERSION = "2"
STORM_BOX_TYPE = "ubuntu/xenial64"
-STORM_ZIP = Dir.glob("../../storm-dist/binary/**/*.zip")
+# The binary build also produces the lean "storm-lite" distribution; the
integration
+# tests run against the full one, so ignore apache-storm-*-lite.zip here.
+STORM_ZIP = Dir.glob("../../storm-dist/binary/**/*.zip").reject { |f|
f.end_with?("-lite.zip") }
if(STORM_ZIP.length != 1)
raise "Expected one storm-binary found: " + STORM_ZIP.join(",") + ". Did you
run : cd ${STORM_SRC_DIR}/storm-dist/binary && mvn clean package
-Dgpg.skip=true"
end
diff --git a/integration-test/run-it.sh b/integration-test/run-it.sh
index 3b1f10e8b..a411456ff 100755
--- a/integration-test/run-it.sh
+++ b/integration-test/run-it.sh
@@ -62,10 +62,12 @@ else
if [[ "${USER}" == "github" ]]; then
( cd "${STORM_SRC_DIR}/storm-dist/binary" && mvn clean package
-Dgpg.skip=true )
fi
- (( $(find "${STORM_SRC_DIR}/storm-dist/binary" -iname 'apache-storm*.zip'
| wc -l) == 1 )) || die "expected exactly one zip file, did you run: cd
${STORM_SRC_DIR}/storm-dist/binary && mvn clean package -Dgpg.skip=true"
+ # The binary build also produces the lean "storm-lite" distribution; the
integration
+ # tests run against the full one, so ignore apache-storm-*-lite.zip here.
+ (( $(find "${STORM_SRC_DIR}/storm-dist/binary" -iname 'apache-storm*.zip'
-not -iname '*-lite.zip' | wc -l) == 1 )) || die "expected exactly one zip
file, did you run: cd ${STORM_SRC_DIR}/storm-dist/binary && mvn clean package
-Dgpg.skip=true"
fi
-storm_binary_zip=$(find "${STORM_SRC_DIR}/storm-dist" -iname '*.zip')
+storm_binary_zip=$(find "${STORM_SRC_DIR}/storm-dist" -iname
'apache-storm*.zip' -not -iname '*-lite.zip')
storm_binary_name=$(basename "${storm_binary_zip}")
export STORM_VERSION=$(grep -oPe '\d.*(?=.zip)' <<<"${storm_binary_name}")
echo "Using storm version:" ${STORM_VERSION}
diff --git a/storm-client/test/py/test_storm_cli.py
b/storm-client/test/py/test_storm_cli.py
index 2815de42f..8322897b0 100644
--- a/storm-client/test/py/test_storm_cli.py
+++ b/storm-client/test/py/test_storm_cli.py
@@ -62,7 +62,7 @@ class TestStormCli(TestCase):
self.java_cmd, '-client', '-Ddaemon.name=',
'-Dstorm.options=+topology.blobstore.map%3D%27%7B%22key1%22%3A%7B%22localname%22%3A%22blob_file%22%2C+%22uncompress%22%3Afalse%7D%2C%22key2%22%3A%7B%7D%7D%27',
'-Dstorm.home=' + self.storm_dir, '-Dstorm.log.dir=' +
self.storm_dir + "/logs",
'-Djava.library.path=', '-Dstorm.conf.file=', '-cp',
- self.storm_dir + '/*:' + self.storm_dir + '/lib-worker:' +
self.storm_dir
+ self.storm_dir + '/*:' + self.storm_dir + '/lib-common:' +
self.storm_dir + '/lib-worker:' + self.storm_dir
+
'/extlib:example/storm-starter/storm-starter-topologies-*.jar:' +
self.storm_dir + '/conf:'
+ self.storm_dir +
'/bin:./external/storm-redis/storm-redis-1.1.0.jar:./external/storm-kafka-client/storm-kafka-client-1.1.0.jar"',
'-Dstorm.jar=example/storm-starter/storm-starter-topologies-*.jar',
'-Dstorm.dependency.jars=./external/storm-redis/storm-redis-1.1.0.jar,./external/storm-kafka-client/storm-kafka-client-1.1.0.jar"',
'-Dstorm.dependency.artifacts={}',
'org.apache.storm.starter.RollingTopWords',
'blobstore-remote2', 'remote'
@@ -80,7 +80,7 @@ class TestStormCli(TestCase):
self.java_cmd, '-client', '-Ddaemon.name=', '-Dstorm.options=',
'-Dstorm.home=' + self.storm_dir, '-Dstorm.log.dir=' +
self.storm_dir + "/logs",
'-Djava.library.path=', '-Dstorm.conf.file=', '-cp',
- self.storm_dir + '/*:' + self.storm_dir + '/lib-worker:' +
self.storm_dir
+ self.storm_dir + '/*:' + self.storm_dir + '/lib-common:' +
self.storm_dir + '/lib-worker:' + self.storm_dir
+ '/extlib:/path/to/jar.jar:' + self.storm_dir + '/conf:' +
self.storm_dir + '/bin:',
'-Dstorm.jar=/path/to/jar.jar', '-Dstorm.dependency.jars=',
'-Dstorm.dependency.artifacts={}',
'some.Topology.Class', '-name', 'run-topology',
'randomArgument', '-randomFlag', 'randomFlagValue',
@@ -93,7 +93,7 @@ class TestStormCli(TestCase):
self.base_test(
["storm", "localconfvalue", "conf_name"], self.mock_popen,
mock.call([
self.java_cmd, '-client', '-Dstorm.options=',
- '-Dstorm.conf.file=', '-cp', self.storm_dir + '/*:' +
self.storm_dir + '/lib:' + self.storm_dir +'/extlib:' + self.storm_dir +
'/extlib-daemon:' + self.storm_dir + '/conf',
+ '-Dstorm.conf.file=', '-cp', self.storm_dir + '/*:' +
self.storm_dir + '/lib-common:' + self.storm_dir + '/lib:' + self.storm_dir
+'/extlib:' + self.storm_dir + '/extlib-daemon:' + self.storm_dir + '/conf',
'org.apache.storm.command.ConfigValue', 'conf_name'
], stdout=-1
)
@@ -103,7 +103,7 @@ class TestStormCli(TestCase):
self.base_test(
["storm", "remoteconfvalue", "conf_name"], self.mock_popen,
mock.call([
self.java_cmd, '-client', '-Dstorm.options=',
- '-Dstorm.conf.file=', '-cp', self.storm_dir + '/*:' +
self.storm_dir + '/lib:' + self.storm_dir + '/extlib:' + self.storm_dir +
'/extlib-daemon:' + self.storm_dir + '/conf',
+ '-Dstorm.conf.file=', '-cp', self.storm_dir + '/*:' +
self.storm_dir + '/lib-common:' + self.storm_dir + '/lib:' + self.storm_dir +
'/extlib:' + self.storm_dir + '/extlib-daemon:' + self.storm_dir + '/conf',
'org.apache.storm.command.ConfigValue', 'conf_name'
], stdout=-1
)
@@ -124,7 +124,7 @@ class TestStormCli(TestCase):
self.java_cmd, '-client','-Ddaemon.name=', '-Dstorm.options=',
'-Dstorm.home=' + self.storm_dir, '-Dstorm.log.dir=' +
self.storm_dir + "/logs",
'-Djava.library.path=', '-Dstorm.conf.file=', '-cp',
- self.storm_dir + '/*:' + self.storm_dir + '/lib:' +
self.storm_dir +
+ self.storm_dir + '/*:' + self.storm_dir + '/lib-common:' +
self.storm_dir + '/lib:' + self.storm_dir +
'/extlib:example/storm-starter/storm-starter-topologies-*.jar:' +
self.storm_dir +
'/conf:' + self.storm_dir +
'/bin:./external/storm-redis/storm-redis-1.1.0.jar:./external/storm-kafka-client/storm-kafka-client-1.1.0.jar"',
@@ -144,7 +144,7 @@ class TestStormCli(TestCase):
self.java_cmd, '-client', '-Ddaemon.name=', '-Dstorm.options=',
'-Dstorm.home=' + self.storm_dir, '-Dstorm.log.dir=' +
self.storm_dir + "/logs",
'-Djava.library.path=', '-Dstorm.conf.file=', '-cp',
- self.storm_dir + '/*:' + self.storm_dir + '/lib:' +
self.storm_dir + '/extlib:' + self.storm_dir +
+ self.storm_dir + '/*:' + self.storm_dir + '/lib-common:' +
self.storm_dir + '/lib:' + self.storm_dir + '/extlib:' + self.storm_dir +
'/extlib-daemon:' + self.storm_dir + '/conf:' + self.storm_dir
+ '/bin', 'org.apache.storm.command.KillTopology', 'doomed_topology'
])
)
@@ -157,7 +157,7 @@ class TestStormCli(TestCase):
self.java_cmd, '-client', '-Ddaemon.name=',
'-Dstorm.options=test%3Dtest',
'-Dstorm.home=' + self.storm_dir, '-Dstorm.log.dir=' +
self.storm_dir + "/logs",
'-Djava.library.path=',
'-Dstorm.conf.file=/some/other/storm.yaml',
- '-cp', self.storm_dir + '/*:' + self.storm_dir + '/lib:' +
+ '-cp', self.storm_dir + '/*:' + self.storm_dir +
'/lib-common:' + self.storm_dir + '/lib:' +
self.storm_dir +
'/extlib:' + self.storm_dir +
'/extlib-daemon:' +
self.storm_dir + '/conf:' +
self.storm_dir +
@@ -173,7 +173,7 @@ class TestStormCli(TestCase):
self.java_cmd, '-client', '-Ddaemon.name=', '-Dstorm.options=',
'-Dstorm.home=' + self.storm_dir, '-Dstorm.log.dir=' +
self.storm_dir + "/logs",
'-Djava.library.path=', '-Dstorm.conf.file=', '-cp',
- self.storm_dir + '/*:' + self.storm_dir + '/lib:' +
self.storm_dir +
+ self.storm_dir + '/*:' + self.storm_dir + '/lib-common:' +
self.storm_dir + '/lib:' + self.storm_dir +
'/extlib:' + self.storm_dir + '/extlib-daemon:' +
self.storm_dir + '/conf:' +
self.storm_dir + '/bin', 'org.apache.storm.command.Blobstore',
'create',
'mytopo:data.tgz', '-f', 'data.tgz', '-a',
'u:alice:rwa,u:bob:rw,o::r'])
@@ -188,7 +188,7 @@ class TestStormCli(TestCase):
'-Dstorm.home=' + self.storm_dir,
'-Dstorm.log.dir=' + self.storm_dir + "/logs",
'-Djava.library.path=',
'-Dstorm.conf.file=', '-cp',
- self.storm_dir + '/*:' + self.storm_dir + '/lib:' +
self.storm_dir +
+ self.storm_dir + '/*:' + self.storm_dir + '/lib-common:' +
self.storm_dir + '/lib:' + self.storm_dir +
'/extlib:' + self.storm_dir + '/extlib-daemon:' +
self.storm_dir + '/conf:' +
self.storm_dir + '/bin', 'org.apache.storm.command.Blobstore',
'list'])
)
@@ -202,7 +202,7 @@ class TestStormCli(TestCase):
'-Dstorm.home=' + self.storm_dir,
'-Dstorm.log.dir=' + self.storm_dir + "/logs",
'-Djava.library.path=',
'-Dstorm.conf.file=', '-cp',
- self.storm_dir + '/*:' + self.storm_dir + '/lib:' +
self.storm_dir +
+ self.storm_dir + '/*:' + self.storm_dir + '/lib-common:' +
self.storm_dir + '/lib:' + self.storm_dir +
'/extlib:' + self.storm_dir + '/extlib-daemon:' +
self.storm_dir + '/conf:' +
self.storm_dir + '/bin', 'org.apache.storm.command.Blobstore',
'list', 'wordstotrack'])
)
@@ -215,7 +215,7 @@ class TestStormCli(TestCase):
self.java_cmd, '-client', '-Ddaemon.name=', '-Dstorm.options=',
'-Dstorm.home=' + self.storm_dir, '-Dstorm.log.dir=' +
self.storm_dir + "/logs",
'-Djava.library.path=', '-Dstorm.conf.file=', '-cp',
- self.storm_dir + '/*:' + self.storm_dir + '/lib:' +
self.storm_dir +
+ self.storm_dir + '/*:' + self.storm_dir + '/lib-common:' +
self.storm_dir + '/lib:' + self.storm_dir +
'/extlib:' + self.storm_dir + '/extlib-daemon:' +
self.storm_dir + '/conf:' +
self.storm_dir + '/bin', 'org.apache.storm.command.Blobstore',
'update', '-f',
'/wordsToTrack.list', 'wordstotrack'])
@@ -229,7 +229,7 @@ class TestStormCli(TestCase):
self.java_cmd, '-client', '-Ddaemon.name=', '-Dstorm.options=',
'-Dstorm.home=' + self.storm_dir, '-Dstorm.log.dir=' +
self.storm_dir + "/logs",
'-Djava.library.path=', '-Dstorm.conf.file=', '-cp',
- self.storm_dir + '/*:' + self.storm_dir + '/lib:' +
self.storm_dir +
+ self.storm_dir + '/*:' + self.storm_dir + '/lib-common:' +
self.storm_dir + '/lib:' + self.storm_dir +
'/extlib:' + self.storm_dir + '/extlib-daemon:' +
self.storm_dir + '/conf:' +
self.storm_dir + '/bin', 'org.apache.storm.command.Blobstore',
'cat', 'wordstotrack'])
)
@@ -242,7 +242,7 @@ class TestStormCli(TestCase):
self.java_cmd, '-client', '-Ddaemon.name=', '-Dstorm.options=',
'-Dstorm.home=' + self.storm_dir, '-Dstorm.log.dir=' +
self.storm_dir + "/logs",
'-Djava.library.path=', '-Dstorm.conf.file=', '-cp',
- self.storm_dir + '/*:' + self.storm_dir + '/lib:' +
self.storm_dir + '/extlib:' + self.storm_dir +
+ self.storm_dir + '/*:' + self.storm_dir + '/lib-common:' +
self.storm_dir + '/lib:' + self.storm_dir + '/extlib:' + self.storm_dir +
'/extlib-daemon:' + self.storm_dir + '/conf:' + self.storm_dir
+ '/bin',
'org.apache.storm.command.Activate', 'doomed_topology'
])
@@ -256,7 +256,7 @@ class TestStormCli(TestCase):
self.java_cmd, '-client', '-Ddaemon.name=', '-Dstorm.options=',
'-Dstorm.home=' + self.storm_dir, '-Dstorm.log.dir=' +
self.storm_dir + "/logs",
'-Djava.library.path=', '-Dstorm.conf.file=', '-cp',
- self.storm_dir + '/*:' + self.storm_dir + '/lib:' +
self.storm_dir + '/extlib:' + self.storm_dir +
+ self.storm_dir + '/*:' + self.storm_dir + '/lib-common:' +
self.storm_dir + '/lib:' + self.storm_dir + '/extlib:' + self.storm_dir +
'/extlib-daemon:' + self.storm_dir + '/conf:' + self.storm_dir
+
'/bin', 'org.apache.storm.command.Deactivate',
'doomed_topology'
])
@@ -270,7 +270,7 @@ class TestStormCli(TestCase):
self.java_cmd, '-client', '-Ddaemon.name=', '-Dstorm.options=',
'-Dstorm.home=' + self.storm_dir, '-Dstorm.log.dir=' +
self.storm_dir + "/logs",
'-Djava.library.path=', '-Dstorm.conf.file=', '-cp',
- self.storm_dir + '/*:' + self.storm_dir + '/lib:' +
self.storm_dir + '/extlib:' + self.storm_dir +
+ self.storm_dir + '/*:' + self.storm_dir + '/lib-common:' +
self.storm_dir + '/lib:' + self.storm_dir + '/extlib:' + self.storm_dir +
'/extlib-daemon:' + self.storm_dir + '/conf:' + self.storm_dir
+
'/bin', 'org.apache.storm.command.Rebalance', 'doomed_topology'
])
@@ -284,7 +284,7 @@ class TestStormCli(TestCase):
self.java_cmd, '-client', '-Ddaemon.name=', '-Dstorm.options=',
'-Dstorm.home=' + self.storm_dir, '-Dstorm.log.dir=' +
self.storm_dir + "/logs",
'-Djava.library.path=', '-Dstorm.conf.file=', '-cp',
- self.storm_dir + '/*:' + self.storm_dir + '/lib:' +
self.storm_dir + '/extlib:' + self.storm_dir +
+ self.storm_dir + '/*:' + self.storm_dir + '/lib-common:' +
self.storm_dir + '/lib:' + self.storm_dir + '/extlib:' + self.storm_dir +
'/extlib-daemon:' + self.storm_dir + '/conf:' + self.storm_dir
+
'/bin', 'org.apache.storm.command.ListTopologies'
])
@@ -298,7 +298,7 @@ class TestStormCli(TestCase):
self.java_cmd, '-server', '-Ddaemon.name=nimbus',
'-Dstorm.options=',
'-Dstorm.home=' + self.storm_dir, '-Dstorm.log.dir=' +
self.storm_dir + "/logs",
'-Djava.library.path=', '-Dstorm.conf.file=', '-cp',
- self.storm_dir + '/*:' + self.storm_dir + '/lib:' +
self.storm_dir +
+ self.storm_dir + '/*:' + self.storm_dir + '/lib-common:' +
self.storm_dir + '/lib:' + self.storm_dir +
'/extlib:' + self.storm_dir + '/extlib-daemon:' +
self.storm_dir + '/conf',
'-Djava.deserialization.disabled=true',
'-Dlogfile.name=nimbus.log',
'-Dlog4j.configurationFile=' + self.storm_dir +
'/log4j2/cluster.xml',
@@ -314,7 +314,7 @@ class TestStormCli(TestCase):
self.java_cmd, '-server', '-Ddaemon.name=supervisor',
'-Dstorm.options=',
'-Dstorm.home=' + self.storm_dir, '-Dstorm.log.dir=' +
self.storm_dir + "/logs",
'-Djava.library.path=', '-Dstorm.conf.file=', '-cp',
- self.storm_dir + '/*:' + self.storm_dir + '/lib:' +
self.storm_dir +
+ self.storm_dir + '/*:' + self.storm_dir + '/lib-common:' +
self.storm_dir + '/lib:' + self.storm_dir +
'/extlib:' + self.storm_dir + '/extlib-daemon:' +
self.storm_dir + '/conf',
'-Djava.deserialization.disabled=true',
'-Dlogfile.name=supervisor.log',
'-Dlog4j.configurationFile=' + self.storm_dir +
'/log4j2/cluster.xml',
@@ -330,7 +330,7 @@ class TestStormCli(TestCase):
self.java_cmd, '-server', '-Ddaemon.name=pacemaker',
'-Dstorm.options=',
'-Dstorm.home=' + self.storm_dir, '-Dstorm.log.dir=' +
self.storm_dir + "/logs",
'-Djava.library.path=', '-Dstorm.conf.file=', '-cp',
- self.storm_dir + '/*:' + self.storm_dir + '/lib:' +
self.storm_dir +
+ self.storm_dir + '/*:' + self.storm_dir + '/lib-common:' +
self.storm_dir + '/lib:' + self.storm_dir +
'/extlib:' + self.storm_dir + '/extlib-daemon:' +
self.storm_dir + '/conf',
'-Djava.deserialization.disabled=true',
'-Dlogfile.name=pacemaker.log',
'-Dlog4j.configurationFile=' + self.storm_dir +
'/log4j2/cluster.xml',
@@ -346,7 +346,7 @@ class TestStormCli(TestCase):
self.java_cmd, '-server', '-Ddaemon.name=ui',
'-Dstorm.options=',
'-Dstorm.home=' + self.storm_dir, '-Dstorm.log.dir=' +
self.storm_dir + "/logs",
'-Djava.library.path=', '-Dstorm.conf.file=', '-cp',
- self.storm_dir + '/*:' + self.storm_dir + '/lib:' +
self.storm_dir +
+ self.storm_dir + '/*:' + self.storm_dir + '/lib-common:' +
self.storm_dir + '/lib:' + self.storm_dir +
'/extlib:' + self.storm_dir + '/extlib-daemon:' +
self.storm_dir +
'/lib-webapp:' + self.storm_dir + '/conf',
'-Djava.deserialization.disabled=true',
'-Dlogfile.name=ui.log',
@@ -363,7 +363,7 @@ class TestStormCli(TestCase):
self.java_cmd, '-server', '-Ddaemon.name=logviewer',
'-Dstorm.options=',
'-Dstorm.home=' + self.storm_dir, '-Dstorm.log.dir=' +
self.storm_dir + "/logs",
'-Djava.library.path=', '-Dstorm.conf.file=', '-cp',
- self.storm_dir + '/*:' + self.storm_dir + '/lib:' +
self.storm_dir +
+ self.storm_dir + '/*:' + self.storm_dir + '/lib-common:' +
self.storm_dir + '/lib:' + self.storm_dir +
'/extlib:' + self.storm_dir + '/extlib-daemon:' +
self.storm_dir +
'/lib-webapp:' + self.storm_dir + '/conf',
'-Djava.deserialization.disabled=true',
'-Dlogfile.name=logviewer.log',
@@ -380,7 +380,7 @@ class TestStormCli(TestCase):
self.java_cmd, '-server', '-Ddaemon.name=drpc',
'-Dstorm.options=',
'-Dstorm.home=' + self.storm_dir, '-Dstorm.log.dir=' +
self.storm_dir + "/logs",
'-Djava.library.path=', '-Dstorm.conf.file=', '-cp',
- self.storm_dir + '/*:' + self.storm_dir + '/lib:' +
self.storm_dir +
+ self.storm_dir + '/*:' + self.storm_dir + '/lib-common:' +
self.storm_dir + '/lib:' + self.storm_dir +
'/extlib:' + self.storm_dir + '/extlib-daemon:' +
self.storm_dir +
'/lib-webapp:' + self.storm_dir + '/conf',
'-Djava.deserialization.disabled=true',
'-Dlogfile.name=drpc.log',
@@ -397,7 +397,7 @@ class TestStormCli(TestCase):
self.java_cmd, '-client', '-Ddaemon.name=', '-Dstorm.options=',
'-Dstorm.home=' + self.storm_dir, '-Dstorm.log.dir=' +
self.storm_dir + "/logs",
'-Djava.library.path=', '-Dstorm.conf.file=', '-cp',
- self.storm_dir + '/*:' + self.storm_dir + '/lib:' +
self.storm_dir + '/extlib:' + self.storm_dir +
+ self.storm_dir + '/*:' + self.storm_dir + '/lib-common:' +
self.storm_dir + '/lib:' + self.storm_dir + '/extlib:' + self.storm_dir +
'/extlib-daemon:' + self.storm_dir + '/conf:' + self.storm_dir
+
'/bin', 'org.apache.storm.command.BasicDrpcClient', 'exclaim',
'a', 'exclaim', 'b', 'test', 'bar'
])
@@ -409,7 +409,7 @@ class TestStormCli(TestCase):
self.java_cmd, '-client', '-Ddaemon.name=', '-Dstorm.options=',
'-Dstorm.home=' + self.storm_dir, '-Dstorm.log.dir=' +
self.storm_dir + "/logs",
'-Djava.library.path=', '-Dstorm.conf.file=', '-cp',
- self.storm_dir + '/*:' + self.storm_dir + '/lib:' +
self.storm_dir + '/extlib:' + self.storm_dir +
+ self.storm_dir + '/*:' + self.storm_dir + '/lib-common:' +
self.storm_dir + '/lib:' + self.storm_dir + '/extlib:' + self.storm_dir +
'/extlib-daemon:' + self.storm_dir + '/conf:' + self.storm_dir
+
'/bin', 'org.apache.storm.command.BasicDrpcClient', '-f',
'exclaim', 'a', 'b'
])
@@ -422,7 +422,7 @@ class TestStormCli(TestCase):
self.java_cmd, [
self.java_cmd, '-client', '-Ddaemon.name=', '-Dstorm.options=',
'-Dstorm.home=' + self.storm_dir, '-Dstorm.log.dir=' +
self.storm_dir + "/logs", '-Djava.library.path=',
- '-Dstorm.conf.file=', '-cp', self.storm_dir + '/*:' +
self.storm_dir + '/lib:' +
+ '-Dstorm.conf.file=', '-cp', self.storm_dir + '/*:' +
self.storm_dir + '/lib-common:' + self.storm_dir + '/lib:' +
self.storm_dir + '/extlib:' + self.storm_dir +
'/extlib-daemon:' + self.storm_dir + '/conf:' +
self.storm_dir + '/bin', 'org.apache.storm.command.HealthCheck'
])
diff --git a/storm-core/src/jvm/org/apache/storm/utils/TopologySpoutLag.java
b/storm-core/src/jvm/org/apache/storm/utils/TopologySpoutLag.java
index 7d8d7bbc8..d6bc36052 100644
--- a/storm-core/src/jvm/org/apache/storm/utils/TopologySpoutLag.java
+++ b/storm-core/src/jvm/org/apache/storm/utils/TopologySpoutLag.java
@@ -54,6 +54,11 @@ public class TopologySpoutLag {
BOOTSTRAP_CONFIG, SECURITY_PROTOCOL_CONFIG));
private static final Logger LOGGER =
LoggerFactory.getLogger(TopologySpoutLag.class);
+ // The storm-kafka-monitor jars ship only in the full binary distribution;
users of the lite
+ // distribution install them on demand (bin/storm-kafka-monitor-fetch).
Log the "not installed"
+ // hint at most once to avoid spamming the UI logs, which poll the lag
endpoint periodically.
+ private static volatile boolean warnedMonitorMissing = false;
+
public static Map<String, Map<String, Object>> lag(StormTopology
stormTopology, Map<String, Object> topologyConf) {
Map<String, Map<String, Object>> result = new HashMap<>();
Map<String, SpoutSpec> spouts = stormTopology.get_spouts();
@@ -69,6 +74,24 @@ public class TopologySpoutLag {
return result;
}
+ /**
+ * Checks whether the storm-kafka-monitor jars (invoked by
bin/storm-kafka-monitor) are present.
+ * They ship only in the full binary distribution and are fetched on
demand on the lite one, so
+ * the UI must degrade gracefully when they are absent rather than failing
the lag shell-out.
+ *
+ * @return true if the monitor appears installed, or if STORM_BASE_DIR is
unknown (in which case
+ * the legacy behavior of attempting the shell-out is preserved).
+ */
+ private static boolean isKafkaMonitorInstalled() {
+ String stormHomeDir = System.getenv("STORM_BASE_DIR");
+ if (stormHomeDir == null) {
+ return true;
+ }
+ File libDir = new File(new File(stormHomeDir, "lib-tools"),
"storm-kafka-monitor");
+ File[] jars = libDir.listFiles((dir, name) -> name.endsWith(".jar"));
+ return jars != null && jars.length > 0;
+ }
+
private static List<String>
getCommandLineOptionsForNewKafkaSpout(Map<String, Object> jsonConf) {
LOGGER.debug("json configuration: {}", jsonConf);
@@ -166,7 +189,19 @@ public class TopologySpoutLag {
LOGGER.debug("Command to run: {}", commands);
// if commands contains one or more null value, spout is compiled
with lower version of storm-kafka-client
- if (!commands.contains(null)) {
+ if (!commands.contains(null) && !isKafkaMonitorInstalled()) {
+ errorMsg = "Kafka spout lag monitoring is unavailable because
the storm-kafka-monitor "
+ + "jars are not installed. They ship only in the full
binary distribution; on the "
+ + "lite distribution run 'bin/storm-kafka-monitor-fetch'
on the UI host (and restart "
+ + "the UI) to enable it.";
+ if (!warnedMonitorMissing) {
+ warnedMonitorMissing = true;
+ LOGGER.info(errorMsg);
+ }
+ if (extraPropertiesFile != null) {
+ extraPropertiesFile.delete();
+ }
+ } else if (!commands.contains(null)) {
try {
String resultFromMonitor = new
ShellCommandRunnerImpl().execCommand(commands.toArray(new String[0]));
diff --git a/storm-dist/binary/final-package/pom.xml
b/storm-dist/binary/final-package/pom.xml
index 8471b307c..240e472f7 100644
--- a/storm-dist/binary/final-package/pom.xml
+++ b/storm-dist/binary/final-package/pom.xml
@@ -43,25 +43,114 @@
<build>
<finalName>apache-storm-${project.version}</finalName>
<plugins>
+ <!--
+ Stage the daemon (lib) and shared (lib-common) jars before
assembly and de-duplicate
+ them. The worker classpath is a byte-identical subset of the
daemon classpath, so the
+ shared jars are kept once in lib-common (referenced by both
classpaths in bin/storm.py)
+ and removed from lib. See src/main/scripts/dedup-libs.py.
+ -->
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>stage-daemon-lib</id>
+ <phase>prepare-package</phase>
+ <goals>
+ <goal>copy-dependencies</goal>
+ </goals>
+ <configuration>
+ <includeScope>runtime</includeScope>
+
<outputDirectory>${project.build.directory}/staging/lib</outputDirectory>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-resources-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>stage-common-lib</id>
+ <phase>prepare-package</phase>
+ <goals>
+ <goal>copy-resources</goal>
+ </goals>
+ <configuration>
+
<outputDirectory>${project.build.directory}/staging/lib-common</outputDirectory>
+ <resources>
+ <resource>
+
<directory>${project.basedir}/../storm-client-bin/target/client/client/lib-worker</directory>
+ <includes>
+ <include>*.jar</include>
+ </includes>
+ </resource>
+ </resources>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>exec-maven-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>dedup-libs</id>
+ <phase>prepare-package</phase>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ <configuration>
+ <executable>python3</executable>
+ <arguments>
+
<argument>${project.basedir}/src/main/scripts/dedup-libs.py</argument>
+
<argument>${project.build.directory}/staging</argument>
+ </arguments>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
+ <!-- Full distribution: apache-storm-${version}.tar.gz /
.zip -->
+ <execution>
+ <id>bin</id>
+ <phase>package</phase>
+ <goals>
+ <goal>single</goal>
+ </goals>
+ <configuration>
+ <descriptors>
+
<descriptor>${project.basedir}/src/main/assembly/binary.xml</descriptor>
+ </descriptors>
+ <appendAssemblyId>false</appendAssemblyId>
+ </configuration>
+ </execution>
+ <!--
+ Lean "storm-lite" distribution:
apache-storm-${version}-lite.tar.gz / .zip
+ (attached with the "lite" classifier). Same contents
as the full
+ distribution minus the bundled storm-autocreds and
storm-kafka-monitor jars.
+ -->
<execution>
+ <id>lite</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
+ <configuration>
+ <descriptors>
+
<descriptor>${project.basedir}/src/main/assembly/binary-lite.xml</descriptor>
+ </descriptors>
+ <appendAssemblyId>true</appendAssemblyId>
+ </configuration>
</execution>
</executions>
<configuration>
<attach>true</attach>
<tarLongFileMode>posix</tarLongFileMode>
<runOnlyAtExecutionRoot>false</runOnlyAtExecutionRoot>
- <descriptors>
-
<descriptor>${project.basedir}/src/main/assembly/binary.xml</descriptor>
- </descriptors>
- <appendAssemblyId>false</appendAssemblyId>
</configuration>
</plugin>
<plugin>
@@ -78,4 +167,43 @@
</plugin>
</plugins>
</build>
+ <profiles>
+ <profile>
+ <id>unix_py_tests</id>
+ <activation>
+ <os>
+ <family>unix</family>
+ </os>
+ </activation>
+ <build>
+ <plugins>
+ <!-- Run the dedup-libs.py unit tests before the script is
used at prepare-package time. -->
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>exec-maven-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>python3-test-dedup-libs</id>
+ <phase>test</phase>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ <configuration>
+ <executable>python3</executable>
+
<workingDirectory>src/main/scripts</workingDirectory>
+ <arguments>
+ <argument>test_dedup_libs.py</argument>
+ </arguments>
+ <environmentVariables>
+
<PYTHONDONTWRITEBYTECODE>true</PYTHONDONTWRITEBYTECODE>
+ </environmentVariables>
+ <skip>${skipTests}</skip>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
</project>
diff --git a/storm-dist/binary/final-package/src/main/assembly/binary-lite.xml
b/storm-dist/binary/final-package/src/main/assembly/binary-lite.xml
new file mode 100644
index 000000000..ce0eb4c3b
--- /dev/null
+++ b/storm-dist/binary/final-package/src/main/assembly/binary-lite.xml
@@ -0,0 +1,63 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!--
+ The "storm-lite" binary distribution: identical to the full distribution
+ (binary.xml) except that the optional plugins carrying heavyweight
+ third-party dependencies are not bundled:
+
+ - storm-autocreds (Hadoop/HBase client tree, secure clusters only)
+ - storm-kafka-monitor (Kafka client, only for spout lag in the UI)
+
+ Only their READMEs ship, consistent with the other external/* connectors.
+ Operators install them on demand with bin/storm-autocreds-fetch and
+ bin/storm-kafka-monitor-fetch, or use the full distribution instead.
+ The UI degrades gracefully when storm-kafka-monitor is absent.
+-->
+<assembly
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0
http://maven.apache.org/xsd/assembly-1.1.0.xsd";>
+ <id>lite</id>
+ <formats>
+ <format>tar.gz</format>
+ <format>zip</format>
+ </formats>
+
+ <componentDescriptors>
+
<componentDescriptor>${project.basedir}/src/main/assembly/common.xml</componentDescriptor>
+ </componentDescriptors>
+
+ <fileSets>
+ <!-- README only; the jars are fetched on demand
(bin/storm-kafka-monitor-fetch). -->
+ <fileSet>
+
<directory>${project.basedir}/../../../external/storm-kafka-monitor</directory>
+ <outputDirectory>external/storm-kafka-monitor</outputDirectory>
+ <includes>
+ <include>README.*</include>
+ </includes>
+ </fileSet>
+
+ <!-- README only; the jars are fetched on demand
(bin/storm-autocreds-fetch). -->
+ <fileSet>
+
<directory>${project.basedir}/../../../external/storm-autocreds</directory>
+ <outputDirectory>external/storm-autocreds</outputDirectory>
+ <includes>
+ <include>README.*</include>
+ </includes>
+ </fileSet>
+ </fileSets>
+</assembly>
diff --git a/storm-dist/binary/final-package/src/main/assembly/binary.xml
b/storm-dist/binary/final-package/src/main/assembly/binary.xml
index ee5cb60e3..32cd51031 100644
--- a/storm-dist/binary/final-package/src/main/assembly/binary.xml
+++ b/storm-dist/binary/final-package/src/main/assembly/binary.xml
@@ -15,6 +15,14 @@
limitations under the License.
-->
+<!--
+ The full binary distribution: everything in common.xml plus the optional
+ plugins that need heavyweight third-party dependencies (Hadoop/HBase for
+ storm-autocreds, the Kafka client for storm-kafka-monitor).
+
+ The storm-lite distribution (binary-lite.xml) omits those jars and ships
only
+ their READMEs; see bin/storm-autocreds-fetch and
bin/storm-kafka-monitor-fetch.
+-->
<assembly
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0
http://maven.apache.org/xsd/assembly-1.1.0.xsd";>
@@ -24,224 +32,29 @@
<format>zip</format>
</formats>
- <!-- put deps in the lib folder -->
- <dependencySets>
- <dependencySet>
- <useProjectArtifact>false</useProjectArtifact>
- <outputDirectory>lib</outputDirectory>
- <unpack>false</unpack>
- </dependencySet>
- </dependencySets>
+ <componentDescriptors>
+
<componentDescriptor>${project.basedir}/src/main/assembly/common.xml</componentDescriptor>
+ </componentDescriptors>
<fileSets>
+ <!-- storm-kafka-monitor: bundled here so Kafka spout lag works out of
the box. -->
<fileSet>
-
<directory>${project.basedir}/../storm-client-bin/target/client/client/</directory>
- <outputDirectory>.</outputDirectory>
- <includes>
- <include>*/**</include>
- </includes>
- </fileSet>
- <fileSet>
-
<directory>${project.basedir}/../storm-webapp-bin/target/webapp/webapp/</directory>
- <outputDirectory>.</outputDirectory>
- <includes>
- <include>*/**</include>
- </includes>
- </fileSet>
- <!--
- $STORM_HOME/bin
- -->
- <fileSet>
- <directory>${project.basedir}/../../../bin</directory>
- <outputDirectory>bin</outputDirectory>
- <includes>
- <include>storm*</include>
- <include>flight.bash</include>
- </includes>
- <lineEnding>unix</lineEnding>
- <fileMode>0755</fileMode>
- </fileSet>
- <!--
- Allow for variable substitution for cache busting in HTML files.
- See storm-dist/binary/pom.xml for custom variables that will be
substituted in.
- The source files should have a ${variable to be replaced} wherever
- maven assembly plugin should inject a value.
- -->
- <fileSet>
-
<directory>${project.basedir}/../../../storm-webapp/target/classes/WEB-INF/</directory>
- <outputDirectory>public</outputDirectory>
- <includes>
- <include>*.html</include>
- </includes>
- <excludes/>
- <filtered>true</filtered>
- </fileSet>
- <!--
- Include rest of public/ directory (webpack bundles, images,
templates)
- without any filtering.
- -->
- <fileSet>
-
<directory>${project.basedir}/../../../storm-webapp/target/classes/WEB-INF/</directory>
- <outputDirectory>public</outputDirectory>
- <includes>
- <include>*/**</include>
- </includes>
- <excludes>
- <exclude>*.html</exclude>
- </excludes>
- </fileSet>
- <!-- EXAMPLES -->
- <fileSet>
- <directory>${project.basedir}/../../../examples</directory>
- <outputDirectory>examples</outputDirectory>
- <excludes>
- <exclude>**/target/**</exclude>
- <exclude>**/dependency-reduced-pom.xml</exclude>
- <exclude>**/*.iml</exclude>
- </excludes>
- </fileSet>
- <fileSet>
-
<directory>${project.basedir}/../../../storm-core/target/native/target/usr/local/bin/</directory>
- <includes>
- <include>worker-launcher</include>
- </includes>
- <outputDirectory>bin</outputDirectory>
- </fileSet>
- <!-- EXTERNAL -->
- <!-- only include the README file -->
- <fileSet>
-
<directory>${project.basedir}/../../../external/storm-kinesis</directory>
- <outputDirectory>external/storm-kinesis</outputDirectory>
- <includes>
- <include>README.*</include>
- </includes>
- </fileSet>
- <fileSet>
-
<directory>${project.basedir}/../../../external/storm-hdfs</directory>
- <outputDirectory>external/storm-hdfs</outputDirectory>
- <includes>
- <include>README.*</include>
- </includes>
- </fileSet>
- <fileSet>
-
<directory>${project.basedir}/../../../external/storm-jdbc</directory>
- <outputDirectory>external/storm-jdbc</outputDirectory>
- <includes>
- <include>README.*</include>
- </includes>
- </fileSet>
- <fileSet>
-
<directory>${project.basedir}/../../../external/storm-eventhubs</directory>
- <outputDirectory>external/storm-eventhubs</outputDirectory>
- <includes>
- <include>README.*</include>
- </includes>
- </fileSet>
-
- <fileSet>
- <directory>${project.basedir}/../../../external/flux</directory>
- <outputDirectory>external/flux</outputDirectory>
- <includes>
- <include>README.*</include>
- </includes>
- </fileSet>
- <fileSet>
-
<directory>${project.basedir}/../../../external/storm-redis</directory>
- <outputDirectory>external/storm-redis</outputDirectory>
- <includes>
- <include>README.*</include>
- </includes>
- </fileSet>
- <fileSet>
-
<directory>${project.basedir}/../../../external/storm-solr</directory>
- <outputDirectory>external/storm-solr</outputDirectory>
- <includes>
- <include>README.*</include>
- </includes>
- </fileSet>
- <fileSet>
-
<directory>${project.basedir}/../../../external/storm-mqtt</directory>
- <outputDirectory>external/storm-mqtt</outputDirectory>
- <includes>
- <include>README.*</include>
- </includes>
- </fileSet>
- <fileSet>
-
<directory>${project.basedir}/../../../external/storm-mongodb</directory>
- <outputDirectory>external/storm-mongodb</outputDirectory>
- <includes>
- <include>README.*</include>
- </includes>
- </fileSet>
- <fileSet>
-
<directory>${project.basedir}/../../../external/storm-kafka-client</directory>
- <outputDirectory>external/storm-kafka-client</outputDirectory>
- <includes>
- <include>README.*</include>
- </includes>
- </fileSet>
- <fileSet>
-
<directory>${project.basedir}/../../../external/storm-opentsdb</directory>
- <outputDirectory>external/storm-opentsdb</outputDirectory>
- <includes>
- <include>README.*</include>
- </includes>
- </fileSet>
- <fileSet>
-
<directory>${project.basedir}/../../../external/storm-druid</directory>
- <outputDirectory>external/storm-druid</outputDirectory>
- <includes>
- <include>README.*</include>
- </includes>
- </fileSet>
- <fileSet>
-
<directory>${project.basedir}/../../../external/storm-jms</directory>
- <outputDirectory>external/storm-jms</outputDirectory>
- <includes>
- <include>README.*</include>
- </includes>
- </fileSet>
- <fileSet>
-
<directory>${project.basedir}/../../../external/storm-pmml</directory>
- <outputDirectory>external/storm-pmml</outputDirectory>
+
<directory>${project.basedir}/../storm-kafka-monitor-bin/target/kafka-monitor/kafka-monitor/lib-kafka-monitor</directory>
+ <outputDirectory>lib-tools/storm-kafka-monitor</outputDirectory>
<includes>
- <include>README.*</include>
+ <include>*jar</include>
</includes>
</fileSet>
<fileSet>
-
<directory>${project.basedir}/../../../external/storm-rocketmq</directory>
- <outputDirectory>external/storm-rocketmq</outputDirectory>
+
<directory>${project.basedir}/../../../external/storm-kafka-monitor</directory>
+ <outputDirectory>external/storm-kafka-monitor</outputDirectory>
<includes>
<include>README.*</include>
</includes>
</fileSet>
- <!-- $STORM_HOME/extlib -->
- <fileSet>
- <directory></directory>
- <outputDirectory>extlib</outputDirectory>
- <includes>
- <include></include>
- </includes>
- </fileSet>
- <!-- $STORM_HOME/extlib-daemon, for daemons only -->
- <fileSet>
- <directory></directory>
- <outputDirectory>extlib-daemon</outputDirectory>
- <includes>
- <include></include>
- </includes>
- </fileSet>
- <!-- $STORM_HOME/toollib -->
- <fileSet>
-
<directory>${project.basedir}/../storm-kafka-monitor-bin/target/kafka-monitor/kafka-monitor/lib-kafka-monitor</directory>
- <outputDirectory>lib-tools/storm-kafka-monitor</outputDirectory>
- <includes>
- <include>*jar</include>
- </includes>
- </fileSet>
-
- <!-- Autocred plugins -->
+ <!-- Autocred plugins: bundled here so secure (Kerberos) clusters can
enable them
+ by copying the jars onto the daemon classpath without fetching
anything. -->
<fileSet>
<directory>${project.basedir}/../storm-autocreds-bin/target/autocreds/autocreds/lib-autocreds</directory>
<outputDirectory>external/storm-autocreds</outputDirectory>
@@ -249,101 +62,12 @@
<include>*jar</include>
</includes>
</fileSet>
- <!-- storm-submit-tools -->
<fileSet>
-
<directory>${project.basedir}/../storm-submit-tools-bin/target/submit-tools/submit-tools/lib-submit-tools</directory>
- <outputDirectory>lib-tools/submit-tools</outputDirectory>
+
<directory>${project.basedir}/../../../external/storm-autocreds</directory>
+ <outputDirectory>external/storm-autocreds</outputDirectory>
<includes>
- <include>*jar</include>
+ <include>README.*</include>
</includes>
</fileSet>
- <!-- Licenses -->
- <fileSet>
- <directory>${project.basedir}/../../../licenses</directory>
- <outputDirectory>licenses</outputDirectory>
- </fileSet>
</fileSets>
-
- <files>
- <!--
- $STORM_HOME/conf
- -->
- <file>
-
<source>${project.basedir}/../../../conf/storm.yaml.example</source>
- <outputDirectory>./conf</outputDirectory>
- <destName>storm.yaml</destName>
- <fileMode>0644</fileMode>
- </file>
- <file>
- <source>${project.basedir}/../../../conf/storm_env.ini</source>
- <outputDirectory>./conf</outputDirectory>
- <destName>storm_env.ini</destName>
- <fileMode>0644</fileMode>
- </file>
- <file>
- <source>${project.basedir}/../../../conf/storm-env.sh</source>
- <outputDirectory>./conf</outputDirectory>
- <destName>storm-env.sh</destName>
- <lineEnding>unix</lineEnding>
- <fileMode>0755</fileMode>
- </file>
- <file>
- <source>${project.basedir}/../../../conf/storm-env.ps1</source>
- <outputDirectory>./conf</outputDirectory>
- <destName>storm-env.ps1</destName>
- <fileMode>0755</fileMode>
- </file>
- <file>
-
<source>${project.basedir}/../../../conf/seccomp.json.example</source>
- <outputDirectory>/conf</outputDirectory>
- <destName>seccomp.json.example</destName>
- <fileMode>0755</fileMode>
- </file>
- <file>
- <source>${project.basedir}/../../../VERSION</source>
- <outputDirectory>.</outputDirectory>
- <destName>RELEASE</destName>
- <filtered>true</filtered>
- </file>
- <!-- TODO this should be a generated file from "target" -->
- <file>
- <source>${project.basedir}/../../../log4j2/cluster.xml</source>
- <outputDirectory>./log4j2</outputDirectory>
- <fileMode>0644</fileMode>
- </file>
-
- <file>
- <source>${project.basedir}/../../../log4j2/worker.xml</source>
- <outputDirectory>./log4j2</outputDirectory>
- <fileMode>0644</fileMode>
- </file>
-
- <file>
- <source>${project.basedir}/../../../LICENSE-binary</source>
- <destName>LICENSE</destName>
- <outputDirectory>.</outputDirectory>
- </file>
-
- <file>
- <source>${project.basedir}/../../../NOTICE-binary</source>
- <destName>NOTICE</destName>
- <outputDirectory>.</outputDirectory>
- </file>
-
- <file>
- <source>${project.basedir}/../../../DEPENDENCY-LICENSES</source>
- <outputDirectory>.</outputDirectory>
- </file>
-
- <file>
- <source>${project.basedir}/../../../README.markdown</source>
- <outputDirectory>.</outputDirectory>
- </file>
-
- <file>
- <source>${project.basedir}/../../../SECURITY.md</source>
- <outputDirectory>.</outputDirectory>
- </file>
- </files>
-
</assembly>
diff --git a/storm-dist/binary/final-package/src/main/assembly/binary.xml
b/storm-dist/binary/final-package/src/main/assembly/common.xml
similarity index 89%
copy from storm-dist/binary/final-package/src/main/assembly/binary.xml
copy to storm-dist/binary/final-package/src/main/assembly/common.xml
index ee5cb60e3..6b50cc8ad 100644
--- a/storm-dist/binary/final-package/src/main/assembly/binary.xml
+++ b/storm-dist/binary/final-package/src/main/assembly/common.xml
@@ -15,30 +15,27 @@
limitations under the License.
-->
-<assembly
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0";
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
-
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0
http://maven.apache.org/xsd/assembly-1.1.0.xsd";>
- <id>bin</id>
- <formats>
- <format>tar.gz</format>
- <format>zip</format>
- </formats>
-
- <!-- put deps in the lib folder -->
- <dependencySets>
- <dependencySet>
- <useProjectArtifact>false</useProjectArtifact>
- <outputDirectory>lib</outputDirectory>
- <unpack>false</unpack>
- </dependencySet>
- </dependencySets>
+<component
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/component/1.1.2";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/component/1.1.2
http://maven.apache.org/xsd/component-1.1.2.xsd";>
<fileSets>
+ <!-- Daemon jars (lib) and shared jars (lib-common) are staged and
de-duplicated by
+ final-package's build (maven-dependency-plugin + dedup-libs.py)
before assembly.
+ The worker jars now live in lib-common, so the storm-client-bin
tree (which only
+ contained lib-worker) is no longer copied here. -->
<fileSet>
-
<directory>${project.basedir}/../storm-client-bin/target/client/client/</directory>
- <outputDirectory>.</outputDirectory>
+ <directory>${project.build.directory}/staging/lib</directory>
+ <outputDirectory>lib</outputDirectory>
<includes>
- <include>*/**</include>
+ <include>*.jar</include>
+ </includes>
+ </fileSet>
+ <fileSet>
+
<directory>${project.build.directory}/staging/lib-common</directory>
+ <outputDirectory>lib-common</outputDirectory>
+ <includes>
+ <include>*.jar</include>
</includes>
</fileSet>
<fileSet>
@@ -232,23 +229,6 @@
<include></include>
</includes>
</fileSet>
- <!-- $STORM_HOME/toollib -->
- <fileSet>
-
<directory>${project.basedir}/../storm-kafka-monitor-bin/target/kafka-monitor/kafka-monitor/lib-kafka-monitor</directory>
- <outputDirectory>lib-tools/storm-kafka-monitor</outputDirectory>
- <includes>
- <include>*jar</include>
- </includes>
- </fileSet>
-
- <!-- Autocred plugins -->
- <fileSet>
-
<directory>${project.basedir}/../storm-autocreds-bin/target/autocreds/autocreds/lib-autocreds</directory>
- <outputDirectory>external/storm-autocreds</outputDirectory>
- <includes>
- <include>*jar</include>
- </includes>
- </fileSet>
<!-- storm-submit-tools -->
<fileSet>
<directory>${project.basedir}/../storm-submit-tools-bin/target/submit-tools/submit-tools/lib-submit-tools</directory>
@@ -345,5 +325,4 @@
<outputDirectory>.</outputDirectory>
</file>
</files>
-
-</assembly>
+</component>
diff --git a/storm-dist/binary/final-package/src/main/scripts/dedup-libs.py
b/storm-dist/binary/final-package/src/main/scripts/dedup-libs.py
new file mode 100755
index 000000000..e0f261405
--- /dev/null
+++ b/storm-dist/binary/final-package/src/main/scripts/dedup-libs.py
@@ -0,0 +1,211 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# De-duplicates the jars shared by the daemon (lib) and worker (lib-worker)
+# classpaths of an assembled Storm distribution into a single lib-common
+# directory, to keep the distribution small.
+#
+# The worker classpath is a strict subset of the daemon classpath, so every
+# lib-worker jar is moved into lib-common; any byte-identical copy in lib is
+# then removed. bin/storm.py adds lib-common to BOTH classpaths, so:
+# daemon classpath = lib-common + lib (unchanged jar set)
+# worker classpath = lib-common (+ lib-worker, now empty)
+#
+# Only byte-identical jars (same name AND same sha-256) are de-duplicated, so a
+# version mismatch is never silently merged. The script fails the build when
+# the two classpaths carry conflicting copies of an artifact, in either form:
+# - same file name, different sha-256 (e.g. a patched rebuild), or
+# - same Maven artifact, different version. The jars are named
+# <artifactId>-<version>[-<classifier>].jar by maven-dependency-plugin, so a
+# version bump changes the FILE NAME and a plain name comparison would let
+# e.g. lib/foo-1.0.jar and lib-worker/foo-1.1.jar pass silently. Jars are
+# therefore compared by their version-stripped artifact base name (the part
+# before the first "-<digit>", the same heuristic Maven uses to start the
+# version).
+# Either way both copies would land on the daemon classpath, with the
+# lib-common copy silently shadowing the lib one. All conflicts are detected in
+# a read-only pass BEFORE any file is moved or removed, so a failing run leaves
+# the distribution tree exactly as it found it. Tool classpaths (lib-tools/*,
+# lib-webapp) are intentionally left untouched: their wrappers do not include
+# lib-common, so their jars must stay in place.
+#
+# The summary also reports the number of worker-only jars (jars promoted to
+# lib-common whose artifact does not appear in lib at all). The worker
+# classpath is a strict subset of the daemon classpath in a single-reactor
+# build, so this count is expected to be 0; a non-zero value is an early
+# warning that the two classpaths have drifted apart.
+
+import hashlib
+import os
+import re
+import shutil
+import sys
+
+
+def sha256(path):
+ h = hashlib.sha256()
+ with open(path, "rb") as fh:
+ for chunk in iter(lambda: fh.read(1 << 20), b""):
+ h.update(chunk)
+ return h.hexdigest()
+
+
+def jars(directory):
+ if not os.path.isdir(directory):
+ return {}
+ return {f: os.path.join(directory, f) for f in os.listdir(directory) if
f.endswith(".jar")}
+
+
+def artifact_base(name):
+ """Version-stripped artifact base of a maven-built jar file name.
+
+ "commons-io-2.13.0.jar" -> "commons-io"; the version starts at the first
+ "-" that is followed by a digit (Maven artifactId segments do not start
+ with a digit). A name without such a separator is returned as-is (minus
+ the .jar suffix)."""
+ stem = name[: -len(".jar")]
+ match = re.match(r"(.+?)-\d", stem)
+ return match.group(1) if match else stem
+
+
+def by_base(jar_names):
+ bases = {}
+ for name in jar_names:
+ bases.setdefault(artifact_base(name), []).append(name)
+ return bases
+
+
+def dedup(dist_root):
+ lib = os.path.join(dist_root, "lib")
+ worker = os.path.join(dist_root, "lib-worker")
+ common = os.path.join(dist_root, "lib-common")
+
+ # Two supported layouts:
+ # - exploded distribution: lib-worker is present, lib-common is created
here;
+ # - staged build: lib-common is already populated, lib-worker is
absent.
+ worker_jars = jars(worker)
+ common_jars = jars(common)
+ lib_jars = jars(lib)
+
+ if not worker_jars and not common_jars:
+ print(f"dedup-libs: no jars in {common} (and none in {worker});
nothing to do")
+ return 0
+
+ # ---- Phase 1: read-only conflict detection. Hash each jar (at most once)
+ # and collect every same-name/different-sha pair BEFORE anything is moved
+ # or removed, so a failing run leaves the tree exactly as it found it.
+ hashes = {}
+
+ def digest(path):
+ if path not in hashes:
+ hashes[path] = sha256(path)
+ return hashes[path]
+
+ # Conflicting jars between two classpath dirs: same file name with a
+ # different sha-256, or same artifact base with a different version (which
+ # a name comparison alone would miss, since the version is part of the
+ # file name). Each entry is a fully formatted message naming both copies.
+ conflicts = []
+ common_bases = by_base(common_jars)
+ for name in sorted(worker_jars):
+ if name in common_jars:
+ if digest(worker_jars[name]) != digest(common_jars[name]):
+ conflicts.append(f"{name}: lib-worker sha-256
{digest(worker_jars[name])} "
+ f"!= lib-common sha-256
{digest(common_jars[name])}")
+ else:
+ for other in sorted(common_bases.get(artifact_base(name), [])):
+ conflicts.append(f"{name}: lib-worker has {name} but
lib-common has {other} "
+ f"(artifact '{artifact_base(name)}',
different version)")
+
+ # The jar set lib-common will hold after absorbing lib-worker. On a
+ # worker-vs-common conflict the winner is ambiguous, but we fail anyway.
+ merged = dict(common_jars)
+ for name, path in worker_jars.items():
+ merged.setdefault(name, path)
+
+ lib_bases = by_base(lib_jars)
+ lib_duplicates = []
+ worker_only = 0
+ for name, merged_copy in sorted(merged.items()):
+ origin = "lib-common" if name in common_jars else "lib-worker"
+ lib_copy = lib_jars.get(name)
+ if lib_copy:
+ if digest(lib_copy) == digest(merged_copy):
+ # Byte-identical copy in the daemon lib dir; removable in
phase 2.
+ lib_duplicates.append(lib_copy)
+ else:
+ # Same name, different bytes: the lib-common copy would
silently
+ # shadow the lib copy on the daemon classpath. Fail the build.
+ conflicts.append(f"{name}: lib sha-256 {digest(lib_copy)} "
+ f"!= {origin} sha-256 {digest(merged_copy)}")
+ else:
+ siblings = sorted(n for n in lib_bases.get(artifact_base(name),
[]))
+ if siblings:
+ # Same artifact, different version: promoting this jar would
put
+ # both versions on the daemon classpath. Fail the build.
+ conflicts.append(f"{name}: {origin} has {name} but lib has {',
'.join(siblings)} "
+ f"(artifact '{artifact_base(name)}',
different version)")
+ else:
+ # Artifact absent from lib entirely: worker-only. Expected to
+ # never happen in a single-reactor build; reported as a canary.
+ worker_only += 1
+
+ if conflicts:
+ for conflict in conflicts:
+ print(f"dedup-libs: ERROR: conflicting jar {conflict}",
file=sys.stderr)
+ print(f"dedup-libs: ERROR: {len(conflicts)} conflict(s) (same name
with different content, or same "
+ "artifact with a different version) between the daemon (lib) and
worker "
+ "(lib-common/lib-worker) classpaths; "
+ "refusing to assemble a distribution with shadowed jars; no
files were changed", file=sys.stderr)
+ return 1
+
+ # ---- Phase 2: mutation. No conflicts exist, so every same-name pair
+ # encountered below is known to be byte-identical.
+ if worker_jars:
+ os.makedirs(common, exist_ok=True)
+ for name, src in sorted(worker_jars.items()):
+ dst = os.path.join(common, name)
+ if os.path.exists(dst):
+ os.remove(src)
+ else:
+ shutil.move(src, dst)
+ # lib-worker is now empty; remove it so the layout is unambiguous.
+ if os.path.isdir(worker) and not os.listdir(worker):
+ os.rmdir(worker)
+
+ # Drop the byte-identical copies from the daemon lib dir.
+ reclaimed = 0
+ for lib_copy in lib_duplicates:
+ reclaimed += os.path.getsize(lib_copy)
+ os.remove(lib_copy)
+
+ print(f"dedup-libs: lib-common has {len(merged)} jar(s); "
+ f"removed {len(lib_duplicates)} duplicate(s) from lib, reclaimed
{reclaimed / (1024 * 1024):.1f} MB; "
+ f"{worker_only} worker-only jar(s) promoted to lib-common (expected
0)")
+ return 0
+
+
+def main(argv):
+ if len(argv) != 2:
+ print("Usage: dedup-libs.py <dist-root>", file=sys.stderr)
+ return 2
+ return dedup(argv[1])
+
+
+if __name__ == "__main__":
+ sys.exit(main(sys.argv))
diff --git
a/storm-dist/binary/final-package/src/main/scripts/test_dedup_libs.py
b/storm-dist/binary/final-package/src/main/scripts/test_dedup_libs.py
new file mode 100644
index 000000000..a9ac652a1
--- /dev/null
+++ b/storm-dist/binary/final-package/src/main/scripts/test_dedup_libs.py
@@ -0,0 +1,265 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Unit tests for dedup-libs.py (same directory). The script name contains a
+# hyphen, so it is loaded via importlib rather than a plain import.
+
+import contextlib
+import importlib.util
+import io
+import os
+import tempfile
+import unittest
+
+_SCRIPT = os.path.join(os.path.dirname(os.path.abspath(__file__)),
"dedup-libs.py")
+_SPEC = importlib.util.spec_from_file_location("dedup_libs", _SCRIPT)
+dedup_libs = importlib.util.module_from_spec(_SPEC)
+_SPEC.loader.exec_module(dedup_libs)
+
+
+class TestDedupLibs(unittest.TestCase):
+
+ def setUp(self):
+ self._tmp = tempfile.TemporaryDirectory()
+ self.addCleanup(self._tmp.cleanup)
+ self.dist_root = self._tmp.name
+
+ def write_jar(self, directory, name, content):
+ path = os.path.join(self.dist_root, directory, name)
+ os.makedirs(os.path.dirname(path), exist_ok=True)
+ with open(path, "wb") as fh:
+ fh.write(content)
+ return path
+
+ def run_dedup(self):
+ """Runs the script's main() and returns (exit_code, stdout, stderr)."""
+ out, err = io.StringIO(), io.StringIO()
+ with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
+ code = dedup_libs.main(["dedup-libs.py", self.dist_root])
+ return code, out.getvalue(), err.getvalue()
+
+ def path(self, *parts):
+ return os.path.join(self.dist_root, *parts)
+
+ def test_identical_jars_are_deduped(self):
+ self.write_jar("lib", "shared.jar", b"same bytes")
+ self.write_jar("lib-worker", "shared.jar", b"same bytes")
+ self.write_jar("lib", "daemon-only.jar", b"daemon only")
+
+ code, out, err = self.run_dedup()
+
+ self.assertEqual(code, 0, err)
+ # The shared jar lives only in lib-common now; lib-worker is gone.
+ self.assertTrue(os.path.exists(self.path("lib-common", "shared.jar")))
+ self.assertFalse(os.path.exists(self.path("lib", "shared.jar")))
+ self.assertFalse(os.path.exists(self.path("lib-worker")))
+ # Daemon-only jars are untouched.
+ self.assertTrue(os.path.exists(self.path("lib", "daemon-only.jar")))
+ self.assertIn("removed 1 duplicate(s) from lib", out)
+ self.assertIn("0 worker-only jar(s)", out)
+
+ def test_diverged_jar_between_lib_and_worker_fails(self):
+ self.write_jar("lib", "diverged.jar", b"daemon version")
+ self.write_jar("lib-worker", "diverged.jar", b"worker version")
+ # An identical shared jar that WOULD be deduped: it must survive a
+ # failing run untouched (conflicts are detected before any mutation).
+ self.write_jar("lib", "shared.jar", b"same bytes")
+ self.write_jar("lib-worker", "shared.jar", b"same bytes")
+
+ code, out, err = self.run_dedup()
+
+ self.assertNotEqual(code, 0)
+ self.assertIn("diverged.jar", err)
+ # Both sha-256 digests are reported.
+ self.assertIn(dedup_libs.sha256(self.path("lib", "diverged.jar")), err)
+ self.assertIn(dedup_libs.sha256(self.path("lib-worker",
"diverged.jar")), err)
+ # The whole tree is untouched: nothing removed from lib, nothing
+ # promoted into lib-common, lib-worker still fully populated.
+ self.assertTrue(os.path.exists(self.path("lib", "diverged.jar")))
+ self.assertTrue(os.path.exists(self.path("lib", "shared.jar")))
+ self.assertTrue(os.path.exists(self.path("lib-worker",
"diverged.jar")))
+ self.assertTrue(os.path.exists(self.path("lib-worker", "shared.jar")))
+ self.assertFalse(os.path.exists(self.path("lib-common")))
+
+ def test_diverged_jar_between_lib_and_common_fails(self):
+ # Staged-build layout: lib-common is pre-populated, lib-worker absent.
+ self.write_jar("lib", "diverged.jar", b"daemon version")
+ self.write_jar("lib-common", "diverged.jar", b"worker version")
+ self.write_jar("lib", "shared.jar", b"same bytes")
+ self.write_jar("lib-common", "shared.jar", b"same bytes")
+
+ code, out, err = self.run_dedup()
+
+ self.assertNotEqual(code, 0)
+ self.assertIn("diverged.jar", err)
+ self.assertTrue(os.path.exists(self.path("lib", "diverged.jar")))
+ self.assertTrue(os.path.exists(self.path("lib-common",
"diverged.jar")))
+ # The identical shared jar was NOT removed from lib on failure.
+ self.assertTrue(os.path.exists(self.path("lib", "shared.jar")))
+
+ def test_diverged_jar_between_worker_and_common_fails(self):
+ # A jar already staged in lib-common conflicts with the lib-worker
copy.
+ self.write_jar("lib-common", "diverged.jar", b"staged version")
+ self.write_jar("lib-worker", "diverged.jar", b"worker version")
+ self.write_jar("lib", "shared.jar", b"same bytes")
+ self.write_jar("lib-worker", "shared.jar", b"same bytes")
+
+ code, out, err = self.run_dedup()
+
+ self.assertNotEqual(code, 0)
+ self.assertIn("diverged.jar", err)
+ # The conflicting worker copy is kept in place for inspection, and
+ # no other jar was moved or removed.
+ self.assertTrue(os.path.exists(self.path("lib-worker",
"diverged.jar")))
+ self.assertTrue(os.path.exists(self.path("lib-common",
"diverged.jar")))
+ self.assertTrue(os.path.exists(self.path("lib", "shared.jar")))
+ self.assertTrue(os.path.exists(self.path("lib-worker", "shared.jar")))
+ self.assertFalse(os.path.exists(self.path("lib-common", "shared.jar")))
+
+ def test_all_conflicts_are_reported(self):
+ self.write_jar("lib", "first.jar", b"daemon 1")
+ self.write_jar("lib-worker", "first.jar", b"worker 1")
+ self.write_jar("lib", "second.jar", b"daemon 2")
+ self.write_jar("lib-worker", "second.jar", b"worker 2")
+
+ code, out, err = self.run_dedup()
+
+ self.assertNotEqual(code, 0)
+ self.assertIn("first.jar", err)
+ self.assertIn("second.jar", err)
+ self.assertIn("2 conflict(s)", err)
+ self.assertIn("no files were changed", err)
+
+ def test_version_drift_between_lib_and_worker_fails(self):
+ # A version bump changes the FILE NAME (artifactId-version.jar), so
+ # drift must be caught on the version-stripped artifact base name.
+ self.write_jar("lib", "commons-io-2.11.0.jar", b"commons-io 2.11.0")
+ self.write_jar("lib-worker", "commons-io-2.13.0.jar", b"commons-io
2.13.0")
+ self.write_jar("lib", "shared-1.0.jar", b"same bytes")
+ self.write_jar("lib-worker", "shared-1.0.jar", b"same bytes")
+
+ code, out, err = self.run_dedup()
+
+ self.assertNotEqual(code, 0)
+ self.assertIn("commons-io-2.11.0.jar", err)
+ self.assertIn("commons-io-2.13.0.jar", err)
+ self.assertIn("different version", err)
+ # The whole tree is untouched on failure.
+ self.assertTrue(os.path.exists(self.path("lib",
"commons-io-2.11.0.jar")))
+ self.assertTrue(os.path.exists(self.path("lib", "shared-1.0.jar")))
+ self.assertTrue(os.path.exists(self.path("lib-worker",
"commons-io-2.13.0.jar")))
+ self.assertTrue(os.path.exists(self.path("lib-worker",
"shared-1.0.jar")))
+ self.assertFalse(os.path.exists(self.path("lib-common")))
+
+ def test_version_drift_between_lib_and_prestaged_common_fails(self):
+ # Staged-build layout: lib-common is pre-populated, lib-worker absent.
+ self.write_jar("lib", "commons-io-2.11.0.jar", b"commons-io 2.11.0")
+ self.write_jar("lib-common", "commons-io-2.13.0.jar", b"commons-io
2.13.0")
+
+ code, out, err = self.run_dedup()
+
+ self.assertNotEqual(code, 0)
+ self.assertIn("commons-io-2.11.0.jar", err)
+ self.assertIn("commons-io-2.13.0.jar", err)
+ self.assertTrue(os.path.exists(self.path("lib",
"commons-io-2.11.0.jar")))
+ self.assertTrue(os.path.exists(self.path("lib-common",
"commons-io-2.13.0.jar")))
+
+ def test_version_drift_between_worker_and_common_fails(self):
+ self.write_jar("lib-common", "commons-io-2.11.0.jar", b"commons-io
2.11.0")
+ self.write_jar("lib-worker", "commons-io-2.13.0.jar", b"commons-io
2.13.0")
+
+ code, out, err = self.run_dedup()
+
+ self.assertNotEqual(code, 0)
+ self.assertIn("commons-io-2.11.0.jar", err)
+ self.assertIn("commons-io-2.13.0.jar", err)
+ self.assertTrue(os.path.exists(self.path("lib-worker",
"commons-io-2.13.0.jar")))
+ self.assertTrue(os.path.exists(self.path("lib-common",
"commons-io-2.11.0.jar")))
+
+ def test_artifact_base(self):
+ for name, base in [
+ ("commons-io-2.13.0.jar", "commons-io"),
+ ("commons-lang3-3.12.0.jar", "commons-lang3"),
+ ("commons-collections4-4.4.jar", "commons-collections4"),
+ ("zookeeper-3.9.2.jar", "zookeeper"),
+ ("slf4j-api-2.0.13.jar", "slf4j-api"),
+ ("kafka-clients-3.7.0.jar", "kafka-clients"),
+ ("javax.servlet-api-3.1.0.jar", "javax.servlet-api"),
+ ("netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar",
+ "netty-transport-native-epoll"),
+ ("storm-shaded-deps.jar", "storm-shaded-deps"),
+ ]:
+ self.assertEqual(dedup_libs.artifact_base(name), base)
+
+ def test_worker_only_jar_is_promoted_and_counted(self):
+ self.write_jar("lib", "shared.jar", b"same bytes")
+ self.write_jar("lib-worker", "shared.jar", b"same bytes")
+ self.write_jar("lib-worker", "worker-only.jar", b"worker only")
+
+ code, out, err = self.run_dedup()
+
+ self.assertEqual(code, 0, err)
+ self.assertTrue(os.path.exists(self.path("lib-common",
"worker-only.jar")))
+ self.assertIn("1 worker-only jar(s)", out)
+
+ def test_nothing_to_do(self):
+ # No lib-worker and no lib-common at all.
+ self.write_jar("lib", "daemon-only.jar", b"daemon only")
+
+ code, out, err = self.run_dedup()
+
+ self.assertEqual(code, 0, err)
+ self.assertIn("nothing to do", out)
+ self.assertTrue(os.path.exists(self.path("lib", "daemon-only.jar")))
+
+ def test_staged_build_layout(self):
+ # Staged build: lib-common is already populated, lib-worker is absent.
+ self.write_jar("lib", "shared.jar", b"same bytes")
+ self.write_jar("lib-common", "shared.jar", b"same bytes")
+
+ code, out, err = self.run_dedup()
+
+ self.assertEqual(code, 0, err)
+ self.assertFalse(os.path.exists(self.path("lib", "shared.jar")))
+ self.assertTrue(os.path.exists(self.path("lib-common", "shared.jar")))
+ self.assertIn("removed 1 duplicate(s) from lib", out)
+ self.assertIn("0 worker-only jar(s)", out)
+
+ def test_rerun_is_idempotent(self):
+ # Running the script twice (e.g. a re-entrant build) must not fail.
+ self.write_jar("lib", "shared.jar", b"same bytes")
+ self.write_jar("lib-worker", "shared.jar", b"same bytes")
+
+ code, out, err = self.run_dedup()
+ self.assertEqual(code, 0, err)
+ code, out, err = self.run_dedup()
+ self.assertEqual(code, 0, err)
+ # On the second pass the lib copy is already gone, so the shared jar
+ # is (correctly) reported as having no counterpart in lib.
+ self.assertIn("1 worker-only jar(s)", out)
+
+ def test_usage_error(self):
+ out, err = io.StringIO(), io.StringIO()
+ with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
+ code = dedup_libs.main(["dedup-libs.py"])
+ self.assertEqual(code, 2)
+ self.assertIn("Usage", err.getvalue())
+
+
+if __name__ == "__main__":
+ unittest.main()
