GitHub user erikdw opened a pull request:
https://github.com/apache/storm/pull/2094
[STORM-2191] shorten classpaths by using wildcards
Instead of fully enumerating all JARs in the lib directories, we just use
a Java classpath wildcard to allow the JVM to autodiscover all JARs in the
classpath. This affects both the Worker and LogWriter processes, as well
as Storm daemons such as the Nimbus, UI, Logviewer, and Supervisor.
This change results in shorter commands, so that you can actually see
the full content of the Worker command in `ps` output on Linux.
Prior to this change Worker commands were easily longer than 4096 bytes,
which is the default Linux kernel limit for commands being recorded into
the process table. Longer commands get truncated, though they do get
executed.
An example of the change in Worker classpath length can be seen here:
Before:
```
-cp
STORM_DIR/lib-worker/asm-5.0.3.jar:STORM_DIR/lib-worker/carbonite-1.5.0.jar:STORM_DIR/lib-worker/chill-java-0.8.0.jar:STORM_DIR/lib-worker/clojure-1.7.0.jar:STORM_DIR/lib-worker/commons-codec-1.6.jar:STORM_DIR/lib-worker/commons-collections-3.2.2.jar:STORM_DIR/lib-worker/commons-io-2.5.jar:STORM_DIR/lib-worker/commons-lang-2.5.jar:STORM_DIR/lib-worker/commons-logging-1.1.3.jar:STORM_DIR/lib-worker/curator-client-2.12.0.jar:STORM_DIR/lib-worker/curator-framework-2.12.0.jar:STORM_DIR/lib-worker/disruptor-3.3.2.jar:STORM_DIR/lib-worker/guava-16.0.1.jar:STORM_DIR/lib-worker/httpclient-4.3.3.jar:STORM_DIR/lib-worker/httpcore-4.4.1.jar:STORM_DIR/lib-worker/jgrapht-core-0.9.0.jar:STORM_DIR/lib-worker/jline-0.9.94.jar:STORM_DIR/lib-worker/json-simple-1.1.jar:STORM_DIR/lib-worker/kryo-3.0.3.jar:STORM_DIR/lib-worker/kryo-shaded-3.0.3.jar:STORM_DIR/lib-worker/libthrift-0.9.3.jar:STORM_DIR/lib-worker/log4j-api-2.8.jar:STORM_DIR/lib-worker/log4j-core-2.8.jar:STORM_DIR/lib-worker/log4j-ove
r-slf4j-1.6.6.jar:STORM_DIR/lib-worker/log4j-slf4j-impl-2.8.jar:STORM_DIR/lib-worker/minlog-1.3.0.jar:STORM_DIR/lib-worker/netty-3.9.0.Final.jar:STORM_DIR/lib-worker/objenesis-2.1.jar:STORM_DIR/lib-worker/reflectasm-1.10.1.jar:STORM_DIR/lib-worker/servlet-api-2.5.jar:STORM_DIR/lib-worker/slf4j-api-1.7.21.jar:STORM_DIR/lib-worker/snakeyaml-1.11.jar:STORM_DIR/lib-worker/storm-client-2.0.0-SNAPSHOT.jar:STORM_DIR/lib-worker/sysout-over-slf4j-1.0.2.jar:STORM_DIR/lib-worker/zookeeper-3.4.6.jar:STORM_DIR/conf:STORM_DIR/storm-local/supervisor/stormdist/foo-topology-1-1-1493359573/stormjar.jar
```
After:
```
-cp
STORM_DIR/lib-worker/*:STORM_DIR/extlib/*:STORM_DIR/conf:STORM_DIR/storm-local/supervisor/stormdist/foo-topology-1-1-1493359573/stormjar.jar
```
This change also includes additional documentation about the use of
classpaths in
Storm and provides some guidance for using the various features for using
external
libraries.
For more details on this problem and a discussion about this solution's
merits, please see
[STORM-2191](https://issues.apache.org/jira/browse/STORM-2191).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/erikdw/storm STORM-2191
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/2094.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2094
----
commit 2ceef0dce21895c4ef0f0b8eb5bc8248fad25e13
Author: Erik Weathers <[email protected]>
Date: 2017-04-28T07:50:18Z
[STORM-2191] shorten classpaths by using wildcards
Instead of fully enumerating all JARs in the lib directories, we just use
a Java classpath wildcard to allow the JVM to autodiscover all JARs in the
classpath. This affects both the Worker and LogWriter processes, as well
as Storm daemons such as the Nimbus, UI, Logviewer, and Supervisor.
This change results in shorter commands, so that you can actually see
the full content of the Worker command in `ps` output on Linux.
Prior to this change Worker commands were easily longer than 4096 bytes,
which is the default Linux kernel limit for commands being recorded into
the process table. Longer commands get truncated, though they do get
executed.
An example of the change in Worker classpath length can be seen here:
Before:
```
-cp
STORM_DIR/lib-worker/asm-5.0.3.jar:STORM_DIR/lib-worker/carbonite-1.5.0.jar:STORM_DIR/lib-worker/chill-java-0.8.0.jar:STORM_DIR/lib-worker/clojure-1.7.0.jar:STORM_DIR/lib-worker/commons-codec-1.6.jar:STORM_DIR/lib-worker/commons-collections-3.2.2.jar:STORM_DIR/lib-worker/commons-io-2.5.jar:STORM_DIR/lib-worker/commons-lang-2.5.jar:STORM_DIR/lib-worker/commons-logging-1.1.3.jar:STORM_DIR/lib-worker/curator-client-2.12.0.jar:STORM_DIR/lib-worker/curator-framework-2.12.0.jar:STORM_DIR/lib-worker/disruptor-3.3.2.jar:STORM_DIR/lib-worker/guava-16.0.1.jar:STORM_DIR/lib-worker/httpclient-4.3.3.jar:STORM_DIR/lib-worker/httpcore-4.4.1.jar:STORM_DIR/lib-worker/jgrapht-core-0.9.0.jar:STORM_DIR/lib-worker/jline-0.9.94.jar:STORM_DIR/lib-worker/json-simple-1.1.jar:STORM_DIR/lib-worker/kryo-3.0.3.jar:STORM_DIR/lib-worker/kryo-shaded-3.0.3.jar:STORM_DIR/lib-worker/libthrift-0.9.3.jar:STORM_DIR/lib-worker/log4j-api-2.8.jar:STORM_DIR/lib-worker/log4j-core-2.8.jar:STORM_DIR/lib-worker/log4j-ove
r-slf4j-1.6.6.jar:STORM_DIR/lib-worker/log4j-slf4j-impl-2.8.jar:STORM_DIR/lib-worker/minlog-1.3.0.jar:STORM_DIR/lib-worker/netty-3.9.0.Final.jar:STORM_DIR/lib-worker/objenesis-2.1.jar:STORM_DIR/lib-worker/reflectasm-1.10.1.jar:STORM_DIR/lib-worker/servlet-api-2.5.jar:STORM_DIR/lib-worker/slf4j-api-1.7.21.jar:STORM_DIR/lib-worker/snakeyaml-1.11.jar:STORM_DIR/lib-worker/storm-client-2.0.0-SNAPSHOT.jar:STORM_DIR/lib-worker/sysout-over-slf4j-1.0.2.jar:STORM_DIR/lib-worker/zookeeper-3.4.6.jar:STORM_DIR/conf:STORM_DIR/storm-local/supervisor/stormdist/foo-topology-1-1-1493359573/stormjar.jar
```
After:
```
-cp
STORM_DIR/lib-worker/*:STORM_DIR/extlib/*:STORM_DIR/conf:STORM_DIR/storm-local/supervisor/stormdist/foo-topology-1-1-1493359573/stormjar.jar
```
This change also includes additional documentation about the use of
classpaths in
Storm and provides some guidance for using the various features for using
external
libraries.
For more details on this problem and a discussion about this solution's
merits, please see
[STORM-2191](https://issues.apache.org/jira/browse/STORM-2191).
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
