[GitHub] storm pull request #2094: [STORM-2191] shorten classpaths by using wildcards

2017-05-05 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2094


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2094: [STORM-2191] shorten classpaths by using wildcards

2017-05-04 Thread erikdw 
Github user erikdw commented on a diff in the pull request:

https://github.com/apache/storm/pull/2094#discussion_r114898011
  
--- Diff: docs/Setting-up-a-Storm-cluster.md ---
@@ -102,9 +102,9 @@ The time to allow any given healthcheck script to run 
before it is marked failed
 storm.health.check.timeout.ms: 5000
 ```
 
-### Configure external libraries and environmental variables (optional)
+### Configure external libraries and environment variables (optional)
 
-If you need support from external libraries or custom plugins, you can 
place such jars into the extlib/ and extlib-daemon/ directories. Note that the 
extlib-daemon/ directory stores jars used only by daemons (Nimbus, Supervisor, 
DRPC, UI, Logviewer), e.g., HDFS and customized scheduling libraries. 
Accordingly, two environmental variables STORM_EXT_CLASSPATH and 
STORM_EXT_CLASSPATH_DAEMON can be configured by users for including the 
external classpath and daemon-only external classpath.
+If you need support from external libraries or custom plugins, you can 
place such jars into the extlib/ and extlib-daemon/ directories. Note that the 
extlib-daemon/ directory stores jars used only by daemons (Nimbus, Supervisor, 
DRPC, UI, Logviewer), e.g., HDFS and customized scheduling libraries. 
Accordingly, two environment variables STORM_EXT_CLASSPATH and 
STORM_EXT_CLASSPATH_DAEMON can be configured by users for including the 
external classpath and daemon-only external classpath. See [Classpath 
handling](Classpath-handling.html)] for more details on using external 
libraries.
--- End diff --

I also just added a reference from the index page.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2094: [STORM-2191] shorten classpaths by using wildcards

2017-05-04 Thread erikdw 
Github user erikdw commented on a diff in the pull request:

https://github.com/apache/storm/pull/2094#discussion_r114897599
  
--- Diff: docs/Setting-up-a-Storm-cluster.md ---
@@ -102,9 +102,9 @@ The time to allow any given healthcheck script to run 
before it is marked failed
 storm.health.check.timeout.ms: 5000
 ```
 
-### Configure external libraries and environmental variables (optional)
+### Configure external libraries and environment variables (optional)
 
-If you need support from external libraries or custom plugins, you can 
place such jars into the extlib/ and extlib-daemon/ directories. Note that the 
extlib-daemon/ directory stores jars used only by daemons (Nimbus, Supervisor, 
DRPC, UI, Logviewer), e.g., HDFS and customized scheduling libraries. 
Accordingly, two environmental variables STORM_EXT_CLASSPATH and 
STORM_EXT_CLASSPATH_DAEMON can be configured by users for including the 
external classpath and daemon-only external classpath.
+If you need support from external libraries or custom plugins, you can 
place such jars into the extlib/ and extlib-daemon/ directories. Note that the 
extlib-daemon/ directory stores jars used only by daemons (Nimbus, Supervisor, 
DRPC, UI, Logviewer), e.g., HDFS and customized scheduling libraries. 
Accordingly, two environment variables STORM_EXT_CLASSPATH and 
STORM_EXT_CLASSPATH_DAEMON can be configured by users for including the 
external classpath and daemon-only external classpath. See [Classpath 
handling](Classpath-handling.html)] for more details on using external 
libraries.
--- End diff --

Thanks for the suggestions @revans2 !  I've pushed a commit that adds 
references to the new page to those 2 pages you pointed at.  I also fixed a 
typo in the markdown.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2094: [STORM-2191] shorten classpaths by using wildcards

2017-04-28 Thread revans2 
Github user revans2 commented on a diff in the pull request:

https://github.com/apache/storm/pull/2094#discussion_r113919131
  
--- Diff: docs/Setting-up-a-Storm-cluster.md ---
@@ -102,9 +102,9 @@ The time to allow any given healthcheck script to run 
before it is marked failed
 storm.health.check.timeout.ms: 5000
 ```
 
-### Configure external libraries and environmental variables (optional)
+### Configure external libraries and environment variables (optional)
 
-If you need support from external libraries or custom plugins, you can 
place such jars into the extlib/ and extlib-daemon/ directories. Note that the 
extlib-daemon/ directory stores jars used only by daemons (Nimbus, Supervisor, 
DRPC, UI, Logviewer), e.g., HDFS and customized scheduling libraries. 
Accordingly, two environmental variables STORM_EXT_CLASSPATH and 
STORM_EXT_CLASSPATH_DAEMON can be configured by users for including the 
external classpath and daemon-only external classpath.
+If you need support from external libraries or custom plugins, you can 
place such jars into the extlib/ and extlib-daemon/ directories. Note that the 
extlib-daemon/ directory stores jars used only by daemons (Nimbus, Supervisor, 
DRPC, UI, Logviewer), e.g., HDFS and customized scheduling libraries. 
Accordingly, two environment variables STORM_EXT_CLASSPATH and 
STORM_EXT_CLASSPATH_DAEMON can be configured by users for including the 
external classpath and daemon-only external classpath. See [Classpath 
handling](Classpath-handling.html)] for more details on using external 
libraries.
--- End diff --

It might be good to include a link to Classpath-handling.html in 
Running-topologies-on-a-production-cluster.md or some other page that talks 
about running a topology.  Perhaps Command-line-client.md would be good to, so 
it can explain how this works for users launching workers.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2094: [STORM-2191] shorten classpaths by using wildcards

2017-04-28 Thread erikdw 
GitHub user erikdw opened a pull request:

https://github.com/apache/storm/pull/2094

[STORM-2191] shorten classpaths by using wildcards

Instead of fully enumerating all JARs in the lib directories, we just use
a Java classpath wildcard to allow the JVM to autodiscover all JARs in the
classpath.  This affects both the Worker and LogWriter processes, as well
as Storm daemons such as the Nimbus, UI, Logviewer, and Supervisor.

This change results in shorter commands, so that you can actually see
the full content of the Worker command in `ps` output on Linux.

Prior to this change Worker commands were easily longer than 4096 bytes,
which is the default Linux kernel limit for commands being recorded into
the process table.  Longer commands get truncated, though they do get
executed.

An example of the change in Worker classpath length can be seen here:

Before:
```
-cp 
STORM_DIR/lib-worker/asm-5.0.3.jar:STORM_DIR/lib-worker/carbonite-1.5.0.jar:STORM_DIR/lib-worker/chill-java-0.8.0.jar:STORM_DIR/lib-worker/clojure-1.7.0.jar:STORM_DIR/lib-worker/commons-codec-1.6.jar:STORM_DIR/lib-worker/commons-collections-3.2.2.jar:STORM_DIR/lib-worker/commons-io-2.5.jar:STORM_DIR/lib-worker/commons-lang-2.5.jar:STORM_DIR/lib-worker/commons-logging-1.1.3.jar:STORM_DIR/lib-worker/curator-client-2.12.0.jar:STORM_DIR/lib-worker/curator-framework-2.12.0.jar:STORM_DIR/lib-worker/disruptor-3.3.2.jar:STORM_DIR/lib-worker/guava-16.0.1.jar:STORM_DIR/lib-worker/httpclient-4.3.3.jar:STORM_DIR/lib-worker/httpcore-4.4.1.jar:STORM_DIR/lib-worker/jgrapht-core-0.9.0.jar:STORM_DIR/lib-worker/jline-0.9.94.jar:STORM_DIR/lib-worker/json-simple-1.1.jar:STORM_DIR/lib-worker/kryo-3.0.3.jar:STORM_DIR/lib-worker/kryo-shaded-3.0.3.jar:STORM_DIR/lib-worker/libthrift-0.9.3.jar:STORM_DIR/lib-worker/log4j-api-2.8.jar:STORM_DIR/lib-worker/log4j-core-2.8.jar:STORM_DIR/lib-worker/log4j-ove
 
r-slf4j-1.6.6.jar:STORM_DIR/lib-worker/log4j-slf4j-impl-2.8.jar:STORM_DIR/lib-worker/minlog-1.3.0.jar:STORM_DIR/lib-worker/netty-3.9.0.Final.jar:STORM_DIR/lib-worker/objenesis-2.1.jar:STORM_DIR/lib-worker/reflectasm-1.10.1.jar:STORM_DIR/lib-worker/servlet-api-2.5.jar:STORM_DIR/lib-worker/slf4j-api-1.7.21.jar:STORM_DIR/lib-worker/snakeyaml-1.11.jar:STORM_DIR/lib-worker/storm-client-2.0.0-SNAPSHOT.jar:STORM_DIR/lib-worker/sysout-over-slf4j-1.0.2.jar:STORM_DIR/lib-worker/zookeeper-3.4.6.jar:STORM_DIR/conf:STORM_DIR/storm-local/supervisor/stormdist/foo-topology-1-1-1493359573/stormjar.jar
```

After:
```
-cp 
STORM_DIR/lib-worker/*:STORM_DIR/extlib/*:STORM_DIR/conf:STORM_DIR/storm-local/supervisor/stormdist/foo-topology-1-1-1493359573/stormjar.jar
```

This change also includes additional documentation about the use of 
classpaths in
Storm and provides some guidance for using the various features for using 
external
libraries.

For more details on this problem and a discussion about this solution's
merits, please see 
[STORM-2191](https://issues.apache.org/jira/browse/STORM-2191).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/erikdw/storm STORM-2191

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2094.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2094


commit 2ceef0dce21895c4ef0f0b8eb5bc8248fad25e13
Author: Erik Weathers 
Date:   2017-04-28T07:50:18Z

[STORM-2191] shorten classpaths by using wildcards

Instead of fully enumerating all JARs in the lib directories, we just use
a Java classpath wildcard to allow the JVM to autodiscover all JARs in the
classpath.  This affects both the Worker and LogWriter processes, as well
as Storm daemons such as the Nimbus, UI, Logviewer, and Supervisor.

This change results in shorter commands, so that you can actually see
the full content of the Worker command in `ps` output on Linux.

Prior to this change Worker commands were easily longer than 4096 bytes,
which is the default Linux kernel limit for commands being recorded into
the process table.  Longer commands get truncated, though they do get
executed.

An example of the change in Worker classpath length can be seen here:

Before:
```
-cp