[
https://issues.apache.org/jira/browse/STORM-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981183#comment-15981183
]
Robert Joseph Evans commented on STORM-2191:
--------------------------------------------
[~erikdw],
Sorry I didn't see your question before. The issue with using * in a java
classpath is that ordering of the jars is not consistent. For Hadoop we have
had all kinds of issues where a failure will only happen on some boxes and not
on others because the jars on the classpath will be written in different orders
and the underlying file system will return the files in different a different
order. We have not had too many issues with this in storm, but the operation
was meant to be defensive.
If we really do want to reduce the command line we can probably play some games
with jars like surefire does with a manifest only jar, but they automatically
disable that in some situations so I don't know enough about why they are doing
that.
> shorten classpaths in worker and LogWriter commands
> ---------------------------------------------------
>
> Key: STORM-2191
> URL: https://issues.apache.org/jira/browse/STORM-2191
> Project: Apache Storm
> Issue Type: Task
> Components: storm-core
> Affects Versions: 1.0.2
> Reporter: Erik Weathers
> Priority: Minor
> Labels: cli, command-line
>
> When launching the worker daemon and its wrapping LogWriter daemon, the
> commands can become so long that they eclipse the default Linux limit of 4096
> bytes. That results in commands that are cut off in {{ps}} output, and
> prevents easily inspecting the system to see even what processes are running.
> The specific scenario in which this problem can be easily triggered: *running
> Storm on Mesos*.
> h5. Details on why it happens:
> # using the default Mesos containerizer instead of Docker containers, which
> causes the storm-mesos package to be unpacked into the Mesos executor sandbox.
> # The ["expand all jars on
> classpath"|https://github.com/apache/storm/blob/6dc6407a01d032483edebb1c1b4d8b69a304d81c/bin/storm.py#L114-L140]
> functionality in the {{bin/storm.py}} script causes every one of the jars
> that storm bundles into its lib directory to be explicitly listed in the
> command.
> #* e.g., say the mesos work dir is {{/var/run/mesos/work_dir/}}
> #* and say that the original classpath argument in the supervisor cmd
> includes the following for the {{lib/}} dir in the binary storm package:
> #**
> {{/var/run/mesos/work_dir/slaves/2357b762-6653-4052-ab9e-f1354d78991b-S12/frameworks/20160509-084241-1086985738-5050-32231-0000/executors/STORM_TOPOLOGY_ID/runs/e6a1407e-73fd-4be4-8d00-e882117b3391/storm-mesos-0.1.7-storm0.9.6-mesos0.28.2/lib/*}}
> #* That leads to a hugely expanded classpath argument for the LogWriter and
> Worker daemons that get launched:
> #**
> {{/var/run/mesos/work_dir/slaves/2357b762-6653-4052-ab9e-f1354d78991b-S12/frameworks/20160509-084241-1086985738-5050-32231-0000/executors/STORM_TOPOLOGY_ID/runs/e6a1407e-73fd-4be4-8d00-e882117b3391/storm-mesos-0.1.7-storm0.9.6-mesos0.28.2/lib/asm-4.0.jar:/var/run/mesos/work_dir/slaves/2357b762-6653-4052-ab9e-f1354d78991b-S12/frameworks/20160509-084241-1086985738-5050-32231-0000/executors/STORM_TOPOLOGY_ID/runs/e6a1407e-73fd-4be4-8d00-e882117b3391/storm-mesos-0.1.7-storm0.9.6-mesos0.28.2/lib/carbonite-1.4.0.jar:...}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
