[
https://issues.apache.org/jira/browse/STORM-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984124#comment-15984124
]
Erik Weathers commented on STORM-2191:
--------------------------------------
[~revans2]: so the existing conflicting classes in {{lib-worker}} are arising
between {{kryo-shaded-3.0.3.jar}} and 3 other jars:
* {{asm-5.0.3.jar}}
* {{kryo-3.0.3.jar}}
* {{reflectasm-1.10.1.jar}}
I'm not sure what the purpose of kryo-shaded *is*. It's including stuff from
all these jars, but most aren't actually shaded into different package paths
than the original package paths (in {{kryo-3.0.3.jar}} and
{{reflectasm-1.10.jar}}). The only classes that are shaded into different
package paths are some of the objectweb ones from {{asm-5.0.3.jar}}, but not
all classes from that jar are included into kryo-shaded... i.e., these aren't
included:
* {{org/objectweb/asm/TypeReference.class}}
* {{org/objectweb/asm/signature/SignatureReader.class}}
* {{org/objectweb/asm/signature/SignatureVisitor.class}}
* {{org/objectweb/asm/signature/SignatureWriter.class}}
This conflicting-classes investigation feels like a diversion from the goal of
this issue, maybe we should open a separate issue to continue discussion of
these conflicts, which should be resolved anyways.
[~sriharsha] : if I'm understanding your proposal correctly, we would still be
"promoting" the Worker and LogWriter java-main arguments into the front of the
commands, we'd just be adding another facility that allows a topology-specific
configurable string to also be "promoted" to the front of the cmds. To be 100%
clear, that topology-specific string would not suffice on its own, we still
need info that is variable between worker processes and topology instances
(worker port, topology ID).
Another wrinkle with this proposal is we should do it for all daemons really,
not just Worker and LogWriter. Nimbus, Logviewer, UI, etc. all can have these
gigantic classpaths because of this classpath enumeration behavior.
Finally, the output of {{ps}} is so much more visually parseable when the lines
aren't 4000+ characters long. Having a command that fills an entire terminal
screen is hard to work with. I personally really dislike looking at huge
hadoop and storm cmds and having to run them through colorized {{grep}} and/or
{{tr}} to be able to see what is going on.
> shorten classpaths in worker and LogWriter commands
> ---------------------------------------------------
>
> Key: STORM-2191
> URL: https://issues.apache.org/jira/browse/STORM-2191
> Project: Apache Storm
> Issue Type: Task
> Components: storm-core
> Affects Versions: 1.0.2
> Reporter: Erik Weathers
> Priority: Minor
> Labels: cli, command-line
>
> When launching the worker daemon and its wrapping LogWriter daemon, the
> commands can become so long that they eclipse the default Linux limit of 4096
> bytes. That results in commands that are cut off in {{ps}} output, and
> prevents easily inspecting the system to see even what processes are running.
> The specific scenario in which this problem can be easily triggered: *running
> Storm on Mesos*.
> h5. Details on why it happens:
> # using the default Mesos containerizer instead of Docker containers, which
> causes the storm-mesos package to be unpacked into the Mesos executor sandbox.
> # The ["expand all jars on
> classpath"|https://github.com/apache/storm/blob/6dc6407a01d032483edebb1c1b4d8b69a304d81c/bin/storm.py#L114-L140]
> functionality in the {{bin/storm.py}} script causes every one of the jars
> that storm bundles into its lib directory to be explicitly listed in the
> command.
> #* e.g., say the mesos work dir is {{/var/run/mesos/work_dir/}}
> #* and say that the original classpath argument in the supervisor cmd
> includes the following for the {{lib/}} dir in the binary storm package:
> #**
> {{/var/run/mesos/work_dir/slaves/2357b762-6653-4052-ab9e-f1354d78991b-S12/frameworks/20160509-084241-1086985738-5050-32231-0000/executors/STORM_TOPOLOGY_ID/runs/e6a1407e-73fd-4be4-8d00-e882117b3391/storm-mesos-0.1.7-storm0.9.6-mesos0.28.2/lib/*}}
> #* That leads to a hugely expanded classpath argument for the LogWriter and
> Worker daemons that get launched:
> #**
> {{/var/run/mesos/work_dir/slaves/2357b762-6653-4052-ab9e-f1354d78991b-S12/frameworks/20160509-084241-1086985738-5050-32231-0000/executors/STORM_TOPOLOGY_ID/runs/e6a1407e-73fd-4be4-8d00-e882117b3391/storm-mesos-0.1.7-storm0.9.6-mesos0.28.2/lib/asm-4.0.jar:/var/run/mesos/work_dir/slaves/2357b762-6653-4052-ab9e-f1354d78991b-S12/frameworks/20160509-084241-1086985738-5050-32231-0000/executors/STORM_TOPOLOGY_ID/runs/e6a1407e-73fd-4be4-8d00-e882117b3391/storm-mesos-0.1.7-storm0.9.6-mesos0.28.2/lib/carbonite-1.4.0.jar:...}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
