[
https://issues.apache.org/jira/browse/STORM-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913084#comment-13913084
]
Robert Joseph Evans commented on STORM-241:
-------------------------------------------
For a single upload to nimbus it should not be a big deal. Even 4GB on gigabit
Ethernet is only going to take 30 secs to a min, and that is a one time
transfer to a single machine. My bigger concern is with downloading to
potentially hundreds of nodes. If Taylor is going to resurrect the bit-torrent
patch, which seems much more like the right solution overall, I don't see a
reason to improve code that will go away in the near future. If we start to
see problems with the upload we can look at adding in throttling then.
Speaking of configs, I am a bit hesitant to even make chunk size a config
option. If 300KB is just plain better than 15KB then why not hard code it,
unless you see a potentially serious problem? Hadoop has hundreds and hundreds
of config options that no one understand or changes, I would like to avoid that
for storm if possible.
> "storm jar" underperforms compared to scp
> -----------------------------------------
>
> Key: STORM-241
> URL: https://issues.apache.org/jira/browse/STORM-241
> Project: Apache Storm (Incubating)
> Issue Type: Improvement
> Environment: Internet connected dev machine deploying to storm
> cluster on AWS. Storm 0.9.0.1
> Reporter: Adam Lewis
> Labels: configuration, newbie
>
> "storm jar" is very slow at sending the jar file, achieving roughly 10% of
> the throughput compared to SCP for the same network configuration. This is
> especially problematic when deploying jars from over the Internet.
> This appears to be related to the chunk size used to transmit the JAR file
> over the thrift protocol. Anecdotally, a 300KB chunk size (compared to the
> currently used default of 15KB), achieved performance in line with scp.
> Default can be overridden in the BufferFileInputStream ctor:
> https://github.com/apache/incubator-storm/blob/master/storm-core/src/jvm/backtype/storm/StormSubmitter.java?source=cc#L154
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
