[
https://issues.apache.org/jira/browse/STORM-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913064#comment-13913064
]
Adam Lewis commented on STORM-241:
----------------------------------
Interesting. Based on the concern you mention, might it need to be a separate
parameter? Controlling chunk size seems like a poor substitute for actual rate
throttling, but I don't want to cause grief for cluster operations with this
patch. My issue appears when there is a high latency link between the
submitter and nimbus (latency was dominating performance since chunks were so
small), presumably the download phase would always be within the cluster where
latency is much lower which implies that using a large chunk might cause the
problem you suggest.
> "storm jar" underperforms compared to scp
> -----------------------------------------
>
> Key: STORM-241
> URL: https://issues.apache.org/jira/browse/STORM-241
> Project: Apache Storm (Incubating)
> Issue Type: Improvement
> Environment: Internet connected dev machine deploying to storm
> cluster on AWS. Storm 0.9.0.1
> Reporter: Adam Lewis
> Labels: configuration, newbie
>
> "storm jar" is very slow at sending the jar file, achieving roughly 10% of
> the throughput compared to SCP for the same network configuration. This is
> especially problematic when deploying jars from over the Internet.
> This appears to be related to the chunk size used to transmit the JAR file
> over the thrift protocol. Anecdotally, a 300KB chunk size (compared to the
> currently used default of 15KB), achieved performance in line with scp.
> Default can be overridden in the BufferFileInputStream ctor:
> https://github.com/apache/incubator-storm/blob/master/storm-core/src/jvm/backtype/storm/StormSubmitter.java?source=cc#L154
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
