[
https://issues.apache.org/jira/browse/STORM-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912989#comment-13912989
]
Robert Joseph Evans commented on STORM-241:
-------------------------------------------
I am not sure if you want to put it into nimbus.clj as part of
beginFileDownload where it creates the BufferFileInputStream. That does
something very similar, but for downloading the file instead of uploading it.
My biggest concern with that part though, is if someone tries to download a
very large file, say 2+GB to many different nodes it could saturate the network
for an extended period of time. During the discussion about using bit-torrent
to download the files instead of thrift, network throttling was a requirement
for it. That has not gone in yet, but I would suspect we would want something
similar here.
> "storm jar" underperforms compared to scp
> -----------------------------------------
>
> Key: STORM-241
> URL: https://issues.apache.org/jira/browse/STORM-241
> Project: Apache Storm (Incubating)
> Issue Type: Improvement
> Environment: Internet connected dev machine deploying to storm
> cluster on AWS. Storm 0.9.0.1
> Reporter: Adam Lewis
> Labels: configuration, newbie
>
> "storm jar" is very slow at sending the jar file, achieving roughly 10% of
> the throughput compared to SCP for the same network configuration. This is
> especially problematic when deploying jars from over the Internet.
> This appears to be related to the chunk size used to transmit the JAR file
> over the thrift protocol. Anecdotally, a 300KB chunk size (compared to the
> currently used default of 15KB), achieved performance in line with scp.
> Default can be overridden in the BufferFileInputStream ctor:
> https://github.com/apache/incubator-storm/blob/master/storm-core/src/jvm/backtype/storm/StormSubmitter.java?source=cc#L154
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
