StaticMian created STORM-3396:
---------------------------------
Summary: uploading dependency jars too slow when StormSubmitter
and Nimbus located in different IDC
Key: STORM-3396
URL: https://issues.apache.org/jira/browse/STORM-3396
Project: Apache Storm
Issue Type: Improvement
Components: storm-core
Affects Versions: 1.2.2
Reporter: StaticMian
Fix For: 2.0.0
when storm client and server is locating in different IDC(one is in
Beijing, while another in Shanghai), uploading dependency jars may take a very
long long time(in my case, 31minutes!)...
when I digged into this, I found that in DependencyUploader, method
"uploadDependencyToBlobStore" using JDK NIO's Files.copy to upload local jars
to remote Blob server. In Files.copy(Path, OutputStream), the buffer size is 8k
by default, given that latency between Beijing and Shanghai is about 20ms, a
dependency fat jar of 360M finally cost me 'a lunch time' to finish uploading!!!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
