I've been working on a Storm topology and recently it has become impossible
to submit my topology. It just times out. The only thing that has changed
since I was last able to successfully submit the topology is the size of
the jar/logic of the topology. It's worth mentioning that it has randomly
succeeded twice but fails 99% of the time. Log messages and details are
below. Any clue what's going on here?
Jar size: 28 MB
Storm version: 0.9.0_wip17_netty
Error message in nimbus.log
014-06-24 18:52:54 b.s.d.nimbus [INFO] Uploading file from client to
var/storm-local/nimbus/inbox/stormjar-b67d69bf-7ff1-4c99-954c-11924268d354.jar
2014-06-24 18:54:11 o.a.t.s.TNonblockingServer [WARN] Got an IOException in
internalRead!
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.7.0_55]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
~[na:1.7.0_55]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
~[na:1.7.0_55]
at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.7.0_55]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
~[na:1.7.0_55]
at
org.apache.thrift7.transport.TNonblockingSocket.read(TNonblockingSocket.java:141)
~[libthrift7-0.7.0.2.jar:0.7.0-2]
at
org.apache.thrift7.server.TNonblockingServer$FrameBuffer.internalRead(TNonblockingServer.java:669)
[libthrift7-0.7.0.2.jar:0.7.0-2]
at
org.apache.thrift7.server.TNonblockingServer$FrameBuffer.read(TNonblockingServer.java:458)
[libthrift7-0.7.0.2.jar:0.7.0-2]
at
org.apache.thrift7.server.TNonblockingServer$SelectThread.handleRead(TNonblockingServer.java:359)
[libthrift7-0.7.0.2.jar:0.7.0-2]
at
org.apache.thrift7.server.TNonblockingServer$SelectThread.select(TNonblockingServer.java:304)
[libthrift7-0.7.0.2.jar:0.7.0-2]
at
org.apache.thrift7.server.TNonblockingServer$SelectThread.run(TNonblockingServer.java:243)
[libthrift7-0.7.0.2.jar:0.7.0-2]
Error message on storm client:
670 [main] INFO backtype.storm.StormSubmitter - Jar not uploaded to
master yet. Submitting jar...
834 [main] INFO backtype.storm.StormSubmitter - Uploading topology jar
build/lib/TestTopology-1.0.jar to assigned location:
var/storm-local/nimbus/inbox/stormjar-b67d69bf-7ff1-4c99-954c-11924268d354.jar
150532 [main] INFO backtype.storm.StormSubmitter - Successfully uploaded
topology jar to assigned location:
var/storm-local/nimbus/inbox/stormjar-b67d69bf-7ff1-4c99-954c-11924268d354.jar
150532 [main] INFO backtype.storm.StormSubmitter - Submitting topology
TestTopology in distributed mode with conf {"topology.workers":12}
Exception in thread "main" java.lang.RuntimeException:
org.apache.thrift7.transport.TTransportException: java.net.SocketException:
Connection reset
at
backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:112)
at
backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:58)
at com.TestTopology.main(TestTopology.java:83)
Caused by: org.apache.thrift7.transport.TTransportException:
java.net.SocketException: Connection reset
at
org.apache.thrift7.transport.TIOStreamTransport.write(TIOStreamTransport.java:147)
at
org.apache.thrift7.transport.TFramedTransport.flush(TFramedTransport.java:156)
at
org.apache.thrift7.TServiceClient.sendBase(TServiceClient.java:65)
at
backtype.storm.generated.Nimbus$Client.send_submitTopology(Nimbus.java:156)
at
backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:145)
at
backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:98)
... 2 more
Caused by: java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at
org.apache.thrift7.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
... 7 more
