Hi,
I've been encountering an out of memory error and it's not because of the
datastructures I'm using. I verified this with htop. The Storm worker seems
to be crashing. Could this be because too many tuples are being sent to the
queue of a bolt? Would it help to introduce a delay or is this a completely
different reason?
I took a heapdump and on analyzing it, I found this:
"Netty-server-localhost-6701-worker-1" prio=5 tid=27 RUNNABLE
at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:48)
at
sun.reflect.GeneratedSerializationConstructorAccessor15.newInstance(<unknown
string>)
Local Variable: java.lang.Object[]#294
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.io.ObjectStreamClass.newInstance(ObjectStreamClass.java:1006)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1792)
Local Variable: java.io.ObjectStreamClass#1
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
Local Variable: java.io.ObjectInputStream#1
at
org.apache.storm.serialization.SerializableSerializer.read(SerializableSerializer.java:56)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
Local Variable: java.util.ArrayList#201
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
Local Variable: com.esotericsoftware.kryo.io.Input#61
Local Variable: org.apache.storm.utils.ListDelegate#1
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689)
Local Variable:
org.apache.storm.serialization.DefaultKryoFactory$KryoSerializableDefault#61
at
org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:37)
at
org.apache.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:50)
Local Variable: java.lang.String#20627
Local Variable: org.apache.storm.tuple.MessageId#95
at
org.apache.storm.messaging.DeserializingConnectionCallback.recv(DeserializingConnectionCallback.java:56)
Local Variable: java.util.ArrayList#11
Local Variable:
org.apache.storm.messaging.DeserializingConnectionCallback#1
Local Variable:
org.apache.storm.serialization.KryoTupleDeserializer#1
at org.apache.storm.messaging.netty.Server.enqueue(Server.java:133)
at org.apache.storm.messaging.netty.Server.received(Server.java:254)
at
org.apache.storm.messaging.netty.StormServerHandler.messageReceived(StormServerHandler.java:61)
Local Variable:
org.apache.storm.messaging.netty.StormServerHandler#17
at
org.apache.storm.shade.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
Local Variable:
org.apache.storm.shade.org.jboss.netty.channel.UpstreamMessageEvent#2
at
org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.apache.storm.shade.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.apache.storm.shade.org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.apache.storm.shade.org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
Local Variable: java.net.InetSocketAddress#150
at
org.apache.storm.shade.org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:310)
Local Variable: org.apache.storm.messaging.netty.MessageDecoder#38
Local Variable:
org.apache.storm.shade.org.jboss.netty.buffer.CompositeChannelBuffer#1
Local Variable:
org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext#115
at
org.apache.storm.shade.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
Local Variable:
org.apache.storm.shade.org.jboss.netty.channel.UpstreamMessageEvent#1
Local Variable:
org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline#39
at
org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.apache.storm.shade.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.apache.storm.shade.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.apache.storm.shade.org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
Local Variable:
org.apache.storm.shade.org.jboss.netty.channel.socket.nio.NioAcceptedSocketChannel#17
at
org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
Local Variable: java.util.HashMap$KeyIterator#1
Local Variable: sun.nio.ch.SelectionKeyImpl#40
at
org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
Local Variable: sun.nio.ch.EPollSelectorImpl#3
at
org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.apache.storm.shade.org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
Local Variable:
org.apache.storm.shade.org.jboss.netty.channel.socket.nio.NioWorker#2
at
org.apache.storm.shade.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
Local Variable: java.lang.String#20742
Local Variable:
org.apache.storm.shade.org.jboss.netty.util.ThreadRenamingRunnable#4
at
org.apache.storm.shade.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
Local Variable: java.util.concurrent.ThreadPoolExecutor#7
Local Variable:
org.apache.storm.shade.org.jboss.netty.util.internal.DeadLockProofWorker$1#2
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#4
at java.lang.Thread.run(Thread.java:745)
There are more errors, but this one was the one highlighted in red.
--
Regards,
Navin
