Hi All, We have started a two node long running MB cluster test to find long running issues. Test started on 2015-03-25. On 2015-03-28 both nodes has gone out of memory. Subscriber machine has gone down due to unknown reason as well.
Following are the OM errors extracted from the heap dump from each node. *Coordinator Node OM Error* - Sat Mar 28 22:57:16 IST 2015 "Thread-23" daemon prio=5 tid=98 RUNNABLE at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:48) at sun.reflect.ByteVectorImpl.resize(ByteVectorImpl.java:84) at sun.reflect.ByteVectorImpl.add(ByteVectorImpl.java:63) at sun.reflect.ClassFileAssembler.emitByte(ClassFileAssembler.java:74) at sun.reflect.ClassFileAssembler.emitConstantPoolUTF8(ClassFileAssembler.java:107) Local Variable: sun.reflect.ClassFileAssembler#1 Local Variable: byte[]#29857 at sun.reflect.AccessorGenerator.emitCommonConstantPoolEntries(AccessorGenerator.java:144) at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:351) Local Variable: java.lang.Class#1 Local Variable: java.lang.String#53810 Local Variable: sun.reflect.ByteVectorImpl#1 Local Variable: java.lang.String#19366 at sun.reflect.MethodAccessorGenerator.generateConstructor(MethodAccessorGenerator.java:94) Local Variable: class com.hazelcast.map.operation.MapKeySetOperation Local Variable: java.lang.Class[]#2 Local Variable: java.lang.Class[]#3 Local Variable: sun.reflect.MethodAccessorGenerator#1 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:48) Local Variable: java.lang.Object[]#25884 Local Variable: sun.reflect.NativeConstructorAccessorImpl#1 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.hazelcast.nio.ClassLoaderUtil.newInstance(ClassLoaderUtil.java:62) at com.hazelcast.nio.serialization.DataSerializer.read(DataSerializer.java:103) Local Variable: java.lang.String#19288 at com.hazelcast.nio.serialization.DataSerializer.read(DataSerializer.java:36) at com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:59) Local Variable: com.hazelcast.nio.serialization.ByteArrayObjectDataInput#2 at com.hazelcast.nio.serialization.SerializationServiceImpl.toObject(SerializationServiceImpl.java:221) Local Variable: com.hazelcast.nio.serialization.SerializationServiceImpl#1 at com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:156) at com.hazelcast.spi.impl.BinaryOperationFactory.createOperation(BinaryOperationFactory.java:52) at com.hazelcast.spi.impl.PartitionIteratingOperation.run(PartitionIteratingOperation.java:51) Local Variable: com.hazelcast.spi.impl.NodeEngineImpl#1 Local Variable: com.hazelcast.spi.impl.PartitionIteratingOperation$ResponseQueue#55 Local Variable: java.util.HashMap#99 at com.hazelcast.spi.impl.BasicOperationService.processOperation(BasicOperationService.java:363) at com.hazelcast.spi.impl.BasicOperationService.runOperation(BasicOperationService.java:228) Local Variable: com.hazelcast.spi.impl.PartitionIteratingOperation#1 at com.hazelcast.spi.impl.BasicInvocation.doInvoke(BasicInvocation.java:371) at com.hazelcast.spi.impl.BasicInvocation.invoke(BasicInvocation.java:190) Local Variable: com.hazelcast.spi.impl.BasicTargetInvocation#1 at com.hazelcast.spi.impl.BasicInvocationBuilder.invoke(BasicInvocationBuilder.java:48) at com.hazelcast.spi.impl.BasicOperationService.invokeOnPartitions(BasicOperationService.java:587) Local Variable: java.util.HashMap#97 Local Variable: java.util.HashMap$EntryIterator#1 Local Variable: com.hazelcast.nio.Address#1 at com.hazelcast.spi.impl.BasicOperationService.invokeOnAllPartitions(BasicOperationService.java:549) Local Variable: com.hazelcast.spi.impl.BasicOperationService#1 Local Variable: com.hazelcast.spi.impl.BinaryOperationFactory#1 Local Variable: java.util.HashMap#30549 Local Variable: java.lang.String#146076 at com.hazelcast.map.proxy.MapProxySupport.keySetInternal(MapProxySupport.java:590) at com.hazelcast.map.proxy.MapProxyImpl.keySet(MapProxyImpl.java:481) Local Variable: com.hazelcast.map.proxy.MapProxyImpl#20 at org.wso2.andes.kernel.slot.SlotManagerClusterMode.getMessagePublishedNodes(SlotManagerClusterMode.java:675) Local Variable: org.wso2.andes.kernel.slot.SlotManagerClusterMode#1 at org.wso2.andes.kernel.slot.SlotDeleteSafeZoneCalc.run(SlotDeleteSafeZoneCalc.java:64) Local Variable: org.wso2.andes.kernel.slot.SlotDeleteSafeZoneCalc#1 at java.lang.Thread.run(Thread.java:745) *Non-coordinator Node* - Sat Mar 28 20:32:15 IST 2015 "SocketAcceptorIoProcessor-0.0" daemon prio=5 tid=301 RUNNABLE at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:48) at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) at java.nio.ByteBuffer.allocate(ByteBuffer.java:331) at org.apache.mina.common.SimpleByteBufferAllocator.allocate(SimpleByteBufferAllocator.java:45) at org.apache.mina.common.ByteBuffer.allocate(ByteBuffer.java:225) at org.apache.mina.common.ByteBuffer.allocate(ByteBuffer.java:214) at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(SocketIoProcessor.java:210) at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(SocketIoProcessor.java:198) Local Variable: org.apache.mina.transport.socket.nio.SocketIoProcessor#1 Local Variable: java.util.HashMap$KeyIterator#2 Local Variable: sun.nio.ch.SelectionKeyImpl#15 Local Variable: org.apache.mina.transport.socket.nio.SocketSessionImpl#9 Local Variable: sun.nio.ch.Util$2#17 at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$400(SocketIoProcessor.java:45) at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProcessor.java:485) Local Variable: org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker#3 Local Variable: sun.nio.ch.EPollSelectorImpl#49 at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51) Local Variable: org.apache.mina.util.NamePreservingRunnable#6 Local Variable: java.lang.String#148830 at java.lang.Thread.run(Thread.java:745) What can be the cause for this issue ? WDYT ? Regards, -- Pumudu Ruhunage Associate Software Engineer | WSO2 Inc M: +94 779 664493 | http://wso2.com
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
