Hi, We found the cause for this OM exception. Since the subscriber is killed while sending messages database table space is exceeded and messages are not written to the database. Therefore all the messages were kept filling in to the memory. Try to enable flow control for memory in broker.xml and run the test. It should control the situation.
Thank you, Sajini On Mon, Mar 30, 2015 at 2:14 PM, Pumudu Ruhunage <[email protected]> wrote: > Hi All, > > We have started a two node long running MB cluster test to find long > running issues. > Test started on 2015-03-25. On 2015-03-28 both nodes has gone out of > memory. Subscriber machine has gone down due to unknown reason as well. > > Following are the OM errors extracted from the heap dump from each node. > > *Coordinator Node OM Error* - Sat Mar 28 22:57:16 IST 2015 > "Thread-23" daemon prio=5 tid=98 RUNNABLE > at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:48) > at sun.reflect.ByteVectorImpl.resize(ByteVectorImpl.java:84) > at sun.reflect.ByteVectorImpl.add(ByteVectorImpl.java:63) > at sun.reflect.ClassFileAssembler.emitByte(ClassFileAssembler.java:74) > at > sun.reflect.ClassFileAssembler.emitConstantPoolUTF8(ClassFileAssembler.java:107) > Local Variable: sun.reflect.ClassFileAssembler#1 > Local Variable: byte[]#29857 > at > sun.reflect.AccessorGenerator.emitCommonConstantPoolEntries(AccessorGenerator.java:144) > at > sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:351) > Local Variable: java.lang.Class#1 > Local Variable: java.lang.String#53810 > Local Variable: sun.reflect.ByteVectorImpl#1 > Local Variable: java.lang.String#19366 > at > sun.reflect.MethodAccessorGenerator.generateConstructor(MethodAccessorGenerator.java:94) > Local Variable: class com.hazelcast.map.operation.MapKeySetOperation > Local Variable: java.lang.Class[]#2 > Local Variable: java.lang.Class[]#3 > Local Variable: sun.reflect.MethodAccessorGenerator#1 > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:48) > Local Variable: java.lang.Object[]#25884 > Local Variable: sun.reflect.NativeConstructorAccessorImpl#1 > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at com.hazelcast.nio.ClassLoaderUtil.newInstance(ClassLoaderUtil.java:62) > at > com.hazelcast.nio.serialization.DataSerializer.read(DataSerializer.java:103) > Local Variable: java.lang.String#19288 > at > com.hazelcast.nio.serialization.DataSerializer.read(DataSerializer.java:36) > at > com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:59) > Local Variable: > com.hazelcast.nio.serialization.ByteArrayObjectDataInput#2 > at > com.hazelcast.nio.serialization.SerializationServiceImpl.toObject(SerializationServiceImpl.java:221) > Local Variable: > com.hazelcast.nio.serialization.SerializationServiceImpl#1 > at com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:156) > at > com.hazelcast.spi.impl.BinaryOperationFactory.createOperation(BinaryOperationFactory.java:52) > at > com.hazelcast.spi.impl.PartitionIteratingOperation.run(PartitionIteratingOperation.java:51) > Local Variable: com.hazelcast.spi.impl.NodeEngineImpl#1 > Local Variable: > com.hazelcast.spi.impl.PartitionIteratingOperation$ResponseQueue#55 > Local Variable: java.util.HashMap#99 > at > com.hazelcast.spi.impl.BasicOperationService.processOperation(BasicOperationService.java:363) > at > com.hazelcast.spi.impl.BasicOperationService.runOperation(BasicOperationService.java:228) > Local Variable: com.hazelcast.spi.impl.PartitionIteratingOperation#1 > at > com.hazelcast.spi.impl.BasicInvocation.doInvoke(BasicInvocation.java:371) > at com.hazelcast.spi.impl.BasicInvocation.invoke(BasicInvocation.java:190) > Local Variable: com.hazelcast.spi.impl.BasicTargetInvocation#1 > at > com.hazelcast.spi.impl.BasicInvocationBuilder.invoke(BasicInvocationBuilder.java:48) > at > com.hazelcast.spi.impl.BasicOperationService.invokeOnPartitions(BasicOperationService.java:587) > Local Variable: java.util.HashMap#97 > Local Variable: java.util.HashMap$EntryIterator#1 > Local Variable: com.hazelcast.nio.Address#1 > at > com.hazelcast.spi.impl.BasicOperationService.invokeOnAllPartitions(BasicOperationService.java:549) > Local Variable: com.hazelcast.spi.impl.BasicOperationService#1 > Local Variable: com.hazelcast.spi.impl.BinaryOperationFactory#1 > Local Variable: java.util.HashMap#30549 > Local Variable: java.lang.String#146076 > at > com.hazelcast.map.proxy.MapProxySupport.keySetInternal(MapProxySupport.java:590) > at com.hazelcast.map.proxy.MapProxyImpl.keySet(MapProxyImpl.java:481) > Local Variable: com.hazelcast.map.proxy.MapProxyImpl#20 > at > org.wso2.andes.kernel.slot.SlotManagerClusterMode.getMessagePublishedNodes(SlotManagerClusterMode.java:675) > Local Variable: org.wso2.andes.kernel.slot.SlotManagerClusterMode#1 > at > org.wso2.andes.kernel.slot.SlotDeleteSafeZoneCalc.run(SlotDeleteSafeZoneCalc.java:64) > Local Variable: org.wso2.andes.kernel.slot.SlotDeleteSafeZoneCalc#1 > at java.lang.Thread.run(Thread.java:745) > > > > *Non-coordinator Node* - Sat Mar 28 20:32:15 IST 2015 > "SocketAcceptorIoProcessor-0.0" daemon prio=5 tid=301 RUNNABLE > at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:48) > at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) > at java.nio.ByteBuffer.allocate(ByteBuffer.java:331) > at > org.apache.mina.common.SimpleByteBufferAllocator.allocate(SimpleByteBufferAllocator.java:45) > at org.apache.mina.common.ByteBuffer.allocate(ByteBuffer.java:225) > at org.apache.mina.common.ByteBuffer.allocate(ByteBuffer.java:214) > at > org.apache.mina.transport.socket.nio.SocketIoProcessor.read(SocketIoProcessor.java:210) > at > org.apache.mina.transport.socket.nio.SocketIoProcessor.process(SocketIoProcessor.java:198) > Local Variable: org.apache.mina.transport.socket.nio.SocketIoProcessor#1 > Local Variable: java.util.HashMap$KeyIterator#2 > Local Variable: sun.nio.ch.SelectionKeyImpl#15 > Local Variable: org.apache.mina.transport.socket.nio.SocketSessionImpl#9 > Local Variable: sun.nio.ch.Util$2#17 > at > org.apache.mina.transport.socket.nio.SocketIoProcessor.access$400(SocketIoProcessor.java:45) > at > org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProcessor.java:485) > Local Variable: > org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker#3 > Local Variable: sun.nio.ch.EPollSelectorImpl#49 > at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51) > Local Variable: org.apache.mina.util.NamePreservingRunnable#6 > Local Variable: java.lang.String#148830 > at java.lang.Thread.run(Thread.java:745) > > What can be the cause for this issue ? WDYT ? > > Regards, > -- > Pumudu Ruhunage > Associate Software Engineer | WSO2 Inc > M: +94 779 664493 | http://wso2.com > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- Sajini De SIlva Software Engineer; WSO2 Inc.; http://wso2.com , Email: [email protected] Blog: http://sajinid.blogspot.com/ Git hub profile: https://github.com/sajinidesilva Phone: +94 712797729
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
