Looks similar to this issue: https://github.com/netty/netty/issues/2950 .
On Wed, Oct 19, 2016 at 2:26 PM, Vladislav Pyatkov <[email protected]> wrote: > Hi, > > This look like in the system does not have enough memory. > > It is difficult to say anything else, without additional information. > Memory consumption, system logs for the moment. > > On Wed, Oct 19, 2016 at 2:00 PM, yfernando <[email protected]> > wrote: > >> Hi, >> >> We've had the multiple instances where a 16 node grid completely crashes >> with the following error. >> >> we use all our caches OFF HEAP caches in our configuration. >> >> We are running Ignite 1.7 and running Java 1.8u60-b27 on Linux >> >> Has anyone seen a similar issue? >> >> Thanks >> >> Yohan >> >> I've attached a few of the hs_err log files. Please let us know if you >> need >> anymore information. >> >> # >> # A fatal error has been detected by the Java Runtime Environment: >> # >> # SIGSEGV (0xb) at pc=0x00007ff13f1c2524, pid=3402054, >> tid=140670445545216 >> # >> # JRE version: Java(TM) SE Runtime Environment (8.0_60-b27) (build >> 1.8.0_60-b27) >> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode >> linux-amd64 compressed oops) >> # Problematic frame: >> # v ~StubRoutines::jint_disjoint_arraycopy >> # >> # Core dump written. Default location: >> /prod/Java/apps/data-grid-server-ignite/2.2.31/bin/core or core.3402054 >> # >> # If you would like to submit a bug report, please visit: >> # http://bugreport.java.com/bugreport/crash.jsp >> # >> >> --------------- T H R E A D --------------- >> >> Current thread (0x00007ff1552f9800): JavaThread >> "pub-#12%DataGridServer-Production%" [_thread_in_Java, id=3402569, >> stack(0x00007ff063dfe000,0x00007ff063eff000)] >> >> siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr: >> 0x000000076a800000 >> >> >> Stack: [0x00007ff063dfe000,0x00007ff063eff000], sp=0x00007ff063efc490, >> free space=1017k >> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native >> code) >> v ~StubRoutines::jint_disjoint_arraycopy >> J 6572 C2 >> org.apache.ignite.internal.util.GridUnsafe.copyMemory(Ljava/ >> lang/Object;JLjava/lang/Object;JJ)V >> (14 bytes) @ 0x00007ff13f52f1a8 [0x00007ff13f52f180+0x28] >> j >> org.apache.ignite.internal.util.io.GridUnsafeDataInput.readS >> hortArray()[S+88 >> j >> org.apache.ignite.marshaller.optimized.OptimizedObjectInputS >> tream.readShortArray()[S+4 >> J 25716 C2 >> org.apache.ignite.marshaller.optimized.OptimizedObjectInputS >> tream.readObjectOverride()Ljava/lang/Object; >> (883 bytes) @ 0x00007ff1437bbc4c [0x00007ff1437ba2a0+0x19ac] >> J 6523 C2 java.io.ObjectInputStream.readObject()Ljava/lang/Object; (124 >> bytes) @ 0x00007ff13f4e0d9c [0x00007ff13f4e0760+0x63c] >> J 12272 C2 >> org.apache.ignite.marshaller.optimized.OptimizedObjectInputS >> tream.readFields(Ljava/lang/Object;Lorg/apache/ignite/ >> marshaller/optimized/OptimizedClassDescriptor$ClassFields;)V >> (336 bytes) @ 0x00007ff140833d78 [0x00007ff140833980+0x3f8] >> J 12508 C2 >> org.apache.ignite.marshaller.optimized.OptimizedObjectInputS >> tream.readSerializable(Ljava/lang/Class;Ljava/util/List; >> Ljava/lang/reflect/Method;Lorg/apache/ignite/marshaller/ >> optimized/OptimizedClassDescriptor$Fields;)Ljava/lang/Object; >> (176 bytes) @ 0x00007ff14178ee74 [0x00007ff14178e780+0x6f4] >> J 16693 C2 >> org.apache.ignite.marshaller.optimized.OptimizedClassDescrip >> tor.read(Lorg/apache/ignite/marshaller/optimized/Optimized >> ObjectInputStream;)Ljava/lang/Object; >> (130 bytes) @ 0x00007ff140804868 [0x00007ff1408044a0+0x3c8] >> J 25716 C2 >> org.apache.ignite.marshaller.optimized.OptimizedObjectInputS >> tream.readObjectOverride()Ljava/lang/Object; >> (883 bytes) @ 0x00007ff1437bbe80 [0x00007ff1437ba2a0+0x1be0] >> J 6523 C2 java.io.ObjectInputStream.readObject()Ljava/lang/Object; (124 >> bytes) @ 0x00007ff13f4e0d9c [0x00007ff13f4e0760+0x63c] >> J 12272 C2 >> org.apache.ignite.marshaller.optimized.OptimizedObjectInputS >> tream.readFields(Ljava/lang/Object;Lorg/apache/ignite/ >> marshaller/optimized/OptimizedClassDescriptor$ClassFields;)V >> (336 bytes) @ 0x00007ff140833d78 [0x00007ff140833980+0x3f8] >> J 12508 C2 >> org.apache.ignite.marshaller.optimized.OptimizedObjectInputS >> tream.readSerializable(Ljava/lang/Class;Ljava/util/List; >> Ljava/lang/reflect/Method;Lorg/apache/ignite/marshaller/ >> optimized/OptimizedClassDescriptor$Fields;)Ljava/lang/Object; >> (176 bytes) @ 0x00007ff14178ee74 [0x00007ff14178e780+0x6f4] >> J 16693 C2 >> org.apache.ignite.marshaller.optimized.OptimizedClassDescrip >> tor.read(Lorg/apache/ignite/marshaller/optimized/Optimized >> ObjectInputStream;)Ljava/lang/Object; >> (130 bytes) @ 0x00007ff140804868 [0x00007ff1408044a0+0x3c8] >> J 25716 C2 >> org.apache.ignite.marshaller.optimized.OptimizedObjectInputS >> tream.readObjectOverride()Ljava/lang/Object; >> (883 bytes) @ 0x00007ff1437bbe80 [0x00007ff1437ba2a0+0x1be0] >> J 6560 C2 >> org.apache.ignite.marshaller.optimized.OptimizedObjectInputS >> tream.readArrayList()Ljava/util/ArrayList; >> (47 bytes) @ 0x00007ff140621050 [0x00007ff140620c80+0x3d0] >> J 25716 C2 >> org.apache.ignite.marshaller.optimized.OptimizedObjectInputS >> tream.readObjectOverride()Ljava/lang/Object; >> (883 bytes) @ 0x00007ff1437bbbac [0x00007ff1437ba2a0+0x190c] >> J 6523 C2 java.io.ObjectInputStream.readObject()Ljava/lang/Object; (124 >> bytes) @ 0x00007ff13f4e0d9c [0x00007ff13f4e0760+0x63c] >> J 12272 C2 >> org.apache.ignite.marshaller.optimized.OptimizedObjectInputS >> tream.readFields(Ljava/lang/Object;Lorg/apache/ignite/ >> marshaller/optimized/OptimizedClassDescriptor$ClassFields;)V >> (336 bytes) @ 0x00007ff140833d78 [0x00007ff140833980+0x3f8] >> J 12508 C2 >> org.apache.ignite.marshaller.optimized.OptimizedObjectInputS >> tream.readSerializable(Ljava/lang/Class;Ljava/util/List; >> Ljava/lang/reflect/Method;Lorg/apache/ignite/marshaller/ >> optimized/OptimizedClassDescriptor$Fields;)Ljava/lang/Object; >> (176 bytes) @ 0x00007ff14178ee74 [0x00007ff14178e780+0x6f4] >> J 16693 C2 >> org.apache.ignite.marshaller.optimized.OptimizedClassDescrip >> tor.read(Lorg/apache/ignite/marshaller/optimized/Optimized >> ObjectInputStream;)Ljava/lang/Object; >> (130 bytes) @ 0x00007ff140804868 [0x00007ff1408044a0+0x3c8] >> J 25716 C2 >> org.apache.ignite.marshaller.optimized.OptimizedObjectInputS >> tream.readObjectOverride()Ljava/lang/Object; >> (883 bytes) @ 0x00007ff1437bbe80 [0x00007ff1437ba2a0+0x1be0] >> J 6523 C2 java.io.ObjectInputStream.readObject()Ljava/lang/Object; (124 >> bytes) @ 0x00007ff13f4e0d9c [0x00007ff13f4e0760+0x63c] >> J 12272 C2 >> org.apache.ignite.marshaller.optimized.OptimizedObjectInputS >> tream.readFields(Ljava/lang/Object;Lorg/apache/ignite/ >> marshaller/optimized/OptimizedClassDescriptor$ClassFields;)V >> (336 bytes) @ 0x00007ff140833d78 [0x00007ff140833980+0x3f8] >> J 12508 C2 >> org.apache.ignite.marshaller.optimized.OptimizedObjectInputS >> tream.readSerializable(Ljava/lang/Class;Ljava/util/List; >> Ljava/lang/reflect/Method;Lorg/apache/ignite/marshaller/ >> optimized/OptimizedClassDescriptor$Fields;)Ljava/lang/Object; >> (176 bytes) @ 0x00007ff14178ee74 [0x00007ff14178e780+0x6f4] >> J 16693 C2 >> org.apache.ignite.marshaller.optimized.OptimizedClassDescrip >> tor.read(Lorg/apache/ignite/marshaller/optimized/Optimized >> ObjectInputStream;)Ljava/lang/Object; >> (130 bytes) @ 0x00007ff140804868 [0x00007ff1408044a0+0x3c8] >> J 25716 C2 >> org.apache.ignite.marshaller.optimized.OptimizedObjectInputS >> tream.readObjectOverride()Ljava/lang/Object; >> (883 bytes) @ 0x00007ff1437bbe80 [0x00007ff1437ba2a0+0x1be0] >> J 6523 C2 java.io.ObjectInputStream.readObject()Ljava/lang/Object; (124 >> bytes) @ 0x00007ff13f4e0d9c [0x00007ff13f4e0760+0x63c] >> J 25841 C2 >> org.apache.ignite.marshaller.optimized.OptimizedMarshaller.u >> nmarshal([BLjava/lang/ClassLoader;)Ljava/lang/Object; >> (148 bytes) @ 0x00007ff140c27724 [0x00007ff140c27300+0x424] >> J 25109 C1 >> org.apache.ignite.internal.processors.job.GridJobWorker.init >> ialize(Lorg/apache/ignite/internal/managers/deployment/GridD >> eployment;Ljava/lang/Class;)Z >> (295 bytes) @ 0x00007ff141e6a83c [0x00007ff141e6a500+0x33c] >> J 24444 C1 >> org.apache.ignite.internal.processors.job.GridJobProcessor.p >> rocessJobExecuteRequest(Lorg/apache/ignite/cluster/ClusterNo >> de;Lorg/apache/ignite/internal/GridJobExecuteRequest;)V >> (1342 bytes) @ 0x00007ff1434879ac [0x00007ff143481280+0x672c] >> J 24443 C1 >> org.apache.ignite.internal.processors.job.GridJobProcessor$J >> obExecutionListener.onMessage(Ljava/util/UUID;Ljava/lang/Object;)V >> (147 bytes) @ 0x00007ff140d08ab4 [0x00007ff140d075a0+0x1514] >> J 14472 C2 >> org.apache.ignite.internal.managers.communication.GridIoMana >> ger.processRegularMessage0(Lorg/apache/ignite/internal/ma >> nagers/communication/GridIoMessage;Ljava/util/UUID;)V >> (55 bytes) @ 0x00007ff141ddc8e4 [0x00007ff141ddc5e0+0x304] >> J 11641 C2 >> org.apache.ignite.internal.managers.communication.GridIoManager$5.run()V >> (52 >> bytes) @ 0x00007ff1414b02cc [0x00007ff1414b0240+0x8c] >> j >> java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/ >> util/concurrent/ThreadPoolExecutor$Worker;)V+95 >> j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+19 >> j java.lang.Thread.run()V+11 >> v ~StubRoutines::call_stub >> V [libjvm.so+0x68bbe6] JavaCalls::call_helper(JavaValue*, >> methodHandle*, >> JavaCallArguments*, Thread*)+0x1056 >> V [libjvm.so+0x68c0f1] JavaCalls::call_virtual(JavaValue*, KlassHandle, >> Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321 >> V [libjvm.so+0x68c597] JavaCalls::call_virtual(JavaValue*, Handle, >> KlassHandle, Symbol*, Symbol*, Thread*)+0x47 >> V [libjvm.so+0x7232d0] thread_entry(JavaThread*, Thread*)+0xa0 >> V [libjvm.so+0xa68f3f] JavaThread::thread_main_inner()+0xdf >> V [libjvm.so+0xa6906c] JavaThread::run()+0x11c >> V [libjvm.so+0x91cb88] java_start(Thread*)+0x108 >> C [libpthread.so.0+0x7aa1] >> hs_err_pid3410800.log >> <http://apache-ignite-users.70518.x6.nabble.com/file/n8356/ >> hs_err_pid3410800.log> >> hs_err_pid3410096.log >> <http://apache-ignite-users.70518.x6.nabble.com/file/n8356/ >> hs_err_pid3410096.log> >> hs_err_pid3408462.log >> <http://apache-ignite-users.70518.x6.nabble.com/file/n8356/ >> hs_err_pid3408462.log> >> >> >> >> -- >> View this message in context: http://apache-ignite-users.705 >> 18.x6.nabble.com/JVM-Crash-SIGSEGV-on-GridUnsafe-copyMemory-tp8356.html >> Sent from the Apache Ignite Users mailing list archive at Nabble.com. >> > > > > -- > Vladislav Pyatkov >
