Which version of RocketMQ was used?
As you are using an ARM CPU, ensure that netty transport, tc-native has
compatible binary lib included.
On Mon, Nov 14, 2022 at 11:09 PM <peng...@linkstec.com> wrote:
> Hello everyone
>
>
>
> I have a strange problem. The rocketMQ java client sometimes crashes
> inexplicably.
>
>
>
> Some JVM crash logs I think are critical as follows:
>
>
>
>
>
>
> *************************************************************************************
>
> --------------- T H R E A D ---------------
>
>
>
> Current thread (0x0000fffef22d1000): JavaThread
> "MQClientFactoryScheduledThread" [_thread_in_Java, id=3247460,
> stack(0x0000fffdcaee0000,0x0000fffdcafe0000)]
>
>
>
> siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr:
> 0x00000008e09fa8e5
>
>
>
> Stack: [0x0000fffdcaee0000,0x0000fffdcafe0000], sp=0x0000fffdcafddce0,
> free space=1015k
>
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
>
> j
> io.netty.channel.nio.AbstractNioChannel.eventLoop()Lio/netty/channel/nio/NioEventLoop;+4
>
> j
> io.netty.channel.nio.AbstractNioChannel.eventLoop()Lio/netty/channel/EventLoop;+1
>
> j
> io.netty.channel.AbstractChannelHandlerContext.executor()Lio/netty/util/concurrent/EventExecutor;+11
>
> j
> io.netty.channel.AbstractChannelHandlerContext.newPromise()Lio/netty/channel/ChannelPromise;+9
>
> j
> io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(Ljava/lang/Object;)Lio/netty/channel/ChannelFuture;+3
>
> j
> io.netty.channel.DefaultChannelPipeline.writeAndFlush(Ljava/lang/Object;)Lio/netty/channel/ChannelFuture;+5
>
> j
> io.netty.channel.AbstractChannel.writeAndFlush(Ljava/lang/Object;)Lio/netty/channel/ChannelFuture;+5
>
> j
> org.apache.rocketmq.remoting.netty.NettyRemotingAbstract.invokeSyncImpl(Lio/netty/channel/Channel;Lorg/apache/rocketmq/remoting/protocol/RemotingCommand;J)Lorg/apache/rocketmq/remoting/protocol/RemotingCommand;+48
>
> J 17751 C2
> org.apache.rocketmq.remoting.netty.NettyRemotingClient.invokeSync(Ljava/lang/String;Lorg/apache/rocketmq/remoting/protocol/RemotingCommand;J)Lorg/apache/rocketmq/remoting/protocol/RemotingCommand;
> (195 bytes) @ 0x0000fffeea65d2b0 [0x0000fffeea65d140+0x170]
>
> J 19050 C2
> org.apache.rocketmq.client.impl.factory.MQClientInstance.updateTopicRouteInfoFromNameServer(Ljava/lang/String;ZLorg/apache/rocketmq/client/producer/DefaultMQProducer;)Z
> (673 bytes) @ 0x0000fffeeab01700 [0x0000fffeeab01480+0x280]
>
> J 19490 C2
> org.apache.rocketmq.client.impl.factory.MQClientInstance.updateTopicRouteInfoFromNameServer()V
> (228 bytes) @ 0x0000fffeeac68368 [0x0000fffeeac666c0+0x1ca8]
>
> J 20110 C2
> org.apache.rocketmq.client.impl.factory.MQClientInstance$3.run()V (27
> bytes) @ 0x0000fffee7b2e738 [0x0000fffee7b2e700+0x38]
>
> J 15811% C2
> java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
> (225 bytes) @ 0x0000fffee9390610 [0x0000fffee9390300+0x310]
>
> j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
>
> j java.lang.Thread.run()V+11
>
> v ~StubRoutines::call_stub
>
> V [libjvm.so+0x5dd6cc] JavaCalls::call_helper(JavaValue*, methodHandle*,
> JavaCallArguments*, Thread*)+0xe3c
>
> V [libjvm.so+0x5dae14] JavaCalls::call_virtual(JavaValue*, KlassHandle,
> Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x28c
>
> V [libjvm.so+0x5db3f4] JavaCalls::call_virtual(JavaValue*, Handle,
> KlassHandle, Symbol*, Symbol*, Thread*)+0x4c
>
> V [libjvm.so+0x648018] thread_entry(JavaThread*, Thread*)+0x68
>
> V [libjvm.so+0x99f0c4] JavaThread::thread_main_inner()+0xcc
>
> V [libjvm.so+0x99f464] JavaThread::run()+0x35c
>
> V [libjvm.so+0x843c00] java_start(Thread*)+0x108
>
> C [libpthread.so.0+0x88cc]
>
> C [libc.so.6+0xda12c]
>
>
>
> VM state:not at safepoint (normal execution)
>
>
>
> VM Mutex/Monitor currently owned by a thread: None
>
>
>
> heap address: 0x00000006c0000000, size: 4096 MB, Compressed Oops mode:
> Zero based, Oop shift amount: 3
>
> Narrow klass base: 0x0000000800000000, Narrow klass shift: 0
>
> Compressed class space size: 1073741824 Address: 0x0000000800000000
>
>
>
> Heap:
>
> par new generation total 873856K, used 191380K [0x00000006c0000000,
> 0x0000000700000000, 0x0000000700000000)
>
> eden space 699136K, 23% used [0x00000006c0000000, 0x00000006c9e4d658,
> 0x00000006eaac0000)
>
> from space 174720K, 16% used [0x00000006f5560000, 0x00000006f71f7a10,
> 0x0000000700000000)
>
> to space 174720K, 0% used [0x00000006eaac0000, 0x00000006eaac0000,
> 0x00000006f5560000)
>
> concurrent mark-sweep generation total 1048576K, used 94096K
> [0x0000000700000000, 0x0000000740000000, 0x00000007c0000000)
>
> Metaspace used 131636K, capacity 138140K, committed 138304K,
> reserved 1171456K
>
> class space used 15499K, capacity 16668K, committed 16704K, reserved
> 1048576K
>
>
>
> Card table byte_map: [0x0000fffee6170000,0x0000fffee6980000]
> byte_map_base: 0x0000fffee2b70000
>
>
>
> Marking Bits: (CMSBitMap*) 0x0000fffef0094bd8
>
> Bits: [0x0000fffec4ff0000, 0x0000fffec7ff0000)
>
>
>
> Mod Union Table: (CMSBitMap*) 0x0000fffef0094c98
>
> Bits: [0x0000fffee5210000, 0x0000fffee52d0000)
>
>
>
> Polling page: 0x0000fffef55c0000
>
>
>
> CodeCache: size=131072Kb used=35285Kb max_used=57935Kb free=95787Kb
>
> bounds [0x0000fffee77f0000, 0x0000fffeeb100000, 0x0000fffeef7f0000]
>
> total_blobs=9048 nmethods=8111 adapters=850
>
> compilation: enabled
>
>
>
> Compilation events (10 events):
>
> Event: 281183.616 Thread 0x0000fffef0163000 24927 4
> org.springframework.scheduling.support.BitsCronField::nextOrSame (137 bytes)
>
> Event: 281183.640 Thread 0x0000fffef0163000 nmethod 24927
> 0x0000fffee9bdccd0 code [0x0000fffee9bdd040, 0x0000fffee9bde910]
>
> Event: 281869.069 Thread 0x0000fffef0169800 24928 3
> io.netty.util.Recycler$DefaultHandle::recycle (27 bytes)
>
> Event: 281869.070 Thread 0x0000fffef0169800 nmethod 24928
> 0x0000fffee80a2790 code [0x0000fffee80a2980, 0x0000fffee80a3140]
>
> Event: 281869.174 Thread 0x0000fffef0169800 24929 3
> io.netty.buffer.PoolThreadCache::allocate (45 bytes)
>
> Event: 281869.174 Thread 0x0000fffef0169800 nmethod 24929
> 0x0000fffeea445910 code [0x0000fffeea445ac0, 0x0000fffeea445f18]
>
> Event: 281898.618 Thread 0x0000fffef0169800 24930 3
> io.netty.util.internal.shaded.org.jctools.queues.MpscChunkedArrayQueue::relaxedOffer
> (6 bytes)
>
> Event: 281898.618 Thread 0x0000fffef0169800 nmethod 24930
> 0x0000fffee9aefe90 code [0x0000fffee9af0000, 0x0000fffee9af01e8]
>
> Event: 282678.618 Thread 0x0000fffef0167000 24931 4
> io.netty.buffer.PoolThreadCache::allocate (45 bytes)
>
> Event: 282678.634 Thread 0x0000fffef0167000 nmethod 24931
> 0x0000fffee8632ed0 code [0x0000fffee8633140, 0x0000fffee8633e18]
>
>
>
> GC Heap History (10 events):
>
> Event: 274419.423 GC heap before
>
> {Heap before GC invocations=118 (full 3):
>
> par new generation total 873856K, used 706451K [0x00000006c0000000,
> 0x0000000700000000, 0x0000000700000000)
>
> eden space 699136K, 100% used [0x00000006c0000000, 0x00000006eaac0000,
> 0x00000006eaac0000)
>
> from space 174720K, 4% used [0x00000006eaac0000, 0x00000006eb1e4de0,
> 0x00000006f5560000)
>
> to space 174720K, 0% used [0x00000006f5560000, 0x00000006f5560000,
> 0x0000000700000000)
>
> concurrent mark-sweep generation total 1048576K, used 94045K
> [0x0000000700000000, 0x0000000740000000, 0x00000007c0000000)
>
> Metaspace used 131616K, capacity 138140K, committed 138304K,
> reserved 1171456K
>
> class space used 15499K, capacity 16668K, committed 16704K, reserved
> 1048576K
>
> Event: 274419.433 GC heap after
>
> Heap after GC invocations=119 (full 3):
>
> par new generation total 873856K, used 3653K [0x00000006c0000000,
> 0x0000000700000000, 0x0000000700000000)
>
> eden space 699136K, 0% used [0x00000006c0000000, 0x00000006c0000000,
> 0x00000006eaac0000)
>
> from space 174720K, 2% used [0x00000006f5560000, 0x00000006f58f1538,
> 0x0000000700000000)
>
> to space 174720K, 0% used [0x00000006eaac0000, 0x00000006eaac0000,
> 0x00000006f5560000)
>
> concurrent mark-sweep generation total 1048576K, used 94069K
> [0x0000000700000000, 0x0000000740000000, 0x00000007c0000000)
>
> Metaspace used 131616K, capacity 138140K, committed 138304K,
> reserved 1171456K
>
> class space used 15499K, capacity 16668K, committed 16704K, reserved
> 1048576K
>
> }
>
>
>
> Memory: 64k page, physical 15871808k(10249728k free), swap 0k(0k free)
>
>
>
> vm_info: Java HotSpot(TM) 64-Bit Server VM (25.321-b07) for linux-aarch64
> JRE (1.8.0_321-b07), built on Dec 15 2021 19:30:02 by "java_re" with gcc
> 7.3.0
>
>
>
> time: Mon Nov 14 01:40:54 2022
>
> timezone: CST
>
> elapsed time: 282708.059187 seconds (3d 6h 31m 48s)
>
>
> *************************************************************************************
>
>
>
> It would be very grateful if someone could give some ideas when
> encountering similar problems.
>
>
>
> Thanks all.
>
>
> *********************************************
> --------------------------------------------------------------------
> 敬请留意：此电子邮件包含保密信息且仅限发送给预期的收件人。如果您不是此电子邮件预期的收件人请立即通知发件人并从您的系统中删除此电子邮件。
> 本公司对于此电子邮件或其任何附件所携带的病毒造成的损失不承担任何责任。
>
> PLEASE READ: The information contained in this e-mail is confidential and
> intended for the named recipient(s) only. If you are not an intended
> recipient of this e-mail please notify the sender immediately and delete your
> copy from your system. Our company accepts no liability for any damage
> sustained as a result of viruses in this e-mail or any attachment(s) to it.
>
>
>
