zhaohaidao opened a new issue, #3638: URL: https://github.com/apache/bookkeeper/issues/3638
**BUG REPORT** ***Describe the bug*** Bookie failed to shutdown with fatal error. fatal error log is followed ``` # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x0000000000000000, pid=17742, tid=19134 # # JRE version: Java(TM) SE Runtime Environment (17.0.5+9) (build 17.0.5+9-LTS-191) # Java VM: Java HotSpot(TM) 64-Bit Server VM (17.0.5+9-LTS-191, mixed mode, sharing, tiered, compressed class ptrs, z gc, linux-amd64) # Problematic frame: # C 0x0000000000000000 # # Core dump will be written. Default location: Core dumps may be processed with "/usr/local/bin/core_pattern_pipe.py %e %p %t %u %g" (or dumping to /opt/pulsar/core.17742) # # If you would like to submit a bug report, please visit: # https://bugreport.java.com/bugreport/crash.jsp # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. # --------------- S U M M A R Y ------------ Command Line: -Dlog4j.configurationFile=log4j2.yaml -Djute.maxbuffer=10485760 -Djava.net.preferIPv4Stack=true -Dzookeeper.clientTcpKeepAlive=true -Dio.netty.tryReflectionSetAccessible=true --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util.zip=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/jdk.internal.misc=ALL-UNNAMED --add-opens=java.base/sun.net=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Xms16g -Xmx16g -XX:MaxDirectMemorySize=48g -XX:MetaspaceSize=96m -XX:+UseZGC -XX:+PerfDisableSharedMem -XX:+AlwaysPreTouch -Xlog:async -Xlog:gc*,safepoint:/opt/pulsar/logs/pulsar_bookie_gc_%p.log:time,uptime,tags:filecount=10,filesize=20M -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.maxCapacityPerThread=4096 -Dpulsar.log.appender=RoutingAppender -Dpulsar.log.dir=/opt/pulsar/logs -Dpulsar.log.level=info -Dpulsar.log.root.level=info -Dpulsa r.routing.appender.default=Console -Dlog4j2.is.webapp=false -Dpulsar.functions.process.container.log.dir=/opt/pulsar/logs -Dpulsar.functions.java.instance.jar=/opt/pulsar/instances/java-instance.jar -Dpulsar.functions.python.instance.file=/opt/pulsar/instances/python-instance/python_instance_main.py -Dpulsar.functions.extra.dependencies.dir=/opt/pulsar/instances/deps -Dpulsar.functions.instance.classpath=/opt/pulsar/conf:::/opt/pulsar/lib/*: -Dpulsar.functions.log.conf=/opt/pulsar/conf/functions_log4j2.xml -Dbookkeeper.metadata.bookie.drivers=org.apache.pulsar.metadata.bookkeeper.PulsarMetadataBookieDriver -Dbookkeeper.metadata.client.drivers=org.apache.pulsar.metadata.bookkeeper.PulsarMetadataClientDriver -Dpulsar.log.file=bookkeeper.log org.apache.bookkeeper.server.Main --conf /opt/pulsar/conf/bookkeeper.conf Host: Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz, 64 cores, 251G, CentOS Linux release 7.4.1708 (Core) Time: Sun Nov 13 23:24:13 2022 CST elapsed time: 404.939532 seconds (0d 0h 6m 44s) --------------- T H R E A D --------------- Current thread (0x00007f37d8000c80): JavaThread "qtp1258103238-126" [_thread_in_native, id=19134, stack(0x00007f3f250f3000,0x00007f3f251f4000)] Stack: [0x00007f3f250f3000,0x00007f3f251f4000], sp=0x00007f3f251f1718, free space=1017k Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) J 7441 org.rocksdb.RocksDB.getLongProperty(JJLjava/lang/String;I)J (0 bytes) @ 0x00007f43957b7c88 [0x00007f43957b7c20+0x0000000000000068] J 7440 c1 org.rocksdb.RocksDB.getLongProperty(Lorg/rocksdb/ColumnFamilyHandle;Ljava/lang/String;)J (26 bytes) @ 0x00007f438e81311c [0x00007f438e812f60+0x00000000000001bc] J 7438 c1 org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.count()J (22 bytes) @ 0x00007f438e89384c [0x00007f438e893760+0x00000000000000ec] J 7436 c1 org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex$$Lambda$165+0x0000000800e98220.get()Ljava/lang/Object; (8 bytes) @ 0x00007f438e3c651c [0x00007f438e3c6400+0x000000000000011c] J 7229 c1 org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndexStats$1.getSample()Ljava/lang/Number; (5 bytes) @ 0x00007f438e92915c [0x00007f438e928fe0+0x000000000000017c] J 7123 c2 org.apache.bookkeeper.stats.prometheus.SimpleGauge.getSample()Ljava/lang/Number; (10 bytes) @ 0x00007f43959adb04 [0x00007f43959adaa0+0x0000000000000064] J 7069 c2 org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider$$Lambda$258+0x0000000800f65490.accept(Ljava/lang/Object;Ljava/lang/Object;)V (20 bytes) @ 0x00007f439599455c [0x00007f4395993b20+0x0000000000000a3c] J 7039 c2 java.util.concurrent.ConcurrentHashMap.forEach(Ljava/util/function/BiConsumer;)V [email protected] (65 bytes) @ 0x00007f4395965220 [0x00007f4395964f80+0x00000000000002a0] j org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider.writeAllMetrics(Ljava/io/Writer;)V+27 j org.apache.bookkeeper.stats.prometheus.PrometheusServlet.doGet(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+29 j javax.servlet.http.HttpServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+35 j javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+39 j org.eclipse.jetty.servlet.ServletHolder.handle(Lorg/eclipse/jetty/server/Request;Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+25 j org.eclipse.jetty.servlet.ServletHandler.doHandle(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+180 j org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+27 j org.eclipse.jetty.server.handler.ContextHandler.doHandle(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+72 j org.eclipse.jetty.server.handler.ScopedHandler.nextScope(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+38 j org.eclipse.jetty.servlet.ServletHandler.doScope(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+179 j org.eclipse.jetty.server.handler.ScopedHandler.nextScope(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+16 j org.eclipse.jetty.server.handler.ContextHandler.doScope(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+382 j org.eclipse.jetty.server.handler.ScopedHandler.handle(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+20 j org.eclipse.jetty.server.handler.HandlerWrapper.handle(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+18 j org.eclipse.jetty.server.Server.handle(Lorg/eclipse/jetty/server/HttpChannel;)V+149 j org.eclipse.jetty.server.HttpChannel.lambda$handle$1()V+69 j org.eclipse.jetty.server.HttpChannel$$Lambda$257+0x0000000800f5ef70.dispatch()V+4 j org.eclipse.jetty.server.HttpChannel.dispatch(Ljavax/servlet/DispatcherType;Lorg/eclipse/jetty/server/HttpChannel$Dispatchable;)V+37 j org.eclipse.jetty.server.HttpChannel.handle()Z+220 j org.eclipse.jetty.server.HttpConnection.onFillable()V+120 j org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded()V+4 j org.eclipse.jetty.io.FillInterest.fillable()Z+57 j org.eclipse.jetty.io.ChannelEndPoint$1.run()V+7 j org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(Ljava/lang/Runnable;)V+1 j org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(Z)Z+507 j org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(Z)V+134 j org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run()V+2 j org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run()V+274 j org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Ljava/lang/Runnable;)V+1 j org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run()V+262 j java.lang.Thread.run()V+11 [email protected] v ~StubRoutines::call_stub ``` ***To Reproduce*** I don't know how to produce thie issue yet. but it does occur frequently during testing ***Expected behavior*** Shutdown with no error ***Screenshots*** If applicable, add screenshots to help explain your problem. ***Additional context*** Add any other context about the problem here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
