donatelloOo opened a new issue, #12376: URL: https://github.com/apache/pinot/issues/12376
While ingesting real-time data in a single table we are getting this SIGSEGV fatal error periodically on all servers. This table is composed of: - a few dimension fields (SV with inverted indexes) - a lot of metric fields (SV with forward index only) - a timestamp field Below is a short extract of the core dump log (full one is attached). ``` # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007f9ba3e89056, pid=1, tid=247 # # JRE version: OpenJDK Runtime Environment Corretto-11.0.20.9.1 (11.0.20.1+9) (build 11.0.20.1+9-LTS) # Java VM: OpenJDK 64-Bit Server VM Corretto-11.0.20.9.1 (11.0.20.1+9-LTS, mixed mode, compressed oops, g1 gc, linux-amd64) # Problematic frame: # J 3816 c2 org.apache.pinot.segment.local.segment.readers.PinotSegmentColumnReader.getValue(I)Ljava/lang/Object; (648 bytes) @ 0x00007f9ba3e89056 [0x00007f9ba3e88f20+0x0000000000000136] # # Core dump will be written. Default location: /opt/pinot/core.1 # # If you would like to submit a bug report, please visit: # https://github.com/corretto/corretto-11/issues/ # --------------- S U M M A R Y ------------ Command Line: -Xms512M -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+ErrorFileToStdout -XX:-TieredCompilation -Xlog:gc*:file=/opt/pinot/gc-pinot-server.log -javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=8008:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -Dlog4j2.configurationFile=/opt/pinot/etc/conf/pinot-server-log4j2.xml -Dplugins.dir=/opt/pinot/plugins -Dplugins.dir=/opt/pinot/plugins -Dapp.name=pinot-admin -Dapp.pid=1 -Dapp.repo=/opt/pinot/lib -Dapp.home=/opt/pinot -Dbasedir=/opt/pinot org.apache.pinot.tools.admin.PinotAdministrator StartServer -clusterName pinot -zkAddress pinot-zookeeper:2181 -configFileName /var/pinot/server/config/pinot-server.conf Host: Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz, 16 cores, 32G, Amazon Linux release 2 (Karoo) Time: Tue Feb 6 12:41:51 2024 UTC elapsed time: 2970.773692 seconds (0d 0h 49m 30s) --------------- T H R E A D --------------- Current thread (0x00007f9a04062800): JavaThread "obf_50d04beb9d1d306c5a5e45167656a565abc036da0763183eb9f074402fbd4f2c__0__3__20240206T1214Z" daemon [_thread_in_Java, id=247, stack(0x00007f9a264f6000,0x00007f9a265f7000)] Stack: [0x00007f9a264f6000,0x00007f9a265f7000], sp=0x00007f9a265f55e0, free space=1021k Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code) J 3816 c2 org.apache.pinot.segment.local.segment.readers.PinotSegmentColumnReader.getValue(I)Ljava/lang/Object; (648 bytes) @ 0x00007f9ba3e89056 [0x00007f9ba3e88f20+0x0000000000000136] J 3819 c2 org.apache.pinot.segment.local.segment.readers.PinotSegmentRecordReader.getRecord(ILorg/apache/pinot/spi/data/readers/GenericRow;)V (106 bytes) @ 0x00007f9ba3e726ac [0x00007f9ba3e72080+0x000000000000062c] J 4599 c2 org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build()V (376 bytes) @ 0x00007f9ba3dc04c4 [0x00007f9ba3dbfc60+0x0000000000000864] j org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(Lorg/apache/pinot/segment/spi/creator/SegmentVersion;Lorg/apache/pinot/common/metrics/ServerMetrics;)V+282 j org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.buildSegmentInternal(Z)Lorg/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager$SegmentBuildDescriptor;+206 j org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.buildSegmentAndReplace()Z+2 j org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager$PartitionConsumer.run()V+561 j java.lang.Thread.run()V+11 java.base@11.0.20.1 v ~StubRoutines::call_stub V [libjvm.so+0x8e13bb] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x39b V [libjvm.so+0x8df37d] JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, Thread*)+0x1ed V [libjvm.so+0x98ae7c] thread_entry(JavaThread*, Thread*)+0x6c V [libjvm.so+0xedf730] JavaThread::run()+0x280 V [libjvm.so+0xedc0ff] Thread::call_run()+0x14f V [libjvm.so+0xc78ea6] thread_native_entry(Thread*)+0xe6 siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007f95f0137ac8 ``` > - Sensitive data has been obfuscated. > - Tiered Compilation was disabled to try fixing this kind of SIGSEGV errors while invoking MutableForwardIndex / OffHeapMutableDictionary See full core dump here: [SIGSEGV-obf.log](https://github.com/apache/pinot/files/14183217/SIGSEGV-obf.log) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
