ankitsultana opened a new issue, #10894: URL: https://github.com/apache/pinot/issues/10894
I have been looking into issues with realtime tables and server restarts. On looking into a recent issue where one of the segments ran into the error state, I found the following stack trace. This looks like a JDK bug. The exception was thrown within ~7 seconds of server process start-up, so I suspect that if we put in a retry with some delay things may work. We could potentially put the retry in `buildCloseableHttpClient`. ``` 2023-06-12 05:54:01.122 [HelixTaskExecutor-message_handle_thread_17] ERROR o.a.helix.messaging.handling.HelixStateTransitionHandler - Exception while executing a state transition task table__134__1339__20230611T0357Z -- java.lang.reflect.InvocationTargetException: null at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:350) at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:278) at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.NullPointerException: null at java.base/sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:133) at java.base/sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:155) at java.base/java.net.URL.openStream(URL.java:1165) at java.base/java.lang.ClassLoader.getResourceAsStream(ClassLoader.java:1739) at org.apache.http.util.VersionInfo.loadVersionInfo(VersionInfo.java:233) at org.apache.http.util.VersionInfo.getUserAgent(VersionInfo.java:319) at org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:1057) at org.apache.pinot.common.utils.http.HttpClient.buildCloseableHttpClient(HttpClient.java:482) at org.apache.pinot.common.utils.http.HttpClient.<init>(HttpClient.java:101) at org.apache.pinot.common.utils.FileUploadDownloadClient.<init>(FileUploadDownloadClient.java:138) at org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler.<init>(ServerSegmentCompletionProtocolHandler.java:82) at org.apache.pinot.core.data.manager.realtime.SegmentBuildTimeLeaseExtender.<init>(SegmentBuildTimeLeaseExtender.java:97) at org.apache.pinot.core.data.manager.realtime.SegmentBuildTimeLeaseExtender.lambda$getOrCreate$0(SegmentBuildTimeLeaseExtender.java:83) at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1947) at org.apache.pinot.core.data.manager.realtime.SegmentBuildTimeLeaseExtender.getOrCreate(SegmentBuildTimeLeaseExtender.java:81) at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.doInit(RealtimeTableDataManager.java:140) at org.apache.pinot.core.data.manager.BaseTableDataManager.init(BaseTableDataManager.java:167) at org.apache.pinot.core.data.manager.offline.TableDataManagerProvider.getTableDataManager(TableDataManagerProvider.java:93) at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.createTableDataManager(HelixInstanceDataManager.java:225) at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.lambda$addRealtimeSegment$3(HelixInstanceDataManager.java:216) at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1737) at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addRealtimeSegment(HelixInstanceDataManager.java:216) at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:168) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
