soullkk opened a new issue, #14242:
URL: https://github.com/apache/druid/issues/14242

   Broker errors when **handshake with** historical or indexer
   
   2023-04-02 04:35:24,831 WARN  
[qtp1319386445-190[groupBy_[xxxx]_7a5c4d33-5a19-41d3-b39c-0d1f4458e68c]][][org.apache.druid.java.util.http.client.NettyHttpClient]
 **Netty faulty channel failed!!!**
   org.jboss.netty.channel.ChannelException: Failed to handshake with 
host[https://x.x.x.x:26204]
           at 
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory$2$1.operationComplete(ChannelResourceFactory.java:157)
 ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:409)
 ~[netty-3.10.6.Final.jar:?]
           at 
org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:395)
 ~[netty-3.10.6.Final.jar:?]
           at 
org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:362)
 ~[netty-3.10.6.Final.jar:?]
           at 
org.jboss.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1461)
 ~[netty-3.10.6.Final.jar:?]
           at 
org.jboss.netty.handler.ssl.SslHandler.access$200(SslHandler.java:180) 
~[netty-3.10.6.Final.jar:?]
           at org.jboss.netty.handler.ssl.SslHandler$1.run(SslHandler.java:372) 
~[netty-3.10.6.Final.jar:?]
           at 
org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:556)
 ~[netty-3.10.6.Final.jar:?]
           at 
org.jboss.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:632)
 ~[netty-3.10.6.Final.jar:?]
           at 
org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:369) 
~[netty-3.10.6.Final.jar:?]
           at 
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
 ~[netty-3.10.6.Final.jar:?]
           at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
   Caused by: javax.net.ssl.SSLException: Handshake did not complete within 
10000ms
           ... 6 more
   2023-04-02 04:35:24,875 WARN  
[ForkJoinPool-1-worker-14][][org.apache.druid.client.JsonParserIterator] Query 
[7a5c4d33-5a19-41d3-b39c-0d1f4458e68c] to host [x.x.x.x:26204] interrupted
   org.jboss.netty.channel.ChannelException: Faulty channel in resource pool
           at 
org.apache.druid.java.util.http.client.NettyHttpClient.go(NettyHttpClient.java:132)
 ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.client.DirectDruidClient.run(DirectDruidClient.java:462) 
~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.client.CachingClusteredClient$SpecificQueryRunnable.getSimpleServerResults(CachingClusteredClient.java:723)
 ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.client.CachingClusteredClient$SpecificQueryRunnable.lambda$addSequencesFromServer$9(CachingClusteredClient.java:685)
 ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
           at java.util.TreeMap.forEach(TreeMap.java:1005) ~[?:1.8.0_362]
           at 
org.apache.druid.client.CachingClusteredClient$SpecificQueryRunnable.addSequencesFromServer(CachingClusteredClient.java:669)
 ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.client.CachingClusteredClient$SpecificQueryRunnable.lambda$run$2(CachingClusteredClient.java:397)
 ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.java.util.common.guava.LazySequence.toYielder(LazySequence.java:46)
 ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
          at 
org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88)
 ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84)
 ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
 ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83)
 ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.java.util.common.guava.MergeSequence.lambda$toYielder$1(MergeSequence.java:65)
 ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44)
 ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.java.util.common.guava.MergeSequence.toYielder(MergeSequence.java:62)
 ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.query.RetryQueryRunner$1.toYielder(RetryQueryRunner.java:134) 
~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.common.guava.CombiningSequence.toYielder(CombiningSequence.java:78)
 ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88)
 ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84)
 ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
 ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83)
 ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49)
 ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88)
 ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84)
 ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:78)
 ~[druid-processing-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83)
 ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.java.util.common.guava.Yielders.each(Yielders.java:32) 
~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.server.QueryResource.doPost(QueryResource.java:230) 
~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
           at sun.reflect.GeneratedMethodAccessor115.invoke(Unknown Source) 
~[?:?]
           at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_362]
           at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_362]
           at 
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
 ~[jersey-server-1.19.3.jar:1.19.3]
           at 
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
 ~[jersey-server-1.19.3.jar:1.19.3]
           at 
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
 ~[jersey-server-1.19.3.jar:1.19.3]
           at 
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
 ~[jersey-server-1.19.3.jar:1.19.3]
           at 
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
 ~[jersey-server-1.19.3.jar:1.19.3]
           at 
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
 ~[jersey-server-1.19.3.jar:1.19.3]
           at 
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
 ~[jersey-server-1.19.3.jar:1.19.3]
         at 
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
 ~[jersey-server-1.19.3.jar:1.19.3]
           at 
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
 ~[jersey-server-1.19.3.jar:1.19.3]
           at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
 ~[jersey-server-1.19.3.jar:1.19.3]
           at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
 ~[jersey-server-1.19.3.jar:1.19.3]
           at 
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
 ~[jersey-servlet-1.19.3.jar:1.19.3]
           at 
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
 ~[jersey-servlet-1.19.3.jar:1.19.3]
           at 
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
 ~[jersey-servlet-1.19.3.jar:1.19.3]
           at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) 
~[javax.servlet-api-3.1.0.jar:3.1.0]
           at 
com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)
 ~[guice-servlet-4.1.0.jar:?]
           at 
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)
 ~[guice-servlet-4.1.0.jar:?]
           at 
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) 
~[guice-servlet-4.1.0.jar:?]
           at 
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
 ~[guice-servlet-4.1.0.jar:?]
           at 
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
 ~[guice-servlet-4.1.0.jar:?]
           at 
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
 ~[guice-servlet-4.1.0.jar:?]
           at 
com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135) 
~[guice-servlet-4.1.0.jar:?]
           at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) 
~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
 ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.apache.druid.server.security.PreResponseAuthorizationCheckFilter.doFilter(PreResponseAuthorizationCheckFilter.java:82)
 ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) 
~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
 ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.apache.druid.server.security.AllowHttpMethodsResourceFilter.doFilter(AllowHttpMethodsResourceFilter.java:78)
 ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) 
~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
 ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.apache.druid.server.security.AllowOptionsResourceFilter.doFilter(AllowOptionsResourceFilter.java:75)
 ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) 
~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
 ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.apache.druid.server.security.AllowAllAuthenticator$1.doFilter(AllowAllAuthenticator.java:84)
 ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59)
 ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) 
~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
 ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.apache.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:77)
 ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) 
~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
 ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) 
~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
 ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
 ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
 ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
 ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
 ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) 
~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
 ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
 ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
 ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) 
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) 
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59) 
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)
 ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) 
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
           at org.eclipse.jetty.server.Server.handle(Server.java:516) 
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) 
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) 
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
           at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) 
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) 
~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
 ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
           at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) 
~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
 ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410) 
~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164) 
~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
           at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) 
~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) 
~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
 ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
 ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
 ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
 ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
 ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
 ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
           at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
 ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
           at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
   Caused by: org.jboss.netty.channel.ChannelException: **Failed to handshake 
with host[https://x.x.x.x:26204]**
           at 
org.apache.druid.java.util.http.client.pool.ChannelResourceFactory$2$1.operationComplete(ChannelResourceFactory.java:157)
 ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
           at 
org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:409)
 ~[netty-3.10.6.Final.jar:?]
           at 
org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:395)
 ~[netty-3.10.6.Final.jar:?]
           at 
org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:362)
 ~[netty-3.10.6.Final.jar:?]
           at 
org.jboss.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1461)
 ~[netty-3.10.6.Final.jar:?]
           at 
org.jboss.netty.handler.ssl.SslHandler.access$200(SslHandler.java:180) 
~[netty-3.10.6.Final.jar:?]
           at org.jboss.netty.handler.ssl.SslHandler$1.run(SslHandler.java:372) 
~[netty-3.10.6.Final.jar:?]
           at 
org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:556)
 ~[netty-3.10.6.Final.jar:?]
           at 
org.jboss.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:632)
 ~[netty-3.10.6.Final.jar:?]
           at 
org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:369) 
~[netty-3.10.6.Final.jar:?]
           at 
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
 ~[netty-3.10.6.Final.jar:?]
           ... 1 more
   Caused by: javax.net.ssl.SSLException: **Handshake did not complete within 
10000ms**
           at org.jboss.netty.handler.ssl.SslHandler$1.run(SslHandler.java:372) 
~[netty-3.10.6.Final.jar:?]
           at 
org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:556)
 ~[netty-3.10.6.Final.jar:?]
           at 
org.jboss.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:632)
 ~[netty-3.10.6.Final.jar:?]
           at 
org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:369) 
~[netty-3.10.6.Final.jar:?]
           at 
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
 ~[netty-3.10.6.Final.jar:?]
   
   ### Affected Version
   
   druid 0.21 version
   
   ### Description
   
   Please include as much detailed information about the problem as possible.
   - total 7 nodes in cluster
   
   This exception occurs frequently.
   I found that the exception occurred in the SslHandler.handshake() when the 
ChannelFuture is null or not done. 
   The source code in netty is as follows:
   
               try {
                   engine.beginHandshake();
                   runDelegatedTasks();
                   handshakeFuture = this.handshakeFuture = future(channel);
                   if (handshakeTimeoutInMillis > 0) {
                       handshakeTimeout = timer.newTimeout(new TimerTask() {
                               public void run(Timeout timeout) throws 
Exception {
                               ChannelFuture future = 
SslHandler.this.handshakeFuture;
                               if (future != null && future.isDone()) {
                                   return;
                               }
                               setHandshakeFailure(channel, new 
SSLException("Handshake did not complete within " +
                                               handshakeTimeoutInMillis + 
"ms"));
                           }
                           }, handshakeTimeoutInMillis, TimeUnit.MILLISECONDS);
                   }
               } catch (Exception e) {
                   handshakeFuture = this.handshakeFuture = 
failedFuture(channel, e);
                   exception = e;
               }
   
   
   
   I checked the server's configuration for druid. server. http. numThreads as 
follows:
   overlord=50
   coordinator=50
   indexer=25
   historical=25
   broker=12
   
   When there is a exception connecting to other 
nodes(coodinator,historical,overlord,indexer), the metric of 
"jetty/numOpenConnections"  are as follows:
   
{"feed":"metrics","timestamp":"2023-05-05T10:00:26.959Z","service":"druid/coordinator","host":"x.x.x.x:26200","version":"","metric":"jetty/numOpenConnections","value":141}
   
{"feed":"metrics","timestamp":"2023-05-06T10:12:44.278Z","service":"druid/historical","host":"x.x.x.x:26202","version":"","metric":"jetty/numOpenConnections","value":49}
   
{"feed":"metrics","timestamp":"2023-05-06T12:26:32.252Z","service":"druid/overlord","host":"x.x.x.x:26203","version":"","metric":"jetty/numOpenConnections","value":158}
   
{"feed":"metrics","timestamp":"2023-05-06T18:10:34.614Z","service":"druid/indexer","host":"x.x.x.x:26204","version":"","metric":"jetty/numOpenConnections","value":44}
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to