gocho1 opened a new issue #8630: Overlord High Availability causing lost of ingestion tasks URL: https://github.com/apache/incubator-druid/issues/8630 ### Affected Version Druid 0.15.1 ### Description I have a cluster running on AWS that is partly composed by 2 overlords. As stated in documentation, 1 is leader and all requests should be forwarded to him by the other overlord. Test run : Ensure 2 overlords are properly running, with logging debug mode activated on both nodes. Identify the leader with API endpoint `/druid/indexer/v1/leader` Send an ingestion spec to the non leader node. Watching the UI to see if the task is running Issue : When I ran the test, no tasks appear running. When analyzing the overlords logs, here is what I found On the non leader node, I find that it forwarded the request to the leader : ``` DEBUG [qtp1472031375-88] org.apache.druid.server.http.RedirectFilter - Forwarding request to [http://leader-overlord:8090/druid/indexer/v1/task] ``` On the leader node, the log indicate that the requests arrives, as I can find a request with the following header `X-Forwarded-For: non-leader-overlord-ip` But then, I can also find some errors : an NPE : ``` 2019-10-04T09:20:10,389 ERROR [qtp1472031375-90] com.sun.jersey.spi.container.ContainerResponse - The RuntimeException could not be mapped to a response, re-throwing to the HTTP container java.lang.NullPointerException at org.apache.druid.indexing.overlord.http.OverlordResource.taskPost(OverlordResource.java:153) ~[druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_222] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_222] at sun.reflect.DelegatingMeth java.lang.NullPointerException at org.apache.druid.indexing.overlord.http.OverlordResource.taskPost(OverlordResource.java:153) ~[druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating] ``` And then a 500 error ``` 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - COMMIT for /druid/indexer/v1/task on HttpChannelOverHttp@7529c4ca{r=1,c=true,a=DISPATCHED,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=57} 500 Server Error HTTP/1.1 Cache-Control: must-revalidate,no-cache,no-store Content-Type: text/html;charset=iso-8859-1 ``` Here is the full stack trace ``` 2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.io.ChannelEndPoint - filled 392 SocketChannelEndPoint@3a62407a{/172.29.182.74:41862<->/overlord-leader-ip:8090,OPEN,fill=-,flush=-,to=1/300000}{io=0/0,kio=0,kro=1}->HttpConnection@64c6429c[p=HttpParser{s=START,0 of -1},g=HttpGenerator@4c8fafee{s=START}]=>HttpChannelOverHttp@7529c4ca{r=0,c=false,a=IDLE,uri=null,age=0} 2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpConnection - HttpConnection@64c6429c[p=HttpParser{s=START,0 of -1},g=HttpGenerator@4c8fafee{s=START}]=>HttpChannelOverHttp@7529c4ca{r=0,c=false,a=IDLE,uri=null,age=0}<-SocketChannelEndPoint@3a62407a{/172.29.182.74:41862<->/overlord-leader-ip:8090,OPEN,fill=-,flush=-,to=0/300000}{io=0/0,kio=0,kro=1}->HttpConnection@64c6429c[p=HttpParser{s=START,0 of -1},g=HttpGenerator@4c8fafee{s=START}]=>HttpChannelOverHttp@7529c4ca{r=0,c=false,a=IDLE,uri=null,age=0} parse HeapByteBuffer@ad74436[p=0,l=392,c=8192,r=392]={<<<POST /druid/index...cation/json\r\n\r\n>>>ation/json\r\nAccep...: {} \n }\x00\x00\x00} {} 2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - parseNext s=START HeapByteBuffer@ad74436[p=0,l=392,c=8192,r=392]={<<<POST /druid/index...cation/json\r\n\r\n>>>ation/json\r\nAccep...: {} \n }\x00\x00\x00} 2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - START --> SPACE1 2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - SPACE1 --> URI 2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - URI --> SPACE2 2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - SPACE2 --> REQUEST_VERSION 2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - REQUEST_VERSION --> HEADER 2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> VALUE 2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> IN_VALUE 2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> FIELD 2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> VALUE 2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> IN_VALUE 2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> FIELD 2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> IN_NAME 2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> VALUE 2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> IN_VALUE 2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> FIELD 2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> VALUE 2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> IN_VALUE 2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> FIELD 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> IN_NAME 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> VALUE 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> IN_VALUE 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> FIELD 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:Content-Length: 0 --> IN_VALUE 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:Content-Length: 0 --> FIELD 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> VALUE 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> IN_VALUE 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> FIELD 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> VALUE 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> IN_VALUE 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> FIELD 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:accept: */* --> IN_VALUE 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:accept: */* --> FIELD 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:Content-Type: application/json --> IN_VALUE 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:Content-Type: application/json --> FIELD 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER --> END 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - REQUEST for //overlord-loadbalancer-url/druid/indexer/v1/task on HttpChannelOverHttp@7529c4ca{r=1,c=false,a=IDLE,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=0} POST //overlord-loadbalancer-url/druid/indexer/v1/task HTTP/1.1 X-Forwarded-For: 172.29.182.70 X-Forwarded-Proto: https X-Forwarded-Port: 443 Host: overlord-loadbalancer-url X-Amzn-Trace-Id: Root=1-5d970eca-416a917c2bee601c58bbbaa0 Content-Length: 0 Authorization: Basic dGVzdDpNMlEwTWpka05ERTROamxtT0RSbE1qZG1PVGRoTWpFeg== User-Agent: curl/7.61.1 Accept: */* Content-Type: application/json 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - HttpChannelOverHttp@7529c4ca{r=1,c=false,a=IDLE,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=0} onContentComplete 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - HttpChannelOverHttp@7529c4ca{r=1,c=false,a=IDLE,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=0} onRequestComplete 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpInput - HttpInputOverHTTP@8985515[c=0,q=0,[0]=null,s=STREAM] addContent EOF 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpConnection - HttpConnection@64c6429c[p=HttpParser{s=END,0 of 0},g=HttpGenerator@4c8fafee{s=START}]=>HttpChannelOverHttp@7529c4ca{r=1,c=false,a=IDLE,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=0}<-SocketChannelEndPoint@3a62407a{/172.29.182.74:41862<->/overlord-leader-ip:8090,OPEN,fill=-,flush=-,to= 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpConnection - releaseRequestBuffer HttpConnection@64c6429c[p=HttpParser{s=END,0 of 0},g=HttpGenerator@4c8fafee{s=START}]=>HttpChannelOverHttp@7529c4ca{r=1,c=false,a=IDLE,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=0}<-SocketChannelEndPoint@3a62407a{/172.29.182.74:41862<->/overlord-leader-ip:8090,OP 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - HttpChannelOverHttp@7529c4ca{r=1,c=false,a=IDLE,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=0} handle //overlord-loadbalancer-url/druid/indexer/v1/task 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannelState - handling HttpChannelState@57964b85{s=IDLE a=NOT_ASYNC i=true r=IDLE w=false} 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - HttpChannelOverHttp@7529c4ca{r=1,c=false,a=DISPATCHED,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=0} action DISPATCH 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.Server - REQUEST POST /druid/indexer/v1/task on HttpChannelOverHttp@7529c4ca{r=1,c=false,a=DISPATCHED,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=0} 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.handler.gzip.GzipHandler - org.eclipse.jetty.server.handler.gzip.GzipHandler@48d739f handle Request(POST //overlord-loadbalancer-url/druid/indexer/v1/task)@2049e2fc in null 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.handler.ContextHandler - scope null||/druid/indexer/v1/task @ o.e.j.s.ServletContextHandler@19fec3d6{/,[jar:file:/opt/druid-0.15.1/lib/druid-console-0.15.1-incubating.jar!/org/apache/druid/console],AVAILABLE} 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.handler.ContextHandler - context=||/druid/indexer/v1/task @ o.e.j.s.ServletContextHandler@19fec3d6{/,[jar:file:/opt/druid-0.15.1/lib/druid-console-0.15.1-incubating.jar!/org/apache/druid/console],AVAILABLE} 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.session - sessionHandler=org.eclipse.jetty.server.session.SessionHandler1384626783==dftMaxIdleSec=-1 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.session - session=null 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.servlet.ServletHandler - servlet |/druid/indexer/v1/task|null -> default@5c13d641==org.eclipse.jetty.servlet.DefaultServlet,jsp=null,order=-1,inst=true 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.servlet.ServletHandler - chain=org.apache.druid.server.security.SecuritySanityCheckFilter-2a0b901c->org.apache.druid.server.security.AuthenticationWrappingFilter-7b96de8d->org.apache.druid.server.security.AllowOptionsResourceFilter-5c3d4f05->org.apache.druid.server.security.PreResponseAuthorizationCheckFilter-7e61e25c->org.apache.d 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.servlet.ServletHandler - call filter org.apache.druid.server.security.SecuritySanityCheckFilter-2a0b901c 2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.servlet.ServletHandler - call filter org.apache.druid.server.security.AuthenticationWrappingFilter-7b96de8d 2019-10-04T09:20:10,387 DEBUG [qtp1472031375-90] org.eclipse.jetty.servlet.ServletHandler - call filter org.apache.druid.server.security.AllowOptionsResourceFilter-5c3d4f05 2019-10-04T09:20:10,387 DEBUG [qtp1472031375-90] org.eclipse.jetty.servlet.ServletHandler - call filter org.apache.druid.server.security.PreResponseAuthorizationCheckFilter-7e61e25c 2019-10-04T09:20:10,388 DEBUG [qtp1472031375-90] org.eclipse.jetty.servlet.ServletHandler - call filter org.apache.druid.server.http.RedirectFilter-7d30007d 2019-10-04T09:20:10,388 DEBUG [qtp1472031375-90] org.eclipse.jetty.servlet.ServletHandler - call filter com.google.inject.servlet.GuiceFilter-382d71c7 2019-10-04T09:20:10,388 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannelState - onEof HttpChannelState@57964b85{s=DISPATCHED a=NOT_ASYNC i=true r=IDLE w=false} 2019-10-04T09:20:10,388 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannelState - onEof HttpChannelState@57964b85{s=DISPATCHED a=NOT_ASYNC i=true r=READY w=false} 2019-10-04T09:20:10,388 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannelState - onEof HttpChannelState@57964b85{s=DISPATCHED a=NOT_ASYNC i=true r=READY w=false} 2019-10-04T09:20:10,389 ERROR [qtp1472031375-90] com.sun.jersey.spi.container.ContainerResponse - The RuntimeException could not be mapped to a response, re-throwing to the HTTP container java.lang.NullPointerException at org.apache.druid.indexing.overlord.http.OverlordResource.taskPost(OverlordResource.java:153) ~[druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_222] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_222] at sun.reflect.DelegatingMeth java.lang.NullPointerException at org.apache.druid.indexing.overlord.http.OverlordResource.taskPost(OverlordResource.java:153) ~[druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_222] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_222] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_222] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222] at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[jersey-server-1.19.3.jar:1.19.3] at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[jersey-server-1.19.3.jar:1.19.3] at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[jersey-server-1.19.3.jar:1.19.3] at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[jersey-server-1.19.3.jar:1.19.3] at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.3.jar:1.19.3] at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) ~[jersey-server-1.19.3.jar:1.19.3] at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.3.jar:1.19.3] at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[jersey-server-1.19.3.jar:1.19.3] at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) ~[jersey-server-1.19.3.jar:1.19.3] at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [jersey-server-1.19.3.jar:1.19.3] at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [jersey-server-1.19.3.jar:1.19.3] at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [jersey-server-1.19.3.jar:1.19.3] at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [jersey-servlet-1.19.3.jar:1.19.3] at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [jersey-servlet-1.19.3.jar:1.19.3] at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [jersey-servlet-1.19.3.jar:1.19.3] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0] at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) [guice-servlet-4.1.0.jar:?] at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) [guice-servlet-4.1.0.jar:?] at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) [guice-servlet-4.1.0.jar:?] at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) [guice-servlet-4.1.0.jar:?] at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) [guice-servlet-4.1.0.jar:?] at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) [guice-servlet-4.1.0.jar:?] at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135) [guice-servlet-4.1.0.jar:?] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503] at org.apache.druid.server.http.RedirectFilter.doFilter(RedirectFilter.java:71) [druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503] at org.apache.druid.server.security.PreResponseAuthorizationCheckFilter.doFilter(PreResponseAuthorizationCheckFilter.java:82) [druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503] at org.apache.druid.server.security.AllowOptionsResourceFilter.doFilter(AllowOptionsResourceFilter.java:75) [druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503] at org.apache.druid.security.basic.authentication.BasicHTTPAuthenticator$BasicHTTPAuthenticationFilter.doFilter(BasicHTTPAuthenticator.java:189) [druid-basic-security-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59) [druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503] at org.apache.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:86) [druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:724) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.server.Server.handle(Server.java:531) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) [jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) [jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) [jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:760) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503] at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:678) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-10-04T09:20:10,391 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.session - FinalSession=null old_session_manager=null this=org.eclipse.jetty.server.session.SessionHandler1384626783==dftMaxIdleSec=-1 2019-10-04T09:20:10,392 WARN [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - /druid/indexer/v1/task java.lang.NullPointerException at org.apache.druid.indexing.overlord.http.OverlordResource.taskPost(OverlordResource.java:153) ~[druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_222] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_222] at sun.reflect.DelegatingMeth 2019-10-04T09:20:10,394 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannelState - onError HttpChannelState@57964b85{s=DISPATCHED a=NOT_ASYNC i=true r=READY w=false} {} 2019-10-04T09:20:10,394 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannelState - unhandle HttpChannelState@57964b85{s=THROWN a=NOT_ASYNC i=true r=READY w=false} 2019-10-04T09:20:10,394 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - HttpChannelOverHttp@7529c4ca{r=1,c=false,a=DISPATCHED,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=38} action ERROR_DISPATCH 2019-10-04T09:20:10,396 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.Server - ERROR POST /druid/indexer/v1/task on HttpChannelOverHttp@7529c4ca{r=1,c=false,a=DISPATCHED,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=40} 2019-10-04T09:20:10,396 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.handler.gzip.GzipHandler - org.eclipse.jetty.server.handler.gzip.GzipHandler@48d739f handle Request(POST //overlord-loadbalancer-url/druid/indexer/v1/task)@2049e2fc in null 2019-10-04T09:20:10,396 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.handler.gzip.GzipHandler - org.eclipse.jetty.server.handler.gzip.GzipHandler@48d739f excluded by dispatcherType ERROR 2019-10-04T09:20:10,396 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.handler.ContextHandler - scope null||/druid/indexer/v1/task @ o.e.j.s.ServletContextHandler@19fec3d6{/,[jar:file:/opt/druid-0.15.1/lib/druid-console-0.15.1-incubating.jar!/org/apache/druid/console],AVAILABLE} 2019-10-04T09:20:10,396 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.handler.ContextHandler - context=||/druid/indexer/v1/task @ o.e.j.s.ServletContextHandler@19fec3d6{/,[jar:file:/opt/druid-0.15.1/lib/druid-console-0.15.1-incubating.jar!/org/apache/druid/console],AVAILABLE} 2019-10-04T09:20:10,396 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.session - sessionHandler=org.eclipse.jetty.server.session.SessionHandler1384626783==dftMaxIdleSec=-1 2019-10-04T09:20:10,396 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.session - session=null 2019-10-04T09:20:10,396 DEBUG [qtp1472031375-90] org.eclipse.jetty.servlet.ServletHandler - servlet |/druid/indexer/v1/task|null -> default@5c13d641==org.eclipse.jetty.servlet.DefaultServlet,jsp=null,order=-1,inst=true 2019-10-04T09:20:10,412 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor - org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor@d26694a exclude by status 500 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - sendResponse info=null content=HeapByteBuffer@5d1ddd15[p=0,l=6488,c=32768,r=6488]={<<<<html>\n<head>\n<me.../body>\n</html>\n>>>:-1},"dataSource"...ation":41536,"l} complete=true committing=true callback=Blocker@7b265756{null} 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - COMMIT for /druid/indexer/v1/task on HttpChannelOverHttp@7529c4ca{r=1,c=true,a=DISPATCHED,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=57} 500 Server Error HTTP/1.1 Cache-Control: must-revalidate,no-cache,no-store Content-Type: text/html;charset=iso-8859-1 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpConnection - org.eclipse.jetty.server.HttpConnection$SendCallback@41aa5ca4[PROCESSING][i=HTTP/1.1{s=500,h=2,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$CommitCallback@39ff5026] generate: NEED_HEADER (null,[p=0,l=6488,c=32768,r=6488],true)@START 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpGenerator - generateHeaders HTTP/1.1{s=500,h=2,cl=-1} last=true content=HeapByteBuffer@5d1ddd15[p=0,l=6488,c=32768,r=6488]={<<<<html>\n<head>\n<me.../body>\n</html>\n>>>:-1},"dataSource"...ation":41536,"l} 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpGenerator - Cache-Control: must-revalidate,no-cache,no-store Content-Type: text/html;charset=iso-8859-1 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpGenerator - EOF_CONTENT 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpConnection - org.eclipse.jetty.server.HttpConnection$SendCallback@41aa5ca4[PROCESSING][i=HTTP/1.1{s=500,h=2,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$CommitCallback@39ff5026] generate: FLUSH ([p=0,l=197,c=8192,r=197],[p=0,l=6488,c=32768,r=6488],true)@COMPLETING 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.io.WriteFlusher - write: WriteFlusher@7ad026a{IDLE}->null [HeapByteBuffer@ad74436[p=0,l=197,c=8192,r=197]={<<<HTTP/1.1 500 Serv....v20180503)\r\n\r\n>>>ca-416a917c2bee60...: {} \n }\x00\x00\x00},HeapByteBuffer@5d1ddd15[p=0,l=6488,c=32768,r=6488]={<<<<html>\n<head>\n<me.../body>\n</html>\n>>>:-1},"dataSource"...ation":41536,"l}] 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.io.WriteFlusher - update WriteFlusher@7ad026a{WRITING}->null:IDLE-->WRITING 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.io.ChannelEndPoint - flushed 6685 SocketChannelEndPoint@3a62407a{/172.29.182.74:41862<->/overlord-leader-ip:8090,OPEN,fill=-,flush=W,to=58/300000}{io=0/0,kio=0,kro=1}->HttpConnection@64c6429c[p=HttpParser{s=END,0 of 0},g=HttpGenerator@4c8fafee{s=COMPLETING}]=>HttpChannelOverHttp@7529c4ca{r=1,c=true,a=DISPATCHED,uri=//druid-overlord.emea 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.io.WriteFlusher - Flushed=true written=6685 remaining=0 WriteFlusher@7ad026a{WRITING}->null 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.io.WriteFlusher - update WriteFlusher@7ad026a{IDLE}->null:WRITING-->IDLE 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpConnection - org.eclipse.jetty.server.HttpConnection$SendCallback@41aa5ca4[PROCESSING][i=HTTP/1.1{s=500,h=2,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$CommitCallback@39ff5026] generate: SHUTDOWN_OUT ([p=197,l=197,c=8192,r=0],[p=6488,l=6488,c=32768,r=0],true)@END 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpConnection - org.eclipse.jetty.server.HttpConnection$SendCallback@41aa5ca4[PROCESSING][i=HTTP/1.1{s=500,h=2,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$CommitCallback@39ff5026] generate: DONE ([p=197,l=197,c=8192,r=0],[p=6488,l=6488,c=32768,r=0],true)@END 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.io.AbstractEndPoint - shutdownOutput SocketChannelEndPoint@3a62407a{/172.29.182.74:41862<->/overlord-leader-ip:8090,OPEN,fill=-,flush=-,to=0/300000}{io=0/0,kio=0,kro=1}->HttpConnection@64c6429c[p=HttpParser{s=END,0 of 0},g=HttpGenerator@4c8fafee{s=END}]=>HttpChannelOverHttp@7529c4ca{r=1,c=true,a=DISPATCHED,uri=//druid-overlord.emea.iods 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.session - FinalSession=null old_session_manager=org.eclipse.jetty.server.session.SessionHandler1384626783==dftMaxIdleSec=-1 this=org.eclipse.jetty.server.session.SessionHandler1384626783==dftMaxIdleSec=-1 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.Server - handled=true async=false committed=true on HttpChannelOverHttp@7529c4ca{r=1,c=true,a=DISPATCHED,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=57} 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannelState - unhandle HttpChannelState@57964b85{s=DISPATCHED a=NOT_ASYNC i=true r=READY w=false} 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - HttpChannelOverHttp@7529c4ca{r=1,c=true,a=COMPLETING,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=57} action COMPLETE 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannelState - onComplete HttpChannelState@57964b85{s=COMPLETING a=NOT_ASYNC i=false r=READY w=false} 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - COMPLETE for /druid/indexer/v1/task written=6488 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - COMPLETE for /druid/indexer/v1/task written=6488 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.apache.druid.jetty.RequestLog - 172.29.182.74 POST //overlord-loadbalancer-url/druid/indexer/v1/task HTTP/1.1 2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannelState - recycle HttpChannelState@57964b85{s=COMPLETED a=NOT_ASYNC i=false r=READY w=false} 2019-10-04T09:20:10,414 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - close HttpParser{s=END,0 of 0} 2019-10-04T09:20:10,414 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - END --> CLOSE 2019-10-04T09:20:10,414 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - HttpChannelOverHttp@7529c4ca{r=1,c=false,a=IDLE,uri=null,age=0} handle exit, result COMPLETE ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
