fantapsody opened a new issue #9201: URL: https://github.com/apache/pulsar/issues/9201
**Describe the bug** I started the broker of version `2.7.0.3` with `functionWorkerEnabled=true` in a K8s cluster and saw error logs like follows during broker start: > 04:54:54.810 [pulsar-web-45-3] WARN org.glassfish.jersey.internal.Errors - The following warnings have been detected: WARNING: Unknown HK2 failure detected: > MultiException stack 1 of 2 > javax.ws.rs.WebApplicationException: HTTP 404 Not Found > at org.apache.pulsar.common.configuration.VipStatus.checkStatus(VipStatus.java:58) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1268) > at org.jvnet.hk2.internal.ClazzCreator.methodMe(ClazzCreator.java:295) > at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:348) > at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463) > at org.glassfish.jersey.inject.hk2.RequestContext.findOrCreate(RequestContext.java:59) > at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102) > at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:758) > at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:721) > at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:691) > at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:160) > at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:30) > at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:105) > at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:260) > at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:51) > at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:86) > at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89) > at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89) > at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89) > at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:69) > at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:38) > at org.glassfish.jersey.process.internal.Stages.process(Stages.java:173) > at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:245) > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) > at org.glassfish.jersey.internal.Errors.process(Errors.java:292) > at org.glassfish.jersey.internal.Errors.process(Errors.java:274) > at org.glassfish.jersey.internal.Errors.process(Errors.java:244) > at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) > at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) > at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) > at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) > at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) > at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366) > at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319) > at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) > at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791) > at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626) > at org.apache.pulsar.broker.web.ResponseHandlerFilter.doFilter(ResponseHandlerFilter.java:58) > at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) > at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) > at org.apache.pulsar.broker.web.PreInterceptFilter.doFilter(PreInterceptFilter.java:65) > at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) > at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) > at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) > at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) > at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) > at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) > at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435) > at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) > at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) > at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) > at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) > at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350) > at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234) > at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) > at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179) > at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > at org.eclipse.jetty.server.Server.handle(Server.java:516) > at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) > at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) > at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) > at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) > at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) > at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) > at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) > at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) > at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) > at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.lang.Thread.run(Thread.java:748) > MultiException stack 2 of 2 > java.lang.IllegalStateException: Unable to perform operation: method inject on org.apache.pulsar.common.configuration.VipStatus > at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:369) > at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463) > at org.glassfish.jersey.inject.hk2.RequestContext.findOrCreate(RequestContext.java:59) > at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102) > at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:758) > at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:721) > at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:691) > at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:160) > at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:30) > at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:105) > at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:260) > at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:51) > at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:86) > at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89) > at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89) > at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89) > at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:69) > at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:38) > at org.glassfish.jersey.process.internal.Stages.process(Stages.java:173) > at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:245) > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) > at org.glassfish.jersey.internal.Errors.process(Errors.java:292) > at org.glassfish.jersey.internal.Errors.process(Errors.java:274) > at org.glassfish.jersey.internal.Errors.process(Errors.java:244) > at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) > at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) > at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) > at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) > at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) > at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366) > at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319) > at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) > at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791) > at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626) > at org.apache.pulsar.broker.web.ResponseHandlerFilter.doFilter(ResponseHandlerFilter.java:58) > at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) > at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) > at org.apache.pulsar.broker.web.PreInterceptFilter.doFilter(PreInterceptFilter.java:65) > at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) > at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) > at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) > at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) > at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) > at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) > at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435) > at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) > at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) > at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) > at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) > at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350) > at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234) > at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) > at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179) > at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > at org.eclipse.jetty.server.Server.handle(Server.java:516) > at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) > at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) > at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) > at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) > at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) > at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) > at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) > at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) > at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) > at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.lang.Thread.run(Thread.java:748) > 04:54:54.829 [pulsar-web-45-3] INFO org.eclipse.jetty.server.RequestLog - 10.88.128.7 - - [13/Jan/2021:04:54:54 +0000] "GET /status.html HTTP/1.1" 404 502 "-" "kube-probe/1.16+" 23 It seems the exceptions were triggered by the liveness/readiness probe of the container during the start. Sometimes the pod failed to become ready and got killed after consecutive failures of liveness probe. Even the pod finally became running, it's annoying and confusing to see error logs like this. These logs didn't show up if the broker was started without `functionWorkerEnabled=true` or if the version of the broker is 2.6.0. **To Reproduce** Start the broker with `functionWorkerEnabled=true` during start and access the `/status.html` for status. **Expected behavior** No error logs like described. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
