| |
|
|
|
|
Hi,
we fixed the jenkins 2.176.3 web interface stop under rather strange circumstances
before stopping, multiple error messages preceded with for various tasks
{code:java}
Oct 18, 2019 9:55:21 AM hudson.model.RunMap retrieveWARNING: could not load /opt/jenkins/jenkins_home/jobs/UCP/jobs/Release_UCP_BROKER/builds/68java.io.IOException: Unable to read /opt/jenkins/jenkins_home/jobs/UCP/jobs/Release_UCP_BROKER/builds/68/build.xml at hudson.XmlFile.unmarshal(XmlFile.java:181) at hudson.XmlFile.unmarshal(XmlFile.java:161) at org.jenkinsci.plugins.workflow.job.WorkflowRun.reload(WorkflowRun.java:723) at hudson.model.Run.<init>(Run.java:330) at org.jenkinsci.plugins.workflow.job.WorkflowRun.<init>(WorkflowRun.java:249) at sun.reflect.GeneratedConstructorAccessor205.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at jenkins.model.lazy.LazyBuildMixIn.loadBuild(LazyBuildMixIn.java:166) at jenkins.model.lazy.LazyBuildMixIn$1.create(LazyBuildMixIn.java:143) at hudson.model.RunMap.retrieve(RunMap.java:224) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:346) at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:74) at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63) at java.util.AbstractMap$2$1.next(AbstractMap.java:418) at com.google.common.collect.Iterators$PeekingImpl.peek(Iterators.java:1186) at com.google.common.collect.Iterators$MergingIterator$1.compare(Iterators.java:1300) at com.google.common.collect.Iterators$MergingIterator$1.compare(Iterators.java:1297) at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:670) at java.util.PriorityQueue.siftUp(PriorityQueue.java:646) at java.util.PriorityQueue.offer(PriorityQueue.java:345) at java.util.PriorityQueue.add(PriorityQueue.java:322) at com.google.common.collect.Iterators$MergingIterator.computeNext(Iterators.java:1323) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at com.google.common.collect.Iterators$7.computeNext(Iterators.java:647) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at hudson.util.Iterators$9.fetch(Iterators.java:388) at hudson.util.Iterators$9.hasNext(Iterators.java:375) at hudson.util.Iterators$9.fetch(Iterators.java:388) at hudson.util.Iterators$9.hasNext(Iterators.java:375) at com.google.common.collect.Iterators$7.computeNext(Iterators.java:647) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at hudson.model.BuildTimelineWidget.doData(BuildTimelineWidget.java:64) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145) at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:219) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:456) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:219) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:676) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:246) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.plugins.locale.LocaleFilter.doFilter(LocaleFilter.java:42) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:92) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1701) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1668) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:502) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.lang.Thread.run(Thread.java:748)Caused by: com.thoughtworks.xstream.converters.ConversionException: : expected: /entry read: iota (position:END_TAG </iota>@2813:21 in java.io.InputStreamReader@b188934) : : expected: /entry read: iota (position:END_TAG </iota>@2813:21 in java.io.InputStreamReader@b188934)---- Debugging information ----message : : expected: /entry read: iota (position:END_TAG </iota>@2813:21 in java.io.InputStreamReader@b188934)cause-exception : com.thoughtworks.xstream.io.StreamExceptioncause-message : : expected: /entry read: iota (position:END_TAG </iota>@2813:21 in java.io.InputStreamReader@b188934)class : org.jenkinsci.plugins.workflow.job.WorkflowRunrequired-type : java.util.HashMapconverter-type : hudson.util.RobustReflectionConverterpath : /flow-build/execution/timings/entry[4]line number : 2813version : not available------------------------------- at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:79) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50) at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189) at hudson.util.XStream2.unmarshal(XStream2.java:161) at hudson.util.XStream2.unmarshal(XStream2.java:132) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173) at hudson.XmlFile.unmarshal(XmlFile.java:178) ... 144 moreCaused by: com.thoughtworks.xstream.io.StreamException: : expected: /entry read: iota (position:END_TAG </iota>@2813:21 in java.io.InputStreamReader@b188934) at com.thoughtworks.xstream.io.xml.XppReader.pullNextEvent(XppReader.java:124) at com.thoughtworks.xstream.io.xml.AbstractPullReader.readRealEvent(AbstractPullReader.java:148) at com.thoughtworks.xstream.io.xml.AbstractPullReader.readEvent(AbstractPullReader.java:141) at com.thoughtworks.xstream.io.xml.AbstractPullReader.move(AbstractPullReader.java:118) at com.thoughtworks.xstream.io.xml.AbstractPullReader.moveUp(AbstractPullReader.java:113) at com.thoughtworks.xstream.io.ReaderWrapper.moveUp(ReaderWrapper.java:40) at com.thoughtworks.xstream.io.path.PathTrackingReader.moveUp(PathTrackingReader.java:42) at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:364) at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:268) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) ... 154 moreCaused by: org.xmlpull.v1.XmlPullParserException: expected: /entry read: iota (position:END_TAG </iota>@2813:21 in java.io.InputStreamReader@b188934) at org.kxml2.io.KXmlParser.exception(Unknown Source) at org.kxml2.io.KXmlParser.error(Unknown Source) at org.kxml2.io.KXmlParser.parseEndTag(Unknown Source) at org.kxml2.io.KXmlParser.nextImpl(Unknown Source) at org.kxml2.io.KXmlParser.next(Unknown Source) at com.thoughtworks.xstream.io.xml.XppReader.pullNextEvent(XppReader.java:109) ... 163 moreWARNING: Failed to retrieve incoming Pull Requests from Bitbucketcom.cloudbees.jenkins.plugins.bitbucket.api.BitbucketRequestException: HTTP request error. Status: 401: .HttpResponseProxy{HTTP/1.1 401 [X-AREQUESTID: *134DEQFx597x100831134x4, X-ASEN: SEN-9068428, Cache-Control: no-cache, no-transform, Vary: X-AUSERNAME, Vary: X-AUSERID, Vary: Cookie, X-Content-Type-Options: nosniff, WWW-Authenticate: OAuth realm="https%3A%2F%2Fstash.delta.sbrf.ru", Content-Type: application/json;charset=UTF-8, Transfer-Encoding: chunked, Date: Fri, 18 Oct 2019 06:57:10 GMT] ResponseEntityProxy{[Content-Type: application/json;charset=UTF-8,Chunked: true]}} at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getRequest(BitbucketServerAPIClient.java:797) at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getResources(BitbucketServerAPIClient.java:749) at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getPullRequests(BitbucketServerAPIClient.java:339) at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getIncomingOpenPullRequests(BitbucketServerAPIClient.java:334) at com.cloudbees.jenkins.plugins.bitbucket.hooks.NativeServerPushHookProcessor$HeadEvent.loadPullRequests(NativeServerPushHookProcessor.java:254) at com.cloudbees.jenkins.plugins.bitbucket.hooks.NativeServerPushHookProcessor$HeadEvent.getPullRequests(NativeServerPushHookProcessor.java:226) at com.cloudbees.jenkins.plugins.bitbucket.hooks.NativeServerPushHookProcessor$HeadEvent.addPullRequests(NativeServerPushHookProcessor.java:186) at com.cloudbees.jenkins.plugins.bitbucket.hooks.NativeServerPushHookProcessor$HeadEvent.heads(NativeServerPushHookProcessor.java:133) at com.cloudbees.jenkins.plugins.bitbucket.hooks.NativeServerHeadEvent.heads(NativeServerHeadEvent.java:84) at jenkins.scm.api.SCMHeadEvent.isMatch(SCMHeadEvent.java:137) at jenkins.branch.MultiBranchProject$SCMEventListenerImpl.processHeadUpdate(MultiBranchProject.java:1406) at jenkins.branch.MultiBranchProject$SCMEventListenerImpl.onSCMHeadEvent(MultiBranchProject.java:1184) at jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:246) at jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:229) at jenkins.scm.api.SCMEvent$Dispatcher.run(SCMEvent.java:505) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
{code}
Further, an interesting message was found in the log, please pay attention to the message time
{code:java}
Oct 18, 2019 9:57:26 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher logLastLinesSEVERE: Error in provisioning; agent=KubernetesSlave name: massive-8g-os-bmb0v, template=PodTemplate{inheritFrom='', name='massive-8G-os', namespace='', label='openshift_8Gb DefaultPool Linux_Default ansible linux masterLin oair rhel7', nodeSelector='', nodeUsageMode=EXCLUSIVE, workspaceVolume=EmptyDirWorkspaceVolume [memory=false], containers=[ContainerTemplate{name='jnlp', image='sbt-gitlab.delta.sbrf.ru:4567/oair/agent-docker:1.28_os', workingDir='/u01/jenkins', command='', args='-noreconnect ${computer.jnlpmac} ${computer.name}', resourceRequestCpu='1', resourceRequestMemory='4Gi', resourceLimitCpu='2', resourceLimitMemory='8Gi', livenessProbe=org.csanchez.jenkins.plugins.kubernetes.ContainerLivenessProbe@636583a}]}. Container jnlp. Logs: INFO: Using Remoting version: 3.29Oct 18, 2019 9:56:22 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDirINFO: Using /u01/jenkins/remoting as a remoting work directoryBoth error and output logs will be printed to /u01/jenkins/remotingOct 18, 2019 9:56:23 AM hudson.remoting.jnlp.Main$CuiListener statusINFO: Locating server among [https://sbt-jenkins.ca.sbrf.ru/jenkins/]Oct 18, 2019 9:57:24 AM hudson.remoting.jnlp.Main$CuiListener error
{code}
Around this time, the web interface stopped responding, connections hung in anticipation, until they fell timeout
Before restarting, I prepared a jstack, but I did not find the reason for the lock in it
|
|
|
|
| |
|
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202647.1571826147000.18073.1571826420505%40Atlassian.JIRA.
|
