[JIRA] (JENKINS-61078) Groovy based parser thread stuck, causing high CPU load on master

[dan...@bachran.de (JIRA)]

Title: Message Title

Daniel Bachran updated an issue   Jenkins / JENKINS-61078 Groovy based parser thread stuck, causing high CPU load on master Change By: Daniel Bachran We have defined custom Groovy based warning parsers, for example to parse custom Matlab Simulink warnings output. When adding the following text snippet to the "Example Log Message" of our custom Matlab Warnings parser, we observe excessively high CPU load on the Jenkins master. This load comes from a thread which seems to be stuck and is displayed by JavaMelody as: * Handling POST /descriptorByName/io.jenkins.plugins.analysis.warnings.groovy.GroovyParser/checkExample from [ip address removed] : qtp2052001577-666216 This thread (and possible sub-threads) is constantly consuming CPU time, easily causing something between 50-90% CPU load on our Jenkins master. Rendering the system almost unable to process something else properly. *Workaround*: Killing the thread via JavaMelody reduces the CPU load to normal (around 5-10%). The thread reappears once the Jenkins configuration page is opened again, e.g. [https://jenkins/configure] This bug probably first occured after upgrading Jenkins LTS and the Warnings NG plugin sometime in October/November 2019. Unfortunately, I cannot tell anymore which versions were involved, as it took until now to realize the root cause of the sporadic high CPU load of our master... And yes, this Groovy script and/or the regex does not work properly. But still, Jenkins should not run into such issues in this case. *Regular _expression_* {code:java} ^\[.(Warning|Info|Caused by):(.*\n.*)*?\].\n\[.>(.*\n.*)*?\]{code} *Mapping Script*   {code:java} import edu.hm.hafner.analysis.Severity String type = matcher.group(1) String message = matcher.group(2) + "\n" + matcher.group(3) // Severity has: ERROR, WARNING_HIGH, WARNING_NORMAL, WARNING_LOW switch (type) { case 'Warning': severity = Severity.ERROR break case 'Info': severity = Severity.WARNING_HIGH break case 'Caused by': severity = Severity.WARNING_LOW break default: severity = Severity.WARNING_NORMAL // unknown, should not happen break } String filename = '' if (message =~ m/file ["'](.+)['"]/i) { filename = $1 } return builder.setSeverity(severity) .setType(type) .setFileName(filename) .setMessage(message) .buildOptional() {code}   *Example Log Message* {code:java} [ Warning: Minimum '0' and maximum '255' from 'Path/ModelElement' are mapped to a portion of the wide signal coming from output port 1 of 'Path/SubPath/Switch' and have been ignored during range checking. To enable the range checking for the minimum and maximum values, insert a Signal Conversion block with its 'Output' parameter set to 'Signal copy' in front of 'Path/ModelElement'.] [ > In slbus_get_struct In sl (line 15) In Simulink.Bus.createObject>createObjectFromBlks (line 169) In Simulink.Bus.createObject (line 149) In CqsModelCreateBusObjectsMissing/createBusObjects (line 301) In CqsModelCreateBusObjectsMissing/processWholeModel (line 86) In run_05_prepareImpModelStart (line 142)] {code} *Java Thread Dump* { { code:java} "Handling POST /descriptorByName/io.jenkins.plugins.analysis.warnings.groovy.GroovyParser/checkExample from [ip address removed] : qtp2052001577-803368" prio=5 RUNNABLE }} {{ java.util.regex.Pattern$GroupTail.match(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match0(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match(Unknown Source) }} {{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match0(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match(Unknown Source) }} {{ java.util.regex.Pattern$GroupHead.match(Unknown Source) }} {{ java.util.regex.Pattern$LazyLoop.match(Unknown Source) }} {{ java.util.regex.Pattern$GroupTail.match(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match0(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match(Unknown Source) }} {{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match0(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match(Unknown Source) }} {{ java.util.regex.Pattern$GroupHead.match(Unknown Source) }} {{ java.util.regex.Pattern$LazyLoop.match(Unknown Source) }} {{ java.util.regex.Pattern$GroupTail.match(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match0(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match(Unknown Source) }} {{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match0(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match(Unknown Source) }} {{ java.util.regex.Pattern$GroupHead.match(Unknown Source) }} {{ java.util.regex.Pattern$LazyLoop.match(Unknown Source) }} {{ java.util.regex.Pattern$GroupTail.match(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match0(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match(Unknown Source) }} {{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match0(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match(Unknown Source) }} {{ java.util.regex.Pattern$GroupHead.match(Unknown Source) }} {{ java.util.regex.Pattern$LazyLoop.match(Unknown Source) }} {{ java.util.regex.Pattern$GroupTail.match(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match0(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match(Unknown Source) }} {{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match0(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match(Unknown Source) }} {{ java.util.regex.Pattern$GroupHead.match(Unknown Source) }} {{ java.util.regex.Pattern$LazyLoop.match(Unknown Source) }} {{ java.util.regex.Pattern$GroupTail.match(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match0(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match(Unknown Source) }} {{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match0(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match(Unknown Source) }} {{ java.util.regex.Pattern$GroupHead.match(Unknown Source) }} {{ java.util.regex.Pattern$LazyLoop.match(Unknown Source) }} {{ java.util.regex.Pattern$GroupTail.match(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match0(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match(Unknown Source) }} {{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match0(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match(Unknown Source) }} {{ java.util.regex.Pattern$GroupHead.match(Unknown Source) }} {{ java.util.regex.Pattern$LazyLoop.match(Unknown Source) }} {{ java.util.regex.Pattern$GroupTail.match(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match0(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match(Unknown Source) }} {{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match0(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match(Unknown Source) }} {{ java.util.regex.Pattern$GroupHead.match(Unknown Source) }} {{ java.util.regex.Pattern$LazyLoop.match(Unknown Source) }} {{ java.util.regex.Pattern$GroupTail.match(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match0(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match(Unknown Source) }} {{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match0(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match(Unknown Source) }} {{ java.util.regex.Pattern$GroupHead.match(Unknown Source) }} {{ java.util.regex.Pattern$LazyLoop.match(Unknown Source) }} {{ java.util.regex.Pattern$GroupTail.match(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match0(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match(Unknown Source) }} {{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match0(Unknown Source) }} {{ java.util.regex.Pattern$Curly.match(Unknown Source) }} {{ java.util.regex.Pattern$GroupHead.match(Unknown Source) }} {{ java.util.regex.Pattern$LazyLoop.matchInit(Unknown Source) }} {{ java.util.regex.Pattern$Prolog.match(Unknown Source) }} {{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source) }} {{ java.util.regex.Pattern$GroupTail.match(Unknown Source) }} {{ java.util.regex.Pattern$BranchConn.match(Unknown Source) }} {{ java.util.regex.Pattern$Slice.match(Unknown Source) }} {{ java.util.regex.Pattern$Branch.match(Unknown Source) }} {{ java.util.regex.Pattern$GroupHead.match(Unknown Source) }} {{ java.util.regex.Pattern$CharProperty.match(Unknown Source) }} {{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source) }} {{ java.util.regex.Pattern$Caret.match(Unknown Source) }} {{ java.util.regex.Pattern$Start.match(Unknown Source) }} {{ java.util.regex.Matcher.search(Unknown Source) }} {{ java.util.regex.Matcher.find(Unknown Source) }} {{ io.jenkins.plugins.analysis.warnings.groovy.GroovyParser$DescriptorImpl.parseExample(GroovyParser.java:381) }} {{ io.jenkins.plugins.analysis.warnings.groovy.GroovyParser$DescriptorImpl.doCheckExample(GroovyParser.java:343) }} {{ java.lang.invoke.LambdaForm$DMH/1622225974.invokeVirtual_L4_L(LambdaForm$DMH) }} {{ java.lang.invoke.LambdaForm$BMH/822603816.reinvoke(LambdaForm$BMH) }} {{ java.lang.invoke.LambdaForm$MH/2096875522.invoker(LambdaForm$MH) }} {{ java.lang.invoke.LambdaForm$MH/2072658356.invokeExact_MT(LambdaForm$MH) }} {{ java.lang.invoke.MethodHandle.invokeWithArguments(Unknown Source) }} {{ org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396) }} {{ org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408) }} {{ org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77) }} {{ org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26) }} {{ org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212) }} {{ org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145) }} {{ org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535) }} {{ org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) }} {{ org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747) }} {{ org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) }} {{ org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:280) }} {{ org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) }} {{ org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747) }} {{ org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) }} {{ org.kohsuke.stapler.Stapler.invoke(Stapler.java:676) }} {{ org.kohsuke.stapler.Stapler.service(Stapler.java:238) }} {{ javax.servlet.http.HttpServlet.service(HttpServlet.java:790) }} {{ org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873) }} {{ org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) }} {{ hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) }} {{ jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76) }} {{ hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) }} {{ com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47) }} {{ hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) }} {{ net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239) }} {{ net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215) }} {{ net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88) }} {{ org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114) }} {{ hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) }} {{ hudson.plugins.locale.LocaleFilter.doFilter(LocaleFilter.java:42) }} {{ hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) }} {{ hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59) }} {{ hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) }} {{ jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128) }} {{ hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) }} {{ hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) }} {{ org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) }} {{ hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99) }} {{ org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) }} {{ hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) }} {{ hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) }} {{ hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) }} {{ jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118) }} {{ hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) }} {{ org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) }} {{ hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) }} {{ org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) }} {{ hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) }} {{ org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) }} {{ hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) }} {{ jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) }} {{ hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) }} {{ org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) }} {{ hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) }} {{ hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) }} {{ hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) }} {{ hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) }} {{ org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) }} {{ org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) }} {{ org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) }} {{ hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) }} {{ org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) }} {{ org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) }} {{ org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) }} {{ org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) }} {{ org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) }} {{ org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) }} {{ org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) }} {{ org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) }} {{ org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700) }} {{ org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) }} {{ org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) }} {{ org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) }} {{ org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) }} {{ org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667) }} {{ org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) }} {{ org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) }} {{ org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) }} {{ org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) }} {{ org.eclipse.jetty.server.Server.handle(Server.java:505) }} {{ org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) }} {{ org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) }} {{ org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) }} {{ org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) }} {{ org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) }} {{ org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) }} {{ org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) }} {{ org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) }} {{ org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) }} {{ org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) }} {{ org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698) }} {{ org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804) }} {{ java.lang.Thread.run(Unknown Source) {code } } Add Comment   This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.204569.1581599465000.8701.1581599580417%40Atlassian.JIRA.