[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics - Allow finishing builds when SubTask badly fail
Title: Message Title Oliver Gondža commented on JENKINS-59793 Re: Possible thread leak 'QueueSubTaskMetrics' in metrics - Allow finishing builds when SubTask badly fail This was backported to 2.204.1. RC is expected to be out on Dec 4th, final release on Dec 18th. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.9320.1575026640525%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics - Allow finishing builds when SubTask badly fail
Title: Message Title Raihaan Shouhell commented on JENKINS-59793 Re: Possible thread leak 'QueueSubTaskMetrics' in metrics - Allow finishing builds when SubTask badly fail We seem to see this issue with metrics. Will this be backported to the LTS release? Oliver Gondža We are currently on 2.176.2 Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.9299.1575023280912%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics - Allow finishing builds when SubTask badly fail
Title: Message Title Ramon Leon commented on JENKINS-59793 Re: Possible thread leak 'QueueSubTaskMetrics' in metrics - Allow finishing builds when SubTask badly fail It will take some time to land, I can give you some feedback by the end of December Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.6988.1574760300398%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics - Allow finishing builds when SubTask badly fail
Title: Message Title Oliver Gondža updated an issue Jenkins / JENKINS-59793 Possible thread leak 'QueueSubTaskMetrics' in metrics - Allow finishing builds when SubTask badly fail Change By: Oliver Gondža Labels: core lts 2.204.1 - candidate fixed core metrics Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.6926.1574759165303%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics - Allow finishing builds when SubTask badly fail
Title: Message Title Oliver Gondža commented on JENKINS-59793 Re: Possible thread leak 'QueueSubTaskMetrics' in metrics - Allow finishing builds when SubTask badly fail Ramon Leon, thanks for the prompt fix of the issue. I would appreciate if you can report back whether this fully mediated the problem in affected production deployment. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.6913.1574758380305%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics - Allow finishing builds when SubTask badly fail
Title: Message Title Daniel Beck updated JENKINS-59793 Jenkins / JENKINS-59793 Possible thread leak 'QueueSubTaskMetrics' in metrics - Allow finishing builds when SubTask badly fail Change By: Daniel Beck Status: Fixed but Unreleased Resolved Released As: jenkins-2.205 Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.5474.1574421361875%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics - Allow finishing builds when SubTask badly fail
Title: Message Title Ramon Leon updated JENKINS-59793 Jenkins / JENKINS-59793 Possible thread leak 'QueueSubTaskMetrics' in metrics - Allow finishing builds when SubTask badly fail Change By: Ramon Leon Status: In Review Fixed but Unreleased Resolution: Fixed Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.13217.1573635721112%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics - Allow finishing builds when SubTask badly fail
Title: Message Title Ramon Leon commented on JENKINS-59793 Re: Possible thread leak 'QueueSubTaskMetrics' in metrics - Allow finishing builds when SubTask badly fail Added what has been done to the title for the sake of searches Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.11825.1573461540334%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics - Allow finishing builds when SubTask badly fail
Title: Message Title Ramon Leon updated an issue Jenkins / JENKINS-59793 Possible thread leak 'QueueSubTaskMetrics' in metrics - Allow finishing builds when SubTask badly fail Change By: Ramon Leon Summary: Possible thread leak 'QueueSubTaskMetrics' in metrics - Allow finishing builds when SubTask badly fail Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.11817.1573461481221%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title Ramon Leon commented on JENKINS-59793 Re: Possible thread leak 'QueueSubTaskMetrics' in metrics I mentioned this because in the instance we detected the metric threads, there were a lot of log messages of this type. But maybe it's not related and it's just a performance issue. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.11205.1573220160328%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title Jesse Glick commented on JENKINS-59793 Re: Possible thread leak 'QueueSubTaskMetrics' in metrics I do not believe the workflow-cps code mentioned has any relation to this issue. Perhaps I am missing something. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.11186.1573219080335%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title Baptiste Mathus updated an issue Jenkins / JENKINS-59793 Possible thread leak 'QueueSubTaskMetrics' in metrics Change By: Baptiste Mathus Labels: core lts-candidate plugin metrics Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.11090.1573204620375%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title Baptiste Mathus updated an issue Jenkins / JENKINS-59793 Possible thread leak 'QueueSubTaskMetrics' in metrics Change By: Baptiste Mathus Component/s: core Component/s: metrics-plugin Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.11078.1573204560590%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title Baptiste Mathus updated an issue Jenkins / JENKINS-59793 Possible thread leak 'QueueSubTaskMetrics' in metrics Change By: Baptiste Mathus Labels: core lts-candidate plugin Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.11076.1573204560566%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title Baptiste Mathus updated an issue Jenkins / JENKINS-59793 Possible thread leak 'QueueSubTaskMetrics' in metrics Change By: Baptiste Mathus Labels: lts-candidate plugin Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.11069.1573204500490%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title
Ramon Leon edited a comment on JENKINS-59793
Re: Possible thread leak 'QueueSubTaskMetrics' in metrics
Besides the code in *Metrics*: [https://github.com/jenkinsci/metrics-plugin/blob/24bf92ebd59095d8f77b2552696b3f210a024bdc/src/main/java/jenkins/metrics/impl/JenkinsMetricProviderImpl.java#L1142] {code:java}try {return future.get();} catch (Throwable t) {sneakyThrow(t);throw new RuntimeException(t);}{code} We have here another affected code in *Pipeline*:[https://github.com/jenkinsci/workflow-cps-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L940]{code:java}try {return getCurrentExecutions(false).get(1, TimeUnit.SECONDS).stream().anyMatch(StepExecution::blocksRestart);} catch (Exception x) {// TODO RestartListener.Default.isReadyToRestart can throw checked exceptions, but AsynchronousExecution.blocksRestart does not currently allow itLevel level = x.getCause() instanceof RejectedExecutionException ? /* stray Executor past program end? */ Level.FINE : Level.WARNING;LOGGER.log(level, "Not blocking restart due to problem checking running steps in " + this, x);return false;}{code}We see in the log: {code:java}2019-10-14 13:16:56.104+ [id=16386139] WARNING o.j.p.w.cps.CpsFlowExecution#blocksRestart: Not blocking restart due to problem checking running steps in CpsFlowExecution[**]]java.util.concurrent.TimeoutException: Timeout waiting for task.at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:259)at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:91)at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.blocksRestart(CpsFlowExecution.java:940)at org.jenkinsci.plugins.workflow.job.WorkflowRun$2.blocksRestart(WorkflowRun.java:372)at org.jenkinsci.plugins.workflow.job.WorkflowRun$2.displayCell(WorkflowRun.java:375)at hudson.model.Executor.isDisplayCell(Executor.java:668)...{code} Fortunately, this code just waits 1 second, so despite populating the code, it doesn't block anything , although it populates the log with these messages . * [|https://cloudbees.atlassian.net/secure/AddComment!default.jspa?id=229191]
Add Comment
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title
Ramon Leon edited a comment on JENKINS-59793
Re: Possible thread leak 'QueueSubTaskMetrics' in metrics
Besides the code in * Metrics * : [ https://github.com/jenkinsci/metrics-plugin/blob/24bf92ebd59095d8f77b2552696b3f210a024bdc/src/main/java/jenkins/metrics/impl/JenkinsMetricProviderImpl.java#L1142 ] {code:java}try {return future.get();} catch (Throwable t) {sneakyThrow(t);throw new RuntimeException(t);}{code} We have here another affected code in *Pipeline* : [ https://github.com/jenkinsci/workflow-cps-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L940 ] {code:java}try {return getCurrentExecutions(false).get(1, TimeUnit.SECONDS).stream().anyMatch(StepExecution::blocksRestart);} catch (Exception x) {// TODO RestartListener.Default.isReadyToRestart can throw checked exceptions, but AsynchronousExecution.blocksRestart does not currently allow itLevel level = x.getCause() instanceof RejectedExecutionException ? /* stray Executor past program end? */ Level.FINE : Level.WARNING;LOGGER.log(level, "Not blocking restart due to problem checking running steps in " + this, x);return false;}{code}We see in the log: {code:java}2019-10-14 13:16:56.104+ [id=16386139] WARNING o.j.p.w.cps.CpsFlowExecution#blocksRestart: Not blocking restart due to problem checking running steps in CpsFlowExecution[**]]java.util.concurrent.TimeoutException: Timeout waiting for task.at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:259)at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:91)at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.blocksRestart(CpsFlowExecution.java:940)at org.jenkinsci.plugins.workflow.job.WorkflowRun$2.blocksRestart(WorkflowRun.java:372)at org.jenkinsci.plugins.workflow.job.WorkflowRun$2.displayCell(WorkflowRun.java:375)at hudson.model.Executor.isDisplayCell(Executor.java:668)...{code} Fortunately, this code just waits 1 second, so despite populating the code, it doesn't block anything. * [|https://cloudbees.atlassian.net/secure/AddComment!default.jspa?id=229191]
Add Comment
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title
Ramon Leon commented on JENKINS-59793
Re: Possible thread leak 'QueueSubTaskMetrics' in metrics
Besides the code in Metrics: https://github.com/jenkinsci/metrics-plugin/blob/24bf92ebd59095d8f77b2552696b3f210a024bdc/src/main/java/jenkins/metrics/impl/JenkinsMetricProviderImpl.java#L1142
try {
return future.get();
} catch (Throwable t) {
sneakyThrow(t);
throw new RuntimeException(t);
}
We have here another affected code in: https://github.com/jenkinsci/workflow-cps-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L940
try {
return getCurrentExecutions(false).get(1, TimeUnit.SECONDS).stream().anyMatch(StepExecution::blocksRestart);
} catch (Exception x) {
// TODO RestartListener.Default.isReadyToRestart can throw checked exceptions, but AsynchronousExecution.blocksRestart does not currently allow it
Level level = x.getCause() instanceof RejectedExecutionException ? /* stray Executor past program end? */ Level.FINE : Level.WARNING;
LOGGER.log(level, "Not blocking restart due to problem checking running steps in " + this, x);
return false;
}
We see in the log:
2019-10-14 13:16:56.104+ [id=16386139] WARNING o.j.p.w.cps.CpsFlowExecution#blocksRestart: Not blocking restart due to problem checking running steps in CpsFlowExecution[**]]java.util.concurrent.TimeoutException: Timeout waiting for task.
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:259)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:91)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.blocksRestart(CpsFlowExecution.java:940)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$2.blocksRestart(WorkflowRun.java:372)
at org.jenkinsci.plugins.workflow.job.WorkflowRun$2.displayCell(WorkflowRun.java:375)
at hudson.model.Executor.isDisplayCell(Executor.java:668)
...
Add Comment
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title Ramon Leon updated JENKINS-59793 Jenkins / JENKINS-59793 Possible thread leak 'QueueSubTaskMetrics' in metrics Change By: Ramon Leon Status: In Progress Review Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.10714.1573146301201%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title Ramon Leon commented on JENKINS-59793 Re: Possible thread leak 'QueueSubTaskMetrics' in metrics PR: https://github.com/jenkinsci/jenkins/pull/4346 Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.10713.1573146301127%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title Ramon Leon commented on JENKINS-59793 Re: Possible thread leak 'QueueSubTaskMetrics' in metrics I've pushed the test failing, after the CI fails I will push the fix Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.10721.1573146302297%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title
Ramon Leon edited a comment on JENKINS-59793
Re: Possible thread leak 'QueueSubTaskMetrics' in metrics
If the exception is thrown in a Builder the wrong behavior is not showed up: {code:java}FreeStyleProject p = j.createProject(FreeStyleProject.class);p.getBuildersList().add(new TestBuilder() { @Override public boolean perform(AbstractBuild build, Launcher launcher, BuildListener listener) throws InterruptedException { listener.getLogger().println("running failing builder"); throw new IndexOutOfBoundsException("Unexpected exception while building"); }});{code} It happens because when it fails and the code enters the _synchronizeEnd_ method, and then the _endLatch.synchronize()_, it exits without throwing an exception, which is different from when you get here from a _SubTask#run_
Add Comment
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.10103.1573050060408%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title
Ramon Leon commented on JENKINS-59793
Re: Possible thread leak 'QueueSubTaskMetrics' in metrics
If the exception is thrown in a Builder the wrong behavior is not showed up:
FreeStyleProject p = j.createProject(FreeStyleProject.class);
p.getBuildersList().add(new TestBuilder() {
@Override
public boolean perform(AbstractBuild build, Launcher launcher, BuildListener listener) throws InterruptedException {
listener.getLogger().println("running failing builder");
throw new IndexOutOfBoundsException("Unexpected exception while building");
}
});
Add Comment
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.10060.1573048980697%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title Ramon Leon commented on JENKINS-59793 Re: Possible thread leak 'QueueSubTaskMetrics' in metrics I've managed to reproduce the problem in a test. I schedule a freestyle project with a long SleepBuilder and you add a SubTaskContributor throwing an exception in the run method of the Executable created. Then the WorkUnitContext#synchronizeEnd method exists with an InterruptedException and the future.set is not called. In addition, the next abort call also exists in if (aborted!=null) return; so the endLatch is not aborted properly. With this scenario, the wuc.future keeps waiting forever in the JenkinsMetricProviderImpl#asSupplier method. From my point of view the synchronizeEnd should guarantee the future is set even when throwing the InterruptedException, or the abort method shouldn't exit without aborting the latches. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.9522.1572977520603%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title Ramon Leon commented on JENKINS-59793 Re: Possible thread leak 'QueueSubTaskMetrics' in metrics I’m trying to cancel jobs while in the Queue (cancel) or once running (getExecutor().interrupt()), but in this way, the job is correctly finished so the metrics thread is freed. I need to figure out a way to kill a job in a way that the future is not set in the WorkUnitContext: https://github.com/jenkinsci/jenkins/blob/264980c707c6cafc01369378b1e90d247561df78/core/src/main/java/hudson/model/queue/WorkUnitContext.java#L132 Most likely playing with several executors and subtasks and interrupting some of them. Still working, any clue on how to achieve that is welcome Daniel Beck Jesse Glick Oleg Nenashev Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.8531.1572889080294%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title Ramon Leon commented on JENKINS-59793 Re: Possible thread leak 'QueueSubTaskMetrics' in metrics I'm working right now on trying to reproduce these leaks playing with the cancellation of the build. Whatever clue or suggestion is welcome. :sweat: Thanks! Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.8432.1572875460627%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title CloudBees Foundation Security updated an issue Jenkins / JENKINS-59793 Possible thread leak 'QueueSubTaskMetrics' in metrics Change By: CloudBees Foundation Security Comment: I'm working right now on trying to reproduce these leaks playing with the cancellation of the build. Whatever clue or suggestion is welcome. :sweat: Thanks! Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.8421.1572875400281%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title CloudBees Foundation Security commented on JENKINS-59793 Re: Possible thread leak 'QueueSubTaskMetrics' in metrics I'm working right now on trying to reproduce these leaks playing with the cancellation of the build. Whatever clue or suggestion is welcome. :sweat: Thanks! Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.8415.1572875340262%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title Oliver Gondža commented on JENKINS-59793 Re: Possible thread leak 'QueueSubTaskMetrics' in metrics We are being impacted by this as well. Ramon Leon I believe the mistake is in the metrics plugin, that seems to presumes that when QueueListener#onLeft is called, its task will start eventually. On a first glance it appears this is not true when the task is canceled while in queue. There might be other reasons I am not aware of, too. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.8408.1572874920189%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title Ramon Leon started work on JENKINS-59793 Change By: Ramon Leon Status: Open In Progress Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.5938.1572506880518%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title Ramon Leon assigned an issue to Ramon Leon Jenkins / JENKINS-59793 Possible thread leak 'QueueSubTaskMetrics' in metrics Change By: Ramon Leon Assignee: Ramon Leon Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.5934.1572506460233%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title Ramon Leon commented on JENKINS-59793 Re: Possible thread leak 'QueueSubTaskMetrics' in metrics Another approach, I think the best candidate, but it requires fixing Jenkins core: I’m pondering whether we could solve the problem of so many QueueSubTaskMetrics threads waiting by ensuring the synchronizeEnd and synchronizeStart always set the future? For example by surrounding the code of these two methods with try-catch and setting the future in the finally clause before rethrowing the exception. I think this because if an exception is thrown in these methods, the result is not set, so the thread is always waiting because the complete flag is not set. WorkUnitContext.java#L132 But this is a fix on Jenkins core, not in Metrics plugin. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.5090.1572425940294%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title Raihaan Shouhell commented on JENKINS-59793 Re: Possible thread leak 'QueueSubTaskMetrics' in metrics Confirming that my instances get the same issue on Jenkins 2.176.2 with Metrics 4.0.2.4 Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.5076.1572423780246%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title Baptiste Mathus updated an issue Jenkins / JENKINS-59793 Possible thread leak 'QueueSubTaskMetrics' in metrics Change By: Baptiste Mathus Priority: Major Critical Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.5054.1572418200840%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title Daniel Beck commented on JENKINS-59793 Re: Possible thread leak 'QueueSubTaskMetrics' in metrics Interesting, that might have been a source of too many threads on ci.jenkins.io recently as well. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202523.1571157685000.8299.1571163360297%40Atlassian.JIRA.
[JIRA] (JENKINS-59793) Possible thread leak 'QueueSubTaskMetrics' in metrics
Title: Message Title Ramon Leon created an issue Jenkins / JENKINS-59793 Possible thread leak 'QueueSubTaskMetrics' in metrics Issue Type: Bug Assignee: Unassigned Components: metrics-plugin Created: 2019-10-15 16:41 Environment: Metrics 4.0.2.5 Labels: plugin Priority: Major Reporter: Ramon Leon In one large instance we can see 3000 threads with this shape: "QueueSubTaskMetrics [#11342]" #5508106 daemon prio=5 os_prio=0 tid=0x7efcf085a800 nid=0x52c7 in Object.wait() [0x7efbccb32000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:75) - locked <0x000512c01d10> (a hudson.model.queue.FutureImpl) at jenkins.metrics.impl.JenkinsMetricProviderImpl.lambda$asSupplier$3(JenkinsMetricProviderImpl.java:1142) at jenkins.metrics.impl.JenkinsMetricProviderImpl$$Lambda$388/1851215464.get(Unknown Source) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Locked ownable synchronizers: - <0x0004dc401fb0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
