[JIRA] (JENKINS-49931) Heap Histogram Collection Destabilizes Masters
Title: Message Title Jesse Glick updated JENKINS-49931 Jenkins / JENKINS-49931 Heap Histogram Collection Destabilizes Masters Change By: Jesse Glick Status: In Review Resolved Resolution: Fixed Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49931) Heap Histogram Collection Destabilizes Masters
Title: Message Title Emilio Escobar commented on JENKINS-49931 Re: Heap Histogram Collection Destabilizes Masters new https://github.com/jenkinsci/support-core-plugin/pull/135 Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49931) Heap Histogram Collection Destabilizes Masters
Title: Message Title SCM/JIRA link daemon commented on JENKINS-49931 Re: Heap Histogram Collection Destabilizes Masters Code changed in jenkins User: Emilio Escobar Path: src/main/java/com/cloudbees/jenkins/support/impl/HeapUsageHistogram.java http://jenkins-ci.org/commit/support-core-plugin/a3ff0c7e985b0ecb9a4460bab94932bc55a17b6a Log: [FIXED JENKINS-49931] DISABLED by default Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49931) Heap Histogram Collection Destabilizes Masters
Title: Message Title Emilio Escobar updated JENKINS-49931 Jenkins / JENKINS-49931 Heap Histogram Collection Destabilizes Masters Change By: Emilio Escobar Status: In Progress Review Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49931) Heap Histogram Collection Destabilizes Masters
Title: Message Title Emilio Escobar commented on JENKINS-49931 Re: Heap Histogram Collection Destabilizes Masters https://github.com/jenkinsci/support-core-plugin/pull/134 already available for review! Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49931) Heap Histogram Collection Destabilizes Masters
Title: Message Title Emilio Escobar started work on JENKINS-49931 Change By: Emilio Escobar Status: Open In Progress Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49931) Heap Histogram Collection Destabilizes Masters
Title: Message Title Sam Van Oort commented on JENKINS-49931 Re: Heap Histogram Collection Destabilizes Masters Assigned to Emilio Escobar because I think he has a fix already almost done for it. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49931) Heap Histogram Collection Destabilizes Masters
Title: Message Title Sam Van Oort created an issue Jenkins / JENKINS-49931 Heap Histogram Collection Destabilizes Masters Issue Type: Bug Assignee: Emilio Escobar Components: support-core-plugin Created: 2018-03-05 19:08 Priority: Critical Reporter: Sam Van Oort When we added Heap Stats collection (https://issues.jenkins-ci.org/browse/JENKINS-22791) in v2.42 it appears that we inadvertently caused a major performance-stability regression if the histogram is collected regularly. How? Well, this gathers a live heap histogram. This appears to triggers a Full GC. This is visible in GC logs because they show the following cause: > [Full GC (Heap Inspection Initiated GC). Now, because this is a FullGC and not a concurrent or young-gen GC, and we're generally using G1 GC, the slow Serial garbage collector is used for FullGC. This is a NON-concurrent GC mode, meaning the application is fully paused until it completes, and it is SINGLE-threaded, meaning rather than 1 GB/s per CPU of GC throughput, we get <1 GB/s total. It also cleans and compacts the entire heap rather than just part of it as with other modes. So, with 15 GB of used heap that means a pause of up ~15s. This matches behavior observed in the wild. I am rating this as critical because on larger-scale production masters a hang that long can cause job failures, visible UI hangs, HTTP request timeouts, and other issues – it should result in Surable Task failures for Pipelines, for example. Proposed solution: only gather the live heap histogram when a user is explicitly requesting a support bundle (disable it by default).
