On Fri, Nov 1, 2019 at 1:13 AM Allan Burdajewicz <[email protected]> wrote:
> About support core, based on what I see in the current bundle, the > instance is running a really recent version of the plugin which is quite > recent and should include most of the improvements made. > yes > > Another thing I can see though is that the support bundles contains all GC > logs since March 2019. That's 244 files, each of them being 20 MB (file > size limit) which means that we have *4.75 GB of GC logs in the support > bundle*. Maybe this could be accounting for the load when generating the > bundle (that happen every hour by default but could be made less frequent > or disabled: > https://github.com/jenkinsci/support-core-plugin/blob/support-core-2.61/src/main/java/com/cloudbees/jenkins/support/SupportPlugin.java#L130-L135 > ) > > * We should maybe setup log rotation on the server ? > > A further reason why the GC logs are included in the bundle is because > they are written right under $JENKINS_HOME/ which is a location that a > particular component is scanning: > https://github.com/jenkinsci/support-core-plugin/blob/support-core-2.61/src/main/java/com/cloudbees/jenkins/support/impl/JenkinsLogs.java#L101-L130 > . > > * So a workaround (*I would say a better practice actually*) is to change > the location of the GC logs to something like `$JENKINS_HOME/gc/` or it > even out them outside the $JENKINS_HOME. > > Maybe this could be improved to not include files older than a certain > number of days or have some limits ( > https://issues.jenkins-ci.org/browse/JENKINS-59030). > > It's what I wanted to fix yesterday and failed by not noticing that the instance is running is running on J11 :-/ I created https://issues.jenkins-ci.org/browse/INFRA-2316 and https://github.com/jenkins-infra/jenkins-infra/pull/1379 and > On Friday, November 1, 2019 at 6:45:09 AM UTC+10, Oleg Nenashev wrote: >> >> One thing to add here is that the Metrics plugin has been disabled by >> Daniel before the restart. >> >> Right now the instance is indeed stable, bit the load was pretty low. But >> now we have a regular Jenkins core massive rebuild, so it should put some >> pressure on CI again. >> If it goes down overnight, we can just restart it tomorrow. >> >> Best regards, >> Oleg >> >> >> On Thu, Oct 31, 2019 at 11:07 PM Baptiste Mathus <[email protected]> wrote: >> >>> I guess in the JIRA Epic as a comment would be good. Especially if you >>> still have seen failures in the last ~2 days. >>> >>> We've applied the change to get back already to G1 from ZGC + bumping to >>> a bigger Xmx value earlier today. But AFAIK the current situation in the >>> last two days since last restart was quite OKish. >>> Hence we're waiting somehow for, if so, the situation to degrade again >>> so we get a new support bundle to analyze, fix things/adjust config, then >>> rinse and repeat. >>> >>> Given things looked ok since restart, we suspect something like thread >>> or memory leak that takes some time to surface. >>> >>> BTW, we had a bit of an impromptu meeting earlier today on this. >>> We do plan to plan recurring public sync-ups (*probably* during weekly >>> infra meeting), so anyone willing to contribute can do it (contrary to >>> today, facepalm). >>> >>> Stay tuned, and feel fully free to ask any question or provide any >>> insight on what misbehaviors you're still seeing. >>> >>> Thanks >>> >>> Le jeu. 31 oct. 2019 à 20:17, Mez Pahlan <[email protected]> a écrit : >>> >>>> Many thanks Baptiste for picking this up. When you say provide a status >>>> check here do you mean to add your stack traces to this forum thread or to >>>> the INFRA ticket you linked to or something else? >>>> >>>> Cheers. >>>> >>>> On Thursday, 31 October 2019 10:28:42 UTC, Baptiste Mathus wrote: >>>>> >>>>> I filed https://issues.jenkins-ci.org/browse/INFRA-2308 after Arnaud >>>>> rightly recommended to have something public and central for us to fix >>>>> this. >>>>> >>>>> Pretty please anyone who's been suffering from ci.jenkins.io recent >>>>> instabilities, do not hesitate to provide a status check here. >>>>> We're especially interested in common stack traces or errors you're >>>>> seeing. >>>>> >>>>> We have started analyzing support bundles. First step is >>>>> https://github.com/jenkins-infra/jenkins-infra/pull/1375 >>>>> >>>>> More to come. >>>>> >>>>> Thanks! >>>>> >>>>> Le mer. 30 oct. 2019 à 14:56, Arnaud Héritier <[email protected]> a >>>>> écrit : >>>>> >>>>>> I still don't have an admin access on the instance but I have a >>>>>> system level access. >>>>>> I was able to grab few bundles and we will see if it helps to start >>>>>> to diagnose the issue >>>>>> >>>>>> On Wed, Oct 30, 2019 at 12:47 AM Daniel Beck <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> > On 27. Oct 2019, at 12:58, Daniel Beck <[email protected]> wrote: >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> >> On 27. Oct 2019, at 11:29, Arnaud Héritier <[email protected]> >>>>>>> wrote: >>>>>>> >> >>>>>>> >> Do we have the support-core plugin installed on this instance ? >>>>>>> If no, can >>>>>>> >> we add it ? >>>>>>> > >>>>>>> > We had it installed for years and nobody cared. After it showed up >>>>>>> recently in a thread dump as the likely cause of excessive load while >>>>>>> generating background bundles, I disabled it. >>>>>>> >>>>>>> And since support-core is disabled, the next restart will disable >>>>>>> metrics plugin too. >>>>>>> >>>>>>> We're affected by JENKINS-59793, with currently (AFAICT ~30 minutes >>>>>>> after restart) more than 700 of these threads. >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Jenkins Developers" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/jenkinsci-dev/38DAE091-615B-48EC-B9BD-904F8A044323%40beckweb.net >>>>>>> . >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> ----- >>>>>> Arnaud Héritier >>>>>> http://aheritier.net >>>>>> Mail/GTalk: aheritier AT gmail DOT com >>>>>> Twitter/Skype : aheritier >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Jenkins Developers" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/jenkinsci-dev/CAFNCU-8uGtbPdyAyVBoHfXzNPLQre7Rp%3DHc1v_BGWqFeuS%3D9mg%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/jenkinsci-dev/CAFNCU-8uGtbPdyAyVBoHfXzNPLQre7Rp%3DHc1v_BGWqFeuS%3D9mg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Jenkins Developers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/jenkinsci-dev/a93ee5ab-305e-49e7-8b38-77f8a09fbac7%40googlegroups.com >>>> <https://groups.google.com/d/msgid/jenkinsci-dev/a93ee5ab-305e-49e7-8b38-77f8a09fbac7%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Jenkins Developers" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/jenkinsci-dev/xqC8vW_emJI/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jenkinsci-dev/CANWgJS42DAD%2BpAK3Hgzv78VRKvv1GmWW1UL9OJsNaxNNYMnUZg%40mail.gmail.com >>> <https://groups.google.com/d/msgid/jenkinsci-dev/CANWgJS42DAD%2BpAK3Hgzv78VRKvv1GmWW1UL9OJsNaxNNYMnUZg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/9a61a4f9-c850-48c1-93de-4d6bccc0b8e5%40googlegroups.com > <https://groups.google.com/d/msgid/jenkinsci-dev/9a61a4f9-c850-48c1-93de-4d6bccc0b8e5%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- ----- Arnaud Héritier http://aheritier.net Mail/GTalk: aheritier AT gmail DOT com Twitter/Skype : aheritier -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAFNCU--X%2BOg-sVnVE67iixYbrXBP6PS1%2Bf7NGEOpk2iK8rg-9w%40mail.gmail.com.
