I guess in the JIRA Epic as a comment would be good. Especially if you still have seen failures in the last ~2 days.
We've applied the change to get back already to G1 from ZGC + bumping to a bigger Xmx value earlier today. But AFAIK the current situation in the last two days since last restart was quite OKish. Hence we're waiting somehow for, if so, the situation to degrade again so we get a new support bundle to analyze, fix things/adjust config, then rinse and repeat. Given things looked ok since restart, we suspect something like thread or memory leak that takes some time to surface. BTW, we had a bit of an impromptu meeting earlier today on this. We do plan to plan recurring public sync-ups (*probably* during weekly infra meeting), so anyone willing to contribute can do it (contrary to today, facepalm). Stay tuned, and feel fully free to ask any question or provide any insight on what misbehaviors you're still seeing. Thanks Le jeu. 31 oct. 2019 à 20:17, Mez Pahlan <[email protected]> a écrit : > Many thanks Baptiste for picking this up. When you say provide a status > check here do you mean to add your stack traces to this forum thread or to > the INFRA ticket you linked to or something else? > > Cheers. > > On Thursday, 31 October 2019 10:28:42 UTC, Baptiste Mathus wrote: >> >> I filed https://issues.jenkins-ci.org/browse/INFRA-2308 after Arnaud >> rightly recommended to have something public and central for us to fix this. >> >> Pretty please anyone who's been suffering from ci.jenkins.io recent >> instabilities, do not hesitate to provide a status check here. >> We're especially interested in common stack traces or errors you're >> seeing. >> >> We have started analyzing support bundles. First step is >> https://github.com/jenkins-infra/jenkins-infra/pull/1375 >> >> More to come. >> >> Thanks! >> >> Le mer. 30 oct. 2019 à 14:56, Arnaud Héritier <[email protected]> a >> écrit : >> >>> I still don't have an admin access on the instance but I have a system >>> level access. >>> I was able to grab few bundles and we will see if it helps to start to >>> diagnose the issue >>> >>> On Wed, Oct 30, 2019 at 12:47 AM Daniel Beck <[email protected]> wrote: >>> >>>> >>>> >>>> > On 27. Oct 2019, at 12:58, Daniel Beck <[email protected]> wrote: >>>> > >>>> > >>>> > >>>> >> On 27. Oct 2019, at 11:29, Arnaud Héritier <[email protected]> >>>> wrote: >>>> >> >>>> >> Do we have the support-core plugin installed on this instance ? If >>>> no, can >>>> >> we add it ? >>>> > >>>> > We had it installed for years and nobody cared. After it showed up >>>> recently in a thread dump as the likely cause of excessive load while >>>> generating background bundles, I disabled it. >>>> >>>> And since support-core is disabled, the next restart will disable >>>> metrics plugin too. >>>> >>>> We're affected by JENKINS-59793, with currently (AFAICT ~30 minutes >>>> after restart) more than 700 of these threads. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Jenkins Developers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/jenkinsci-dev/38DAE091-615B-48EC-B9BD-904F8A044323%40beckweb.net >>>> . >>>> >>> >>> >>> -- >>> ----- >>> Arnaud Héritier >>> http://aheritier.net >>> Mail/GTalk: aheritier AT gmail DOT com >>> Twitter/Skype : aheritier >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jenkinsci-dev/CAFNCU-8uGtbPdyAyVBoHfXzNPLQre7Rp%3DHc1v_BGWqFeuS%3D9mg%40mail.gmail.com >>> <https://groups.google.com/d/msgid/jenkinsci-dev/CAFNCU-8uGtbPdyAyVBoHfXzNPLQre7Rp%3DHc1v_BGWqFeuS%3D9mg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/a93ee5ab-305e-49e7-8b38-77f8a09fbac7%40googlegroups.com > <https://groups.google.com/d/msgid/jenkinsci-dev/a93ee5ab-305e-49e7-8b38-77f8a09fbac7%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANWgJS42DAD%2BpAK3Hgzv78VRKvv1GmWW1UL9OJsNaxNNYMnUZg%40mail.gmail.com.
