Danny, Yi, Thank you for taking care of this!
— Alexey > On 7 Nov 2023, at 17:10, Yi Hu via dev <dev@beam.apache.org> wrote: > > Hi Alexey, > > > all Jenkins jobs are stuck and there is a big Build Queue on > > https://ci-beam.apache.org/ > > This is not intentional. This is likely due to INFRA's routine Jenkins > upgrade on Nov 5 and caused this outage. Have created > https://issues.apache.org/jira/projects/INFRA/issues/INFRA-25147?filter=allopenissues > > > Regards, > Yi > > On Tue, Nov 7, 2023 at 10:35 AM Danny McCormick via dev <dev@beam.apache.org > <mailto:dev@beam.apache.org>> wrote: >> I don't think it's related. I noticed the problem half an hour ago; it seems >> there's an expired cert on the Jenkins machines. I'm hoping >> https://github.com/apache/beam/actions/runs/6786537134/job/18447281366 will >> fix this since the IO-Datastores cert is the problematic piece I think (and >> that has fixed a similar problem before). I'm not totally confident it will >> though given that the job succeeded last week. >> >> On Tue, Nov 7, 2023 at 10:18 AM Alexey Romanenko <aromanenko....@gmail.com >> <mailto:aromanenko....@gmail.com>> wrote: >>> Not sure if it’s related but I see that, seems, all Jenkins jobs are stuck >>> and there is a big Build Queue on https://ci-beam.apache.org/ >>> >>> Random clicks on jobs show that “"All nodes of label ‘beam’ are offline” >>> message. >>> >>> Is it known problem? >>> >>> — >>> Alexey >>> >>>> On 24 Oct 2023, at 21:50, Yi Hu via dev <dev@beam.apache.org >>>> <mailto:dev@beam.apache.org>> wrote: >>>> >>>> Hi all, >>>> >>>> We have shut down most tests in the Jenkins Load Tests and Performance >>>> Tests categories [1, 2], as they have been migrated to GitHub Actions for >>>> a while and are continuously publishing the metrics as expected. Please >>>> refer to https://github.com/apache/beam/pull/29092 for these tests. Note >>>> that pull requests mostly do not involve these tests so that it does not >>>> affect the development and release process. >>>> >>>> For the queueing issue mentioned before, after the self-hosted runners >>>> switched back from github webhook scaling to load based scaling, it is >>>> back to being stable. The issue was likely due to the webhook scaling on >>>> GitHub side. >>>> >>>> Regards, >>>> Yi >>>> >>>> [1] https://ci-beam.apache.org/view/LoadTests/ >>>> [2] https://ci-beam.apache.org/view/PerformanceTests/ >>>> >>>> >>>> >>>
