Yep, thanks Ed for a detailed comment and my apologies for the spike... When I catch up with the cherrypicks then it will be hopefully better. Still tweaking the process...
--a > On 26 Sep 2019, at 15:57, Ed Kern via Lists.Fd.Io <[email protected]> > wrote: > > This has nothing to do with the LF (outside of jobs getting stuck (which I > haven’t seen since the upgrade yesterday)). > > The queue has been ballooning and shrinking the last couple of days for a few > reasons: > a. a very large number of 1908 verify and then merge jobs. > b. increase recently in the number and in some cases long duration of csit > jobs > c. merge jobs that have to wait for the previous so they just (correctly) sit > in the build queue. > d. multi hour shutdown yesterday which always causes a bubble… > > There is no current way to prioritize master runs over 1908 cherrypicks so > for that wait I apologize. Current longest > wait time I’ve seen on a verify based job was around 80 minutes to start. > > Andrew has said that he had a backlog of cherrypicks and he expects the > number at any one time to reduce. > > We have more than doubled the amount of jobs running at any one time over the > last three weeks. The system is fine > but certainly the amount of ’slack’ is considerably less then there used to > be… > > Currently thought is still to repurpose ‘retiring’ virl hardware thats in > place into generic x86 infra build to get us a lot more > ’slack’. > > Ed > > > >> On Sep 26, 2019, at 6:56 AM, Jan Gelety via Lists.Fd.Io >> <[email protected]> wrote: >> >> Hello, >> >> Could you, please, check FD.io Jenkins? There’s a quite a huge queue of >> wating jobs (87 at the moment) there. >> >> Thanks, >> Jan >> -=-=-=-=-=-=-=-=-=-=-=- >> Links: You receive all messages sent to this group. >> >> View/Reply Online (#14059): https://lists.fd.io/g/vpp-dev/message/14059 >> Mute This Topic: https://lists.fd.io/mt/34298430/675649 >> Group Owner: [email protected] >> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]] >> -=-=-=-=-=-=-=-=-=-=-=- > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#14064): https://lists.fd.io/g/vpp-dev/message/14064 > Mute This Topic: https://lists.fd.io/mt/34298430/675608 > Group Owner: [email protected] > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]] > -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14065): https://lists.fd.io/g/vpp-dev/message/14065 Mute This Topic: https://lists.fd.io/mt/34298430/21656 Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
