update: mnaser (vex host ) did a ‘live migration’ of the jenkins.fd.io<http://jenkins.fd.io> ’server’ taking RTT from >6 seconds back down to the @100ms range. As a result jobs are starting up again and the queue is grinding through the backlog and new jobs.
Im not currently seeing the same delay on/towards other hosted services gerrit/nexus etc. Ill be watching it closely the next few hours but please let me know if you see any unusual errors. thanks, Ed On Feb 27, 2019, at 8:34 AM, Ed Kern via Lists.Fd.Io <[email protected]<mailto:[email protected]>> wrote: fyi to vpp-dev as well.. debugging with vexxhost Begin forwarded message: From: "Peter Mikus via Lists.Fd.Io" <[email protected]<mailto:[email protected]>> Subject: [csit-dev] Jenkins infra outage - 100% jobs queued, VPN access very slow Date: February 27, 2019 at 8:32:50 AM MST To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> Reply-To: [email protected]<mailto:[email protected]> Hello team, We are observing jobs queueing in Jenkins.fd.io<http://jenkins.fd.io/> and also the VPN access to infra is very slow. Can you please investigate? ping 10.30.51.50 PING 10.30.51.50 (10.30.51.50) 56(84) bytes of data. 64 bytes from 10.30.51.50: icmp_seq=1 ttl=62 time=9961 ms 64 bytes from 10.30.51.50: icmp_seq=2 ttl=62 time=8928 ms 64 bytes from 10.30.51.50: icmp_seq=3 ttl=62 time=7907 ms 64 bytes from 10.30.51.50: icmp_seq=4 ttl=62 time=6887 ms 64 bytes from 10.30.51.50: icmp_seq=5 ttl=62 time=5863 ms 64 bytes from 10.30.51.50: icmp_seq=6 ttl=62 time=4834 ms 64 bytes from 10.30.51.50: icmp_seq=7 ttl=62 time=3809 ms 64 bytes from 10.30.51.50: icmp_seq=8 ttl=62 time=2790 ms 64 bytes from 10.30.51.50: icmp_seq=9 ttl=62 time=1762 ms Thank you. Peter Mikus Engineer – Software Cisco Systems Limited -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#3348): https://lists.fd.io/g/csit-dev/message/3348 Mute This Topic: https://lists.fd.io/mt/30153144/675649 Group Owner: [email protected]<mailto:[email protected]> Unsubscribe: https://lists.fd.io/g/csit-dev/unsub [[email protected]<mailto:[email protected]>] -=-=-=-=-=-=-=-=-=-=-=- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12363): https://lists.fd.io/g/vpp-dev/message/12363 Mute This Topic: https://lists.fd.io/mt/30153159/675649 Group Owner: [email protected]<mailto:[email protected]> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]<mailto:[email protected]>] -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12364): https://lists.fd.io/g/vpp-dev/message/12364 Mute This Topic: https://lists.fd.io/mt/30153159/21656 Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
