Hi Maciek, A quick acknowledgment, we have seen this ticket. The vexhost CSIT team has been notified to prioritize and troubleshoot.
Have asked Mohammed for an update for 11am TSC call too. Regards, Trishan On Thu, May 2, 2019, 5:13 AM Maciek Konstantynowicz via RT < [email protected]> wrote: > > <URL: https://rt.linuxfoundation.org/Ticket/Display.html?id=74794 > > > Hello, > > CRITICAL ISSUE: ALL CSIT PERFORMANCE TESTING JOBS ARE EFFECTED. 1904 RLS > SHOWSTOPPER. > > This is now a critical issue as all CSIT performance testing jobs > are failing, preventing FD.io<http://FD.io> CSIT project to complete the > CSIT-1904 release. > > This ticket #74794, is related to the same underlying jnlp connectivity > issue > reported in other recent tickets #74795, #73486, #73487, but apparently > issue > was observed earlier #70022, and much earlier too.. > > It was attributed to haproxy issues (Layer4 timeouts), but the issue is > still > happening. Below is the sorted list of timestamps from Peter Mikus script > as per earlier note in this case. One can see that multiple jobs get > affected > at the same time. > > Please advise on resolution, and if any assistance is needed from FD.io< > http://FD.io> CSIT > project. > > Thank you for your attention! > > Cheers, > -Maciek > > 2019.04.27-10:54:40 java.nio.channels.ClosedChannelException > 2019.04.27-10:55:06 java.nio.channels.ClosedChannelException > 2019.04.27-10:55:10 java.nio.channels.ClosedChannelException > 2019.04.28-09:32:20 java.nio.channels.ClosedChannelException > 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException > 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException > 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException > 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException > 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException > 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException > 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException > 2019.05.01-13:48:18 java.nio.channels.ClosedChannelException > 2019.05.01-13:48:18 java.nio.channels.ClosedChannelException > 2019.05.01-13:48:18 java.nio.channels.ClosedChannelException > 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException > 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException > 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException > 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException > 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException > 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException > 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException > 2019.05.01-13:48:27 java.nio.channels.ClosedChannelException > 2019.05.01-17:59:20 java.nio.channels.ClosedChannelException > > On 2 May 2019, at 07:47, Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at > Cisco) <[email protected]<mailto:[email protected]>> wrote: > > Hello, > > This issue is starting to be very serious and impacting deliverables for > CSIT project. > > Can you please investigate with priority. > > Subset of failed jobs (just few examples from past days): > https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/38/ > https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/37/ > https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/36/ > https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/35/ > https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/34/ > https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/20/ > https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/19/ > https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/18/ > https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/17/ > https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/15/ > https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/13/ > https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/16/ > https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/17/ > https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/18/ > https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/19/ > https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/10/ > https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/13/ > https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/14/ > https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/15/ > https://jenkins.fd.io/job/csit-dpdk-perf-verify-1904-3n-skx/8/ > https://jenkins.fd.io/job/csit-dpdk-perf-verify-1904-3n-skx/10/ > https://jenkins.fd.io/job/csit-vpp-perf-mrr-daily-master/666/ > https://jenkins.fd.io/job/csit-vpp-perf-mrr-daily-master/665/ > > Thank you. > > Note: From logs, it can be quite easily extracted dates and times of > outages and correlation in time. > > > jobs=('https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/38/' > 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/37/' > 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/36/' > 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/35/' > 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/34/' > 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/20/' > 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/19/' > 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/18/' > 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/17/' > 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/15/' > 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/13/' > 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/16/' > 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/17/' > 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/18/' > 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/19/' > 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/10/' > 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/13/' > 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/14/' > 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/15/' > 'https://jenkins.fd.io/job/csit-dpdk-perf-verify-1904-3n-skx/8/' > 'https://jenkins.fd.io/job/csit-dpdk-perf-verify-1904-3n-skx/10/' > 'https://jenkins.fd.io/job/csit-vpp-perf-mrr-daily-master/666/' > 'https://jenkins.fd.io/job/csit-vpp-perf-mrr-daily-master/665/' > ' > https://jenkins.fd.io/job/csit-vpp-perf-mrr-daily-master-3n-skx/416/') > for job in "${jobs[@]}"; do > curl -s > "${job}/timestamps/?time=yyyy.MM.dd-HH:mm:ss&appendLog&locale=en_US" | grep > -m 1 "java.nio.channels.ClosedChannelException" > done > > which results in: > > 2019.05.01-13:48:27 java.nio.channels.ClosedChannelException > 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException > 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException > 2019.05.01-13:48:18 java.nio.channels.ClosedChannelException > 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException > 2019.05.01-13:48:18 java.nio.channels.ClosedChannelException > 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException > 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException > 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException > 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException > 2019.04.27-10:55:06 java.nio.channels.ClosedChannelException > 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException > 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException > 2019.05.01-17:59:20 java.nio.channels.ClosedChannelException > 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException > 2019.04.27-10:54:40 java.nio.channels.ClosedChannelException > 2019.04.28-09:32:20 java.nio.channels.ClosedChannelException > 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException > 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException > 2019.05.01-13:48:18 java.nio.channels.ClosedChannelException > 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException > 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException > 2019.04.27-10:55:10 java.nio.channels.ClosedChannelException > > Peter Mikus > Engineer – Software > Cisco Systems Limited > > From: [email protected]<mailto:[email protected]> <[email protected] > <mailto:[email protected]>> On Behalf Of Jan Gelety via Lists.Fd.Io > Sent: Monday, April 29, 2019 8:58 AM > To: [email protected]<mailto:[email protected]>; > [email protected]<mailto: > [email protected]> > Cc: [email protected]<mailto:[email protected]> > Subject: [vpp-dev] Jenkins.fd.io network issues > > Hello, > > We are experiencing quite a lot of network issues when running CSIT tests > for 19.04 report: > > Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote > call on JNLP4-connect connection from > vex-yul-rot-ingress-1.ci.codeaurora.org/10.30.48.3:41068< > http://vex-yul-rot-ingress-1.ci.codeaurora.org/10.30.48.3:41068> failed. > The channel is closing down or has closed down > > https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/13/console > > Could you, please, have a look on it? > > Thank you very much. > > Regards, > Jan > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12903): https://lists.fd.io/g/vpp-dev/message/12903 Mute This Topic: https://lists.fd.io/mt/31454815/21656 Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
