Hello, This issue is starting to be very serious and impacting deliverables for CSIT project.
Can you please investigate with priority. Subset of failed jobs (just few examples from past days): https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/38/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/37/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/36/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/35/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/34/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/20/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/19/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/18/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/17/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/15/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/13/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/16/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/17/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/18/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/19/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/10/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/13/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/14/ https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/15/ https://jenkins.fd.io/job/csit-dpdk-perf-verify-1904-3n-skx/8/ https://jenkins.fd.io/job/csit-dpdk-perf-verify-1904-3n-skx/10/ https://jenkins.fd.io/job/csit-vpp-perf-mrr-daily-master/666/ https://jenkins.fd.io/job/csit-vpp-perf-mrr-daily-master/665/ Thank you. Note: From logs, it can be quite easily extracted dates and times of outages and correlation in time. jobs=('https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/38/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/37/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/36/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/35/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-2n-skx/34/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/20/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/19/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/18/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/17/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/15/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-skx/13/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/16/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/17/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/18/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/19/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/10/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/13/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/14/' 'https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/15/' 'https://jenkins.fd.io/job/csit-dpdk-perf-verify-1904-3n-skx/8/' 'https://jenkins.fd.io/job/csit-dpdk-perf-verify-1904-3n-skx/10/' 'https://jenkins.fd.io/job/csit-vpp-perf-mrr-daily-master/666/' 'https://jenkins.fd.io/job/csit-vpp-perf-mrr-daily-master/665/' ' https://jenkins.fd.io/job/csit-vpp-perf-mrr-daily-master-3n-skx/416/') for job in "${jobs[@]}"; do curl -s "${job}/timestamps/?time=yyyy.MM.dd-HH:mm:ss&appendLog&locale=en_US" | grep -m 1 "java.nio.channels.ClosedChannelException" done which results in: 2019.05.01-13:48:27 java.nio.channels.ClosedChannelException 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException 2019.05.01-13:48:18 java.nio.channels.ClosedChannelException 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException 2019.05.01-13:48:18 java.nio.channels.ClosedChannelException 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException 2019.04.27-10:55:06 java.nio.channels.ClosedChannelException 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException 2019.05.01-17:59:20 java.nio.channels.ClosedChannelException 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException 2019.04.27-10:54:40 java.nio.channels.ClosedChannelException 2019.04.28-09:32:20 java.nio.channels.ClosedChannelException 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException 2019.05.01-13:48:18 java.nio.channels.ClosedChannelException 2019.05.01-13:48:26 java.nio.channels.ClosedChannelException 2019.04.30-16:16:29 java.nio.channels.ClosedChannelException 2019.04.27-10:55:10 java.nio.channels.ClosedChannelException Peter Mikus Engineer - Software Cisco Systems Limited From: [email protected] <[email protected]> On Behalf Of Jan Gelety via Lists.Fd.Io Sent: Monday, April 29, 2019 8:58 AM To: [email protected]; [email protected] Cc: [email protected] Subject: [vpp-dev] Jenkins.fd.io network issues Hello, We are experiencing quite a lot of network issues when running CSIT tests for 19.04 report: Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on JNLP4-connect connection from vex-yul-rot-ingress-1.ci.codeaurora.org/10.30.48.3:41068 failed. The channel is closing down or has closed down https://jenkins.fd.io/job/csit-vpp-perf-verify-1904-3n-hsw/13/console Could you, please, have a look on it? Thank you very much. Regards, Jan
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12894): https://lists.fd.io/g/vpp-dev/message/12894 Mute This Topic: https://lists.fd.io/mt/31380682/21656 Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
