Adding Anil. On Tue, Nov 20, 2018 at 4:10 AM Michael Vorburger <vorbur...@redhat.com> wrote:
> On Mon, Nov 19, 2018 at 7:34 PM Robert Varga <n...@hq.sk> wrote: > >> On 19/11/2018 15:30, Michael Vorburger wrote: >> > On Mon, Nov 19, 2018 at 3:24 PM Faseela K <faseel...@ericsson.com >> > <mailto:faseel...@ericsson.com>> wrote: >> > >> > Are these issues resolved? >> > I am still getting some job timeouts intermittently. >> > Sometimes merge jobs, sometimes CSIT jobs. >> > >> > >> > same here, something NOK on Jenkins slaves, >> > e.g. >> https://jenkins.opendaylight.org/releng/job/openflowplugin-maven-verify-neon-mvn35-openjdk8/146/console >> . >> > >> > I've opened helpdesk ticket #63907 about this. >> > > > Caused by: java.net.UnknownHostException: nexus.opendaylight.org > > > seen on > https://jenkins.opendaylight.org/releng/job/infrautils-maven-javadoc-verify-neon/189/console > about 5h ago ... > > This is very weird, as the slaves seem to be pausing for quite some >> time, quoteth >> >> https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-maven-verify-neon-mvn35-openjdk8/478/console-timestamp.log.gz >> : >> >> > 13:44:30 [INFO] --- maven-bundle-plugin:4.0.0:deploy (default-deploy) >> @ ipv6service-api --- >> > 13:44:30 [INFO] Remote OBR update disabled (enable with -DremoteOBR) >> > 13:44:30 [INFO] >> > 13:44:30 [INFO] --------------< >> org.opendaylight.netvirt:odl-netvirt-api >-------------- >> > 13:44:30 [INFO] Building OpenDaylight :: NetVirt :: api >> 0.8.0-SNAPSHOT [33/64] >> > 13:44:30 [INFO] ------------------------------[ feature >> ]------------------------------- >> > 14:02:19 [INFO] >> > 14:02:19 [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ >> odl-netvirt-api --- >> >> And load indicates all CPU activity pretty much halted for 17 minutes: >> >> > 14:19:16 01:42:01 PM CPU %user %nice %system %iowait >> %steal %idle >> > 14:19:16 01:43:01 PM all 35.17 0.00 1.01 0.00 >> 0.02 63.80 >> > 14:19:16 01:44:01 PM all 7.15 0.00 0.45 0.00 >> 0.01 92.39 >> > 14:19:16 01:45:01 PM all 8.81 0.00 0.49 0.00 >> 0.00 90.69 >> > 14:19:16 01:46:01 PM all 0.18 0.00 0.03 0.00 >> 0.00 99.79 >> > 14:19:16 01:47:01 PM all 0.57 0.00 0.04 0.00 >> 0.00 99.39 >> > 14:19:16 01:48:01 PM all 0.32 0.00 0.03 0.00 >> 0.00 99.64 >> > 14:19:16 01:49:01 PM all 0.14 0.00 0.03 0.00 >> 0.00 99.83 >> > 14:19:16 01:50:01 PM all 0.17 0.00 0.03 0.00 >> 0.00 99.79 >> > 14:19:16 01:51:01 PM all 0.22 0.00 0.04 0.01 >> 0.00 99.73 >> > 14:19:16 01:52:02 PM all 0.18 0.00 0.03 0.00 >> 0.00 99.79 >> > 14:19:16 01:53:01 PM all 0.23 0.00 0.03 0.00 >> 0.00 99.74 >> > 14:19:16 01:54:01 PM all 0.34 0.00 0.05 0.00 >> 0.00 99.61 >> > 14:19:16 01:55:01 PM all 0.38 0.00 0.11 0.01 >> 0.00 99.49 >> > 14:19:16 01:56:01 PM all 0.32 0.00 0.10 0.00 >> 0.00 99.58 >> > 14:19:16 01:57:01 PM all 0.17 0.00 0.05 0.00 >> 0.00 99.78 >> > 14:19:16 01:58:01 PM all 0.16 0.00 0.04 0.00 >> 0.00 99.80 >> > 14:19:16 01:59:01 PM all 0.14 0.00 0.04 0.00 >> 0.00 99.82 >> > 14:19:16 02:00:01 PM all 0.28 0.00 0.09 0.00 >> 0.00 99.63 >> > 14:19:16 02:01:01 PM all 0.30 0.00 0.10 0.00 >> 0.00 99.59 >> > 14:19:16 02:02:01 PM all 0.36 0.00 0.09 0.00 >> 0.00 99.55 >> > 14:19:16 02:03:01 PM all 6.68 0.00 0.83 0.03 >> 0.00 92.45 >> > 14:19:16 02:04:01 PM all 38.29 0.00 1.83 0.05 >> 0.04 59.79 >> >> while network activity went down for ~12 minutes: >> >> > 14:19:16 01:42:01 PM eth0 201.27 193.89 439.31 >> 13.10 0.00 0.00 0.00 >> > 14:19:16 01:43:01 PM eth0 6.63 4.76 2.29 >> 3.20 0.00 0.00 0.00 >> > 14:19:16 01:44:01 PM eth0 1.15 0.67 0.22 >> 0.07 0.00 0.00 0.00 >> > 14:19:16 01:45:01 PM eth0 9.83 7.48 3.94 >> 3.15 0.00 0.00 0.00 >> > 14:19:16 01:46:01 PM eth0 3.41 2.48 2.51 >> 0.74 0.00 0.00 0.00 >> > 14:19:16 01:47:01 PM eth0 8.56 3.33 4.80 >> 0.89 0.00 0.00 0.00 >> > 14:19:16 01:48:01 PM eth0 4.71 3.40 3.48 >> 1.42 0.00 0.00 0.00 >> > 14:19:16 01:49:01 PM eth0 3.81 2.68 2.52 >> 1.10 0.00 0.00 0.00 >> > 14:19:16 01:50:01 PM eth0 5.23 3.46 5.58 >> 0.83 0.00 0.00 0.00 >> > 14:19:16 01:51:01 PM eth0 6.09 4.41 7.75 >> 0.92 0.00 0.00 0.00 >> > 14:19:16 01:52:02 PM eth0 5.68 3.76 6.22 >> 0.82 0.00 0.00 0.00 >> > 14:19:16 01:53:01 PM eth0 5.15 3.88 5.31 >> 0.98 0.00 0.00 0.00 >> > 14:19:16 01:54:01 PM eth0 26.00 19.09 39.96 >> 2.84 0.00 0.00 0.00 >> > 14:19:16 01:55:01 PM eth0 153.66 143.29 341.57 >> 10.78 0.00 0.00 0.00 >> > 14:19:16 01:56:01 PM eth0 205.63 200.37 429.25 >> 13.79 0.00 0.00 0.00 >> > 14:19:16 01:57:01 PM eth0 108.76 102.56 237.71 >> 6.80 0.00 0.00 0.00 >> > 14:19:16 01:58:01 PM eth0 73.50 70.35 142.76 >> 5.56 0.00 0.00 0.00 >> >> and IO was mostly silenced, too: >> >> > 14:19:16 12:47:01 PM tps rtps wtps bread/s bwrtn/s >> > 14:19:16 01:38:01 PM 19.29 0.03 19.25 0.80 1005.33 >> > 14:19:16 01:39:01 PM 9.80 0.00 9.80 0.00 710.09 >> > 14:19:16 01:40:01 PM 6.86 0.03 6.83 0.67 437.74 >> > 14:19:16 01:41:01 PM 245.23 221.90 23.33 3130.10 1035.75 >> > 14:19:16 01:42:01 PM 7.38 0.00 7.38 0.00 965.43 >> > 14:19:16 01:43:01 PM 7.89 0.05 7.84 0.80 407.00 >> > 14:19:16 01:44:01 PM 2.22 0.00 2.22 0.00 94.69 >> > 14:19:16 01:45:01 PM 10.78 0.00 10.78 0.00 670.92 >> > 14:19:16 01:46:01 PM 4.40 0.00 4.40 0.00 65.95 >> > 14:19:16 01:47:01 PM 5.31 0.00 5.31 0.00 86.04 >> > 14:19:16 01:48:01 PM 6.30 0.00 6.30 0.00 104.20 >> > 14:19:16 01:49:01 PM 5.06 0.00 5.06 0.00 82.18 >> > 14:19:16 01:50:01 PM 5.76 0.00 5.76 0.00 95.94 >> > 14:19:16 01:51:01 PM 5.26 0.00 5.26 0.00 90.86 >> > 14:19:16 01:52:02 PM 5.75 0.00 5.75 0.00 85.94 >> > 14:19:16 01:53:01 PM 4.96 0.00 4.96 0.00 96.58 >> > 14:19:16 01:54:01 PM 2.85 0.00 2.85 0.00 56.69 >> > 14:19:16 01:55:01 PM 16.44 0.00 16.44 0.00 736.68 >> > 14:19:16 01:56:01 PM 6.11 0.00 6.11 0.00 1012.36 >> > 14:19:16 01:57:01 PM 4.55 0.00 4.55 0.00 777.04 >> > 14:19:16 01:58:01 PM 1.63 0.00 1.63 0.00 311.32 >> > 14:19:16 01:59:01 PM 0.70 0.00 0.70 0.00 255.83 >> > 14:19:16 02:00:01 PM 0.98 0.00 0.98 0.00 423.66 >> > 14:19:16 02:01:01 PM 1.13 0.00 1.13 0.00 421.36 >> > 14:19:16 02:02:01 PM 1.01 0.00 1.01 0.00 390.00 >> > 14:19:16 02:03:01 PM 10.27 0.20 10.07 137.22 360.95 >> > 14:19:16 02:04:01 PM 57.46 0.05 57.41 41.43 10262.12 >> >> So something was very much amiss... a retriggered >> >> https://jenkins.opendaylight.org/releng/job/netvirt-maven-verify-neon-mvn35-openjdk8/482/console >> shows this mulling for about seconds: >> >> > 00:14:26.156 [INFO] Remote OBR update disabled (enable with -DremoteOBR) >> > 00:14:26.157 [INFO] >> > 00:14:26.157 [INFO] --------------< >> org.opendaylight.netvirt:odl-netvirt-api >-------------- >> > 00:14:26.157 [INFO] Building OpenDaylight :: NetVirt :: api >> 0.8.0-SNAPSHOT [33/64] >> > 00:14:26.157 [INFO] ------------------------------[ feature >> ]------------------------------- >> > 00:14:46.642 [INFO] >> > 00:14:46.642 [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) >> @ odl-netvirt-api --- >> >> Looking at Michael's build at >> >> https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/openflowplugin-maven-verify-neon-mvn35-openjdk8/146/console-timestamp.log.gz >> : >> >> > 12:06:34 [INFO] --< >> org.opendaylight.openflowplugin:odl-openflowplugin-flow-services-rest >-- >> > 12:06:34 [INFO] Building OpenDaylight :: Openflow Plugin :: Flow >> Services :: REST 0.8.0-SNAPSHOT [70/81] >> > 12:06:34 [INFO] ------------------------------[ feature >> ]------------------------------- >> > 12:10:45 [INFO] >> > 12:10:45 [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ >> odl-openflowplugin-flow-services-rest --- >> >> and this is definitely nexus: >> >> > 11:32:03 [INFO] --- maven-bundle-plugin:4.0.0:deploy (default-deploy) >> @ openflowplugin-blueprint-config --- >> > 11:32:03 [INFO] Remote OBR update disabled (enable with -DremoteOBR) >> > 11:32:03 [INFO] >> > 11:32:03 [INFO] ---< >> org.opendaylight.openflowplugin:odl-openflowplugin-southbound >---- >> > 11:32:03 [INFO] Building OpenDaylight :: Openflow Plugin :: Li >> southbound API implementation 0.8.0-SNAPSHOT [34/81] >> > 11:32:03 [INFO] ------------------------------[ feature >> ]------------------------------- >> > 11:35:22 [WARNING] Could not validate integrity of download from >> https://nexus.opendaylight.org/content/repositories/public/com/bea/xml/jsr173-ri/1.0/jsr173-ri-1.0.jar: >> Checksum validation failed, no checksums available >> > 11:35:22 [WARNING] Checksum validation failed, no checksums available >> from opendaylight-mirror for >> https://nexus.opendaylight.org/content/repositories/public/com/bea/xml/jsr173-ri/1.0/jsr173-ri-1.0.jar >> > 11:35:25 [INFO] >> > 11:35:25 [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ >> odl-openflowplugin-southbound --- >> >> Regards, >> Robert >> >>
_______________________________________________ Discuss mailing list Discuss@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/discuss
