On 19/11/2018 15:30, Michael Vorburger wrote: > On Mon, Nov 19, 2018 at 3:24 PM Faseela K <faseel...@ericsson.com > <mailto:faseel...@ericsson.com>> wrote: > > Are these issues resolved? > I am still getting some job timeouts intermittently. > Sometimes merge jobs, sometimes CSIT jobs. > > > same here, something NOK on Jenkins slaves, > e.g. > https://jenkins.opendaylight.org/releng/job/openflowplugin-maven-verify-neon-mvn35-openjdk8/146/console. > > I've opened helpdesk ticket #63907 about this.
This is very weird, as the slaves seem to be pausing for quite some time, quoteth https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-maven-verify-neon-mvn35-openjdk8/478/console-timestamp.log.gz : > 13:44:30 [INFO] --- maven-bundle-plugin:4.0.0:deploy (default-deploy) @ > ipv6service-api --- > 13:44:30 [INFO] Remote OBR update disabled (enable with -DremoteOBR) > 13:44:30 [INFO] > 13:44:30 [INFO] --------------< org.opendaylight.netvirt:odl-netvirt-api > >-------------- > 13:44:30 [INFO] Building OpenDaylight :: NetVirt :: api 0.8.0-SNAPSHOT > [33/64] > 13:44:30 [INFO] ------------------------------[ feature > ]------------------------------- > 14:02:19 [INFO] > 14:02:19 [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ > odl-netvirt-api --- And load indicates all CPU activity pretty much halted for 17 minutes: > 14:19:16 01:42:01 PM CPU %user %nice %system %iowait > %steal %idle > 14:19:16 01:43:01 PM all 35.17 0.00 1.01 0.00 > 0.02 63.80 > 14:19:16 01:44:01 PM all 7.15 0.00 0.45 0.00 > 0.01 92.39 > 14:19:16 01:45:01 PM all 8.81 0.00 0.49 0.00 > 0.00 90.69 > 14:19:16 01:46:01 PM all 0.18 0.00 0.03 0.00 > 0.00 99.79 > 14:19:16 01:47:01 PM all 0.57 0.00 0.04 0.00 > 0.00 99.39 > 14:19:16 01:48:01 PM all 0.32 0.00 0.03 0.00 > 0.00 99.64 > 14:19:16 01:49:01 PM all 0.14 0.00 0.03 0.00 > 0.00 99.83 > 14:19:16 01:50:01 PM all 0.17 0.00 0.03 0.00 > 0.00 99.79 > 14:19:16 01:51:01 PM all 0.22 0.00 0.04 0.01 > 0.00 99.73 > 14:19:16 01:52:02 PM all 0.18 0.00 0.03 0.00 > 0.00 99.79 > 14:19:16 01:53:01 PM all 0.23 0.00 0.03 0.00 > 0.00 99.74 > 14:19:16 01:54:01 PM all 0.34 0.00 0.05 0.00 > 0.00 99.61 > 14:19:16 01:55:01 PM all 0.38 0.00 0.11 0.01 > 0.00 99.49 > 14:19:16 01:56:01 PM all 0.32 0.00 0.10 0.00 > 0.00 99.58 > 14:19:16 01:57:01 PM all 0.17 0.00 0.05 0.00 > 0.00 99.78 > 14:19:16 01:58:01 PM all 0.16 0.00 0.04 0.00 > 0.00 99.80 > 14:19:16 01:59:01 PM all 0.14 0.00 0.04 0.00 > 0.00 99.82 > 14:19:16 02:00:01 PM all 0.28 0.00 0.09 0.00 > 0.00 99.63 > 14:19:16 02:01:01 PM all 0.30 0.00 0.10 0.00 > 0.00 99.59 > 14:19:16 02:02:01 PM all 0.36 0.00 0.09 0.00 > 0.00 99.55 > 14:19:16 02:03:01 PM all 6.68 0.00 0.83 0.03 > 0.00 92.45 > 14:19:16 02:04:01 PM all 38.29 0.00 1.83 0.05 > 0.04 59.79 while network activity went down for ~12 minutes: > 14:19:16 01:42:01 PM eth0 201.27 193.89 439.31 13.10 > 0.00 0.00 0.00 > 14:19:16 01:43:01 PM eth0 6.63 4.76 2.29 3.20 > 0.00 0.00 0.00 > 14:19:16 01:44:01 PM eth0 1.15 0.67 0.22 0.07 > 0.00 0.00 0.00 > 14:19:16 01:45:01 PM eth0 9.83 7.48 3.94 3.15 > 0.00 0.00 0.00 > 14:19:16 01:46:01 PM eth0 3.41 2.48 2.51 0.74 > 0.00 0.00 0.00 > 14:19:16 01:47:01 PM eth0 8.56 3.33 4.80 0.89 > 0.00 0.00 0.00 > 14:19:16 01:48:01 PM eth0 4.71 3.40 3.48 1.42 > 0.00 0.00 0.00 > 14:19:16 01:49:01 PM eth0 3.81 2.68 2.52 1.10 > 0.00 0.00 0.00 > 14:19:16 01:50:01 PM eth0 5.23 3.46 5.58 0.83 > 0.00 0.00 0.00 > 14:19:16 01:51:01 PM eth0 6.09 4.41 7.75 0.92 > 0.00 0.00 0.00 > 14:19:16 01:52:02 PM eth0 5.68 3.76 6.22 0.82 > 0.00 0.00 0.00 > 14:19:16 01:53:01 PM eth0 5.15 3.88 5.31 0.98 > 0.00 0.00 0.00 > 14:19:16 01:54:01 PM eth0 26.00 19.09 39.96 2.84 > 0.00 0.00 0.00 > 14:19:16 01:55:01 PM eth0 153.66 143.29 341.57 10.78 > 0.00 0.00 0.00 > 14:19:16 01:56:01 PM eth0 205.63 200.37 429.25 13.79 > 0.00 0.00 0.00 > 14:19:16 01:57:01 PM eth0 108.76 102.56 237.71 6.80 > 0.00 0.00 0.00 > 14:19:16 01:58:01 PM eth0 73.50 70.35 142.76 5.56 > 0.00 0.00 0.00 and IO was mostly silenced, too: > 14:19:16 12:47:01 PM tps rtps wtps bread/s bwrtn/s > 14:19:16 01:38:01 PM 19.29 0.03 19.25 0.80 1005.33 > 14:19:16 01:39:01 PM 9.80 0.00 9.80 0.00 710.09 > 14:19:16 01:40:01 PM 6.86 0.03 6.83 0.67 437.74 > 14:19:16 01:41:01 PM 245.23 221.90 23.33 3130.10 1035.75 > 14:19:16 01:42:01 PM 7.38 0.00 7.38 0.00 965.43 > 14:19:16 01:43:01 PM 7.89 0.05 7.84 0.80 407.00 > 14:19:16 01:44:01 PM 2.22 0.00 2.22 0.00 94.69 > 14:19:16 01:45:01 PM 10.78 0.00 10.78 0.00 670.92 > 14:19:16 01:46:01 PM 4.40 0.00 4.40 0.00 65.95 > 14:19:16 01:47:01 PM 5.31 0.00 5.31 0.00 86.04 > 14:19:16 01:48:01 PM 6.30 0.00 6.30 0.00 104.20 > 14:19:16 01:49:01 PM 5.06 0.00 5.06 0.00 82.18 > 14:19:16 01:50:01 PM 5.76 0.00 5.76 0.00 95.94 > 14:19:16 01:51:01 PM 5.26 0.00 5.26 0.00 90.86 > 14:19:16 01:52:02 PM 5.75 0.00 5.75 0.00 85.94 > 14:19:16 01:53:01 PM 4.96 0.00 4.96 0.00 96.58 > 14:19:16 01:54:01 PM 2.85 0.00 2.85 0.00 56.69 > 14:19:16 01:55:01 PM 16.44 0.00 16.44 0.00 736.68 > 14:19:16 01:56:01 PM 6.11 0.00 6.11 0.00 1012.36 > 14:19:16 01:57:01 PM 4.55 0.00 4.55 0.00 777.04 > 14:19:16 01:58:01 PM 1.63 0.00 1.63 0.00 311.32 > 14:19:16 01:59:01 PM 0.70 0.00 0.70 0.00 255.83 > 14:19:16 02:00:01 PM 0.98 0.00 0.98 0.00 423.66 > 14:19:16 02:01:01 PM 1.13 0.00 1.13 0.00 421.36 > 14:19:16 02:02:01 PM 1.01 0.00 1.01 0.00 390.00 > 14:19:16 02:03:01 PM 10.27 0.20 10.07 137.22 360.95 > 14:19:16 02:04:01 PM 57.46 0.05 57.41 41.43 10262.12 So something was very much amiss... a retriggered https://jenkins.opendaylight.org/releng/job/netvirt-maven-verify-neon-mvn35-openjdk8/482/console shows this mulling for about seconds: > 00:14:26.156 [INFO] Remote OBR update disabled (enable with -DremoteOBR) > 00:14:26.157 [INFO] > 00:14:26.157 [INFO] --------------< org.opendaylight.netvirt:odl-netvirt-api > >-------------- > 00:14:26.157 [INFO] Building OpenDaylight :: NetVirt :: api 0.8.0-SNAPSHOT > [33/64] > 00:14:26.157 [INFO] ------------------------------[ feature > ]------------------------------- > 00:14:46.642 [INFO] > 00:14:46.642 [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ > odl-netvirt-api --- Looking at Michael's build at https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/openflowplugin-maven-verify-neon-mvn35-openjdk8/146/console-timestamp.log.gz: > 12:06:34 [INFO] --< > org.opendaylight.openflowplugin:odl-openflowplugin-flow-services-rest >-- > 12:06:34 [INFO] Building OpenDaylight :: Openflow Plugin :: Flow Services :: > REST 0.8.0-SNAPSHOT [70/81] > 12:06:34 [INFO] ------------------------------[ feature > ]------------------------------- > 12:10:45 [INFO] > 12:10:45 [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ > odl-openflowplugin-flow-services-rest --- and this is definitely nexus: > 11:32:03 [INFO] --- maven-bundle-plugin:4.0.0:deploy (default-deploy) @ > openflowplugin-blueprint-config --- > 11:32:03 [INFO] Remote OBR update disabled (enable with -DremoteOBR) > 11:32:03 [INFO] > 11:32:03 [INFO] ---< > org.opendaylight.openflowplugin:odl-openflowplugin-southbound >---- > 11:32:03 [INFO] Building OpenDaylight :: Openflow Plugin :: Li southbound > API implementation 0.8.0-SNAPSHOT [34/81] > 11:32:03 [INFO] ------------------------------[ feature > ]------------------------------- > 11:35:22 [WARNING] Could not validate integrity of download from > https://nexus.opendaylight.org/content/repositories/public/com/bea/xml/jsr173-ri/1.0/jsr173-ri-1.0.jar: > Checksum validation failed, no checksums available > 11:35:22 [WARNING] Checksum validation failed, no checksums available from > opendaylight-mirror for > https://nexus.opendaylight.org/content/repositories/public/com/bea/xml/jsr173-ri/1.0/jsr173-ri-1.0.jar > 11:35:25 [INFO] > 11:35:25 [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ > odl-openflowplugin-southbound --- Regards, Robert
signature.asc
Description: OpenPGP digital signature
_______________________________________________ infrastructure mailing list infrastructure@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/infrastructure
