Heads-up - resolution in progress..

-Maciek

Begin forwarded message:

From: Maciek Konstantynowicz <mkons...@cisco.com<mailto:mkons...@cisco.com>>
Subject: number of csit-vpp jobs failing on vpp master branch - RCA wip
Date: 6 March 2018 at 14:54:39 GMT
To: csit-...@lists.fd.io<mailto:csit-...@lists.fd.io>
Cc: Peter Mikus <pmi...@cisco.com<mailto:pmi...@cisco.com>>

RCA Update from Peter Mikus on two issues behind the failing csit-vpp jobs due 
to recent VPP patches:-
Side note - honeycomb jobs are also affected.

1. errors during VPP install due to new vpp-lib dependencies - observed from 
last Friday

Root cause
----------

VPP patch [1] introduce new external package dependencies (libmbedcrypto0 
libmbedtls10 libmbedx509-0). Without these packages VPP service is not starting 
results in following log messages:
      vpp[30760]: /usr/bin/vpp: error while loading shared libraries: 
libmbedtls.so.10: cannot open shared object file: No such file or directory

This was not caught by vpp-csit-verify-virl-master job due to the bug in 
evaluating the robot framework results from multiple parallel simulation (see 
full log in [2]). Root cause is error in writing into /run/shm/rc_test_run${N} 
where CSIT bootstrap.sh is storing the results of partial runs. This results 
into faulty evaluation of total number of passed/failed test cases.

There are several steps needed to fix this issue:
#1 - Fix bootstrap to proper evaluate the results 
https://gerrit.fd.io/r/#/c/10981/
#2 – Install the missing dependencies in VIRL Ubuntu image .. TODO (patch 
https://gerrit.fd.io/r/#/c/10980/)
-   Rebuild of Ubuntu image needed
#3 – Install the missing dependencies in VIRL Centos image .. TODO (patch 
https://gerrit.fd.io/r/#/c/10980/)
-   Rebuild of Ubuntu image needed
#4 – Install the missing dependencies on PhyTB .. DONE (patch 
https://gerrit.fd.io/r/#/c/10980/)

[1] https://gerrit.fd.io/r/10768
[2] 
https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-csit-verify-virl-master/9755/console.log.gz


2. startup.conf broken - observed from last Friday
    impacting all csit-vpp jobs and hc-vpp jobs
        wasn't detected in vpp-csit-verify

Root cause
----------

This issue is caused by api-segment { gid vpp } to be not recognized in default 
startup.conf installed by VPP, resulting into VPP service not started. Root 
cause to be confirmed. Work in progress. Need to fix the vpp-lib dependencies 
first.

  • [vpp-dev] number... Maciek Konstantynowicz (mkonstan)

Reply via email to