The problem is almost definitely in VIRL. I noticed it on the 26th while
load testing VIRL3 by doing rechecks of CSIT patches:
https://gerrit.fd.io/r/r/8759
https://gerrit.fd.io/r/9890
https://gerrit.fd.io/r/9900
I think Ed Kern will need to look at this.
I noticed while it was failing nova list showed nothing. Then I noticed
that there was nothing listed on the other VIRL servers despite
rechecking patch 9904.
Today, I failed to launch a local simulation on virl3:
virl@t4-virl3:~/tfh/csit/resources/tools/disk-image-builder/centos$ !719
virl_std_client -u $VIRL_USER -p $VIRL_PASSWORD simengine-launch -f
listmaker/virl-listmaker-centos-7.3-1611.yaml
INFO 2017-12-31 16:37:48,788 virl.std.client Client.simengine_launch
called args=(<open file 'listmaker/virl-listmaker-centos-7.3-1611.yaml',
mode 'rb' at 0x7f038b6ed5d0>,
None,
False,
None,
None,
None,
None,
None,
None) kargs={}
INFO 2017-12-31 16:37:48,788 virl.std.client simengine_launch POST
on URL "http://localhost:19399/simengine/rest/launch";
INFO 2017-12-31 16:37:48,966 virl.std.client simengine_launch
response 500 to POST on URL
"http://localhost:19399/simengine/rest/launch?file=virl-listmaker-centos-7.3-1611.yaml";
ERROR 2017-12-31 16:37:48,967 virl.std.client STD client call to
Client.simengine_launch received invalid response status 500 (Cisco
contact was not established. This may be temporary.)
----------------
Exception cause:
----------------
STD simengine-launch request received invalid response: 500 - Cisco
contact was not established. This may be temporary.
I grabbed the logs and saw this in the std.server.log
virl_std_client -u $VIRL_USER -p $VIRL_PASSWORD simengine-systemlogs >
logs.zip
I saw the following in std_server.log
vi std_server.log
quest: "GET /simengine/rest/events/session" user "tb4-virl"
ERROR 2017-12-31 16:23:18,788 PID=45336 virl.std.implementation
Simulation "session" not found.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line
1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line
1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "<decorator-gen-554>", line 2, in simengine_events
File
"/var/jenkins/workspace/VIRL_CORE_build/test-virl-repo/virl/std/implementation.py",
line 330, in middleware
File "<decorator-gen-553>", line 2, in simengine_events
File
"/var/jenkins/workspace/VIRL_CORE_build/test-virl-repo/virl/common/utils.py",
line 94, in measurer
File
"/var/jenkins/workspace/VIRL_CORE_build/test-virl-repo/virl/std/implementation.py",
line 2177, in simengine_events
HttpException: Simulation "session" not found.
--Tom
On 12/27/2017 09:48 AM, Thomas F Herbert wrote:
On 12/27/2017 09:02 AM, Neale Ranns (nranns) wrote:
Hi Nitin,
Hit the ‘reply’ button and post a review comment of:
recheck
that will poke Jenkins to redo the verification.
/neale
-----Original Message-----
From:<[email protected]> on behalf of "Saxena,
Nitin"<[email protected]>
Date: Wednesday, 27 December 2017 at 14:38
To: "Dave Barach (dbarach)"<[email protected]>,"[email protected]"
<[email protected]>
Subject: [vpp-dev] gerrit 9904 VIRL verification is failing
Hi,
I sent a patch (https://gerrit.fd.io/r/#/c/9904/) for review in which "vpp-csit-verify-virl-master" job is failing.
Console logs (https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-csit-verify-virl-master/8798/console.log.gz) shows following error.
====================
call_home\nFlmClientException: Cisco contact was not established. This may be
temporary.\nPlease make sure the VIRL server is connected to the Internet and
capable of reaching the configured Cisco master.\nAlso make sure that the minion key
provided to you matches your minion ID and domain, and remains valid.\nCurrent
status is: Last successful contact was more than 7 days ago.\nLast call home check
result was: Call has timed out; failed to connect or minion key not accepted.\n"
I have also seen what looks like the same thing on VIRL3 which is
currently not in production. I reported it yesterday to the CSIT
mailing list.
https://jenkins.fd.io/job/csit-vpp-functional-master-ubuntu1604-virl/3290/
}
+ VIRL_SID[${index}]=
+ retval=1
+ '[' 1 -ne 0 ']'
+ echo 'VIRL simulation start failed on 10.30.51.29'
VIRL simulation start failed on 10.30.51.29
=======================
Seems like a temporary problem. What is the gerrit command such that Jenkins again start doing verification.
Thanks,
Nitin
_______________________________________________
vpp-dev mailing list
[email protected]
https://lists.fd.io/mailman/listinfo/vpp-dev
_______________________________________________
vpp-dev mailing list
[email protected]
https://lists.fd.io/mailman/listinfo/vpp-dev
--
*Thomas F Herbert*
NFV and Fast Data Planes
Networking Group Office of the CTO
*Red Hat*
--
*Thomas F Herbert*
NFV and Fast Data Planes
Networking Group Office of the CTO
*Red Hat*
_______________________________________________
vpp-dev mailing list
[email protected]
https://lists.fd.io/mailman/listinfo/vpp-dev
