Re: [vpp-dev] VRRP Unit Tests failing on Master Ubuntu

[Dave Wallace]

Hi Paul,

Good point.

Do you think it's worth adding an env var for running tests with 
intermittent failures on the LF infra?  In my experience, it is often 
hard to reproduce intermittent failures locally so I'm thinking that in 
may be useful to have a "penalty box" for flaky tests.


Or I can push a patch to just skip the VRRP tests instead of using 
extended tests.


Thanks,
-daw-

On 3/13/2020 1:40 PM, Paul Vinciguerra wrote:


Hi Dave,

Please don't add broken tests to extended tests.

See https://gerrit.fd.io/r/c/vpp/+/24567/12/test/test_bfd.py

Thanks!

On Fri, Mar 13, 2020 at 1:33 PM Dave Wallace > wrote:


Matt,

I will keep an eye on this gerrit and merge it once the verify
jobs have completed.
If there are other tests which fail, are you ok if I add them to
this patch and turn it into a generic 'disable failing tests'
gerrit change?

The other possibility is that this is due to the recent disabling
of the Naginator retry plugin.

I'm going to investigate if this issue may have been masked by
Naginator...

Thanks for your help on keeping the CI operational!
-daw-

On 3/12/2020 12:09 PM, Matthew Smith via Lists.Fd.Io
 wrote:


Change submitted - https://gerrit.fd.io/r/c/vpp/+/25834.
Verification jobs are running. Hopefully they won't fail :)

-Matt


On Thu, Mar 12, 2020 at 10:22 AM Matthew Smith via Lists.Fd.Io
 mailto:netgate@lists.fd.io>> wrote:


I don't have a solution yet, but one observation has popped
up quickly

In the 2 failed jobs Ray sent links for, one of them had a
test fail which was not related to VRRP. There is a BFD6 test
failure for the NAT change https://gerrit.fd.io/r/c/vpp/+/25462:


https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2678/archives/

Looking back through a couple of recent failed runs of that
job, there is also a DHCP6 PD test failure for rdma change
https://gerrit.fd.io/r/c/vpp/+/25823:


https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2682/archives/

The most obvious common thread between BFD6, DHCP6 and VRRP
to me seems to be that they all maintain state which is
dependent on timers. There could be a more general issue with
timing-sensitive tests. I am going to submit a change which
will prevent the VRRP tests from running temporarily while I
can figure out a proper solution. Based on the above, other
tests may need the same treatment.

-Matt



On Thu, Mar 12, 2020 at 8:57 AM Matthew Smith
mailto:mgsm...@netgate.com>> wrote:

Hi Ray,

Thanks for bringing it to my attention. I'll look into it.

-Matt


On Thu, Mar 12, 2020 at 8:24 AM Ray Kinsella
mailto:m...@ashroe.eu>> wrote:

Anyone else noticing seeming spurious failures
related to the VRRP plugin's unit tests.
Some examples from un-related commits.

Ray K

nat: timed out session scavenging upgrade
(https://gerrit.fd.io/r/c/vpp/+/25462)

https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2678/console.log.gz


==
TEST RESULTS:
     Scheduled tests: 1138
      Executed tests: 1138
        Passed tests: 1021
       Skipped tests: 112
            Failures: 3
              Errors: 2
FAILURES AND ERRORS IN TESTS:
  Testcase name: IPv4 VRRP Test Case
    FAILURE: IPv4 Master VR does not reply for VIP w/
accept mode off
[test_vrrp.TestVRRP4.test_vrrp4_accept_mode_disabled]
    FAILURE: IPv4 Master VR preempted by higher
priority backup
[test_vrrp.TestVRRP4.test_vrrp4_master_preempted]
  Testcase name: IPv6 VRRP Test Case
    FAILURE: IPv6 Master VR preempted by higher
priority backup
[test_vrrp.TestVRRP6.test_vrrp6_master_preempted]
      ERROR: IPv6 Backup VR preempts lower priority
master [test_vrrp.TestVRRP6.test_vrrp6_backup_preempts]
  Testcase name: Bidirectional Forwarding Detection
(BFD) (IPv6)
      ERROR: echo function
[test_bfd.BFD6TestCase.test_echo]

==

vlib: startup multi-arch variant configuration

Re: [vpp-dev] VRRP Unit Tests failing on Master Ubuntu

[Paul Vinciguerra]

Hi Dave,

Please don't add broken tests to extended tests.

See https://gerrit.fd.io/r/c/vpp/+/24567/12/test/test_bfd.py

Thanks!

On Fri, Mar 13, 2020 at 1:33 PM Dave Wallace  wrote:

> Matt,
>
> I will keep an eye on this gerrit and merge it once the verify jobs have
> completed.
> If there are other tests which fail, are you ok if I add them to this
> patch and turn it into a generic 'disable failing tests' gerrit change?
>
> The other possibility is that this is due to the recent disabling of the
> Naginator retry plugin.
>
> I'm going to investigate if this issue may have been masked by Naginator...
>
> Thanks for your help on keeping the CI operational!
> -daw-
>
> On 3/12/2020 12:09 PM, Matthew Smith via Lists.Fd.Io wrote:
>
>
> Change submitted - https://gerrit.fd.io/r/c/vpp/+/25834. Verification
> jobs are running. Hopefully they won't fail :)
>
> -Matt
>
>
> On Thu, Mar 12, 2020 at 10:22 AM Matthew Smith via Lists.Fd.Io  netgate@lists.fd.io> wrote:
>
>>
>> I don't have a solution yet, but one observation has popped up quickly
>>
>> In the 2 failed jobs Ray sent links for, one of them had a test fail
>> which was not related to VRRP. There is a BFD6 test failure for the NAT
>> change https://gerrit.fd.io/r/c/vpp/+/25462:
>>
>>
>> https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2678/archives/
>>
>> Looking back through a couple of recent failed runs of that job, there is
>> also a DHCP6 PD test failure for rdma change
>> https://gerrit.fd.io/r/c/vpp/+/25823:
>>
>>
>> https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2682/archives/
>>
>> The most obvious common thread between BFD6, DHCP6 and VRRP to me seems
>> to be that they all maintain state which is dependent on timers. There
>> could be a more general issue with timing-sensitive tests. I am going to
>> submit a change which will prevent the VRRP tests from running temporarily
>> while I can figure out a proper solution. Based on the above, other tests
>> may need the same treatment.
>>
>> -Matt
>>
>>
>>
>>
>> On Thu, Mar 12, 2020 at 8:57 AM Matthew Smith 
>> wrote:
>>
>>> Hi Ray,
>>>
>>> Thanks for bringing it to my attention. I'll look into it.
>>>
>>> -Matt
>>>
>>>
>>> On Thu, Mar 12, 2020 at 8:24 AM Ray Kinsella  wrote:
>>>
 Anyone else noticing seeming spurious failures related to the VRRP
 plugin's unit tests.
 Some examples from un-related commits.

 Ray K

 nat: timed out session scavenging upgrade (
 https://gerrit.fd.io/r/c/vpp/+/25462)

 https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2678/console.log.gz


 ==
 TEST RESULTS:
  Scheduled tests: 1138
   Executed tests: 1138
 Passed tests: 1021
Skipped tests: 112
 Failures: 3
   Errors: 2
 FAILURES AND ERRORS IN TESTS:
   Testcase name: IPv4 VRRP Test Case
 FAILURE: IPv4 Master VR does not reply for VIP w/ accept mode off
 [test_vrrp.TestVRRP4.test_vrrp4_accept_mode_disabled]
 FAILURE: IPv4 Master VR preempted by higher priority backup
 [test_vrrp.TestVRRP4.test_vrrp4_master_preempted]
   Testcase name: IPv6 VRRP Test Case
 FAILURE: IPv6 Master VR preempted by higher priority backup
 [test_vrrp.TestVRRP6.test_vrrp6_master_preempted]
   ERROR: IPv6 Backup VR preempts lower priority master
 [test_vrrp.TestVRRP6.test_vrrp6_backup_preempts]
   Testcase name: Bidirectional Forwarding Detection (BFD) (IPv6)
   ERROR: echo function [test_bfd.BFD6TestCase.test_echo]

 ==

 vlib: startup multi-arch variant configuration (
 https://gerrit.fd.io/r/c/vpp/+/25798_

 https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2675/console.log.gz


 ==
 TEST RESULTS:
  Scheduled tests: 22
   Executed tests: 22
 Passed tests: 21
 Failures: 1
 FAILURES AND ERRORS IN TESTS:
   Testcase name: IPv4 VRRP Test Case
 FAILURE: IPv4 Master VR preempted by higher priority backup
 [test_vrrp.TestVRRP4.test_vrrp4_master_preempted]

 ==




>>
>
> 
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15782): https://lists.fd.io/g/vpp-dev/message/15782
Mute This Topic: https://lists.fd.io/mt/71901798/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] VRRP Unit Tests failing on Master Ubuntu

[Klement Sekera via Lists.Fd.Io]

There is also test/scripts/test-loop.sh which might some users better.

Regards,
Klement

> On 12 Mar 2020, at 19:08, Dave Wallace  wrote:
> 
> Hi Matt,
> 
> Your patch [0] verified, Ray +1'd it, and I merged it.
> 
> In my investigation on Naginator retries, I found an unrelated gerrit change 
> [1] where there was a VRRP test failure [2] on which failed the 
> vpp-arm-verify-master-ubuntu1804 job but subsequently passed on both the 
> Naginator retry [3] as well as the verify of the next patch [4] to the gerrit 
> change.
> 
> This failure occurred on March 02, 2020 prior to the recent timekeeping 
> related changes.
> 
> In case you are not aware, I wrote a bash function [5] which allows iterative 
> running of make test until it encounters a failure. This function has been 
> helpful in tracking down and fixing intermittent test failures in the quic 
> tests which were very hard to reproduce outside of 'make test'. Note that in 
> particular, I have seen many more intermittent failures with 'make test' 
> running tests in parallel (make test TEST_JOBS=auto) when running them 
> serially. Also, the grep (-g) option is most useful for detecting 
> clib_warning() instrumentation of suspected errant conditions in release 
> images.
> 
> Hope this helps,
> -daw-
> 
> [0] https://gerrit.fd.io/r/c/vpp/+/25834
> [1] https://gerrit.fd.io/r/c/vpp/+/25581
> [2] https://gerrit.fd.io/r/c/vpp/+/25581#message-cb3ca555_cb3c5e63
>   
> https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-arm-verify-master-ubuntu1804/8899/console-timestamp.log.gz
> [3] https://gerrit.fd.io/r/c/vpp/+/25581#message-b01de4c2_560ef9c6
> [4] https://gerrit.fd.io/r/c/vpp/+/25581#message-d2ecb27d_a9d52cc9
> [5] https://git.fd.io/vpp/tree/extras/bash/functions.bash
> - %< -
> Usage: vpp-make-test [-a][-d][-f][-g ][-r ]  
> []
>  -aRun extended tests
>  -dRun vpp debug image (i.e. with ASSERTS)
>  -fTestcase is a feature set (e.g. tcp)
>  -g  Text to grep for in log, FAIL on match.
>Enclose  in single quotes when it contains 
> any dashes:
>e.g.  vpp-make-test -g 'goof-bad-' test_xyz
>  -r   Retry Count (default = 100 for individual | 1 for 
> feature)
> - %< -
> 
> 
> On 3/12/2020 12:41 PM, Matthew Smith wrote:
>> Hi Dave,
>> 
>> That sounds fine to me.
>> 
>> Thanks,
>> -Matt
>> 
>> 
>> On Thu, Mar 12, 2020 at 11:32 AM Dave Wallace  wrote:
>> Matt,
>> 
>> I will keep an eye on this gerrit and merge it once the verify jobs have 
>> completed.
>> If there are other tests which fail, are you ok if I add them to this patch 
>> and turn it into a generic 'disable failing tests' gerrit change?
>> 
>> The other possibility is that this is due to the recent disabling of the 
>> Naginator retry plugin.
>> 
>> I'm going to investigate if this issue may have been masked by Naginator...
>> 
>> Thanks for your help on keeping the CI operational!
>> -daw-
>> 
>> On 3/12/2020 12:09 PM, Matthew Smith via Lists.Fd.Io wrote:
>>> 
>>> Change submitted - https://gerrit.fd.io/r/c/vpp/+/25834. Verification jobs 
>>> are running. Hopefully they won't   fail :)
>>> 
>>> -Matt
>>> 
>>> 
>>> On Thu, Mar 12, 2020 at 10:22 AM Matthew Smith via Lists.Fd.Io 
>>>  wrote:
>>> 
>>> I don't have a solution yet, but one observation has popped up quickly
>>> 
>>> In the 2 failed jobs Ray sent links for, one of them had a test fail which 
>>> was not related to VRRP. There is a BFD6 test failure for the NAT change 
>>> https://gerrit.fd.io/r/c/vpp/+/25462:
>>> 
>>> https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2678/archives/
>>> 
>>> Looking back through a couple of recent failed runs of that job, there is 
>>> also a DHCP6 PD test failure for rdma change 
>>> https://gerrit.fd.io/r/c/vpp/+/25823:
>>> 
>>> https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2682/archives/
>>> 
>>> The most obvious common thread between BFD6, DHCP6 and VRRP to me seems to 
>>> be that they all maintain state which is dependent on timers. There could 
>>> be a more general issue with timing-sensitive tests. I am going to submit a 
>>> change which will prevent the VRRP tests from running temporarily while I 
>>> can figure out a proper solution. Based on the above, other tests may need 
>>> the same treatment.
>>> 
>>> -Matt
>>>
>>> 
>>> 
>>> 
>>> On Thu, Mar 12, 2020 at 8:57 AM Matthew Smith  wrote:
>>> Hi Ray,
>>> 
>>> Thanks for bringing it to my attention. I'll look into it.
>>> 
>>> -Matt
>>> 
>>> 
>>> On Thu, Mar 12, 2020 at 8:24 AM Ray Kinsella  wrote:
>>> Anyone else noticing seeming spurious failures related to the VRRP plugin's 
>>> unit tests.
>>> Some examples from un-related commits.
>>> 
>>> Ray K
>>> 
>>> nat: timed out session scavenging upgrade 
>>> (https://gerrit.fd.io/r/c/vpp/+/25462)
>>> 

Re: [vpp-dev] VRRP Unit Tests failing on Master Ubuntu

[Matthew Smith via Lists.Fd.Io]

Hi Dave,

That sounds fine to me.

Thanks,
-Matt


On Thu, Mar 12, 2020 at 11:32 AM Dave Wallace  wrote:

> Matt,
>
> I will keep an eye on this gerrit and merge it once the verify jobs have
> completed.
> If there are other tests which fail, are you ok if I add them to this
> patch and turn it into a generic 'disable failing tests' gerrit change?
>
> The other possibility is that this is due to the recent disabling of the
> Naginator retry plugin.
>
> I'm going to investigate if this issue may have been masked by Naginator...
>
> Thanks for your help on keeping the CI operational!
> -daw-
>
> On 3/12/2020 12:09 PM, Matthew Smith via Lists.Fd.Io wrote:
>
>
> Change submitted - https://gerrit.fd.io/r/c/vpp/+/25834. Verification
> jobs are running. Hopefully they won't fail :)
>
> -Matt
>
>
> On Thu, Mar 12, 2020 at 10:22 AM Matthew Smith via Lists.Fd.Io  netgate@lists.fd.io> wrote:
>
>>
>> I don't have a solution yet, but one observation has popped up quickly
>>
>> In the 2 failed jobs Ray sent links for, one of them had a test fail
>> which was not related to VRRP. There is a BFD6 test failure for the NAT
>> change https://gerrit.fd.io/r/c/vpp/+/25462:
>>
>>
>> https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2678/archives/
>>
>> Looking back through a couple of recent failed runs of that job, there is
>> also a DHCP6 PD test failure for rdma change
>> https://gerrit.fd.io/r/c/vpp/+/25823:
>>
>>
>> https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2682/archives/
>>
>> The most obvious common thread between BFD6, DHCP6 and VRRP to me seems
>> to be that they all maintain state which is dependent on timers. There
>> could be a more general issue with timing-sensitive tests. I am going to
>> submit a change which will prevent the VRRP tests from running temporarily
>> while I can figure out a proper solution. Based on the above, other tests
>> may need the same treatment.
>>
>> -Matt
>>
>>
>>
>>
>> On Thu, Mar 12, 2020 at 8:57 AM Matthew Smith 
>> wrote:
>>
>>> Hi Ray,
>>>
>>> Thanks for bringing it to my attention. I'll look into it.
>>>
>>> -Matt
>>>
>>>
>>> On Thu, Mar 12, 2020 at 8:24 AM Ray Kinsella  wrote:
>>>
 Anyone else noticing seeming spurious failures related to the VRRP
 plugin's unit tests.
 Some examples from un-related commits.

 Ray K

 nat: timed out session scavenging upgrade (
 https://gerrit.fd.io/r/c/vpp/+/25462)

 https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2678/console.log.gz


 ==
 TEST RESULTS:
  Scheduled tests: 1138
   Executed tests: 1138
 Passed tests: 1021
Skipped tests: 112
 Failures: 3
   Errors: 2
 FAILURES AND ERRORS IN TESTS:
   Testcase name: IPv4 VRRP Test Case
 FAILURE: IPv4 Master VR does not reply for VIP w/ accept mode off
 [test_vrrp.TestVRRP4.test_vrrp4_accept_mode_disabled]
 FAILURE: IPv4 Master VR preempted by higher priority backup
 [test_vrrp.TestVRRP4.test_vrrp4_master_preempted]
   Testcase name: IPv6 VRRP Test Case
 FAILURE: IPv6 Master VR preempted by higher priority backup
 [test_vrrp.TestVRRP6.test_vrrp6_master_preempted]
   ERROR: IPv6 Backup VR preempts lower priority master
 [test_vrrp.TestVRRP6.test_vrrp6_backup_preempts]
   Testcase name: Bidirectional Forwarding Detection (BFD) (IPv6)
   ERROR: echo function [test_bfd.BFD6TestCase.test_echo]

 ==

 vlib: startup multi-arch variant configuration (
 https://gerrit.fd.io/r/c/vpp/+/25798_

 https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2675/console.log.gz


 ==
 TEST RESULTS:
  Scheduled tests: 22
   Executed tests: 22
 Passed tests: 21
 Failures: 1
 FAILURES AND ERRORS IN TESTS:
   Testcase name: IPv4 VRRP Test Case
 FAILURE: IPv4 Master VR preempted by higher priority backup
 [test_vrrp.TestVRRP4.test_vrrp4_master_preempted]

 ==




>>
> 
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15764): https://lists.fd.io/g/vpp-dev/message/15764
Mute This Topic: https://lists.fd.io/mt/71901798/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] VRRP Unit Tests failing on Master Ubuntu

[Dave Wallace]

Matt,

I will keep an eye on this gerrit and merge it once the verify jobs have 
completed.
If there are other tests which fail, are you ok if I add them to this 
patch and turn it into a generic 'disable failing tests' gerrit change?


The other possibility is that this is due to the recent disabling of the 
Naginator retry plugin.


I'm going to investigate if this issue may have been masked by Naginator...

Thanks for your help on keeping the CI operational!
-daw-

On 3/12/2020 12:09 PM, Matthew Smith via Lists.Fd.Io wrote:


Change submitted - https://gerrit.fd.io/r/c/vpp/+/25834. Verification 
jobs are running. Hopefully they won't fail :)


-Matt


On Thu, Mar 12, 2020 at 10:22 AM Matthew Smith via Lists.Fd.Io 
 > wrote:



I don't have a solution yet, but one observation has popped up
quickly

In the 2 failed jobs Ray sent links for, one of them had a test
fail which was not related to VRRP. There is a BFD6 test failure
for the NAT change https://gerrit.fd.io/r/c/vpp/+/25462:


https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2678/archives/

Looking back through a couple of recent failed runs of that job,
there is also a DHCP6 PD test failure for rdma change
https://gerrit.fd.io/r/c/vpp/+/25823:


https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2682/archives/

The most obvious common thread between BFD6, DHCP6 and VRRP to me
seems to be that they all maintain state which is dependent on
timers. There could be a more general issue with timing-sensitive
tests. I am going to submit a change which will prevent the VRRP
tests from running temporarily while I can figure out a proper
solution. Based on the above, other tests may need the same treatment.

-Matt



On Thu, Mar 12, 2020 at 8:57 AM Matthew Smith mailto:mgsm...@netgate.com>> wrote:

Hi Ray,

Thanks for bringing it to my attention. I'll look into it.

-Matt


On Thu, Mar 12, 2020 at 8:24 AM Ray Kinsella mailto:m...@ashroe.eu>> wrote:

Anyone else noticing seeming spurious failures related to
the VRRP plugin's unit tests.
Some examples from un-related commits.

Ray K

nat: timed out session scavenging upgrade
(https://gerrit.fd.io/r/c/vpp/+/25462)

https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2678/console.log.gz


==
TEST RESULTS:
     Scheduled tests: 1138
      Executed tests: 1138
        Passed tests: 1021
       Skipped tests: 112
            Failures: 3
              Errors: 2
FAILURES AND ERRORS IN TESTS:
  Testcase name: IPv4 VRRP Test Case
    FAILURE: IPv4 Master VR does not reply for VIP w/
accept mode off
[test_vrrp.TestVRRP4.test_vrrp4_accept_mode_disabled]
    FAILURE: IPv4 Master VR preempted by higher priority
backup [test_vrrp.TestVRRP4.test_vrrp4_master_preempted]
  Testcase name: IPv6 VRRP Test Case
    FAILURE: IPv6 Master VR preempted by higher priority
backup [test_vrrp.TestVRRP6.test_vrrp6_master_preempted]
      ERROR: IPv6 Backup VR preempts lower priority master
[test_vrrp.TestVRRP6.test_vrrp6_backup_preempts]
  Testcase name: Bidirectional Forwarding Detection (BFD)
(IPv6)
      ERROR: echo function [test_bfd.BFD6TestCase.test_echo]

==

vlib: startup multi-arch variant configuration
(https://gerrit.fd.io/r/c/vpp/+/25798_

https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2675/console.log.gz


==
TEST RESULTS:
     Scheduled tests: 22
      Executed tests: 22
        Passed tests: 21
            Failures: 1
FAILURES AND ERRORS IN TESTS:
  Testcase name: IPv4 VRRP Test Case
    FAILURE: IPv4 Master VR preempted by higher priority
backup [test_vrrp.TestVRRP4.test_vrrp4_master_preempted]

==








-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15763): https://lists.fd.io/g/vpp-dev/message/15763
Mute This Topic: https://lists.fd.io/mt/71901798/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  

Re: [vpp-dev] VRRP Unit Tests failing on Master Ubuntu

[Matthew Smith via Lists.Fd.Io]

Change submitted - https://gerrit.fd.io/r/c/vpp/+/25834. Verification jobs
are running. Hopefully they won't fail :)

-Matt


On Thu, Mar 12, 2020 at 10:22 AM Matthew Smith via Lists.Fd.Io  wrote:

>
> I don't have a solution yet, but one observation has popped up quickly
>
> In the 2 failed jobs Ray sent links for, one of them had a test fail which
> was not related to VRRP. There is a BFD6 test failure for the NAT change
> https://gerrit.fd.io/r/c/vpp/+/25462:
>
>
> https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2678/archives/
>
> Looking back through a couple of recent failed runs of that job, there is
> also a DHCP6 PD test failure for rdma change
> https://gerrit.fd.io/r/c/vpp/+/25823:
>
>
> https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2682/archives/
>
> The most obvious common thread between BFD6, DHCP6 and VRRP to me seems to
> be that they all maintain state which is dependent on timers. There could
> be a more general issue with timing-sensitive tests. I am going to submit a
> change which will prevent the VRRP tests from running temporarily while I
> can figure out a proper solution. Based on the above, other tests may need
> the same treatment.
>
> -Matt
>
>
>
>
> On Thu, Mar 12, 2020 at 8:57 AM Matthew Smith  wrote:
>
>> Hi Ray,
>>
>> Thanks for bringing it to my attention. I'll look into it.
>>
>> -Matt
>>
>>
>> On Thu, Mar 12, 2020 at 8:24 AM Ray Kinsella  wrote:
>>
>>> Anyone else noticing seeming spurious failures related to the VRRP
>>> plugin's unit tests.
>>> Some examples from un-related commits.
>>>
>>> Ray K
>>>
>>> nat: timed out session scavenging upgrade (
>>> https://gerrit.fd.io/r/c/vpp/+/25462)
>>>
>>> https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2678/console.log.gz
>>>
>>>
>>> ==
>>> TEST RESULTS:
>>>  Scheduled tests: 1138
>>>   Executed tests: 1138
>>> Passed tests: 1021
>>>Skipped tests: 112
>>> Failures: 3
>>>   Errors: 2
>>> FAILURES AND ERRORS IN TESTS:
>>>   Testcase name: IPv4 VRRP Test Case
>>> FAILURE: IPv4 Master VR does not reply for VIP w/ accept mode off
>>> [test_vrrp.TestVRRP4.test_vrrp4_accept_mode_disabled]
>>> FAILURE: IPv4 Master VR preempted by higher priority backup
>>> [test_vrrp.TestVRRP4.test_vrrp4_master_preempted]
>>>   Testcase name: IPv6 VRRP Test Case
>>> FAILURE: IPv6 Master VR preempted by higher priority backup
>>> [test_vrrp.TestVRRP6.test_vrrp6_master_preempted]
>>>   ERROR: IPv6 Backup VR preempts lower priority master
>>> [test_vrrp.TestVRRP6.test_vrrp6_backup_preempts]
>>>   Testcase name: Bidirectional Forwarding Detection (BFD) (IPv6)
>>>   ERROR: echo function [test_bfd.BFD6TestCase.test_echo]
>>>
>>> ==
>>>
>>> vlib: startup multi-arch variant configuration (
>>> https://gerrit.fd.io/r/c/vpp/+/25798_
>>>
>>> https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2675/console.log.gz
>>>
>>>
>>> ==
>>> TEST RESULTS:
>>>  Scheduled tests: 22
>>>   Executed tests: 22
>>> Passed tests: 21
>>> Failures: 1
>>> FAILURES AND ERRORS IN TESTS:
>>>   Testcase name: IPv4 VRRP Test Case
>>> FAILURE: IPv4 Master VR preempted by higher priority backup
>>> [test_vrrp.TestVRRP4.test_vrrp4_master_preempted]
>>>
>>> ==
>>>
>>>
>>>
>>> 
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15762): https://lists.fd.io/g/vpp-dev/message/15762
Mute This Topic: https://lists.fd.io/mt/71901798/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] VRRP Unit Tests failing on Master Ubuntu

[Matthew Smith via Lists.Fd.Io]

I don't have a solution yet, but one observation has popped up quickly

In the 2 failed jobs Ray sent links for, one of them had a test fail which
was not related to VRRP. There is a BFD6 test failure for the NAT change
https://gerrit.fd.io/r/c/vpp/+/25462:

https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2678/archives/

Looking back through a couple of recent failed runs of that job, there is
also a DHCP6 PD test failure for rdma change
https://gerrit.fd.io/r/c/vpp/+/25823:

https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2682/archives/

The most obvious common thread between BFD6, DHCP6 and VRRP to me seems to
be that they all maintain state which is dependent on timers. There could
be a more general issue with timing-sensitive tests. I am going to submit a
change which will prevent the VRRP tests from running temporarily while I
can figure out a proper solution. Based on the above, other tests may need
the same treatment.

-Matt




On Thu, Mar 12, 2020 at 8:57 AM Matthew Smith  wrote:

> Hi Ray,
>
> Thanks for bringing it to my attention. I'll look into it.
>
> -Matt
>
>
> On Thu, Mar 12, 2020 at 8:24 AM Ray Kinsella  wrote:
>
>> Anyone else noticing seeming spurious failures related to the VRRP
>> plugin's unit tests.
>> Some examples from un-related commits.
>>
>> Ray K
>>
>> nat: timed out session scavenging upgrade (
>> https://gerrit.fd.io/r/c/vpp/+/25462)
>>
>> https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2678/console.log.gz
>>
>>
>> ==
>> TEST RESULTS:
>>  Scheduled tests: 1138
>>   Executed tests: 1138
>> Passed tests: 1021
>>Skipped tests: 112
>> Failures: 3
>>   Errors: 2
>> FAILURES AND ERRORS IN TESTS:
>>   Testcase name: IPv4 VRRP Test Case
>> FAILURE: IPv4 Master VR does not reply for VIP w/ accept mode off
>> [test_vrrp.TestVRRP4.test_vrrp4_accept_mode_disabled]
>> FAILURE: IPv4 Master VR preempted by higher priority backup
>> [test_vrrp.TestVRRP4.test_vrrp4_master_preempted]
>>   Testcase name: IPv6 VRRP Test Case
>> FAILURE: IPv6 Master VR preempted by higher priority backup
>> [test_vrrp.TestVRRP6.test_vrrp6_master_preempted]
>>   ERROR: IPv6 Backup VR preempts lower priority master
>> [test_vrrp.TestVRRP6.test_vrrp6_backup_preempts]
>>   Testcase name: Bidirectional Forwarding Detection (BFD) (IPv6)
>>   ERROR: echo function [test_bfd.BFD6TestCase.test_echo]
>>
>> ==
>>
>> vlib: startup multi-arch variant configuration (
>> https://gerrit.fd.io/r/c/vpp/+/25798_
>>
>> https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2675/console.log.gz
>>
>>
>> ==
>> TEST RESULTS:
>>  Scheduled tests: 22
>>   Executed tests: 22
>> Passed tests: 21
>> Failures: 1
>> FAILURES AND ERRORS IN TESTS:
>>   Testcase name: IPv4 VRRP Test Case
>> FAILURE: IPv4 Master VR preempted by higher priority backup
>> [test_vrrp.TestVRRP4.test_vrrp4_master_preempted]
>>
>> ==
>>
>>
>>
>>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15761): https://lists.fd.io/g/vpp-dev/message/15761
Mute This Topic: https://lists.fd.io/mt/71901798/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] VRRP Unit Tests failing on Master Ubuntu

[Paul Vinciguerra]

Yes.
Has been for a few days.

On Thu, Mar 12, 2020 at 9:25 AM Ray Kinsella  wrote:

> Anyone else noticing seeming spurious failures related to the VRRP
> plugin's unit tests.
> Some examples from un-related commits.
>
> Ray K
>
> nat: timed out session scavenging upgrade (
> https://gerrit.fd.io/r/c/vpp/+/25462)
>
> https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2678/console.log.gz
>
>
> ==
> TEST RESULTS:
>  Scheduled tests: 1138
>   Executed tests: 1138
> Passed tests: 1021
>Skipped tests: 112
> Failures: 3
>   Errors: 2
> FAILURES AND ERRORS IN TESTS:
>   Testcase name: IPv4 VRRP Test Case
> FAILURE: IPv4 Master VR does not reply for VIP w/ accept mode off
> [test_vrrp.TestVRRP4.test_vrrp4_accept_mode_disabled]
> FAILURE: IPv4 Master VR preempted by higher priority backup
> [test_vrrp.TestVRRP4.test_vrrp4_master_preempted]
>   Testcase name: IPv6 VRRP Test Case
> FAILURE: IPv6 Master VR preempted by higher priority backup
> [test_vrrp.TestVRRP6.test_vrrp6_master_preempted]
>   ERROR: IPv6 Backup VR preempts lower priority master
> [test_vrrp.TestVRRP6.test_vrrp6_backup_preempts]
>   Testcase name: Bidirectional Forwarding Detection (BFD) (IPv6)
>   ERROR: echo function [test_bfd.BFD6TestCase.test_echo]
>
> ==
>
> vlib: startup multi-arch variant configuration (
> https://gerrit.fd.io/r/c/vpp/+/25798_
>
> https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2675/console.log.gz
>
>
> ==
> TEST RESULTS:
>  Scheduled tests: 22
>   Executed tests: 22
> Passed tests: 21
> Failures: 1
> FAILURES AND ERRORS IN TESTS:
>   Testcase name: IPv4 VRRP Test Case
> FAILURE: IPv4 Master VR preempted by higher priority backup
> [test_vrrp.TestVRRP4.test_vrrp4_master_preempted]
>
> ==
>
>
>
> 
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15757): https://lists.fd.io/g/vpp-dev/message/15757
Mute This Topic: https://lists.fd.io/mt/71901798/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] VRRP Unit Tests failing on Master Ubuntu

[Matthew Smith via Lists.Fd.Io]

Hi Ray,

Thanks for bringing it to my attention. I'll look into it.

-Matt


On Thu, Mar 12, 2020 at 8:24 AM Ray Kinsella  wrote:

> Anyone else noticing seeming spurious failures related to the VRRP
> plugin's unit tests.
> Some examples from un-related commits.
>
> Ray K
>
> nat: timed out session scavenging upgrade (
> https://gerrit.fd.io/r/c/vpp/+/25462)
>
> https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2678/console.log.gz
>
>
> ==
> TEST RESULTS:
>  Scheduled tests: 1138
>   Executed tests: 1138
> Passed tests: 1021
>Skipped tests: 112
> Failures: 3
>   Errors: 2
> FAILURES AND ERRORS IN TESTS:
>   Testcase name: IPv4 VRRP Test Case
> FAILURE: IPv4 Master VR does not reply for VIP w/ accept mode off
> [test_vrrp.TestVRRP4.test_vrrp4_accept_mode_disabled]
> FAILURE: IPv4 Master VR preempted by higher priority backup
> [test_vrrp.TestVRRP4.test_vrrp4_master_preempted]
>   Testcase name: IPv6 VRRP Test Case
> FAILURE: IPv6 Master VR preempted by higher priority backup
> [test_vrrp.TestVRRP6.test_vrrp6_master_preempted]
>   ERROR: IPv6 Backup VR preempts lower priority master
> [test_vrrp.TestVRRP6.test_vrrp6_backup_preempts]
>   Testcase name: Bidirectional Forwarding Detection (BFD) (IPv6)
>   ERROR: echo function [test_bfd.BFD6TestCase.test_echo]
>
> ==
>
> vlib: startup multi-arch variant configuration (
> https://gerrit.fd.io/r/c/vpp/+/25798_
>
> https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2675/console.log.gz
>
>
> ==
> TEST RESULTS:
>  Scheduled tests: 22
>   Executed tests: 22
> Passed tests: 21
> Failures: 1
> FAILURES AND ERRORS IN TESTS:
>   Testcase name: IPv4 VRRP Test Case
> FAILURE: IPv4 Master VR preempted by higher priority backup
> [test_vrrp.TestVRRP4.test_vrrp4_master_preempted]
>
> ==
>
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15754): https://lists.fd.io/g/vpp-dev/message/15754
Mute This Topic: https://lists.fd.io/mt/71901798/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] VRRP Unit Tests failing on Master Ubuntu

[Ray Kinsella]

Anyone else noticing seeming spurious failures related to the VRRP plugin's 
unit tests.
Some examples from un-related commits.

Ray K

nat: timed out session scavenging upgrade (https://gerrit.fd.io/r/c/vpp/+/25462)
https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2678/console.log.gz

==
TEST RESULTS:
 Scheduled tests: 1138
  Executed tests: 1138
Passed tests: 1021
   Skipped tests: 112
Failures: 3
  Errors: 2
FAILURES AND ERRORS IN TESTS:
  Testcase name: IPv4 VRRP Test Case 
FAILURE: IPv4 Master VR does not reply for VIP w/ accept mode off 
[test_vrrp.TestVRRP4.test_vrrp4_accept_mode_disabled]
FAILURE: IPv4 Master VR preempted by higher priority backup 
[test_vrrp.TestVRRP4.test_vrrp4_master_preempted]
  Testcase name: IPv6 VRRP Test Case 
FAILURE: IPv6 Master VR preempted by higher priority backup 
[test_vrrp.TestVRRP6.test_vrrp6_master_preempted]
  ERROR: IPv6 Backup VR preempts lower priority master 
[test_vrrp.TestVRRP6.test_vrrp6_backup_preempts]
  Testcase name: Bidirectional Forwarding Detection (BFD) (IPv6) 
  ERROR: echo function [test_bfd.BFD6TestCase.test_echo]
==

vlib: startup multi-arch variant configuration 
(https://gerrit.fd.io/r/c/vpp/+/25798_
https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1804/2675/console.log.gz

==
TEST RESULTS:
 Scheduled tests: 22
  Executed tests: 22
Passed tests: 21
Failures: 1
FAILURES AND ERRORS IN TESTS:
  Testcase name: IPv4 VRRP Test Case 
FAILURE: IPv4 Master VR preempted by higher priority backup 
[test_vrrp.TestVRRP4.test_vrrp4_master_preempted]
==





signature.asc
Description: OpenPGP digital signature
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15751): https://lists.fd.io/g/vpp-dev/message/15751
Mute This Topic: https://lists.fd.io/mt/71901798/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-