Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
Both aarch64 & x86_64 jobs completed successfully: https://jenkins.fd.io/sandbox/job/daw_30773-vpp-verify-master-ubuntu2004-aarch64/5/ https://jenkins.fd.io/sandbox/job/daw_30773-vpp-verify-master-ubuntu2004-x86_64/4/ Thanks, -daw- On 2/12/2021 8:50 AM, Dave Wallace via lists.fd.io wrote: Great! I will retest with the latest changes and report back. Thanks, -daw- On 2/12/2021 8:41 AM, Benoit Ganne (bganne) wrote: Hi Dave, Your patch fixed the issue [0] (x86_64 passes too)! Great! I updated it with more fixes and debug checks:https://gerrit.fd.io/r/c/vpp/+/31240 and abandoned the change 31241 (which I have merged with 31240 instead). Best ben Thanks, -daw- [0]https://jenkins.fd.io/sandbox/job/daw_30773-vpp-verify-master- ubuntu2004-aarch64/4/ https://jenkins.fd.io/sandbox/job/daw_30773-vpp-verify-master- ubuntu2004-x86_64/3/ -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18741): https://lists.fd.io/g/vpp-dev/message/18741 Mute This Topic: https://lists.fd.io/mt/80495347/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
Great! I will retest with the latest changes and report back. Thanks, -daw- On 2/12/2021 8:41 AM, Benoit Ganne (bganne) wrote: Hi Dave, Your patch fixed the issue [0] (x86_64 passes too)! Great! I updated it with more fixes and debug checks: https://gerrit.fd.io/r/c/vpp/+/31240 and abandoned the change 31241 (which I have merged with 31240 instead). Best ben Thanks, -daw- [0] https://jenkins.fd.io/sandbox/job/daw_30773-vpp-verify-master- ubuntu2004-aarch64/4/ https://jenkins.fd.io/sandbox/job/daw_30773-vpp-verify-master- ubuntu2004-x86_64/3/ -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18740): https://lists.fd.io/g/vpp-dev/message/18740 Mute This Topic: https://lists.fd.io/mt/80495347/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
Hi Dave, > Your patch fixed the issue [0] (x86_64 passes too)! Great! I updated it with more fixes and debug checks: https://gerrit.fd.io/r/c/vpp/+/31240 and abandoned the change 31241 (which I have merged with 31240 instead). Best ben > Thanks, > -daw- > > [0] https://jenkins.fd.io/sandbox/job/daw_30773-vpp-verify-master- > ubuntu2004-aarch64/4/ > https://jenkins.fd.io/sandbox/job/daw_30773-vpp-verify-master- > ubuntu2004-x86_64/3/ > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18739): https://lists.fd.io/g/vpp-dev/message/18739 Mute This Topic: https://lists.fd.io/mt/80495347/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
Ben, Your patch fixed the issue [0] (x86_64 passes too)! Thanks, -daw- [0] https://jenkins.fd.io/sandbox/job/daw_30773-vpp-verify-master-ubuntu2004-aarch64/4/ https://jenkins.fd.io/sandbox/job/daw_30773-vpp-verify-master-ubuntu2004-x86_64/3/ On 2/11/2021 3:06 PM, Dave Wallace via lists.fd.io wrote: Ben, Excellent debugging work! I will rebase my gerrit change onto yours, retest in the Jenkins Sandbox, and report back. Thanks, -daw- On 2/11/21 2:18 PM, Benoit Ganne (bganne) wrote: Hi Dave, There are still 6 ikev2 test failures from this test run that need to be resolved. I identified the issue: undefined behaviour + aggressive gcc optimization. The tentative fix is here:https://gerrit.fd.io/r/c/vpp/+/31240 Note that for this to happen, you need both: - compile with gcc - use gcc memcpy builtin This is the case for ARM64 but not x86, as we do not use gcc memcpy builtin on x86. It looks like there are other cases where we call memcpy with NULL pointers:https://gerrit.fd.io/r/c/vpp/+/31241 I am going to chase them down. Best ben -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18738): https://lists.fd.io/g/vpp-dev/message/18738 Mute This Topic: https://lists.fd.io/mt/80495347/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
Ben, Excellent debugging work! I will rebase my gerrit change onto yours, retest in the Jenkins Sandbox, and report back. Thanks, -daw- On 2/11/21 2:18 PM, Benoit Ganne (bganne) wrote: Hi Dave, There are still 6 ikev2 test failures from this test run that need to be resolved. I identified the issue: undefined behaviour + aggressive gcc optimization. The tentative fix is here: https://gerrit.fd.io/r/c/vpp/+/31240 Note that for this to happen, you need both: - compile with gcc - use gcc memcpy builtin This is the case for ARM64 but not x86, as we do not use gcc memcpy builtin on x86. It looks like there are other cases where we call memcpy with NULL pointers: https://gerrit.fd.io/r/c/vpp/+/31241 I am going to chase them down. Best ben -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18734): https://lists.fd.io/g/vpp-dev/message/18734 Mute This Topic: https://lists.fd.io/mt/80495347/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
Wow, this is a great catch, Ben! Kudos! --a > On 11 Feb 2021, at 20:18, Benoit Ganne (bganne) via lists.fd.io > wrote: > > Hi Dave, > >> There are still 6 ikev2 test failures from this test run that need to be >> resolved. > > I identified the issue: undefined behaviour + aggressive gcc optimization. > The tentative fix is here: https://gerrit.fd.io/r/c/vpp/+/31240 > > Note that for this to happen, you need both: > - compile with gcc > - use gcc memcpy builtin > This is the case for ARM64 but not x86, as we do not use gcc memcpy builtin > on x86. > > It looks like there are other cases where we call memcpy with NULL pointers: > https://gerrit.fd.io/r/c/vpp/+/31241 > I am going to chase them down. > > Best > ben > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18733): https://lists.fd.io/g/vpp-dev/message/18733 Mute This Topic: https://lists.fd.io/mt/80495347/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
Hi Dave, > There are still 6 ikev2 test failures from this test run that need to be > resolved. I identified the issue: undefined behaviour + aggressive gcc optimization. The tentative fix is here: https://gerrit.fd.io/r/c/vpp/+/31240 Note that for this to happen, you need both: - compile with gcc - use gcc memcpy builtin This is the case for ARM64 but not x86, as we do not use gcc memcpy builtin on x86. It looks like there are other cases where we call memcpy with NULL pointers: https://gerrit.fd.io/r/c/vpp/+/31241 I am going to chase them down. Best ben -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18732): https://lists.fd.io/g/vpp-dev/message/18732 Mute This Topic: https://lists.fd.io/mt/80495347/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
Excellent! Thanks Ben :)
-daw-
On 2/11/2021 11:06 AM, Benoit Ganne (bganne) wrote:
Hi Dave,
I am working on the ikev2 crashes, I think I identified the issue, stay tuned.
ben
-Original Message-
From: Dave Wallace
Sent: mercredi 10 février 2021 22:49
To: v...@barachs.net; 'Paul Vinciguerra'
Cc: Benoit Ganne (bganne) ; 'Ole Troan'
; 'vpp-dev' ; 'Juraj Linkeš'
Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
Verified that the fix resolves the Bihash test failure (rebased 30734):
https://logs.fd.io/sandbox/vex-yul-rot-jenkins-2/daw_30773-vpp-verify-
master-ubuntu2004-aarch64/2/
There are still 6 ikev2 test failures from this test run that need to be
resolved. These all have core file stack traces available (see
core.traceback.gz in the test directories. All of these tests appear to
fail with this in the stacktrace:
#5 0x699683dc in ikev2_payload_add_data (data=data@entry=0x0,
c=, c=) at /w/workspace/daw_30773-vpp-
verify-master-ubuntu2004-aarch64/src/plugins/ikev2/ikev2_payload.c:136
Thanks,
-daw-
On 2/10/21 10:18 AM, Dave Wallace via lists.fd.io wrote:
Thanks Dave!
-daw-
On 2/10/2021 7:54 AM, v...@barachs.net <mailto:v...@barachs.net>
wrote:
To make the pain go away, I quadrupled the vapi timeout for
the bihash test vectors. Turns out that a couple of tests run for almost 5
seconds (on aarch64, debug) which caused sporadic failures.
That’s probably about the right size hammer for this specific
problem.
FWIW... Dave
From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> <mailto:vpp-dev@lists.fd.io> On Behalf Of Paul
Vinciguerra
Sent: Tuesday, February 9, 2021 1:31 PM
To: Dave Barach <mailto:v...@barachs.net>
Cc: Benoit Ganne (bganne)
<mailto:bga...@cisco.com> ; Ole Troan
<mailto:otr...@employees.org> ; Dave Wallace
<mailto:dwallac...@gmail.com> ; vpp-dev <mailto:vpp-
d...@lists.fd.io> ; Juraj Linkeš
<mailto:juraj.lin...@pantheon.tech>
Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04
AARCH64
Hi Dave,
test_cli.py verifies the timeout checking code is working.
The groundwork to clean up the framework can be found here.
[0]
The test framework cannot distinguish between a timeout and a
more serious problem, so for now we assume vpp has died.
For test, I would have liked to do something similar to what
you do with growing your vectors. If it is a true Timeout, rerun the test
with say 3/2 the failing value and output the timeout with the run.
We could have the testcases use different values for arm, but
when load or hw specs change, the tests will break again.
Paul
[0] https://gerrit.fd.io/r/c/vpp/+/24085
On Tue, Feb 9, 2021 at 8:37 AM Dave Barach mailto:v...@barachs.net> > wrote:
On aarch64, "make TEST=test_bihash test-debug" involves
a debug CLI timeout which is right on the hairy edge of being too short.
05:29:45,470 Calling cli_inband('cmd':'test bihash
threads 2 nbuckets 64000 careful 0 verbose
0\n','context':2,'_vl_msg_id':574)
05:29:45,471 TIMEOUT:: 5
This seems like the right incantation to fix it in
src/vppinfra/test/test_bihash.py, but apparently it is not:
error = self.vapi.cli("test bihash threads 2
nbuckets" +
" 64000 careful 0 verbose
0", timeout=15)
Please advise... Thanks... Dave
-Original Message-
From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>
mailto:vpp-dev@lists.fd.io> > On Behalf Of Benoit
Ganne (bganne) via lists.fd.io <http://lists.fd.io>
Sent: Tuesday, February 9, 2021 8:06 AM
To: Dave Barach mailto:v...@barachs.net> >; 'Ole Troan' mailto:otr...@employees.org> >; 'Dave Wallace' mailto:dwallac...@gmail.com> >; vpp-dev@lists.fd.io <mailto:vpp-
d...@lists.fd.io>
Cc: 'Juraj Linkeš' mailto:juraj.lin...@pantheon.tech> >
Subject: Re: [vpp-dev] Make test failures on ubuntu-
20.04 AARCH64
I am going to have a look at IKE.
In case it is useful to others, thanks to QEMU you can
run a Ubuntu 20.04 ARM docker container on your x86 workstation:
# step 0: add support for multiarch (must be done once
after reboot) docker run --rm --privileged multiarch/qemu-user-static --
reset --persistent yes --credentia
Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
Hi Dave,
I am working on the ikev2 crashes, I think I identified the issue, stay tuned.
ben
> -Original Message-
> From: Dave Wallace
> Sent: mercredi 10 février 2021 22:49
> To: v...@barachs.net; 'Paul Vinciguerra'
> Cc: Benoit Ganne (bganne) ; 'Ole Troan'
> ; 'vpp-dev' ; 'Juraj Linkeš'
>
> Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
>
> Verified that the fix resolves the Bihash test failure (rebased 30734):
>
> https://logs.fd.io/sandbox/vex-yul-rot-jenkins-2/daw_30773-vpp-verify-
> master-ubuntu2004-aarch64/2/
>
> There are still 6 ikev2 test failures from this test run that need to be
> resolved. These all have core file stack traces available (see
> core.traceback.gz in the test directories. All of these tests appear to
> fail with this in the stacktrace:
>
> #5 0x699683dc in ikev2_payload_add_data (data=data@entry=0x0,
> c=, c=) at /w/workspace/daw_30773-vpp-
> verify-master-ubuntu2004-aarch64/src/plugins/ikev2/ikev2_payload.c:136
>
> Thanks,
> -daw-
>
>
>
> On 2/10/21 10:18 AM, Dave Wallace via lists.fd.io wrote:
>
>
> Thanks Dave!
> -daw-
>
>
> On 2/10/2021 7:54 AM, v...@barachs.net <mailto:v...@barachs.net>
> wrote:
>
>
> To make the pain go away, I quadrupled the vapi timeout for
> the bihash test vectors. Turns out that a couple of tests run for almost 5
> seconds (on aarch64, debug) which caused sporadic failures.
>
>
>
> That’s probably about the right size hammer for this specific
> problem.
>
>
>
> FWIW... Dave
>
>
>
> From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> d...@lists.fd.io> <mailto:vpp-dev@lists.fd.io> On Behalf Of Paul
> Vinciguerra
> Sent: Tuesday, February 9, 2021 1:31 PM
> To: Dave Barach <mailto:v...@barachs.net>
> Cc: Benoit Ganne (bganne)
> <mailto:bga...@cisco.com> ; Ole Troan
> <mailto:otr...@employees.org> ; Dave Wallace
> <mailto:dwallac...@gmail.com> ; vpp-dev <mailto:vpp-
> d...@lists.fd.io> ; Juraj Linkeš
> <mailto:juraj.lin...@pantheon.tech>
> Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04
> AARCH64
>
>
>
> Hi Dave,
>
>
>
> test_cli.py verifies the timeout checking code is working.
>
>
>
> The groundwork to clean up the framework can be found here.
> [0]
>
>
>
> The test framework cannot distinguish between a timeout and a
> more serious problem, so for now we assume vpp has died.
>
>
>
> For test, I would have liked to do something similar to what
> you do with growing your vectors. If it is a true Timeout, rerun the test
> with say 3/2 the failing value and output the timeout with the run.
>
>
>
> We could have the testcases use different values for arm, but
> when load or hw specs change, the tests will break again.
>
>
>
> Paul
>
>
>
> [0] https://gerrit.fd.io/r/c/vpp/+/24085
>
>
>
>
>
>
>
> On Tue, Feb 9, 2021 at 8:37 AM Dave Barach <mailto:v...@barachs.net> > wrote:
>
> On aarch64, "make TEST=test_bihash test-debug" involves
> a debug CLI timeout which is right on the hairy edge of being too short.
>
>
>
> 05:29:45,470 Calling cli_inband('cmd':'test bihash
> threads 2 nbuckets 64000 careful 0 verbose
> 0\n','context':2,'_vl_msg_id':574)
>
> 05:29:45,471 TIMEOUT:: 5
>
>
>
> This seems like the right incantation to fix it in
> src/vppinfra/test/test_bihash.py, but apparently it is not:
>
>
>
> error = self.vapi.cli("test bihash threads 2
> nbuckets" +
>
> " 64000 careful 0 verbose
> 0", timeout=15)
>
>
>
> Please advise... Thanks... Dave
>
>
>
> -Original Message-
> From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>
> mailto:vpp-dev@lists.fd.io> > On Behalf Of Benoit
> Ganne (bganne) via lists.fd.io <http://lists.fd.io>
> Sent: Tuesday, February 9, 2021 8:06 AM
> To: Dave Barach <mailto:v...@barachs.net> >; 'Ole Troan' <mailto:otr...@employees.org> >; 'Dave Wallace' <mailto:dwallac...@gmail.com> >
Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
Merged. Thanks Dave! :-)
--a
> On 10 Feb 2021, at 16:17, Dave Wallace wrote:
>
> Thanks Dave!
> -daw-
>
> On 2/10/2021 7:54 AM, v...@barachs.net wrote:
>> To make the pain go away, I quadrupled the vapi timeout for the bihash test
>> vectors. Turns out that a couple of tests run for almost 5 seconds (on
>> aarch64, debug) which caused sporadic failures.
>>
>> That’s probably about the right size hammer for this specific problem.
>>
>> FWIW... Dave
>>
>> From: vpp-dev@lists.fd.io On Behalf Of Paul Vinciguerra
>> Sent: Tuesday, February 9, 2021 1:31 PM
>> To: Dave Barach
>> Cc: Benoit Ganne (bganne) ; Ole Troan
>> ; Dave Wallace ; vpp-dev
>> ; Juraj Linkeš
>> Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
>>
>> Hi Dave,
>>
>> test_cli.py verifies the timeout checking code is working.
>>
>> The groundwork to clean up the framework can be found here. [0]
>>
>> The test framework cannot distinguish between a timeout and a more serious
>> problem, so for now we assume vpp has died.
>>
>> For test, I would have liked to do something similar to what you do with
>> growing your vectors. If it is a true Timeout, rerun the test with say 3/2
>> the failing value and output the timeout with the run.
>>
>> We could have the testcases use different values for arm, but when load or
>> hw specs change, the tests will break again.
>>
>> Paul
>>
>> [0] https://gerrit.fd.io/r/c/vpp/+/24085
>>
>>
>>
>> On Tue, Feb 9, 2021 at 8:37 AM Dave Barach wrote:
>> On aarch64, "make TEST=test_bihash test-debug" involves a debug CLI timeout
>> which is right on the hairy edge of being too short.
>>
>>
>>
>> 05:29:45,470 Calling cli_inband('cmd':'test bihash threads 2 nbuckets 64000
>> careful 0 verbose 0\n','context':2,'_vl_msg_id':574)
>>
>> 05:29:45,471 TIMEOUT:: 5
>>
>>
>>
>> This seems like the right incantation to fix it in
>> src/vppinfra/test/test_bihash.py, but apparently it is not:
>>
>>
>>
>> error = self.vapi.cli("test bihash threads 2 nbuckets" +
>>
>> " 64000 careful 0 verbose 0", timeout=15)
>>
>>
>>
>> Please advise... Thanks... Dave
>>
>>
>>
>> -Original Message-
>> From: vpp-dev@lists.fd.io On Behalf Of Benoit Ganne
>> (bganne) via lists.fd.io
>> Sent: Tuesday, February 9, 2021 8:06 AM
>> To: Dave Barach ; 'Ole Troan' ;
>> 'Dave Wallace' ; vpp-dev@lists.fd.io
>> Cc: 'Juraj Linkeš'
>> Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
>>
>>
>>
>> I am going to have a look at IKE.
>>
>>
>>
>> In case it is useful to others, thanks to QEMU you can run a Ubuntu 20.04
>> ARM docker container on your x86 workstation:
>>
>>
>>
>> # step 0: add support for multiarch (must be done once after reboot) docker
>> run --rm --privileged multiarch/qemu-user-static --reset --persistent yes
>> --credential yes
>>
>>
>>
>> # step 1: create your chroot (must be done once - I am sharing my homedir
>> with my chroot and same UID/GID) docker run --name aarch64_u2004
>> --privileged --net host -v $HOME:$HOME -v /dev:/dev -td arm64v8/ubuntu:20.04
>> /bin/bash
>>
>>
>>
>> # step 2: update and add user inside container docker container exec
>> aarch64_u2004 sh -c "apt -qy update && apt dist-upgrade -qy && apt install
>> -qy vim sudo make git && groupadd -g $(id -rg) $USER && useradd -u $(id -ru)
>> -g $(id -rg) -M -d $HOME -s /bin/bash $USER && echo '$USER ALL=(ALL)
>> NOPASSWD:ALL' >> /etc/sudoers && echo aarch64_u2004 > /etc/debian_chroot"
>>
>>
>>
>> # step 3: log into ARM chroot
>>
>> docker container exec -it aarch64_deb10 /bin/login -f $USER
>>
>>
>>
>> Best
>>
>> ben
>>
>>
>>
>> > -Original Message-
>>
>> > From: vpp-dev@lists.fd.io On Behalf Of Dave
>>
>> > Barach
>>
>> > Sent: mardi 9 février 2021 12:47
>>
>> > To: 'Ole Troan' ; 'Dave Wallace'
>>
>> > ; vpp-dev@lists.fd.io
>>
>> > Cc: 'Juraj Linkeš'
>>
>> > Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
>>
>> >
>>
>> > Would be glad to take a
Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
Thanks Dave!
-daw-
On 2/10/2021 7:54 AM, v...@barachs.net wrote:
To make the pain go away, I quadrupled the vapi timeout for the bihash
test vectors. Turns out that a couple of tests run for /almost/ 5
seconds (on aarch64, debug) which caused sporadic failures.
That’s probably about the right size hammer for this specific problem.
FWIW... Dave
*From:* vpp-dev@lists.fd.io *On Behalf Of *Paul
Vinciguerra
*Sent:* Tuesday, February 9, 2021 1:31 PM
*To:* Dave Barach
*Cc:* Benoit Ganne (bganne) ; Ole Troan
; Dave Wallace ; vpp-dev
; Juraj Linkeš
*Subject:* Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
Hi Dave,
test_cli.py verifies the timeout checking code is working.
The groundwork to clean up the framework can be found here. [0]
The test framework cannot distinguish between a timeout and a more
serious problem, so for now we assume vpp has died.
For test, I would have liked to do something similar to what you do
with growing your vectors. If it is a true Timeout, rerun the test
with say 3/2 the failing value and output the timeout with the run.
We could have the testcases use different values for arm, but when
load or hw specs change, the tests will break again.
Paul
[0] https://gerrit.fd.io/r/c/vpp/+/24085
<https://gerrit.fd.io/r/c/vpp/+/24085>
On Tue, Feb 9, 2021 at 8:37 AM Dave Barach <mailto:v...@barachs.net>> wrote:
On aarch64, "make TEST=test_bihash test-debug" involves a debug
CLI timeout which is right on the hairy edge of being too short.
05:29:45,470 Calling cli_inband('cmd':'test bihash threads 2
nbuckets 64000 careful 0 verbose 0\n','context':2,'_vl_msg_id':574)
05:29:45,471 TIMEOUT:: 5
This seems like the right incantation to fix it in
src/vppinfra/test/test_bihash.py, but apparently it is not:
error = self.vapi.cli("test bihash threads 2 nbuckets" +
" 64000 careful 0 verbose 0", timeout=15)
Please advise... Thanks... Dave
-Original Message-
From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>
mailto:vpp-dev@lists.fd.io>> On Behalf Of
Benoit Ganne (bganne) via lists.fd.io <http://lists.fd.io>
Sent: Tuesday, February 9, 2021 8:06 AM
To: Dave Barach mailto:v...@barachs.net>>; 'Ole
Troan' mailto:otr...@employees.org>>; 'Dave
Wallace' mailto:dwallac...@gmail.com>>;
vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>
Cc: 'Juraj Linkeš' mailto:juraj.lin...@pantheon.tech>>
Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
I am going to have a look at IKE.
In case it is useful to others, thanks to QEMU you can run a
Ubuntu 20.04 ARM docker container on your x86 workstation:
# step 0: add support for multiarch (must be done once after
reboot) docker run --rm --privileged multiarch/qemu-user-static
--reset --persistent yes --credential yes
# step 1: create your chroot (must be done once - I am sharing my
homedir with my chroot and same UID/GID) docker run --name
aarch64_u2004 --privileged --net host -v $HOME:$HOME -v /dev:/dev
-td arm64v8/ubuntu:20.04 /bin/bash
# step 2: update and add user inside container docker container
exec aarch64_u2004 sh -c "apt -qy update && apt dist-upgrade -qy
&& apt install -qy vim sudo make git && groupadd -g $(id -rg)
$USER && useradd -u $(id -ru) -g $(id -rg) -M -d $HOME -s
/bin/bash $USER && echo '$USER ALL=(ALL) NOPASSWD:ALL' >>
/etc/sudoers && echo aarch64_u2004 > /etc/debian_chroot"
# step 3: log into ARM chroot
docker container exec -it aarch64_deb10 /bin/login -f $USER
Best
ben
> -Original Message-
> From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>
mailto:vpp-dev@lists.fd.io>> On Behalf Of Dave
> Barach
> Sent: mardi 9 février 2021 12:47
> To: 'Ole Troan' mailto:otr...@employees.org>>; 'Dave Wallace'
> mailto:dwallac...@gmail.com>>;
vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>
> Cc: 'Juraj Linkeš' mailto:juraj.lin...@pantheon.tech>>
> Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
>
> Would be glad to take a look at the aarch64 bihash failure,
modulo the
> fact that the LF data center resources I've used before seem to
have
> had their credentials changed.
>
> Please unicast appropriate (ip, userid, password) tuples.
>
> Thanks... Dave
>
> ssh vppdev@10.30.51.65 <mailto:vppdev@10.30.51.65>
> Warning: Permanently added '10.30.51.65' (ECDSA) to the list of
known
> hosts.
> vppdev@10.30.51.65's <mailto:vppdev@10.30.51.65's> password:
> Permission denied, please try again.
Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
To make the pain go away, I quadrupled the vapi timeout for the bihash test
vectors. Turns out that a couple of tests run for almost 5 seconds (on aarch64,
debug) which caused sporadic failures.
That’s probably about the right size hammer for this specific problem.
FWIW... Dave
From: vpp-dev@lists.fd.io On Behalf Of Paul Vinciguerra
Sent: Tuesday, February 9, 2021 1:31 PM
To: Dave Barach
Cc: Benoit Ganne (bganne) ; Ole Troan ;
Dave Wallace ; vpp-dev ; Juraj
Linkeš
Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
Hi Dave,
test_cli.py verifies the timeout checking code is working.
The groundwork to clean up the framework can be found here. [0]
The test framework cannot distinguish between a timeout and a more serious
problem, so for now we assume vpp has died.
For test, I would have liked to do something similar to what you do with
growing your vectors. If it is a true Timeout, rerun the test with say 3/2 the
failing value and output the timeout with the run.
We could have the testcases use different values for arm, but when load or hw
specs change, the tests will break again.
Paul
[0] https://gerrit.fd.io/r/c/vpp/+/24085
On Tue, Feb 9, 2021 at 8:37 AM Dave Barach mailto:v...@barachs.net> > wrote:
On aarch64, "make TEST=test_bihash test-debug" involves a debug CLI timeout
which is right on the hairy edge of being too short.
05:29:45,470 Calling cli_inband('cmd':'test bihash threads 2 nbuckets 64000
careful 0 verbose 0\n','context':2,'_vl_msg_id':574)
05:29:45,471 TIMEOUT:: 5
This seems like the right incantation to fix it in
src/vppinfra/test/test_bihash.py, but apparently it is not:
error = self.vapi.cli("test bihash threads 2 nbuckets" +
" 64000 careful 0 verbose 0", timeout=15)
Please advise... Thanks... Dave
-Original Message-
From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io> > On Behalf Of Benoit Ganne (bganne) via
lists.fd.io <http://lists.fd.io>
Sent: Tuesday, February 9, 2021 8:06 AM
To: Dave Barach mailto:v...@barachs.net> >; 'Ole Troan'
mailto:otr...@employees.org> >; 'Dave Wallace'
mailto:dwallac...@gmail.com> >; vpp-dev@lists.fd.io
<mailto:vpp-dev@lists.fd.io>
Cc: 'Juraj Linkeš' mailto:juraj.lin...@pantheon.tech> >
Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
I am going to have a look at IKE.
In case it is useful to others, thanks to QEMU you can run a Ubuntu 20.04 ARM
docker container on your x86 workstation:
# step 0: add support for multiarch (must be done once after reboot) docker run
--rm --privileged multiarch/qemu-user-static --reset --persistent yes
--credential yes
# step 1: create your chroot (must be done once - I am sharing my homedir with
my chroot and same UID/GID) docker run --name aarch64_u2004 --privileged --net
host -v $HOME:$HOME -v /dev:/dev -td arm64v8/ubuntu:20.04 /bin/bash
# step 2: update and add user inside container docker container exec
aarch64_u2004 sh -c "apt -qy update && apt dist-upgrade -qy && apt install -qy
vim sudo make git && groupadd -g $(id -rg) $USER && useradd -u $(id -ru) -g
$(id -rg) -M -d $HOME -s /bin/bash $USER && echo '$USER ALL=(ALL) NOPASSWD:ALL'
>> /etc/sudoers && echo aarch64_u2004 > /etc/debian_chroot"
# step 3: log into ARM chroot
docker container exec -it aarch64_deb10 /bin/login -f $USER
Best
ben
> -Original Message-
> From: <mailto:vpp-dev@lists.fd.io> vpp-dev@lists.fd.io <
> <mailto:vpp-dev@lists.fd.io> vpp-dev@lists.fd.io> On Behalf Of Dave
> Barach
> Sent: mardi 9 février 2021 12:47
> To: 'Ole Troan' < <mailto:otr...@employees.org> otr...@employees.org>; 'Dave
> Wallace'
> < <mailto:dwallac...@gmail.com> dwallac...@gmail.com>;
> <mailto:vpp-dev@lists.fd.io> vpp-dev@lists.fd.io
> Cc: 'Juraj Linkeš' < <mailto:juraj.lin...@pantheon.tech>
> juraj.lin...@pantheon.tech>
> Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
>
> Would be glad to take a look at the aarch64 bihash failure, modulo the
> fact that the LF data center resources I've used before seem to have
> had their credentials changed.
>
> Please unicast appropriate (ip, userid, password) tuples.
>
> Thanks... Dave
>
> ssh <mailto:vppdev@10.30.51.65> vppdev@10.30.51.65
> Warning: Permanently added '10.30.51.65' (ECDSA) to the list of known
> hosts.
> <mailto:vppdev@10.30.51.65's> vppdev@10.30.51.65's password:
> Permission denied, please try again.
> <mailto:vppdev@10.30.51.65's> vppdev@10.30.51.65's password:
> Permission denied, please try again.
Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
Hi Dave,
test_cli.py verifies the timeout checking code is working.
The groundwork to clean up the framework can be found here. [0]
The test framework cannot distinguish between a timeout and a more
serious problem, so for now we assume vpp has died.
For test, I would have liked to do something similar to what you do with
growing your vectors. If it is a true Timeout, rerun the test with say 3/2
the failing value and output the timeout with the run.
We could have the testcases use different values for arm, but when load or
hw specs change, the tests will break again.
Paul
[0] https://gerrit.fd.io/r/c/vpp/+/24085
On Tue, Feb 9, 2021 at 8:37 AM Dave Barach wrote:
> On aarch64, "make TEST=test_bihash test-debug" involves a debug CLI
> timeout which is right on the hairy edge of being too short.
>
>
>
> 05:29:45,470 Calling cli_inband('cmd':'test bihash threads 2 nbuckets
> 64000 careful 0 verbose 0\n','context':2,'_vl_msg_id':574)
>
> 05:29:45,471 TIMEOUT:: 5
>
>
>
> This seems like the right incantation to fix it in
> src/vppinfra/test/test_bihash.py, but apparently it is not:
>
>
>
> error = self.vapi.cli("test bihash threads 2 nbuckets" +
>
> " 64000 careful 0 verbose 0", timeout=15)
>
>
>
> Please advise... Thanks... Dave
>
>
>
> -Original Message-
> From: vpp-dev@lists.fd.io On Behalf Of Benoit Ganne
> (bganne) via lists.fd.io
> Sent: Tuesday, February 9, 2021 8:06 AM
> To: Dave Barach ; 'Ole Troan' ;
> 'Dave Wallace' ; vpp-dev@lists.fd.io
> Cc: 'Juraj Linkeš'
> Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
>
>
>
> I am going to have a look at IKE.
>
>
>
> In case it is useful to others, thanks to QEMU you can run a Ubuntu 20.04
> ARM docker container on your x86 workstation:
>
>
>
> # step 0: add support for multiarch (must be done once after reboot)
> docker run --rm --privileged multiarch/qemu-user-static --reset
> --persistent yes --credential yes
>
>
>
> # step 1: create your chroot (must be done once - I am sharing my homedir
> with my chroot and same UID/GID) docker run --name aarch64_u2004
> --privileged --net host -v $HOME:$HOME -v /dev:/dev -td
> arm64v8/ubuntu:20.04 /bin/bash
>
>
>
> # step 2: update and add user inside container docker container exec
> aarch64_u2004 sh -c "apt -qy update && apt dist-upgrade -qy && apt install
> -qy vim sudo make git && groupadd -g $(id -rg) $USER && useradd -u $(id
> -ru) -g $(id -rg) -M -d $HOME -s /bin/bash $USER && echo '$USER ALL=(ALL)
> NOPASSWD:ALL' >> /etc/sudoers && echo aarch64_u2004 > /etc/debian_chroot"
>
>
>
> # step 3: log into ARM chroot
>
> docker container exec -it aarch64_deb10 /bin/login -f $USER
>
>
>
> Best
>
> ben
>
>
>
> > -Original Message-
>
> > From: vpp-dev@lists.fd.io On Behalf Of Dave
>
> > Barach
>
> > Sent: mardi 9 février 2021 12:47
>
> > To: 'Ole Troan' ; 'Dave Wallace'
>
> > ; vpp-dev@lists.fd.io
>
> > Cc: 'Juraj Linkeš'
>
> > Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
>
> >
>
> > Would be glad to take a look at the aarch64 bihash failure, modulo the
>
> > fact that the LF data center resources I've used before seem to have
>
> > had their credentials changed.
>
> >
>
> > Please unicast appropriate (ip, userid, password) tuples.
>
> >
>
> > Thanks... Dave
>
> >
>
> > ssh vppdev@10.30.51.65
>
> > Warning: Permanently added '10.30.51.65' (ECDSA) to the list of known
>
> > hosts.
>
> > vppdev@10.30.51.65's password:
>
> > Permission denied, please try again.
>
> > vppdev@10.30.51.65's password:
>
> > Permission denied, please try again.
>
> > vppdev@10.30.51.65's password:
>
> > Received disconnect from 10.30.51.65 port 22:2: Too many
>
> > authentication failures Disconnected from 10.30.51.65 port 22 $ ssh
>
> > vppdev@10.30.51.68
>
> > Warning: Permanently added '10.30.51.68' (ECDSA) to the list of known
>
> > hosts.
>
> > vppdev@10.30.51.68's password:
>
> > Permission denied, please try again.
>
> > vppdev@10.30.51.68's password:
>
> > Permission denied, please try again.
>
> > vppdev@10.30.51.68's password:
>
> > Received disconnect from 10.30.51.68 port 22:2: Too many
>
> > authentication failures Disconnected from 10.30.51.68 port 22
>
> >
>
> > -Original Message-
>
> > From: vpp-dev@lists.fd.io On Behalf Of Ole Tro
Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
On aarch64, "make TEST=test_bihash test-debug" involves a debug CLI timeout
which is right on the hairy edge of being too short.
05:29:45,470 Calling cli_inband('cmd':'test bihash threads 2 nbuckets 64000
careful 0 verbose 0\n','context':2,'_vl_msg_id':574)
05:29:45,471 TIMEOUT:: 5
This seems like the right incantation to fix it in
src/vppinfra/test/test_bihash.py, but apparently it is not:
error = self.vapi.cli("test bihash threads 2 nbuckets" +
" 64000 careful 0 verbose 0", timeout=15)
Please advise... Thanks... Dave
-Original Message-
From: vpp-dev@lists.fd.io On Behalf Of Benoit Ganne
(bganne) via lists.fd.io
Sent: Tuesday, February 9, 2021 8:06 AM
To: Dave Barach ; 'Ole Troan' ; 'Dave
Wallace' ; vpp-dev@lists.fd.io
Cc: 'Juraj Linkeš'
Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
I am going to have a look at IKE.
In case it is useful to others, thanks to QEMU you can run a Ubuntu 20.04
ARM docker container on your x86 workstation:
# step 0: add support for multiarch (must be done once after reboot) docker
run --rm --privileged multiarch/qemu-user-static --reset --persistent yes
--credential yes
# step 1: create your chroot (must be done once - I am sharing my homedir
with my chroot and same UID/GID) docker run --name aarch64_u2004
--privileged --net host -v $HOME:$HOME -v /dev:/dev -td arm64v8/ubuntu:20.04
/bin/bash
# step 2: update and add user inside container docker container exec
aarch64_u2004 sh -c "apt -qy update && apt dist-upgrade -qy && apt install
-qy vim sudo make git && groupadd -g $(id -rg) $USER && useradd -u $(id -ru)
-g $(id -rg) -M -d $HOME -s /bin/bash $USER && echo '$USER ALL=(ALL)
NOPASSWD:ALL' >> /etc/sudoers && echo aarch64_u2004 > /etc/debian_chroot"
# step 3: log into ARM chroot
docker container exec -it aarch64_deb10 /bin/login -f $USER
Best
ben
> -Original Message-
> From: <mailto:vpp-dev@lists.fd.io> vpp-dev@lists.fd.io <
<mailto:vpp-dev@lists.fd.io> vpp-dev@lists.fd.io> On Behalf Of Dave
> Barach
> Sent: mardi 9 février 2021 12:47
> To: 'Ole Troan' < <mailto:otr...@employees.org> otr...@employees.org>;
'Dave Wallace'
> < <mailto:dwallac...@gmail.com> dwallac...@gmail.com>;
<mailto:vpp-dev@lists.fd.io> vpp-dev@lists.fd.io
> Cc: 'Juraj Linkeš' < <mailto:juraj.lin...@pantheon.tech>
juraj.lin...@pantheon.tech>
> Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
>
> Would be glad to take a look at the aarch64 bihash failure, modulo the
> fact that the LF data center resources I've used before seem to have
> had their credentials changed.
>
> Please unicast appropriate (ip, userid, password) tuples.
>
> Thanks... Dave
>
> ssh <mailto:vppdev@10.30.51.65> vppdev@10.30.51.65
> Warning: Permanently added '10.30.51.65' (ECDSA) to the list of known
> hosts.
> <mailto:vppdev@10.30.51.65's> vppdev@10.30.51.65's password:
> Permission denied, please try again.
> <mailto:vppdev@10.30.51.65's> vppdev@10.30.51.65's password:
> Permission denied, please try again.
> <mailto:vppdev@10.30.51.65's> vppdev@10.30.51.65's password:
> Received disconnect from 10.30.51.65 port 22:2: Too many
> authentication failures Disconnected from 10.30.51.65 port 22 $ ssh
> <mailto:vppdev@10.30.51.68> vppdev@10.30.51.68
> Warning: Permanently added '10.30.51.68' (ECDSA) to the list of known
> hosts.
> <mailto:vppdev@10.30.51.68's> vppdev@10.30.51.68's password:
> Permission denied, please try again.
> <mailto:vppdev@10.30.51.68's> vppdev@10.30.51.68's password:
> Permission denied, please try again.
> <mailto:vppdev@10.30.51.68's> vppdev@10.30.51.68's password:
> Received disconnect from 10.30.51.68 port 22:2: Too many
> authentication failures Disconnected from 10.30.51.68 port 22
>
> -Original Message-
> From: <mailto:vpp-dev@lists.fd.io> vpp-dev@lists.fd.io <
<mailto:vpp-dev@lists.fd.io> vpp-dev@lists.fd.io> On Behalf Of Ole Troan
> Sent: Tuesday, February 9, 2021 5:20 AM
> To: Dave Wallace < <mailto:dwallac...@gmail.com> dwallac...@gmail.com>;
<mailto:vpp-dev@lists.fd.io> vpp-dev@lists.fd.io
> Cc: Juraj Linkeš < <mailto:juraj.lin...@pantheon.tech>
juraj.lin...@pantheon.tech>
> Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
>
> And just to be clear, the failing tests are in the QUIC, bihash and
> IKEv2 features.
>
> > [4]
> <https://logs.fd.io/sandbox/vex-yul-rot-jenkins-2/daw_30773_31019-vpp->
https://logs.fd.io/sandbox/vex-yul-rot-jenkins-2/daw_30773_
Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
I am going to have a look at IKE. In case it is useful to others, thanks to QEMU you can run a Ubuntu 20.04 ARM docker container on your x86 workstation: # step 0: add support for multiarch (must be done once after reboot) docker run --rm --privileged multiarch/qemu-user-static --reset --persistent yes --credential yes # step 1: create your chroot (must be done once - I am sharing my homedir with my chroot and same UID/GID) docker run --name aarch64_u2004 --privileged --net host -v $HOME:$HOME -v /dev:/dev -td arm64v8/ubuntu:20.04 /bin/bash # step 2: update and add user inside container docker container exec aarch64_u2004 sh -c "apt -qy update && apt dist-upgrade -qy && apt install -qy vim sudo make git && groupadd -g $(id -rg) $USER && useradd -u $(id -ru) -g $(id -rg) -M -d $HOME -s /bin/bash $USER && echo '$USER ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers && echo aarch64_u2004 > /etc/debian_chroot" # step 3: log into ARM chroot docker container exec -it aarch64_deb10 /bin/login -f $USER Best ben > -Original Message- > From: vpp-dev@lists.fd.io On Behalf Of Dave Barach > Sent: mardi 9 février 2021 12:47 > To: 'Ole Troan' ; 'Dave Wallace' > ; vpp-dev@lists.fd.io > Cc: 'Juraj Linkeš' > Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64 > > Would be glad to take a look at the aarch64 bihash failure, modulo the > fact > that the LF data center resources I've used before seem to have had their > credentials changed. > > Please unicast appropriate (ip, userid, password) tuples. > > Thanks... Dave > > ssh vppdev@10.30.51.65 > Warning: Permanently added '10.30.51.65' (ECDSA) to the list of known > hosts. > vppdev@10.30.51.65's password: > Permission denied, please try again. > vppdev@10.30.51.65's password: > Permission denied, please try again. > vppdev@10.30.51.65's password: > Received disconnect from 10.30.51.65 port 22:2: Too many authentication > failures > Disconnected from 10.30.51.65 port 22 > $ ssh vppdev@10.30.51.68 > Warning: Permanently added '10.30.51.68' (ECDSA) to the list of known > hosts. > vppdev@10.30.51.68's password: > Permission denied, please try again. > vppdev@10.30.51.68's password: > Permission denied, please try again. > vppdev@10.30.51.68's password: > Received disconnect from 10.30.51.68 port 22:2: Too many authentication > failures > Disconnected from 10.30.51.68 port 22 > > -Original Message- > From: vpp-dev@lists.fd.io On Behalf Of Ole Troan > Sent: Tuesday, February 9, 2021 5:20 AM > To: Dave Wallace ; vpp-dev@lists.fd.io > Cc: Juraj Linkeš > Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64 > > And just to be clear, the failing tests are in the QUIC, bihash and IKEv2 > features. > > > [4] > https://logs.fd.io/sandbox/vex-yul-rot-jenkins-2/daw_30773_31019-vpp- > verify- > master-ubuntu2004-aarch64/4/ > > vpp-unittest-QUICEchoIntMStreamTestCase-f5ywwo0l-FAILED > > vpp-unittest-TestBihash-_iohsbtr-FAILED > > vpp-unittest-TestInitiatorDelSAFromResponder-dgqdg78o-FAILED > > vpp-unittest-TestInitiatorKeepaliveMsg-0vo66yj2-FAILED > > vpp-unittest-TestInitiatorNATT-edyneyju-FAILED > > vpp-unittest-TestInitiatorPsk-m3vrysyi-FAILED > > vpp-unittest-TestInitiatorRekey-7lrb0f8t-FAILED > > vpp-unittest-TestInitiatorRequestWindowSize-4y4stlww-FAILED > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18707): https://lists.fd.io/g/vpp-dev/message/18707 Mute This Topic: https://lists.fd.io/mt/80495347/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
Would be glad to take a look at the aarch64 bihash failure, modulo the fact that the LF data center resources I've used before seem to have had their credentials changed. Please unicast appropriate (ip, userid, password) tuples. Thanks... Dave ssh vppdev@10.30.51.65 Warning: Permanently added '10.30.51.65' (ECDSA) to the list of known hosts. vppdev@10.30.51.65's password: Permission denied, please try again. vppdev@10.30.51.65's password: Permission denied, please try again. vppdev@10.30.51.65's password: Received disconnect from 10.30.51.65 port 22:2: Too many authentication failures Disconnected from 10.30.51.65 port 22 $ ssh vppdev@10.30.51.68 Warning: Permanently added '10.30.51.68' (ECDSA) to the list of known hosts. vppdev@10.30.51.68's password: Permission denied, please try again. vppdev@10.30.51.68's password: Permission denied, please try again. vppdev@10.30.51.68's password: Received disconnect from 10.30.51.68 port 22:2: Too many authentication failures Disconnected from 10.30.51.68 port 22 -Original Message- From: vpp-dev@lists.fd.io On Behalf Of Ole Troan Sent: Tuesday, February 9, 2021 5:20 AM To: Dave Wallace ; vpp-dev@lists.fd.io Cc: Juraj Linkeš Subject: Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64 And just to be clear, the failing tests are in the QUIC, bihash and IKEv2 features. > [4] https://logs.fd.io/sandbox/vex-yul-rot-jenkins-2/daw_30773_31019-vpp-verify- master-ubuntu2004-aarch64/4/ > vpp-unittest-QUICEchoIntMStreamTestCase-f5ywwo0l-FAILED > vpp-unittest-TestBihash-_iohsbtr-FAILED > vpp-unittest-TestInitiatorDelSAFromResponder-dgqdg78o-FAILED > vpp-unittest-TestInitiatorKeepaliveMsg-0vo66yj2-FAILED > vpp-unittest-TestInitiatorNATT-edyneyju-FAILED > vpp-unittest-TestInitiatorPsk-m3vrysyi-FAILED > vpp-unittest-TestInitiatorRekey-7lrb0f8t-FAILED > vpp-unittest-TestInitiatorRequestWindowSize-4y4stlww-FAILED > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18705): https://lists.fd.io/g/vpp-dev/message/18705 Mute This Topic: https://lists.fd.io/mt/80495347/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Make test failures on ubuntu-20.04 AARCH64
And just to be clear, the failing tests are in the QUIC, bihash and IKEv2 features. > [4] > https://logs.fd.io/sandbox/vex-yul-rot-jenkins-2/daw_30773_31019-vpp-verify-master-ubuntu2004-aarch64/4/ > vpp-unittest-QUICEchoIntMStreamTestCase-f5ywwo0l-FAILED > vpp-unittest-TestBihash-_iohsbtr-FAILED > vpp-unittest-TestInitiatorDelSAFromResponder-dgqdg78o-FAILED > vpp-unittest-TestInitiatorKeepaliveMsg-0vo66yj2-FAILED > vpp-unittest-TestInitiatorNATT-edyneyju-FAILED > vpp-unittest-TestInitiatorPsk-m3vrysyi-FAILED > vpp-unittest-TestInitiatorRekey-7lrb0f8t-FAILED > vpp-unittest-TestInitiatorRequestWindowSize-4y4stlww-FAILED > signature.asc Description: Message signed with OpenPGP -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18704): https://lists.fd.io/g/vpp-dev/message/18704 Mute This Topic: https://lists.fd.io/mt/80495347/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] Make test failures on ubuntu-20.04 AARCH64
Folks, There are a number of VPP make test jobs that are failing when running on ubuntu-20.04 AARCH64. I am testing the [promoted] vpp-verify-master-ubuntu2004-aarch64 job [0] in the jenkins sandbox against the VPP gerrit change [1] which changes the default 'make verify' OS to ubuntu-20.04. Andrew & I have also enhanced the CI post build scripts to dump the VPP stacktrace instead of archiving the core file [2] which is also applied to the sandbox test jobs. This can be found both in the console output as well as in the 'make test' FAILED archive directory in the file 'core.stacktrace.gz' All of the production 'make test' testcases passed on X86_64 [3]. I have included the list of FAILED make test testcases below from the logs archive [4] of the test run. Note: If there are less than 4 FAILED testcases in the archive then the test is failing intermittently and passed on a subsequent retry. Please investigate and report back at your earliest convenience for any feature that you maintain. These failures are holding up the deployment of [1] which will complete the change to ubuntu-20.04 as the primary VPP OS distro. Thanks, -daw- [0] https://jenkins.fd.io/sandbox/job/daw_30773_31019-vpp-verify-master-ubuntu2004-aarch64/ [1] https://gerrit.fd.io/r/c/vpp/+/30734 [2] https://gerrit.fd.io/r/c/ci-management/+/31019 [3] https://jenkins.fd.io/sandbox/job/daw_30773_31019-vpp-verify-master-ubuntu2004-x86_64 https://logs.fd.io/sandbox/vex-yul-rot-jenkins-2/daw_30773_31019-vpp-verify-master-ubuntu2004-x86_64/4/ [4] https://logs.fd.io/sandbox/vex-yul-rot-jenkins-2/daw_30773_31019-vpp-verify-master-ubuntu2004-aarch64/4/ vpp-unittest-QUICEchoIntMStreamTestCase-f5ywwo0l-FAILED vpp-unittest-TestBihash-_iohsbtr-FAILED vpp-unittest-TestInitiatorDelSAFromResponder-dgqdg78o-FAILED vpp-unittest-TestInitiatorKeepaliveMsg-0vo66yj2-FAILED vpp-unittest-TestInitiatorNATT-edyneyju-FAILED vpp-unittest-TestInitiatorPsk-m3vrysyi-FAILED vpp-unittest-TestInitiatorRekey-7lrb0f8t-FAILED vpp-unittest-TestInitiatorRequestWindowSize-4y4stlww-FAILED -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18701): https://lists.fd.io/g/vpp-dev/message/18701 Mute This Topic: https://lists.fd.io/mt/80495347/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
