On Thu, 9 Mar 2017, Andrew Cagney wrote: Note: These problems listed will have minor test cases issues that are nice small problems for GSOC students to get familiar with our code base. But some might also be real and difficult bugs. If you want to have a go and some of these, do join the #swan channel and ask for help. If you want to know how to run tests, see:
https://libreswan.org/wiki/Test_Suite
Looking at http://testing.libreswan.org/results/testing/ and excluding WIP we seem to consistently have 100 failing tests (you can eyeball this from the graph) so I did a little analysis using the script testing/web/never-passed.sh: - 45 are 'original' and have never ever passed - 16 are more recent yet also show no sign of passing
It might be worth redoing the list you made with the PEXPECT removed? Some comments from the top of my head on your list:
0 230 230 ikev2-allow-narrow-01 0 230 230 ikev2-allow-narrow-02 0 230 230 ikev2-allow-narrow-03
These used to pass, but they are testing failure path. I think the logging in the path changed.
0 230 230 interop-ikev2-racoon-02-psk-responder
We have no good racoon sanitizer.
0 230 230 interop-ikev2-strongswan-19-x509-res-certreq 0 230 230 interop-ikev2-strongswan-21-transport-02 0 230 230 interop-ikev2-strongswan-21-transport-03 0 230 230 interop-ikev2-strongswan-26-ke-mismatch-responder 0 230 230 interop-ikev2-strongswan-27-fragmentation
Should all pass, but might be kernel/strongswan version specific.
0 230 230 newoe-07-ike-replace-initiator-esp 0 230 230 newoe-15-portpass 0 230 230 newoe-18-poc-block 0 230 230 newoe-18-poc-blockall 0 230 230 newoe-18-private-clear 0 230 230 newoe-18-private-clearall 0 230 230 newoe-19-poc-poc-clear 0 230 230 newoe-20-ipv6 0 230 230 newoe-21-liveness-clear
Should all pass.
0 230 230 nss-cert-08-mismatch
Recently got a code fix to get it to pass, but needs updatin.
0 230 230 nss-cert-nosecret 0 230 230 nss-cert-ocsp-01-strict 0 230 230 nss-cert-ocsp-02 0 230 230 nss-cert-ocsp-02-ikev2 0 230 230 nss-cert-ocsp-03-strict 0 230 230 nss-cert-ocsp-04 0 230 230 nss-cert-ocsp-05-strict 0 230 230 nss-cert-ocsp-06 0 230 230 nss-cert-ocsp-07-nourl
Some people show a weird ocspd problem on nic. Some people have no ocspd on nic :P
0 218 218 dnssec-pluto-01
I'm working on this code and will pick it up
0 218 218 dpd-01 0 218 218 dpd-04 0 218 218 dpd-06
the dpd and liveness tests are awful to get consistent :( Usually all these are false positives.
0 218 218 l2tp-01 0 218 218 l2tp-02
I've looked at this a few times. It seems a real bug, but I haven't been able to locate it yet.
0 177 177 seccomp-01-enabled 0 177 177 seccomp-02-tolerant
experimental code needs fixing
0 152 152 nss-cert-09-notyetvalid-initiator 0 152 152 nss-cert-09-notyetvalid-initiator-ikev2 0 152 152 nss-cert-10-notyetvalid-responder 0 152 152 nss-cert-10-notyetvalid-responder-ikev2
These should work, but do require libfaketime and faketime seems to not work properly when using docker. I thought these should pass using kvm though.
0 152 152 nss-cert-ocsp-08-post
same ocspd issue?
0 75 75 addconn-01
A recent add for testing a parser bug. The bug is still there :)
0 53 53 certoe-01-whack
should work?
0 53 53 certoe-07-nat-2-clients 0 53 53 certoe-08-nat-packet-cop-restart 0 49 49 nss-cert-chain-01-ikev2 0 49 49 nss-cert-chain-03-ikev2 0 49 49 nss-cert-chain-04-ikev2
Should also work.
0 37 37 ikev1-aggr-sendcert-01
Recently added test case to show a bug where we did not properly send CERT payloads in Aggressive Mode. I think this bug still needs fixing. It was fixed recently for Main Mode I think.
0 18 18 netkey-vti-06c
I think this one shows a known kernel bug.
5 225 230 nflog-01-global
This is a weird one too. I looked at it a few times. It should work.
6 224 230 ipsec-hostkey-ckaid-02-fips 8 0 8 ikev2-algo-ike-dh-ecp-01 8 0 8 interop-ikev2-strongswan-33-ike-dh-ecp-responder
I assume you know best about these? :)
33 197 230 nss-cert-ocsp-01 33 197 230 nss-cert-ocsp-01-chain 33 197 230 nss-cert-ocsp-01-ikev2 33 197 230 nss-cert-ocsp-03 33 197 230 nss-cert-ocsp-03-ikev2 33 197 230 nss-cert-ocsp-05 33 197 230 nss-cert-ocsp-05-ikev2
same ocspd?
41 6 47 delete-sa-09-rhbz1311360-c32
This shows a bug that is fixed, so it should work. Perhaps test just needs updating.
45 8 53 ikev2-asymmetric-01-parsing
This is a new one to ensure we parse all auth/authby properly. Probably needs updating.
47 1 48 delete-sa-08-rhbz1311360
Also shows a bug that is fixed so should work or might need an update.
48 1 49 nss-cert-chain-02-ikev2 48 1 49 nss-cert-chain-04
Recent work should have fixed these too.
50 3 53 ikev2-asymmetric-04-null-rsaraw 50 3 53 ikev2-asymmetric-07-psk-rsaraw 50 3 53 ikev2-asymmetric-08-psk-rsacert 50 3 53 ikev2-asymmetric-09-rsaraw-null 50 3 53 ikev2-asymmetric-12-rsacert-null 50 3 53 ikev2-asymmetric-13-rsacert-psk 50 3 53 ikev2-asymmetric-15-rsaraw-rsaraw
Should all work. There might be a bug in finding the right connection in some combinations. I forgot what the current status was here.
63 167 230 ikev2-01-fallback-ikev1
The code tested here is slated to be removed, once we make conns to be either ikev1 or ikev2 but not both. Similar with the bidown attack test case. But it should currently work with the fallback.
66 39 105 klips-algo-twofish-01 71 34 105 klips-algo-serpent-01
should work?
72 100 172 ikev1-impair-gx-02
some of these were the pexect problems that got resolved two days ago.
89 16 105 klips-algo-cast-01
should work ?
113 105 218 netkey-audit-01
Requires proper USE_LINUX_AUDIT=true which is not the default. I think if debian/ubuntu support the auditd now, we can change the default to be enabled.
159 59 218 netkey-algo-twofish-01
should work.
160 58 218 netkey-algo-aes_ccm-01 160 58 218 netkey-algo-aes_gcm-01 160 58 218 netkey-algo-aes_gcm-02 160 58 218 netkey-algo-aes_gcm-03
should work?
167 63 230 ikev2-delete-02
We changed some of our delete/restart bhaviour, and we are not done with the changes yet. So some of these delete test cases might be in flux for a bit.
167 63 230 netkey-tfc-01 167 63 230 netkey-tfc-02 171 59 230 netkey-tfc-03
Requires new enough kernel (4.x maybe?)
172 58 230 ikev2-invalid-ke-02-wrong-modp 174 56 230 ikev2-11-simple-psk 174 56 230 ikev2-algo-ike-sha2-03
should work?
174 56 230 xauth-pluto-05 174 56 230 xauth-pluto-06 174 56 230 xauth-pluto-09 174 56 230 xauth-pluto-19 175 55 230 basic-pluto-06
these might need fixing.
177 53 230 ikev2-delete-01
See above.
179 39 218 netkey-algo-null-01 179 39 218 netkey-algo-serpent-01 180 50 230 ikev2-12-transport-psk
should work.
181 49 230 x509-pluto-02 181 49 230 x509-pluto-03
might be obsolete ones, will need to look.
181 37 218 ikev1-isakmp-reserved-flags-01 181 37 218 ikev1-isakmp-reserved-flags-02
I think the failure path changed output.
184 46 230 algo-pluto-13-invalid-3des
I think this tested a bogus keysize, but then we switched to rejecting to load these. Might require a new impair to send bogus keysize value over the wire.
184 46 230 ikev2-21-mode-mismatch-02
This might have changed. The RFC says if client requests transport mode, but server refuses transport mode, client needs to use tunnel mode. We might insist on exact match, or changed to following the spec without updating the test case.
201 29 230 ikev2-invalid-ke-01-invalid-modp 201 29 230 ikev2-invalid-ke-03-default-initiator 201 29 230 ikev2-invalid-ke-04-default-responder
Should work but failure path output might have changed?
212 18 230 ikev2-isakmp-reserved-flags-01 212 18 230 ikev2-isakmp-reserved-flags-02 212 18 230 ikev2-minor-version-initiator 212 18 230 ikev2-minor-version-responder 212 18 230 ikev2-payload-reserved-flags-01
Same.
213 17 230 ikev2-algo-esn-01 213 17 230 ikev2-algo-esn-03 213 17 230 ikev2-algo-esn-05 213 17 230 ikev2-algo-esn-06
Might depend on kernel version. Might need 4.x
213 5 218 l2tp-02-netkey
A real concern / bug like the other two l2tp test cases.
214 4 218 netkey-vti-02
I believe these first two VTI testcases predated some of the VTI features and need updating. I thought I did that though?
229 1 230 newoe-18-*
And all the others. These should pass, but are known to be sensitive to machine speeds with false positives on whether it eats 1 or 2 packets. Paul _______________________________________________ Swan-dev mailing list [email protected] https://lists.libreswan.org/mailman/listinfo/swan-dev
