On 06/04/2021 13:59, Flavio Leitner wrote: > On Tue, Apr 06, 2021 at 12:30:42PM +0100, Mark Gray wrote: >> On 05/04/2021 20:56, Flavio Leitner wrote: >>> On Thu, Apr 01, 2021 at 07:48:19PM +0100, Mark Gray wrote: >>>> On 01/04/2021 19:29, Ilya Maximets wrote: >>>>> On 4/1/21 3:58 PM, Mark Gray wrote: >>>>>> This series fixes IPv6 support for Libreswan and introduces >>>>>> IPsec system tests for Libreswan. >>>>>> >>>>>> Mark Gray (5): >>>>>> ipsec: IPv6 default route support for Libreswan >>>>>> system-common-macros: clean up veth device on test failure >>>>>> ipsec: Allow custom file locations >>>>>> ipsec: Introduce IPsec system tests for Libreswan >>>>>> ipsec: Update ordering of imports >>>>>> >>>>>> ipsec/ovs-monitor-ipsec.in | 144 ++++++++++-- >>>>>> tests/automake.mk | 3 +- >>>>>> tests/system-common-macros.at | 2 +- >>>>>> tests/system-ipsec.at | 406 +++++++++++++++++++++++++++++++++ >>>>>> tests/system-kmod-testsuite.at | 1 + >>>>>> 5 files changed, 529 insertions(+), 27 deletions(-) >>>>>> create mode 100644 tests/system-ipsec.at >>>>>> >>>>> >>>>> Thanks! Applied to master. >>>>> First patch with the fix also backported down to 2.13. >>>>> >>>>> Best regards, Ilya Maximets. >>>>> >>>> >>>> Thanks Ilya. >>>> >>>> Flavio was having some issues with the system test patch which appears >>>> to have a flake. We have not received his Ack yet. I will follow-up with >>>> a fix if it doesn't work for him or you can revert. >>> >>> It seems left side doesn't load the config fast enough, so the >>> replied proposal is none, AES_GCM_C_256-NONE-NONE-DISABLE. Then >> >> So this gets rejected? > > Yes. > >>> the retransmission happens in 60 seconds by default, but the >>> script timeout is 30s. Obviously the timeout always wins failing >>> the test. >>> >>> I used the following patch to test that and then the tunnel >>> succeeds on the second attempt retransmitting in 10s. >>> >>> I got 0 failures in 100 tests. >>> >>> diff --git a/ipsec/ovs-monitor-ipsec.in b/ipsec/ovs-monitor-ipsec.in >>> index 89a36fe17..132db4dce 100755 >>> --- a/ipsec/ovs-monitor-ipsec.in >>> +++ b/ipsec/ovs-monitor-ipsec.in >>> @@ -383,6 +383,7 @@ config setup >>> conn %%default >>> keyingtries=%%forever >>> type=transport >>> + retransmit-timeout=10 >>> auto=route >>> ike=aes_gcm256-sha2_256 >>> esp=aes_gcm256 >>> >>> Another thing I tried is to add a simple sleep between the adding >>> the ports: >>> --- a/tests/system-ipsec.at >>> +++ b/tests/system-ipsec.at >>> @@ -166,6 +166,7 @@ dnl Set up IPsec tunnel on 'left' host >>> OVS_VSCTL_LEFT(add-port br-ipsec tun -- set Interface tun type=geneve \ >>> options:remote_ip=10.1.1.2 options:psk=swordfish) >>> >>> +sleep 1 >>> dnl Set up IPsec tunnel on 'right' host >>> >>> OVS_VSCTL_RIGHT(add-port br-ipsec tun -- set Interface tun type=geneve \ >>> >>> I got 0 failures in 100 tests with the above patch. >>> >>> What would be a good way to tell if a tunnel port is ready? Perhaps >>> we could poll for status similar to what it is done at CHECK_ESP_TRAFFIC >>> to make sure both sides are ready. >> >> You are saying is that one side comes up quick and one does not so it >> misses the negotiation window causing the test to timeout while it waits >> for IKE to re-transmit. I am surprised that the "slow" side does not >> trigger negotiation when it comes up. >> >> Does it timeout on waiting for the connection to be active? i.e. >> >> https://github.com/openvswitch/ovs/blob/f8be30acf2eb60d567bb7386b98f5cb58ddb9119/tests/system-ipsec.at#L135 > > Yes. > > This is the tunnel state while waiting for it to establish: > > # for i in > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/ovs-monitor-ipsec.*.ctl; > do ovs-appctl -t $i tunnels/show; done > Interface name: tun v1 (CONFIGURED) > Tunnel Type: geneve > Local IP: %defaultroute > Remote IP: 10.1.1.2 > Address Family: IPv4 > SKB mark: None > Local cert: None > Local name: None > Local key: None > Remote cert: None > Remote name: None > CA cert: None > PSK: swordfish > Ofport: 1 > CFM state: Disabled > Kernel policies installed: > Kernel security associations installed: > sel src 10.1.1.2/32 dst 10.1.1.1/32 > IPsec connections that are active: > 000 #2: "tun-in-1":500 STATE_PARENT_I2 (sent IKE_AUTH request); > EVENT_RETRANSMIT in 39s; idle; > > Interface name: tun v1 (CONFIGURED) > Tunnel Type: geneve > Local IP: %defaultroute > Remote IP: 10.1.1.1 > Address Family: IPv4 > SKB mark: None > Local cert: None > Local name: None > Local key: None > Remote cert: None > Remote name: None > CA cert: None > PSK: swordfish > Ofport: 1 > CFM state: Disabled > Kernel policies installed: > Kernel security associations installed: > sel src 10.1.1.1/32 dst 10.1.1.2/32 > IPsec connections that are active: > 000 #4: "tun-in-1":500 STATE_PARENT_I2 (sent IKE_AUTH request); > EVENT_RETRANSMIT in 39s; idle; > > > Note the EVENT_RETRANSMIT on both. > > This is the test log. > > > # -*- compilation -*- > 140. system-ipsec.at:152: testing IPsec -- Libreswan (ipv4, geneve, > defaultroute, psk) ... > Linux Libreswan 4.3 (netkey) on 5.11.11-200.fc33.x86_64 > ./system-ipsec.at:158: modprobe openvswitch > Module vport_lisp not loaded. > Module vport_stt not loaded. > ./system-ipsec.at:158: ovsdb-tool create conf.db > $abs_top_srcdir/vswitchd/vswitch.ovsschema > ./system-ipsec.at:158: ovsdb-server --detach --no-chdir --pidfile --log-file > --remote=punix:$OVS_RUNDIR/db.sock > stderr: > 2021-04-06T12:52:07Z|00001|vlog|INFO|opened log file > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/ovsdb-server.log > ./system-ipsec.at:158: sed < stderr ' > /vlog|INFO|opened log file/d > /ovsdb_server|INFO|ovsdb-server (Open vSwitch)/d' > ./system-ipsec.at:158: ovs-vsctl --no-wait init > ./system-ipsec.at:158: ovs-vswitchd --detach --no-chdir --pidfile --log-file > -vvconn -vofproto_dpif -vunixctl > stderr: > 2021-04-06T12:52:07Z|00001|vlog|INFO|opened log file > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/ovs-vswitchd.log > 2021-04-06T12:52:07Z|00002|ovs_numa|INFO|Discovered 4 CPU cores on NUMA node 0 > 2021-04-06T12:52:07Z|00003|ovs_numa|INFO|Discovered 1 NUMA nodes and 4 CPU > cores > 2021-04-06T12:52:07Z|00004|reconnect|INFO|unix:/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/db.sock: > connecting... > 2021-04-06T12:52:07Z|00005|reconnect|INFO|unix:/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/db.sock: > connected > ./system-ipsec.at:158: sed < stderr ' > /ovs_numa|INFO|Discovered /d > /vlog|INFO|opened log file/d > /vswitchd|INFO|ovs-vswitchd (Open vSwitch)/d > /reconnect|INFO|/d > /dpif_netlink|INFO|Generic Netlink family .ovs_datapath. does not exist/d > /ofproto|INFO|using datapath ID/d > /netdev_linux|INFO|.*device has unknown hardware address family/d > /ofproto|INFO|datapath ID changed to fedcba9876543210/d > /dpdk|INFO|DPDK Disabled - Use other_config:dpdk-init to enable/d > /netlink_socket|INFO|netlink: could not enable listening to all nsid/d > /probe tc:/d > /tc: Using policy/d' > ./system-ipsec.at:158: ovs-vsctl -- add-br br0 -- set Bridge br0 > protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 > fail-mode=secure -- > ./system-ipsec.at:159: cp ${abs_top_srcdir}/vswitchd/vswitch.ovsschema > vswitch.ovsschema > ./system-ipsec.at:159: ovs-ofctl add-flow br0 "actions=normal" > Cannot remove namespace file "/var/run/netns/left": No such file or directory > ./system-ipsec.at:162: ip netns add left || return 77 > net.netfilter.nf_conntrack_helper = 0 > net.ipv6.conf.all.accept_dad = 0 > net.ipv6.conf.default.accept_dad = 0 > ./system-ipsec.at:162: ip link add p0 type veth peer name ovs-p0 || return 77 > ./system-ipsec.at:162: ip link set p0 netns left > ./system-ipsec.at:162: ip link set dev ovs-p0 up > ./system-ipsec.at:162: ovs-vsctl add-port br0 ovs-p0 -- \ > set interface ovs-p0 external-ids:iface-id="p0" > ./system-ipsec.at:162: ip netns exec left sh << NS_EXEC_HEREDOC > ip addr add 10.1.1.1/24 dev p0 > NS_EXEC_HEREDOC > ./system-ipsec.at:162: ip netns exec left sh << NS_EXEC_HEREDOC > ip link set dev p0 up > NS_EXEC_HEREDOC > 2021-04-06T12:52:07Z|00001|vlog|INFO|opened log file > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left/ovsdb.log > 2021-04-06T12:52:07Z|00001|vlog|INFO|opened log file > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left/vswitchd.log > 2021-04-06T12:52:07Z|00002|ovs_numa|INFO|Discovered 4 CPU cores on NUMA node 0 > 2021-04-06T12:52:07Z|00003|ovs_numa|INFO|Discovered 1 NUMA nodes and 4 CPU > cores > 2021-04-06T12:52:07Z|00004|reconnect|INFO|unix:/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left/db.sock: > connecting... > 2021-04-06T12:52:07Z|00005|reconnect|INFO|unix:/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left/db.sock: > connected > Initializing NSS database > > ./system-ipsec.at:162: ip netns exec left sh << NS_EXEC_HEREDOC > ipsec pluto --config $ovs_base/left/ipsec.conf \ > --ipsecdir $ovs_base/left --nssdir $ovs_base/left/ipsec.d \ > --logfile $ovs_base/left/pluto.log --secretsfile > $ovs_base/left/secrets \ > --rundir $ovs_base/left > NS_EXEC_HEREDOC > stderr: > /usr/libexec/ipsec/pluto: adjusting ipsec.d to > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left > /usr/libexec/ipsec/pluto: adjusting nssdir to > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left/ipsec.d > ./system-ipsec.at:162: ip netns exec left sh << NS_EXEC_HEREDOC > ovs-monitor-ipsec unix:${OVS_RUNDIR}/left/db.sock\ > --pidfile=${OVS_RUNDIR}/left/ovs-monitor-ipsec.pid > --ike-daemon=libreswan\ > --ipsec-conf=$ovs_base/left/ipsec.conf > --ipsec-d=$ovs_base/left/ipsec.d \ > --ipsec-secrets=$ovs_base/left/secrets \ > --log-file=$ovs_base/left/ovs-monitor-ipsec.log \ > --ipsec-ctl=$ovs_base/left/pluto.ctl \ > --no-restart-ike-daemon --detach > NS_EXEC_HEREDOC > stderr: > Cannot remove namespace file "/var/run/netns/right": No such file or directory > ./system-ipsec.at:163: ip netns add right || return 77 > net.netfilter.nf_conntrack_helper = 0 > net.ipv6.conf.all.accept_dad = 0 > net.ipv6.conf.default.accept_dad = 0 > ./system-ipsec.at:163: ip link add p1 type veth peer name ovs-p1 || return 77 > ./system-ipsec.at:163: ip link set p1 netns right > ./system-ipsec.at:163: ip link set dev ovs-p1 up > ./system-ipsec.at:163: ovs-vsctl add-port br0 ovs-p1 -- \ > set interface ovs-p1 external-ids:iface-id="p1" > ./system-ipsec.at:163: ip netns exec right sh << NS_EXEC_HEREDOC > ip addr add 10.1.1.2/24 dev p1 > NS_EXEC_HEREDOC > ./system-ipsec.at:163: ip netns exec right sh << NS_EXEC_HEREDOC > ip link set dev p1 up > NS_EXEC_HEREDOC > 2021-04-06T12:52:07Z|00001|vlog|INFO|opened log file > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/right/ovsdb.log > 2021-04-06T12:52:07Z|00001|vlog|INFO|opened log file > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/right/vswitchd.log > 2021-04-06T12:52:07Z|00002|ovs_numa|INFO|Discovered 4 CPU cores on NUMA node 0 > 2021-04-06T12:52:07Z|00003|ovs_numa|INFO|Discovered 1 NUMA nodes and 4 CPU > cores > 2021-04-06T12:52:07Z|00004|reconnect|INFO|unix:/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/right/db.sock: > connecting... > 2021-04-06T12:52:07Z|00005|reconnect|INFO|unix:/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/right/db.sock: > connected > Initializing NSS database > > ./system-ipsec.at:163: ip netns exec right sh << NS_EXEC_HEREDOC > ipsec pluto --config $ovs_base/right/ipsec.conf \ > --ipsecdir $ovs_base/right --nssdir $ovs_base/right/ipsec.d \ > --logfile $ovs_base/right/pluto.log --secretsfile > $ovs_base/right/secrets \ > --rundir $ovs_base/right > NS_EXEC_HEREDOC > stderr: > /usr/libexec/ipsec/pluto: adjusting ipsec.d to > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/right > /usr/libexec/ipsec/pluto: adjusting nssdir to > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/right/ipsec.d > ./system-ipsec.at:163: ip netns exec right sh << NS_EXEC_HEREDOC > ovs-monitor-ipsec unix:${OVS_RUNDIR}/right/db.sock\ > --pidfile=${OVS_RUNDIR}/right/ovs-monitor-ipsec.pid > --ike-daemon=libreswan\ > --ipsec-conf=$ovs_base/right/ipsec.conf > --ipsec-d=$ovs_base/right/ipsec.d \ > --ipsec-secrets=$ovs_base/right/secrets \ > --log-file=$ovs_base/right/ovs-monitor-ipsec.log \ > --ipsec-ctl=$ovs_base/right/pluto.ctl \ > --no-restart-ike-daemon --detach > NS_EXEC_HEREDOC > stderr: > system-ipsec.at:175: waiting until test `ipsec status --rundir > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left | grep > Total... > dropped privs to tcpdump > tcpdump: verbose output suppressed, use -v[v]... for full protocol decode > listening on ovs-p0, link-type EN10MB (Ethernet), snapshot length 262144 bytes > dropped privs to tcpdump > tcpdump: verbose output suppressed, use -v[v]... for full protocol decode > listening on ovs-p1, link-type EN10MB (Ethernet), snapshot length 262144 bytes > system-ipsec.at:175: wait succeeded quickly > system-ipsec.at:175: waiting until test `ipsec status --rundir > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/right | > grep Total... > system-ipsec.at:175: wait succeeded immediately > system-ipsec.at:175: waiting until test `ipsec status --rundir > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left | grep > Total... > system-ipsec.at:175: wait failed after 30 seconds > ./ovs-macros.at:242: hard failure > 0 packets captured > 0 packets received by filter > 0 packets dropped by kernel > 0 packets captured > 0 packets received by filter > 0 packets dropped by kernel > 2021-04-06T12:52:41Z|00001|daemon_unix|WARN|/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/ovs-vswitchd.pid: > open: No such file or directory > ovs-appctl: cannot read pidfile > "/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/ovs-vswitchd.pid" (No > such file or directory) > ovs-dpctl: opening datapath (No such device) > 140. system-ipsec.at:152: 140. IPsec -- Libreswan (ipv4, geneve, > defaultroute, psk) (system-ipsec.at:152): FAILED (ovs-macros.at:242) >
Could you post pluto.log from both left and right side? _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
