Ilya Maximets <[email protected]> writes: > On 6/16/23 14:56, Aaron Conole wrote: >> Ilya Maximets <[email protected]> writes: >> >>> On 6/15/23 19:49, Paolo Valerio wrote: >>>> Ilya Maximets <[email protected]> writes: >>>> >>>>> On 6/14/23 21:08, Ilya Maximets wrote: >>>>>> On 6/14/23 20:11, Paolo Valerio wrote: >>>>>>> Ilya Maximets <[email protected]> writes: >>>>>>> >>>>>>>> On 6/12/23 16:57, Aaron Conole wrote: >>>>>>>>> Paolo Valerio <[email protected]> writes: >>>>>>>>> >>>>>>>>>> since a27d70a89 ("conntrack: add generic IP protocol support") all >>>>>>>>>> the unrecognized IP protocols get handled using ct_proto_other ops >>>>>>>>>> and are managed as L3 using 3 tuples. >>>>>>>>>> >>>>>>>>>> This patch stores L4 information for SCTP in the conn_key so that >>>>>>>>>> multiple conn instances, instead of one with ports zeroed, will be >>>>>>>>>> created when there are multiple SCTP connections between two hosts. >>>>>>>>>> It also performs crc32c check when not offloaded, and adds SCTP to >>>>>>>>>> pat_enabled. >>>>>>>>>> >>>>>>>>>> With this patch, given two SCTP association between two hosts, >>>>>>>>>> tracking the connection will result in: >>>>>>>>>> >>>>>>>>>> sctp,orig=(src=10.1.1.2,dst=10.1.1.1,sport=55884,dport=5201),reply=(src=10.1.1.1,dst=10.1.1.2,sport=5201,dport=12345),zone=1 >>>>>>>>>> sctp,orig=(src=10.1.1.2,dst=10.1.1.1,sport=59874,dport=5202),reply=(src=10.1.1.1,dst=10.1.1.2,sport=5202,dport=12346),zone=1 >>>>>>>>>> >>>>>>>>>> instead of: >>>>>>>>>> >>>>>>>>>> sctp,orig=(src=10.1.1.2,dst=10.1.1.1,sport=0,dport=0),reply=(src=10.1.1.1,dst=10.1.1.2,sport=0,dport=0),zone=1 >>>>>>>>>> >>>>>>>>>> Signed-off-by: Paolo Valerio <[email protected]> >>>>>>>>>> --- >>>>>>>>> >>>>>>>>> Thanks for this work - I think it looks good. >>>>>>>>> >>>>>>>>> Perhaps it should have a NEWS item mentioned that the userspace >>>>>>>>> conntrack now supports matching SCTP l4 data. >>>>>>>>> >>>>>>>>> If you do spin a v4 with that change, you can keep my: >>>>>>>>> >>>>>>>>> Acked-by: Aaron Conole <[email protected]> >>>>>>>> >>>>>>>> Hi, Paolo and Aaron. >>>>>>>> >>>>>>>> I'm getting a consistent test failure while running check-kernel >>>>>>>> on Ubuntu 22.10 with 5.19 kernel: >>>>>>>> >>>>>>>> >>>>>>>> ./system-traffic.at:4754: cat ofctl_monitor.log >>>>>>>> --- - 2023-06-14 11:26:41.958591125 +0000 >>>>>>>> +++ >>>>>>>> /root/ovs/tests/system-kmod-testsuite.dir/at-groups/105/stdout >>>>>>>> 2023-06-14 11:26:41.952000000 +0000 >>>>>>>> @@ -12,8 +12,6 @@ >>>>>>>> >>>>>>>> sctp,vlan_tci=0x0000,dl_src=e6:66:c1:22:22:22,dl_dst=e6:66:c1:11:11:11,nw_src=10.1.1.2,nw_dst=10.1.1.1,nw_tos=0,nw_ecn=2,nw_ttl=64,nw_frag=no,tp_src=12345,tp_dst=54969 >>>>>>>> sctp_csum:9b67e853 >>>>>>>> NXT_PACKET_IN2 (xid=0x0): cookie=0x0 total_len=54 in_port=1 >>>>>>>> (via action) data_len=54 (unbuffered) >>>>>>>> >>>>>>>> sctp,vlan_tci=0x0000,dl_src=e6:66:c1:11:11:11,dl_dst=e6:66:c1:22:22:22,nw_src=10.1.1.240,nw_dst=10.1.1.2,nw_tos=0,nw_ecn=2,nw_ttl=64,nw_frag=no,tp_src=34567,tp_dst=12345 >>>>>>>> sctp_csum:bc0e5463 >>>>>>>> -NXT_PACKET_IN2 (xid=0x0): table_id=1 cookie=0x0 total_len=50 >>>>>>>> ct_state=est|rpl|trk|dnat,ct_zone=1,ct_nw_src=10.1.1.1,ct_nw_dst=10.1.1.2,ct_nw_proto=132,ct_tp_src=54969,ct_tp_dst=12345,ip,in_port=2 >>>>>>>> (via action) data_len=50 (unbuffered) >>>>>>>> -sctp,vlan_tci=0x0000,dl_src=e6:66:c1:22:22:22,dl_dst=e6:66:c1:11:11:11,nw_src=10.1.1.2,nw_dst=10.1.1.1,nw_tos=0,nw_ecn=2,nw_ttl=64,nw_frag=no,tp_src=12345,tp_dst=54969 >>>>>>>> sctp_csum:d6ce6b9e >>>>>>>> NXT_PACKET_IN2 (xid=0x0): cookie=0x0 total_len=50 in_port=1 >>>>>>>> (via action) data_len=50 (unbuffered) >>>>>>>> -sctp,vlan_tci=0x0000,dl_src=e6:66:c1:11:11:11,dl_dst=e6:66:c1:22:22:22,nw_src=10.1.1.240,nw_dst=10.1.1.2,nw_tos=0,nw_ecn=2,nw_ttl=64,nw_frag=no,tp_src=34567,tp_dst=12345 >>>>>>>> sctp_csum:add7db93 >>>>>>>> +sctp,vlan_tci=0x0000,dl_src=e6:66:c1:11:11:11,dl_dst=e6:66:c1:22:22:22,nw_src=10.1.1.1,nw_dst=10.1.1.2,nw_tos=0,nw_ecn=2,nw_ttl=64,nw_frag=no,tp_src=54969,tp_dst=12345 >>>>>>>> sctp_csum:5db68ce >>>>>>>> >>>>>>>> >>>>>>>> Do you know what can be a problem here? >>>>>>>> >>>>>>>> Test is passing on Fedora 38 with 6.3 kernel and on rhel 9.2. >>>>>>>> >>>>>>> >>>>>>> Hi Ilya, >>>>>>> >>>>>>> Uhm, it seems there's a problem with the shutdown sequence. >>>>>>> I just ran the on a VM: >>>>>>> >>>>>>> vagrant@ubuntu2210:~/ovs$ grep CONFIG_NF_CT_PROTO_SCTP >>>>>>> /boot/config-5.19.0-38-generic >>>>>>> CONFIG_NF_CT_PROTO_SCTP=y >>>>>>> >>>>>>> vagrant@ubuntu2210:~/ovs$ grep VERSION /etc/os-release >>>>>>> VERSION_ID="22.10" >>>>>>> VERSION="22.10 (Kinetic Kudu)" >>>>>>> VERSION_CODENAME=kinetic >>>>>>> >>>>>>> vagrant@ubuntu2210:~/ovs$ uname -r >>>>>>> 5.19.0-38-generic >>>>>> >>>>>> The only difference with my VM is that I have -43-generic kernel. >>>>>> >>>>>>> >>>>>>> but I can't see the failure. >>>>>>> Any chance to see if they are marked for some reason as invalid? >>>>>> >>>>>> I dumped conntrack after every packet and here is what I see: >>>>>> >>>>>> On RHEL9, where test is working: >>>>>> >>>>>> 1. sctp 132 9 CLOSED src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>> dport=12345 [UNREPLIED] src=10.1.1.2 dst=10.1.1.240 sport=12345 >>>>>> dport=34567 mark=0 zone=1 use=1 >>>>>> 2. sctp 132 2 COOKIE_WAIT src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>> dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 dport=34567 >>>>>> mark=0 zone=1 use=1 >>>>>> 3. sctp 132 2 COOKIE_ECHOED src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>> dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 dport=34567 >>>>>> mark=0 zone=1 use=1 >>>>>> 4. sctp 132 431999 ESTABLISHED src=10.1.1.1 dst=10.1.1.2 >>>>>> sport=54969 dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 >>>>>> dport=34567 [ASSURED] mark=0 zone=1 use=1 >>>>>> 5. sctp 132 431999 ESTABLISHED src=10.1.1.1 dst=10.1.1.2 >>>>>> sport=54969 dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 >>>>>> dport=34567 [ASSURED] mark=0 zone=1 use=1 >>>>>> 6. sctp 132 431999 ESTABLISHED src=10.1.1.1 dst=10.1.1.2 >>>>>> sport=54969 dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 >>>>>> dport=34567 [ASSURED] mark=0 zone=1 use=1 >>>>>> 7. sctp 132 0 SHUTDOWN_SENT src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>> dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 dport=34567 >>>>>> [ASSURED] mark=0 zone=1 use=1 >>>>>> 8. sctp 132 2 SHUTDOWN_ACK_SENT src=10.1.1.1 dst=10.1.1.2 >>>>>> sport=54969 dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 >>>>>> dport=34567 [ASSURED] mark=0 zone=1 use=1 >>>>>> 9. sctp 132 9 CLOSED src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>> dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 dport=34567 >>>>>> [ASSURED] mark=0 zone=1 use=1 >>>>> >>>>> Here if I monitor conntrack during the test, I get: >>>>> >>>>> # conntrack -E --proto=sctp >>>>> [NEW] sctp 132 10 CLOSED src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>> dport=12345 [UNREPLIED] src=10.1.1.2 dst=10.1.1.240 sport=12345 >>>>> dport=34567 zone=1 >>>>> [DESTROY] sctp 132 9 CLOSED src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>> dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 dport=34567 >>>>> [ASSURED] zone=1 [USERSPACE] portid=3715 >>>>> >>>>> >>>>>> >>>>>> On Ubuntu, where it doesn't work: >>>>>> >>>>>> 1. sctp 132 9 CLOSED src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>> dport=12345 [UNREPLIED] src=10.1.1.2 dst=10.1.1.240 sport=12345 >>>>>> dport=34567 zone=1 use=1 >>>>>> 2. sctp 132 2 COOKIE_WAIT src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>> dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 dport=34567 >>>>>> zone=1 use=1 >>>>>> 3. sctp 132 2 COOKIE_ECHOED src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>> dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 dport=34567 >>>>>> zone=1 use=1 >>>>>> 4. sctp 132 209 ESTABLISHED src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>> dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 dport=34567 >>>>>> [ASSURED] zone=1 use=1 >>>>>> 5. sctp 132 209 ESTABLISHED src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>> dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 dport=34567 >>>>>> [ASSURED] zone=1 use=1 >>>>>> 6. sctp 132 209 ESTABLISHED src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>> dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 dport=34567 >>>>>> [ASSURED] zone=1 use=1 >>>>>> 7. sctp 132 0 SHUTDOWN_SENT src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>> dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 dport=34567 >>>>>> [ASSURED] zone=1 use=1 >>>>>> 8. NO ENTRY! >>>>>> 9. NO ENTRY! >>>>> >>>>> But here I have: >>>>> >>>>> # conntrack -E --proto=sctp >>>>> [NEW] sctp 132 10 CLOSED src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>> dport=12345 [UNREPLIED] src=10.1.1.2 dst=10.1.1.240 sport=12345 >>>>> dport=34567 zone=1 >>>>> [DESTROY] sctp 132 SHUTDOWN_SENT src=10.1.1.1 dst=10.1.1.2 >>>>> sport=54969 dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 >>>>> dport=34567 [ASSURED] zone=1 >>>>> >>>>> So, the connection indeed is getting destroyed while in SHUTDOWN_SENT >>>>> state. >>>>> Sounds like a kernel bug in Ubuntu... >>>>> >>>> >>>> Thanks Ilya for digging more into it. >>>> It seemed so to me as well, but looking at the logs you provided, the >>>> timeout for SHUTDOWN_SENT is 0 (third column of the dump), so it seems >>>> it's a matter of speed in the reply. >>>> >>>> ❯ cat /proc/sys/net/netfilter/nf_conntrack_sctp_timeout_shutdown_sent >>>> 0 >>>> >>>> I'm a bit surprised by this. >>>> >>>> Just to confirm that, I upgraded the kernel of my vm to >>>> 5.19.0-43-generic and the test succeeded. Sleeping for 1 second in >>>> SHUTDOWN_SENT before sending the SHUTDOWN_ACK_SENT make the test fail. I >>>> would expect the same on RHEL 9 and Fedora. >>> >>> Hmm, good point. >>> The difference between my tests on Ubuntu and RHEL is that I tested >>> with -O1 and sanitizers on Ubuntu, so it was a tiny bit slower. >>> I just tried to run with sanitizers on RHEL and I'm getting the same >>> failure as I have in Ubuntu. >>> >>> So, the test seems to be extremely time-sensitive. Is there a way >>> to make it more stable? >> >> I guess one alternative could be change from trying to dump the ct >> information directly, check for the 'conntrack' utility, and use that to >> log the events - then sweep the ct events log. It seems the ofctl >> monitor is showing that it is a bit racy, but maybe relying on the ct >> events log could still give us the confidence that it is working without >> the raciness of the ofctl monitor? > > The problem is that we will get different events based on the reaction time. > If the test is slow for any reason, the [DESTROY] event is happening in the > SHUTDOWN_SENT state and SHUTDOWN_ACK and SHUTDOWN_COMPLETE packets are > not coming through correctly. If the test is fast, then the [DESTROY] event > is happening in the CLOSED state and the last two packets are reaching the > destination. > > So, we will have different output regardless of what monitoring tool we're > using. Or am I missing something? >
Probably an interesting solution would have been applying a timeout policy for zone 1 for the kernel dp, but that is not implemented for sctp. One last chance to exploit would be aborting [0] the association instead of gracefully terminate it with the shutdown sequence. If my reading is correct, that conntrack entry would transitions to the CLOSED state and the last timeout (10 seconds) should not matter anymore. I didn't test it, so it needs to be confirmed. Given the very narrow timeout, I had to rule out all the other alternatives that came to mind. https://www.rfc-editor.org/rfc/rfc4960#section-9.1 >> >>>> >>>> Paolo >>>> >>>>>> >>>>>> So, after sending SHUTDOWN_ACK, there is no conntrack entry in the >>>>>> kernel anymore. >>>>>> >>>>>> >>>>>>> >>>>>>>> Best regards, Ilya Maximets. >>>>>>> >>>>>> >>>> >> _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
