On 6/16/23 19:41, Paolo Valerio wrote: > Ilya Maximets <[email protected]> writes: > >> On 6/16/23 14:56, Aaron Conole wrote: >>> Ilya Maximets <[email protected]> writes: >>> >>>> On 6/15/23 19:49, Paolo Valerio wrote: >>>>> Ilya Maximets <[email protected]> writes: >>>>> >>>>>> On 6/14/23 21:08, Ilya Maximets wrote: >>>>>>> On 6/14/23 20:11, Paolo Valerio wrote: >>>>>>>> Ilya Maximets <[email protected]> writes: >>>>>>>> >>>>>>>>> On 6/12/23 16:57, Aaron Conole wrote: >>>>>>>>>> Paolo Valerio <[email protected]> writes: >>>>>>>>>> >>>>>>>>>>> since a27d70a89 ("conntrack: add generic IP protocol support") all >>>>>>>>>>> the unrecognized IP protocols get handled using ct_proto_other ops >>>>>>>>>>> and are managed as L3 using 3 tuples. >>>>>>>>>>> >>>>>>>>>>> This patch stores L4 information for SCTP in the conn_key so that >>>>>>>>>>> multiple conn instances, instead of one with ports zeroed, will be >>>>>>>>>>> created when there are multiple SCTP connections between two hosts. >>>>>>>>>>> It also performs crc32c check when not offloaded, and adds SCTP to >>>>>>>>>>> pat_enabled. >>>>>>>>>>> >>>>>>>>>>> With this patch, given two SCTP association between two hosts, >>>>>>>>>>> tracking the connection will result in: >>>>>>>>>>> >>>>>>>>>>> sctp,orig=(src=10.1.1.2,dst=10.1.1.1,sport=55884,dport=5201),reply=(src=10.1.1.1,dst=10.1.1.2,sport=5201,dport=12345),zone=1 >>>>>>>>>>> sctp,orig=(src=10.1.1.2,dst=10.1.1.1,sport=59874,dport=5202),reply=(src=10.1.1.1,dst=10.1.1.2,sport=5202,dport=12346),zone=1 >>>>>>>>>>> >>>>>>>>>>> instead of: >>>>>>>>>>> >>>>>>>>>>> sctp,orig=(src=10.1.1.2,dst=10.1.1.1,sport=0,dport=0),reply=(src=10.1.1.1,dst=10.1.1.2,sport=0,dport=0),zone=1 >>>>>>>>>>> >>>>>>>>>>> Signed-off-by: Paolo Valerio <[email protected]> >>>>>>>>>>> --- >>>>>>>>>> >>>>>>>>>> Thanks for this work - I think it looks good. >>>>>>>>>> >>>>>>>>>> Perhaps it should have a NEWS item mentioned that the userspace >>>>>>>>>> conntrack now supports matching SCTP l4 data. >>>>>>>>>> >>>>>>>>>> If you do spin a v4 with that change, you can keep my: >>>>>>>>>> >>>>>>>>>> Acked-by: Aaron Conole <[email protected]> >>>>>>>>> >>>>>>>>> Hi, Paolo and Aaron. >>>>>>>>> >>>>>>>>> I'm getting a consistent test failure while running check-kernel >>>>>>>>> on Ubuntu 22.10 with 5.19 kernel: >>>>>>>>> >>>>>>>>> >>>>>>>>> ./system-traffic.at:4754: cat ofctl_monitor.log >>>>>>>>> --- - 2023-06-14 11:26:41.958591125 +0000 >>>>>>>>> +++ >>>>>>>>> /root/ovs/tests/system-kmod-testsuite.dir/at-groups/105/stdout >>>>>>>>> 2023-06-14 11:26:41.952000000 +0000 >>>>>>>>> @@ -12,8 +12,6 @@ >>>>>>>>> >>>>>>>>> sctp,vlan_tci=0x0000,dl_src=e6:66:c1:22:22:22,dl_dst=e6:66:c1:11:11:11,nw_src=10.1.1.2,nw_dst=10.1.1.1,nw_tos=0,nw_ecn=2,nw_ttl=64,nw_frag=no,tp_src=12345,tp_dst=54969 >>>>>>>>> sctp_csum:9b67e853 >>>>>>>>> NXT_PACKET_IN2 (xid=0x0): cookie=0x0 total_len=54 in_port=1 >>>>>>>>> (via action) data_len=54 (unbuffered) >>>>>>>>> >>>>>>>>> sctp,vlan_tci=0x0000,dl_src=e6:66:c1:11:11:11,dl_dst=e6:66:c1:22:22:22,nw_src=10.1.1.240,nw_dst=10.1.1.2,nw_tos=0,nw_ecn=2,nw_ttl=64,nw_frag=no,tp_src=34567,tp_dst=12345 >>>>>>>>> sctp_csum:bc0e5463 >>>>>>>>> -NXT_PACKET_IN2 (xid=0x0): table_id=1 cookie=0x0 total_len=50 >>>>>>>>> ct_state=est|rpl|trk|dnat,ct_zone=1,ct_nw_src=10.1.1.1,ct_nw_dst=10.1.1.2,ct_nw_proto=132,ct_tp_src=54969,ct_tp_dst=12345,ip,in_port=2 >>>>>>>>> (via action) data_len=50 (unbuffered) >>>>>>>>> -sctp,vlan_tci=0x0000,dl_src=e6:66:c1:22:22:22,dl_dst=e6:66:c1:11:11:11,nw_src=10.1.1.2,nw_dst=10.1.1.1,nw_tos=0,nw_ecn=2,nw_ttl=64,nw_frag=no,tp_src=12345,tp_dst=54969 >>>>>>>>> sctp_csum:d6ce6b9e >>>>>>>>> NXT_PACKET_IN2 (xid=0x0): cookie=0x0 total_len=50 in_port=1 >>>>>>>>> (via action) data_len=50 (unbuffered) >>>>>>>>> -sctp,vlan_tci=0x0000,dl_src=e6:66:c1:11:11:11,dl_dst=e6:66:c1:22:22:22,nw_src=10.1.1.240,nw_dst=10.1.1.2,nw_tos=0,nw_ecn=2,nw_ttl=64,nw_frag=no,tp_src=34567,tp_dst=12345 >>>>>>>>> sctp_csum:add7db93 >>>>>>>>> +sctp,vlan_tci=0x0000,dl_src=e6:66:c1:11:11:11,dl_dst=e6:66:c1:22:22:22,nw_src=10.1.1.1,nw_dst=10.1.1.2,nw_tos=0,nw_ecn=2,nw_ttl=64,nw_frag=no,tp_src=54969,tp_dst=12345 >>>>>>>>> sctp_csum:5db68ce >>>>>>>>> >>>>>>>>> >>>>>>>>> Do you know what can be a problem here? >>>>>>>>> >>>>>>>>> Test is passing on Fedora 38 with 6.3 kernel and on rhel 9.2. >>>>>>>>> >>>>>>>> >>>>>>>> Hi Ilya, >>>>>>>> >>>>>>>> Uhm, it seems there's a problem with the shutdown sequence. >>>>>>>> I just ran the on a VM: >>>>>>>> >>>>>>>> vagrant@ubuntu2210:~/ovs$ grep CONFIG_NF_CT_PROTO_SCTP >>>>>>>> /boot/config-5.19.0-38-generic >>>>>>>> CONFIG_NF_CT_PROTO_SCTP=y >>>>>>>> >>>>>>>> vagrant@ubuntu2210:~/ovs$ grep VERSION /etc/os-release >>>>>>>> VERSION_ID="22.10" >>>>>>>> VERSION="22.10 (Kinetic Kudu)" >>>>>>>> VERSION_CODENAME=kinetic >>>>>>>> >>>>>>>> vagrant@ubuntu2210:~/ovs$ uname -r >>>>>>>> 5.19.0-38-generic >>>>>>> >>>>>>> The only difference with my VM is that I have -43-generic kernel. >>>>>>> >>>>>>>> >>>>>>>> but I can't see the failure. >>>>>>>> Any chance to see if they are marked for some reason as invalid? >>>>>>> >>>>>>> I dumped conntrack after every packet and here is what I see: >>>>>>> >>>>>>> On RHEL9, where test is working: >>>>>>> >>>>>>> 1. sctp 132 9 CLOSED src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>>> dport=12345 [UNREPLIED] src=10.1.1.2 dst=10.1.1.240 sport=12345 >>>>>>> dport=34567 mark=0 zone=1 use=1 >>>>>>> 2. sctp 132 2 COOKIE_WAIT src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>>> dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 dport=34567 >>>>>>> mark=0 zone=1 use=1 >>>>>>> 3. sctp 132 2 COOKIE_ECHOED src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>>> dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 dport=34567 >>>>>>> mark=0 zone=1 use=1 >>>>>>> 4. sctp 132 431999 ESTABLISHED src=10.1.1.1 dst=10.1.1.2 >>>>>>> sport=54969 dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 >>>>>>> dport=34567 [ASSURED] mark=0 zone=1 use=1 >>>>>>> 5. sctp 132 431999 ESTABLISHED src=10.1.1.1 dst=10.1.1.2 >>>>>>> sport=54969 dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 >>>>>>> dport=34567 [ASSURED] mark=0 zone=1 use=1 >>>>>>> 6. sctp 132 431999 ESTABLISHED src=10.1.1.1 dst=10.1.1.2 >>>>>>> sport=54969 dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 >>>>>>> dport=34567 [ASSURED] mark=0 zone=1 use=1 >>>>>>> 7. sctp 132 0 SHUTDOWN_SENT src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>>> dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 dport=34567 >>>>>>> [ASSURED] mark=0 zone=1 use=1 >>>>>>> 8. sctp 132 2 SHUTDOWN_ACK_SENT src=10.1.1.1 dst=10.1.1.2 >>>>>>> sport=54969 dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 >>>>>>> dport=34567 [ASSURED] mark=0 zone=1 use=1 >>>>>>> 9. sctp 132 9 CLOSED src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>>> dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 dport=34567 >>>>>>> [ASSURED] mark=0 zone=1 use=1 >>>>>> >>>>>> Here if I monitor conntrack during the test, I get: >>>>>> >>>>>> # conntrack -E --proto=sctp >>>>>> [NEW] sctp 132 10 CLOSED src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>> dport=12345 [UNREPLIED] src=10.1.1.2 dst=10.1.1.240 sport=12345 >>>>>> dport=34567 zone=1 >>>>>> [DESTROY] sctp 132 9 CLOSED src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>> dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 dport=34567 >>>>>> [ASSURED] zone=1 [USERSPACE] portid=3715 >>>>>> >>>>>> >>>>>>> >>>>>>> On Ubuntu, where it doesn't work: >>>>>>> >>>>>>> 1. sctp 132 9 CLOSED src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>>> dport=12345 [UNREPLIED] src=10.1.1.2 dst=10.1.1.240 sport=12345 >>>>>>> dport=34567 zone=1 use=1 >>>>>>> 2. sctp 132 2 COOKIE_WAIT src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>>> dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 dport=34567 >>>>>>> zone=1 use=1 >>>>>>> 3. sctp 132 2 COOKIE_ECHOED src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>>> dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 dport=34567 >>>>>>> zone=1 use=1 >>>>>>> 4. sctp 132 209 ESTABLISHED src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>>> dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 dport=34567 >>>>>>> [ASSURED] zone=1 use=1 >>>>>>> 5. sctp 132 209 ESTABLISHED src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>>> dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 dport=34567 >>>>>>> [ASSURED] zone=1 use=1 >>>>>>> 6. sctp 132 209 ESTABLISHED src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>>> dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 dport=34567 >>>>>>> [ASSURED] zone=1 use=1 >>>>>>> 7. sctp 132 0 SHUTDOWN_SENT src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>>> dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 dport=34567 >>>>>>> [ASSURED] zone=1 use=1 >>>>>>> 8. NO ENTRY! >>>>>>> 9. NO ENTRY! >>>>>> >>>>>> But here I have: >>>>>> >>>>>> # conntrack -E --proto=sctp >>>>>> [NEW] sctp 132 10 CLOSED src=10.1.1.1 dst=10.1.1.2 sport=54969 >>>>>> dport=12345 [UNREPLIED] src=10.1.1.2 dst=10.1.1.240 sport=12345 >>>>>> dport=34567 zone=1 >>>>>> [DESTROY] sctp 132 SHUTDOWN_SENT src=10.1.1.1 dst=10.1.1.2 >>>>>> sport=54969 dport=12345 src=10.1.1.2 dst=10.1.1.240 sport=12345 >>>>>> dport=34567 [ASSURED] zone=1 >>>>>> >>>>>> So, the connection indeed is getting destroyed while in SHUTDOWN_SENT >>>>>> state. >>>>>> Sounds like a kernel bug in Ubuntu... >>>>>> >>>>> >>>>> Thanks Ilya for digging more into it. >>>>> It seemed so to me as well, but looking at the logs you provided, the >>>>> timeout for SHUTDOWN_SENT is 0 (third column of the dump), so it seems >>>>> it's a matter of speed in the reply. >>>>> >>>>> ❯ cat /proc/sys/net/netfilter/nf_conntrack_sctp_timeout_shutdown_sent >>>>> 0 >>>>> >>>>> I'm a bit surprised by this. >>>>> >>>>> Just to confirm that, I upgraded the kernel of my vm to >>>>> 5.19.0-43-generic and the test succeeded. Sleeping for 1 second in >>>>> SHUTDOWN_SENT before sending the SHUTDOWN_ACK_SENT make the test fail. I >>>>> would expect the same on RHEL 9 and Fedora. >>>> >>>> Hmm, good point. >>>> The difference between my tests on Ubuntu and RHEL is that I tested >>>> with -O1 and sanitizers on Ubuntu, so it was a tiny bit slower. >>>> I just tried to run with sanitizers on RHEL and I'm getting the same >>>> failure as I have in Ubuntu. >>>> >>>> So, the test seems to be extremely time-sensitive. Is there a way >>>> to make it more stable? >>> >>> I guess one alternative could be change from trying to dump the ct >>> information directly, check for the 'conntrack' utility, and use that to >>> log the events - then sweep the ct events log. It seems the ofctl >>> monitor is showing that it is a bit racy, but maybe relying on the ct >>> events log could still give us the confidence that it is working without >>> the raciness of the ofctl monitor? >> >> The problem is that we will get different events based on the reaction time. >> If the test is slow for any reason, the [DESTROY] event is happening in the >> SHUTDOWN_SENT state and SHUTDOWN_ACK and SHUTDOWN_COMPLETE packets are >> not coming through correctly. If the test is fast, then the [DESTROY] event >> is happening in the CLOSED state and the last two packets are reaching the >> destination. >> >> So, we will have different output regardless of what monitoring tool we're >> using. Or am I missing something? >> > > Probably an interesting solution would have been applying a timeout > policy for zone 1 for the kernel dp, but that is not implemented for > sctp. > > One last chance to exploit would be aborting [0] the association instead > of gracefully terminate it with the shutdown sequence. > > If my reading is correct, that conntrack entry would transitions to the > CLOSED state and the last timeout (10 seconds) should not matter > anymore. I didn't test it, so it needs to be confirmed. > > Given the very narrow timeout, I had to rule out all the other > alternatives that came to mind. > > https://www.rfc-editor.org/rfc/rfc4960#section-9.1
Sounds fine to me. Since it's a valid way to close the connection, we may test that instead. Best regards, Ilya Maximets. > >>> >>>>> >>>>> Paolo >>>>> >>>>>>> >>>>>>> So, after sending SHUTDOWN_ACK, there is no conntrack entry in the >>>>>>> kernel anymore. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>>> Best regards, Ilya Maximets. >>>>>>>> >>>>>>> >>>>> >>> > _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
