Hi Eyle, Excellent, thanks for letting me know ! :-)
--a > On 6 Sep 2019, at 12:00, Eyle Brinkhuis <[email protected]> wrote: > > Hi Andrew, > > Awesome, thanks so much for your explanation! I have fixed the problem. > Apparently, these rules are set from openstack, which applies these rules on > the ports. That means that adding the right ranges to this port will allow > the traffic, which it does. > > I thought I had it all arranged by disabling port security in the openstack > environment, but apparently, port-security is only disabled in openstack when > one does not have any security group nor allowed-address pair assigned to the > instance. I guess we'll have to sort security on a different place then. > > Thanks so much once again! > > Regards, > > Eyle > > On 06/09/2019, 11:18, "Andrew 👽 Yourtchenko" <[email protected]> wrote: > > Ok so I will explain the logic a bit more and maybe this will solve the > puzzle. > > Assumption: that the trace in your first mail and the show output in > the last one correspond to the same state of the VPP and the stuff > attahed to it (sw_if_index, etc.) > > The purpose in life of the MACIP ACLs is to give a stateless way to > enforce the correspondence of IP and MAC, as well as to ensure the > hosts on a given VPP interface are using the MAC address they are > supposed to use. > > the ingress packet in the frame 3 (the dropped one) in the trace > received on VirtualEthernet0/0/3 is: > l2-input: sw_if_index 9 dst fa:16:3e:93:0c:50 src fa:16:3e:26:3e:0e > > so we need to have a MACIP ACL that tells tells that > fa:16:3e:26:3e:0e is okay to hang off that interface. > > We can indeed find that it is the macip acl #2: > > MACIP acl_index: 2, count: 2 (true len 2) tag {} is free pool slot: 0 > > ip4_table_index 17, ip6_table_index 17, l2_table_index 17 > > out_ip4_table_index -1, out_ip6_table_index -1, out_l2_table_index -1 > > rule 0: ipv4 action 1 ip 0.0.0.0/32 mac fa:16:3e:26:3e:0e mask > ff:ff:ff:ff:ff:ff > > rule 1: ipv4 action 1 ip 145.144.1.78/32 mac fa:16:3e:26:3e:0e > mask ff:ff:ff:ff:ff:ff > > applied on sw_if_index(s): 9 > > so, it has the good MAC address and it is applied, but this ACL says > "I can either allow the source IP of 0.0.0.0 or a source IP of > 145.144.1.78 incoming on VirtualEthernet0/0/3 with the source of > fa:16:3e:26:3e:0e. > > So from the trace, the packet is: > > lc_index 0 l3 ip4 145.144.1.53 -> 145.144.1.84 l4 > lsb_of_sw_if_index 9 proto 1 l4_is_input 1 l4_slow_path 1 l4_flags > 0x03 port 0 -> 0 tcp flags (invalid) 00 rsvd 0 > > The source of 145.144.1.53 is *not* the 145.144.1.78 which MACIP ACL > permits to appear on that interface, so from this I can conclude it > all WorksAsConfigured(tm). > > I think by default the OpenStack assumes the endpoint connected to VPP > is a host, which in your case it isn't. So you would need to relax the > whichever policies govern that. I am not sure where, since I deal with > the VPP side, all I see is the API calls :-) - maybe Naveen might give > a pointer or two here... > > This all is fairly old code, so I think tweaking the config will work. > > But, for curiosity/if my assumption in the beginning is wrong, or in > case some curious reader finds this mail later, I will go a bit > further and tell what the next steps would have been if we *did* find > that the MACIP ACL was configured correctly to permit the source > IP+MAC that we saw in frame 3 of the trace. For that we need to know > how MACIP ACL functions under the hood. > > ACL plugin reuses the pre-existing core feature of VPP called "IPACL > with classifier tables". > A bit confusing, but that's why you see it in the trace. > Classifier tables are a pretty cool thing - you can match up to 5*u32 > worth of contiguous bits with an arbitrary bitmask pretty much > anywhere in the packet, with a fixed offset. But the drawback is that > all the possible values matched need to be the *same* place and the > same bitmask. So to deal with that, the classifier tables can be > chained - and the lookup made sequentially. > > If you do "show classify tables verbose" you will see the tables in > all the glory, including the contents. There one can verify at a lower > level that what the ACL plugin programs does make sense. > > hope this helps! > > --a > > > > > >> On 9/6/19, Eyle Brinkhuis <[email protected]> wrote: >> Okay.. so when I pull up the macip acl’s: >> >> vpp# show acl-plugin macip acl >> MACIP acl_index: 0, count: 2 (true len 2) tag {} is free pool slot: 0 >> ip4_table_index 5, ip6_table_index 5, l2_table_index 5 >> out_ip4_table_index -1, out_ip6_table_index -1, out_l2_table_index -1 >> rule 0: ipv4 action 1 ip 0.0.0.0/32 mac fa:16:3e:79:87:05 mask >> ff:ff:ff:ff:ff:ff >> rule 1: ipv4 action 1 ip 145.144.1.5/32 mac fa:16:3e:79:87:05 mask >> ff:ff:ff:ff:ff:ff >> applied on sw_if_index(s): >> MACIP acl_index: 1, count: 2 (true len 2) tag {} is free pool slot: 0 >> ip4_table_index 11, ip6_table_index 11, l2_table_index 11 >> out_ip4_table_index -1, out_ip6_table_index -1, out_l2_table_index -1 >> rule 0: ipv4 action 1 ip 0.0.0.0/32 mac fa:16:3e:79:87:05 mask >> ff:ff:ff:ff:ff:ff >> rule 1: ipv4 action 1 ip 145.144.1.5/32 mac fa:16:3e:79:87:05 mask >> ff:ff:ff:ff:ff:ff >> applied on sw_if_index(s): 4 >> MACIP acl_index: 2, count: 2 (true len 2) tag {} is free pool slot: 0 >> ip4_table_index 17, ip6_table_index 17, l2_table_index 17 >> out_ip4_table_index -1, out_ip6_table_index -1, out_l2_table_index -1 >> rule 0: ipv4 action 1 ip 0.0.0.0/32 mac fa:16:3e:26:3e:0e mask >> ff:ff:ff:ff:ff:ff >> rule 1: ipv4 action 1 ip 145.144.1.78/32 mac fa:16:3e:26:3e:0e mask >> ff:ff:ff:ff:ff:ff >> applied on sw_if_index(s): 9 >> MACIP acl_index: 3, count: 2 (true len 2) tag {} is free pool slot: 0 >> ip4_table_index 23, ip6_table_index 23, l2_table_index 23 >> out_ip4_table_index -1, out_ip6_table_index -1, out_l2_table_index -1 >> rule 0: ipv4 action 1 ip 0.0.0.0/32 mac fa:16:3e:10:04:3e mask >> ff:ff:ff:ff:ff:ff >> rule 1: ipv4 action 1 ip 145.144.1.29/32 mac fa:16:3e:10:04:3e mask >> ff:ff:ff:ff:ff:ff >> applied on sw_if_index(s): 5 >> MACIP acl_index: 4, count: 2 (true len 2) tag {} is free pool slot: 0 >> ip4_table_index 29, ip6_table_index 29, l2_table_index 29 >> out_ip4_table_index -1, out_ip6_table_index -1, out_l2_table_index -1 >> rule 0: ipv4 action 1 ip 0.0.0.0/32 mac fa:16:3e:7c:96:d0 mask >> ff:ff:ff:ff:ff:ff >> rule 1: ipv4 action 1 ip 145.144.1.53/32 mac fa:16:3e:7c:96:d0 mask >> ff:ff:ff:ff:ff:ff >> applied on sw_if_index(s): 7 >> MACIP acl_index: 6, count: 2 (true len 2) tag {} is free pool slot: 0 >> ip4_table_index 41, ip6_table_index 41, l2_table_index 41 >> out_ip4_table_index -1, out_ip6_table_index -1, out_l2_table_index -1 >> rule 0: ipv4 action 1 ip 0.0.0.0/32 mac fa:16:3e:93:0c:50 mask >> ff:ff:ff:ff:ff:ff >> rule 1: ipv4 action 1 ip 145.144.1.84/32 mac fa:16:3e:93:0c:50 mask >> ff:ff:ff:ff:ff:ff >> applied on sw_if_index(s): 10 >> >> >> >> But then compare that to the show hardware: >> vpp# sh hard >> Name Idx Link Hardware >> VirtualEthernet0/0/0 3 up VirtualEthernet0/0/0 >> Link speed: unknown >> Ethernet address fa:16:3c:c9:a8:50 >> VirtualEthernet0/0/1 4 up VirtualEthernet0/0/1 >> Link speed: unknown >> Ethernet address fa:16:3c:70:15:b3 >> VirtualEthernet0/0/2 5 up VirtualEthernet0/0/2 >> Link speed: unknown >> Ethernet address fa:16:3c:05:66:7c >> VirtualEthernet0/0/3 6 up VirtualEthernet0/0/3 >> Link speed: unknown >> Ethernet address fa:16:3c:f0:21:0a >> VirtualEthernet0/0/4 7 up VirtualEthernet0/0/4 >> Link speed: unknown >> Ethernet address fa:16:3c:0f:9d:5d >> local0 0 down local0 >> Link speed: unknown >> local >> rdma0 1 up rdma0 >> Link speed: 40 Gbps >> Ethernet address 02:fe:99:32:82:4f >> flags: admin-up promiscuous >> rdma1 2 up rdma1 >> Link speed: 40 Gbps >> Ethernet address 02:fe:27:ea:09:82 >> flags: admin-up >> >> It looks like there doesn’t even exist an acl for VirtualEthernet0/0/3? Is >> that why it is dropped? >> >> Eyle >> >> From: Andrew 👽 Yourtchenko <[email protected]> >> Date: Thursday, 5 September 2019 at 19:20 >> To: "Naveen Joy (najoy)" <[email protected]> >> Cc: Eyle Brinkhuis <[email protected]>, "[email protected]" >> <[email protected]> >> Subject: Re: [vpp-dev] ACL drops while pinging another interface >> >> It hits the session, so it does pass the L3 acl. Just before ... >> --a >> >> On 5 Sep 2019, at 18:52, Naveen Joy (najoy) >> <[email protected]<mailto:[email protected]>> wrote: >> From the trace, it appears like ICMP Echo reply is not permitted by the >> security-group applied on neutron’s port corresponding to >> VirtualEthernet0/0/3. >> This could be causing the ICMP reply packet from the firewall to drop. >> >> lc_index 0 l3 ip4 145.144.1.53 -> 145.144.1.84 l4 lsb_of_sw_if_index 9 >> proto 1 l4_is_input 1 l4_slow_path 1 l4_flags 0x03 port 0 -> 0 tcp flags >> (invalid) 00 rsvd 0 >> >> 00:53:47:316359: l2-input-feat-arc-end >> >> IN-FEAT-ARC: head 0 feature_bitmap 100525 ethertype 0 sw_if_index -1, >> next_index 17 >> >> 00:53:47:316360: l2-input-acl >> >> INACL: sw_if_index 9, next_index 0, table 12, offset -1 >> >> 00:53:47:316361: error-drop >> >> rx:VirtualEthernet0/0/3 >> >> -Naveen >> >> From: <[email protected]<mailto:[email protected]>> on behalf of Andrew >> Yourtchenko <[email protected]<mailto:[email protected]>> >> Date: Thursday, September 5, 2019 at 7:20 AM >> To: Eyle Brinkhuis >> <[email protected]<mailto:[email protected]>> >> Cc: "[email protected]<mailto:[email protected]>" >> <[email protected]<mailto:[email protected]>> >> Subject: Re: [vpp-dev] ACL drops while pinging another interface >> >> Thanks for the traces ! >> >> MACIP acl uses the classifier-bases “ip-acl”; so it sounds like it is not >> programmed with the source Mac of your packets. >> >> “Show acl-plugin macip” will help to see what the acl plugin sees, and if it >> looks legit, then you can check the classifier tables applied as input acl >> to verify those are programmed correctly. >> >> Shout if you get stuck :) >> >> --a >> >> On 5 Sep 2019, at 14:18, Eyle Brinkhuis >> <[email protected]<mailto:[email protected]>> wrote: >> Hi guys, >> >> I’m using VPP 19.08 with networking-vpp in an openstack stein environment, >> where we are busy building an open environment that is specifically built >> for NFV applications. One of those functions is a firewall setup, where we >> firewall a customer’s traffic and provide said customer with a ‘clean and >> safe’ internet connection. >> >> As such, I am evaluating a VPP setup, which looks very promising. However: >> in the following scenario, I run into an issue: >> >> I have a compute host on which I have a firewall running ánd a guest (cirros >> for now). Setup is as follows: >> >> 145.144.1.53-fa:16:3e:7c:96:d0 – VirtualEthernet0/0/2 | firewall instance | >> VirtualEthernet0/03 145.144.1.78 - fa:16:3e:26:3e:0e <–> vlan 69 <–> >> 145.144.1.84 - fa:16:3e:93:0c:50- VirtualEthernet0/0/4 | cirros instance | >> >> From the cirros instance pingin the inside interface of the firewall (0/0/3) >> works like a charm, I wouldn’t have expected any different. >> >> When I try to ping the outside interface of the firewall (0/0/2), traces >> show the following: >> >> >> 0:53:47:316205: vhost-user-input >> >> VirtualEthernet0/0/4 queue 0 >> >> virtio flags: >> >> INDIRECT Indirect descriptor >> >> virtio_net_hdr first_desc_len 12 >> >> flags 0x00 gso_type 0 >> >> num_buff 0 >> >> 00:53:47:316208: ethernet-input >> >> frame: flags 0x1, hw-if-index 7, sw-if-index 10 >> >> IP4: fa:16:3e:93:0c:50 -> fa:16:3e:26:3e:0e >> >> 00:53:47:316209: l2-input >> >> l2-input: sw_if_index 10 dst fa:16:3e:26:3e:0e src fa:16:3e:93:0c:50 >> >> 00:53:47:316210: l2-input-feat-arc >> >> IN-FEAT-ARC: head 1 feature_bitmap 500525 ethertype 800 sw_if_index 10, >> next_index 22 >> >> 00:53:47:316211: acl-plugin-in-ip4-l2 >> >> acl-plugin: lc_index: -1, sw_if_index 10, next index 1, action: 3, match: >> acl -1 rule 44 trace_bits 80000000 >> >> pkt info 0000000000000000 0000000000000000 0000000000000000 >> 3501909154019091 000a030100000008 0200ffff00000000 >> >> lc_index 0 l3 ip4 145.144.1.84 -> 145.144.1.53 l4 lsb_of_sw_if_index 10 >> proto 1 l4_is_input 1 l4_slow_path 1 l4_flags 0x03 port 8 -> 0 tcp flags >> (invalid) 00 rsvd 0 >> >> 00:53:47:316214: l2-input-feat-arc-end >> >> IN-FEAT-ARC: head 0 feature_bitmap 100525 ethertype 0 sw_if_index -1, >> next_index 17 >> >> 00:53:47:316215: l2-input-acl >> >> INACL: sw_if_index 10, next_index 9, table 41, offset 1392 >> >> 00:53:47:316216: l2-learn >> >> l2-learn: sw_if_index 10 dst fa:16:3e:26:3e:0e src fa:16:3e:93:0c:50 >> bd_index 3 >> >> 00:53:47:316218: l2-fwd >> >> l2-fwd: sw_if_index 10 dst fa:16:3e:26:3e:0e src fa:16:3e:93:0c:50 >> bd_index 3 result [0x5d50000000009, 9] none >> >> 00:53:47:316219: l2-output >> >> l2-output: sw_if_index 9 dst fa:16:3e:26:3e:0e src fa:16:3e:93:0c:50 data >> 08 00 45 00 00 54 33 9c 40 00 40 01 >> >> 00:53:47:316220: l2-output-feat-arc >> >> OUT-FEAT-ARC: head 1 feature_bitmap 4001 ethertype 800 sw_if_index 9, >> next_index 11 >> >> 00:53:47:316220: acl-plugin-out-ip4-l2 >> >> acl-plugin: lc_index: 6, sw_if_index 9, next index 1, action: 1, match: >> acl 4 rule 2 trace_bits 00000000 >> >> pkt info 0000000000000000 0000000000000000 0000000000000000 >> 3501909154019091 0009020100000008 0200ffff00000006 >> >> lc_index 6 l3 ip4 145.144.1.84 -> 145.144.1.53 l4 lsb_of_sw_if_index 9 >> proto 1 l4_is_input 0 l4_slow_path 1 l4_flags 0x02 port 8 -> 0 tcp flags >> (invalid) 00 rsvd 0 >> >> 00:53:47:316223: l2-output-feat-arc-end >> >> OUT-FEAT-ARC: head 0 feature_bitmap 1 ethertype 0 sw_if_index -1, >> next_index 0 >> >> 00:53:47:316224: VirtualEthernet0/0/3-output >> >> VirtualEthernet0/0/3 l2_hdr_offset_valid l3_hdr_offset_valid >> >> IP4: fa:16:3e:93:0c:50 -> fa:16:3e:26:3e:0e >> >> ICMP: 145.144.1.84 -> 145.144.1.53 >> >> tos 0x00, ttl 64, length 84, checksum 0xe163 >> >> fragment id 0x339c, flags DONT_FRAGMENT >> >> ICMP echo_request checksum 0x9914 >> >> 00:53:47:316225: VirtualEthernet0/0/3-tx >> >> VirtualEthernet0/0/3 queue 0 >> >> virtio flags: >> >> SINGLE_DESC Single descriptor packet >> >> virtio_net_hdr first_desc_len 4096 >> >> flags 0x00 gso_type 0 >> >> num_buff 1 >> >> >> >> Packet 3 >> >> >> >> 00:53:47:316357: vhost-user-input >> >> VirtualEthernet0/0/3 queue 0 >> >> virtio flags: >> >> INDIRECT Indirect descriptor >> >> virtio_net_hdr first_desc_len 12 >> >> flags 0x00 gso_type 0 >> >> num_buff 0 >> >> 00:53:47:316358: ethernet-input >> >> frame: flags 0x1, hw-if-index 6, sw-if-index 9 >> >> IP4: fa:16:3e:26:3e:0e -> fa:16:3e:93:0c:50 >> >> 00:53:47:316358: l2-input >> >> l2-input: sw_if_index 9 dst fa:16:3e:93:0c:50 src fa:16:3e:26:3e:0e >> >> 00:53:47:316359: l2-input-feat-arc >> >> IN-FEAT-ARC: head 1 feature_bitmap 500525 ethertype 800 sw_if_index 9, >> next_index 22 >> >> 00:53:47:316359: acl-plugin-in-ip4-l2 >> >> acl-plugin: lc_index: -1, sw_if_index 9, next index 1, action: 3, match: >> acl -1 rule 97 trace_bits 80000000 >> >> pkt info 0000000000000000 0000000000000000 0000000000000000 >> 5401909135019091 0009030100000000 0200ffff00000000 >> >> lc_index 0 l3 ip4 145.144.1.53 -> 145.144.1.84 l4 lsb_of_sw_if_index 9 >> proto 1 l4_is_input 1 l4_slow_path 1 l4_flags 0x03 port 0 -> 0 tcp flags >> (invalid) 00 rsvd 0 >> >> 00:53:47:316359: l2-input-feat-arc-end >> >> IN-FEAT-ARC: head 0 feature_bitmap 100525 ethertype 0 sw_if_index -1, >> next_index 17 >> >> 00:53:47:316360: l2-input-acl >> >> INACL: sw_if_index 9, next_index 0, table 12, offset -1 >> >> 00:53:47:316361: error-drop >> >> rx:VirtualEthernet0/0/3 >> >> 00:53:47:316362: drop >> >> l2-input-acl: input ACL table-miss drops >> >> >> The packets get dropped.. I have found all the acl’s in vpp, but I cannot >> really see what would be wrong here, or what causes the packets to be >> dropped. Has anyone else had any experiences like this, or a solution? >> >> Regards, >> >> Eyle >> >> -=-=-=-=-=-=-=-=-=-=-=- >> Links: You receive all messages sent to this group. >> >> View/Reply Online (#13909): https://lists.fd.io/g/vpp-dev/message/13909 >> Mute This Topic: https://lists.fd.io/mt/33152601/675608 >> Group Owner: [email protected]<mailto:[email protected]> >> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub >> [[email protected]<mailto:[email protected]>] >> -=-=-=-=-=-=-=-=-=-=-=- >> > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13918): https://lists.fd.io/g/vpp-dev/message/13918 Mute This Topic: https://lists.fd.io/mt/33152601/21656 Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
