Hi Andrew, Awesome, thanks so much for your explanation! I have fixed the problem. Apparently, these rules are set from openstack, which applies these rules on the ports. That means that adding the right ranges to this port will allow the traffic, which it does.
I thought I had it all arranged by disabling port security in the openstack environment, but apparently, port-security is only disabled in openstack when one does not have any security group nor allowed-address pair assigned to the instance. I guess we'll have to sort security on a different place then. Thanks so much once again! Regards, Eyle On 06/09/2019, 11:18, "Andrew 👽 Yourtchenko" <[email protected]> wrote: Ok so I will explain the logic a bit more and maybe this will solve the puzzle. Assumption: that the trace in your first mail and the show output in the last one correspond to the same state of the VPP and the stuff attahed to it (sw_if_index, etc.) The purpose in life of the MACIP ACLs is to give a stateless way to enforce the correspondence of IP and MAC, as well as to ensure the hosts on a given VPP interface are using the MAC address they are supposed to use. the ingress packet in the frame 3 (the dropped one) in the trace received on VirtualEthernet0/0/3 is: l2-input: sw_if_index 9 dst fa:16:3e:93:0c:50 src fa:16:3e:26:3e:0e so we need to have a MACIP ACL that tells tells that fa:16:3e:26:3e:0e is okay to hang off that interface. We can indeed find that it is the macip acl #2: MACIP acl_index: 2, count: 2 (true len 2) tag {} is free pool slot: 0 ip4_table_index 17, ip6_table_index 17, l2_table_index 17 out_ip4_table_index -1, out_ip6_table_index -1, out_l2_table_index -1 rule 0: ipv4 action 1 ip 0.0.0.0/32 mac fa:16:3e:26:3e:0e mask ff:ff:ff:ff:ff:ff rule 1: ipv4 action 1 ip 145.144.1.78/32 mac fa:16:3e:26:3e:0e mask ff:ff:ff:ff:ff:ff applied on sw_if_index(s): 9 so, it has the good MAC address and it is applied, but this ACL says "I can either allow the source IP of 0.0.0.0 or a source IP of 145.144.1.78 incoming on VirtualEthernet0/0/3 with the source of fa:16:3e:26:3e:0e. So from the trace, the packet is: lc_index 0 l3 ip4 145.144.1.53 -> 145.144.1.84 l4 lsb_of_sw_if_index 9 proto 1 l4_is_input 1 l4_slow_path 1 l4_flags 0x03 port 0 -> 0 tcp flags (invalid) 00 rsvd 0 The source of 145.144.1.53 is *not* the 145.144.1.78 which MACIP ACL permits to appear on that interface, so from this I can conclude it all WorksAsConfigured(tm). I think by default the OpenStack assumes the endpoint connected to VPP is a host, which in your case it isn't. So you would need to relax the whichever policies govern that. I am not sure where, since I deal with the VPP side, all I see is the API calls :-) - maybe Naveen might give a pointer or two here... This all is fairly old code, so I think tweaking the config will work. But, for curiosity/if my assumption in the beginning is wrong, or in case some curious reader finds this mail later, I will go a bit further and tell what the next steps would have been if we *did* find that the MACIP ACL was configured correctly to permit the source IP+MAC that we saw in frame 3 of the trace. For that we need to know how MACIP ACL functions under the hood. ACL plugin reuses the pre-existing core feature of VPP called "IPACL with classifier tables". A bit confusing, but that's why you see it in the trace. Classifier tables are a pretty cool thing - you can match up to 5*u32 worth of contiguous bits with an arbitrary bitmask pretty much anywhere in the packet, with a fixed offset. But the drawback is that all the possible values matched need to be the *same* place and the same bitmask. So to deal with that, the classifier tables can be chained - and the lookup made sequentially. If you do "show classify tables verbose" you will see the tables in all the glory, including the contents. There one can verify at a lower level that what the ACL plugin programs does make sense. hope this helps! --a On 9/6/19, Eyle Brinkhuis <[email protected]> wrote: > Okay.. so when I pull up the macip acl’s: > > vpp# show acl-plugin macip acl > MACIP acl_index: 0, count: 2 (true len 2) tag {} is free pool slot: 0 > ip4_table_index 5, ip6_table_index 5, l2_table_index 5 > out_ip4_table_index -1, out_ip6_table_index -1, out_l2_table_index -1 > rule 0: ipv4 action 1 ip 0.0.0.0/32 mac fa:16:3e:79:87:05 mask > ff:ff:ff:ff:ff:ff > rule 1: ipv4 action 1 ip 145.144.1.5/32 mac fa:16:3e:79:87:05 mask > ff:ff:ff:ff:ff:ff > applied on sw_if_index(s): > MACIP acl_index: 1, count: 2 (true len 2) tag {} is free pool slot: 0 > ip4_table_index 11, ip6_table_index 11, l2_table_index 11 > out_ip4_table_index -1, out_ip6_table_index -1, out_l2_table_index -1 > rule 0: ipv4 action 1 ip 0.0.0.0/32 mac fa:16:3e:79:87:05 mask > ff:ff:ff:ff:ff:ff > rule 1: ipv4 action 1 ip 145.144.1.5/32 mac fa:16:3e:79:87:05 mask > ff:ff:ff:ff:ff:ff > applied on sw_if_index(s): 4 > MACIP acl_index: 2, count: 2 (true len 2) tag {} is free pool slot: 0 > ip4_table_index 17, ip6_table_index 17, l2_table_index 17 > out_ip4_table_index -1, out_ip6_table_index -1, out_l2_table_index -1 > rule 0: ipv4 action 1 ip 0.0.0.0/32 mac fa:16:3e:26:3e:0e mask > ff:ff:ff:ff:ff:ff > rule 1: ipv4 action 1 ip 145.144.1.78/32 mac fa:16:3e:26:3e:0e mask > ff:ff:ff:ff:ff:ff > applied on sw_if_index(s): 9 > MACIP acl_index: 3, count: 2 (true len 2) tag {} is free pool slot: 0 > ip4_table_index 23, ip6_table_index 23, l2_table_index 23 > out_ip4_table_index -1, out_ip6_table_index -1, out_l2_table_index -1 > rule 0: ipv4 action 1 ip 0.0.0.0/32 mac fa:16:3e:10:04:3e mask > ff:ff:ff:ff:ff:ff > rule 1: ipv4 action 1 ip 145.144.1.29/32 mac fa:16:3e:10:04:3e mask > ff:ff:ff:ff:ff:ff > applied on sw_if_index(s): 5 > MACIP acl_index: 4, count: 2 (true len 2) tag {} is free pool slot: 0 > ip4_table_index 29, ip6_table_index 29, l2_table_index 29 > out_ip4_table_index -1, out_ip6_table_index -1, out_l2_table_index -1 > rule 0: ipv4 action 1 ip 0.0.0.0/32 mac fa:16:3e:7c:96:d0 mask > ff:ff:ff:ff:ff:ff > rule 1: ipv4 action 1 ip 145.144.1.53/32 mac fa:16:3e:7c:96:d0 mask > ff:ff:ff:ff:ff:ff > applied on sw_if_index(s): 7 > MACIP acl_index: 6, count: 2 (true len 2) tag {} is free pool slot: 0 > ip4_table_index 41, ip6_table_index 41, l2_table_index 41 > out_ip4_table_index -1, out_ip6_table_index -1, out_l2_table_index -1 > rule 0: ipv4 action 1 ip 0.0.0.0/32 mac fa:16:3e:93:0c:50 mask > ff:ff:ff:ff:ff:ff > rule 1: ipv4 action 1 ip 145.144.1.84/32 mac fa:16:3e:93:0c:50 mask > ff:ff:ff:ff:ff:ff > applied on sw_if_index(s): 10 > > > > But then compare that to the show hardware: > vpp# sh hard > Name Idx Link Hardware > VirtualEthernet0/0/0 3 up VirtualEthernet0/0/0 > Link speed: unknown > Ethernet address fa:16:3c:c9:a8:50 > VirtualEthernet0/0/1 4 up VirtualEthernet0/0/1 > Link speed: unknown > Ethernet address fa:16:3c:70:15:b3 > VirtualEthernet0/0/2 5 up VirtualEthernet0/0/2 > Link speed: unknown > Ethernet address fa:16:3c:05:66:7c > VirtualEthernet0/0/3 6 up VirtualEthernet0/0/3 > Link speed: unknown > Ethernet address fa:16:3c:f0:21:0a > VirtualEthernet0/0/4 7 up VirtualEthernet0/0/4 > Link speed: unknown > Ethernet address fa:16:3c:0f:9d:5d > local0 0 down local0 > Link speed: unknown > local > rdma0 1 up rdma0 > Link speed: 40 Gbps > Ethernet address 02:fe:99:32:82:4f > flags: admin-up promiscuous > rdma1 2 up rdma1 > Link speed: 40 Gbps > Ethernet address 02:fe:27:ea:09:82 > flags: admin-up > > It looks like there doesn’t even exist an acl for VirtualEthernet0/0/3? Is > that why it is dropped? > > Eyle > > From: Andrew 👽 Yourtchenko <[email protected]> > Date: Thursday, 5 September 2019 at 19:20 > To: "Naveen Joy (najoy)" <[email protected]> > Cc: Eyle Brinkhuis <[email protected]>, "[email protected]" > <[email protected]> > Subject: Re: [vpp-dev] ACL drops while pinging another interface > > It hits the session, so it does pass the L3 acl. Just before ... > --a > > On 5 Sep 2019, at 18:52, Naveen Joy (najoy) > <[email protected]<mailto:[email protected]>> wrote: > From the trace, it appears like ICMP Echo reply is not permitted by the > security-group applied on neutron’s port corresponding to > VirtualEthernet0/0/3. > This could be causing the ICMP reply packet from the firewall to drop. > > lc_index 0 l3 ip4 145.144.1.53 -> 145.144.1.84 l4 lsb_of_sw_if_index 9 > proto 1 l4_is_input 1 l4_slow_path 1 l4_flags 0x03 port 0 -> 0 tcp flags > (invalid) 00 rsvd 0 > > 00:53:47:316359: l2-input-feat-arc-end > > IN-FEAT-ARC: head 0 feature_bitmap 100525 ethertype 0 sw_if_index -1, > next_index 17 > > 00:53:47:316360: l2-input-acl > > INACL: sw_if_index 9, next_index 0, table 12, offset -1 > > 00:53:47:316361: error-drop > > rx:VirtualEthernet0/0/3 > > -Naveen > > From: <[email protected]<mailto:[email protected]>> on behalf of Andrew > Yourtchenko <[email protected]<mailto:[email protected]>> > Date: Thursday, September 5, 2019 at 7:20 AM > To: Eyle Brinkhuis > <[email protected]<mailto:[email protected]>> > Cc: "[email protected]<mailto:[email protected]>" > <[email protected]<mailto:[email protected]>> > Subject: Re: [vpp-dev] ACL drops while pinging another interface > > Thanks for the traces ! > > MACIP acl uses the classifier-bases “ip-acl”; so it sounds like it is not > programmed with the source Mac of your packets. > > “Show acl-plugin macip” will help to see what the acl plugin sees, and if it > looks legit, then you can check the classifier tables applied as input acl > to verify those are programmed correctly. > > Shout if you get stuck :) > > --a > > On 5 Sep 2019, at 14:18, Eyle Brinkhuis > <[email protected]<mailto:[email protected]>> wrote: > Hi guys, > > I’m using VPP 19.08 with networking-vpp in an openstack stein environment, > where we are busy building an open environment that is specifically built > for NFV applications. One of those functions is a firewall setup, where we > firewall a customer’s traffic and provide said customer with a ‘clean and > safe’ internet connection. > > As such, I am evaluating a VPP setup, which looks very promising. However: > in the following scenario, I run into an issue: > > I have a compute host on which I have a firewall running ánd a guest (cirros > for now). Setup is as follows: > > 145.144.1.53-fa:16:3e:7c:96:d0 – VirtualEthernet0/0/2 | firewall instance | > VirtualEthernet0/03 145.144.1.78 - fa:16:3e:26:3e:0e <–> vlan 69 <–> > 145.144.1.84 - fa:16:3e:93:0c:50- VirtualEthernet0/0/4 | cirros instance | > > From the cirros instance pingin the inside interface of the firewall (0/0/3) > works like a charm, I wouldn’t have expected any different. > > When I try to ping the outside interface of the firewall (0/0/2), traces > show the following: > > > 0:53:47:316205: vhost-user-input > > VirtualEthernet0/0/4 queue 0 > > virtio flags: > > INDIRECT Indirect descriptor > > virtio_net_hdr first_desc_len 12 > > flags 0x00 gso_type 0 > > num_buff 0 > > 00:53:47:316208: ethernet-input > > frame: flags 0x1, hw-if-index 7, sw-if-index 10 > > IP4: fa:16:3e:93:0c:50 -> fa:16:3e:26:3e:0e > > 00:53:47:316209: l2-input > > l2-input: sw_if_index 10 dst fa:16:3e:26:3e:0e src fa:16:3e:93:0c:50 > > 00:53:47:316210: l2-input-feat-arc > > IN-FEAT-ARC: head 1 feature_bitmap 500525 ethertype 800 sw_if_index 10, > next_index 22 > > 00:53:47:316211: acl-plugin-in-ip4-l2 > > acl-plugin: lc_index: -1, sw_if_index 10, next index 1, action: 3, match: > acl -1 rule 44 trace_bits 80000000 > > pkt info 0000000000000000 0000000000000000 0000000000000000 > 3501909154019091 000a030100000008 0200ffff00000000 > > lc_index 0 l3 ip4 145.144.1.84 -> 145.144.1.53 l4 lsb_of_sw_if_index 10 > proto 1 l4_is_input 1 l4_slow_path 1 l4_flags 0x03 port 8 -> 0 tcp flags > (invalid) 00 rsvd 0 > > 00:53:47:316214: l2-input-feat-arc-end > > IN-FEAT-ARC: head 0 feature_bitmap 100525 ethertype 0 sw_if_index -1, > next_index 17 > > 00:53:47:316215: l2-input-acl > > INACL: sw_if_index 10, next_index 9, table 41, offset 1392 > > 00:53:47:316216: l2-learn > > l2-learn: sw_if_index 10 dst fa:16:3e:26:3e:0e src fa:16:3e:93:0c:50 > bd_index 3 > > 00:53:47:316218: l2-fwd > > l2-fwd: sw_if_index 10 dst fa:16:3e:26:3e:0e src fa:16:3e:93:0c:50 > bd_index 3 result [0x5d50000000009, 9] none > > 00:53:47:316219: l2-output > > l2-output: sw_if_index 9 dst fa:16:3e:26:3e:0e src fa:16:3e:93:0c:50 data > 08 00 45 00 00 54 33 9c 40 00 40 01 > > 00:53:47:316220: l2-output-feat-arc > > OUT-FEAT-ARC: head 1 feature_bitmap 4001 ethertype 800 sw_if_index 9, > next_index 11 > > 00:53:47:316220: acl-plugin-out-ip4-l2 > > acl-plugin: lc_index: 6, sw_if_index 9, next index 1, action: 1, match: > acl 4 rule 2 trace_bits 00000000 > > pkt info 0000000000000000 0000000000000000 0000000000000000 > 3501909154019091 0009020100000008 0200ffff00000006 > > lc_index 6 l3 ip4 145.144.1.84 -> 145.144.1.53 l4 lsb_of_sw_if_index 9 > proto 1 l4_is_input 0 l4_slow_path 1 l4_flags 0x02 port 8 -> 0 tcp flags > (invalid) 00 rsvd 0 > > 00:53:47:316223: l2-output-feat-arc-end > > OUT-FEAT-ARC: head 0 feature_bitmap 1 ethertype 0 sw_if_index -1, > next_index 0 > > 00:53:47:316224: VirtualEthernet0/0/3-output > > VirtualEthernet0/0/3 l2_hdr_offset_valid l3_hdr_offset_valid > > IP4: fa:16:3e:93:0c:50 -> fa:16:3e:26:3e:0e > > ICMP: 145.144.1.84 -> 145.144.1.53 > > tos 0x00, ttl 64, length 84, checksum 0xe163 > > fragment id 0x339c, flags DONT_FRAGMENT > > ICMP echo_request checksum 0x9914 > > 00:53:47:316225: VirtualEthernet0/0/3-tx > > VirtualEthernet0/0/3 queue 0 > > virtio flags: > > SINGLE_DESC Single descriptor packet > > virtio_net_hdr first_desc_len 4096 > > flags 0x00 gso_type 0 > > num_buff 1 > > > > Packet 3 > > > > 00:53:47:316357: vhost-user-input > > VirtualEthernet0/0/3 queue 0 > > virtio flags: > > INDIRECT Indirect descriptor > > virtio_net_hdr first_desc_len 12 > > flags 0x00 gso_type 0 > > num_buff 0 > > 00:53:47:316358: ethernet-input > > frame: flags 0x1, hw-if-index 6, sw-if-index 9 > > IP4: fa:16:3e:26:3e:0e -> fa:16:3e:93:0c:50 > > 00:53:47:316358: l2-input > > l2-input: sw_if_index 9 dst fa:16:3e:93:0c:50 src fa:16:3e:26:3e:0e > > 00:53:47:316359: l2-input-feat-arc > > IN-FEAT-ARC: head 1 feature_bitmap 500525 ethertype 800 sw_if_index 9, > next_index 22 > > 00:53:47:316359: acl-plugin-in-ip4-l2 > > acl-plugin: lc_index: -1, sw_if_index 9, next index 1, action: 3, match: > acl -1 rule 97 trace_bits 80000000 > > pkt info 0000000000000000 0000000000000000 0000000000000000 > 5401909135019091 0009030100000000 0200ffff00000000 > > lc_index 0 l3 ip4 145.144.1.53 -> 145.144.1.84 l4 lsb_of_sw_if_index 9 > proto 1 l4_is_input 1 l4_slow_path 1 l4_flags 0x03 port 0 -> 0 tcp flags > (invalid) 00 rsvd 0 > > 00:53:47:316359: l2-input-feat-arc-end > > IN-FEAT-ARC: head 0 feature_bitmap 100525 ethertype 0 sw_if_index -1, > next_index 17 > > 00:53:47:316360: l2-input-acl > > INACL: sw_if_index 9, next_index 0, table 12, offset -1 > > 00:53:47:316361: error-drop > > rx:VirtualEthernet0/0/3 > > 00:53:47:316362: drop > > l2-input-acl: input ACL table-miss drops > > > The packets get dropped.. I have found all the acl’s in vpp, but I cannot > really see what would be wrong here, or what causes the packets to be > dropped. Has anyone else had any experiences like this, or a solution? > > Regards, > > Eyle > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#13909): https://lists.fd.io/g/vpp-dev/message/13909 > Mute This Topic: https://lists.fd.io/mt/33152601/675608 > Group Owner: [email protected]<mailto:[email protected]> > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub > [[email protected]<mailto:[email protected]>] > -=-=-=-=-=-=-=-=-=-=-=- >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13917): https://lists.fd.io/g/vpp-dev/message/13917 Mute This Topic: https://lists.fd.io/mt/33152601/21656 Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
