Ok so I will explain the logic a bit more and maybe this will solve the puzzle.
Assumption: that the trace in your first mail and the show output in
the last one correspond to the same state of the VPP and the stuff
attahed to it (sw_if_index, etc.)
The purpose in life of the MACIP ACLs is to give a stateless way to
enforce the correspondence of IP and MAC, as well as to ensure the
hosts on a given VPP interface are using the MAC address they are
supposed to use.
the ingress packet in the frame 3 (the dropped one) in the trace
received on VirtualEthernet0/0/3 is:
l2-input: sw_if_index 9 dst fa:16:3e:93:0c:50 src fa:16:3e:26:3e:0e
so we need to have a MACIP ACL that tells tells that
fa:16:3e:26:3e:0e is okay to hang off that interface.
We can indeed find that it is the macip acl #2:
MACIP acl_index: 2, count: 2 (true len 2) tag {} is free pool slot: 0
ip4_table_index 17, ip6_table_index 17, l2_table_index 17
out_ip4_table_index -1, out_ip6_table_index -1, out_l2_table_index -1
rule 0: ipv4 action 1 ip 0.0.0.0/32 mac fa:16:3e:26:3e:0e mask
ff:ff:ff:ff:ff:ff
rule 1: ipv4 action 1 ip 145.144.1.78/32 mac fa:16:3e:26:3e:0e
mask ff:ff:ff:ff:ff:ff
applied on sw_if_index(s): 9
so, it has the good MAC address and it is applied, but this ACL says
"I can either allow the source IP of 0.0.0.0 or a source IP of
145.144.1.78 incoming on VirtualEthernet0/0/3 with the source of
fa:16:3e:26:3e:0e.
So from the trace, the packet is:
lc_index 0 l3 ip4 145.144.1.53 -> 145.144.1.84 l4
lsb_of_sw_if_index 9 proto 1 l4_is_input 1 l4_slow_path 1 l4_flags
0x03 port 0 -> 0 tcp flags (invalid) 00 rsvd 0
The source of 145.144.1.53 is *not* the 145.144.1.78 which MACIP ACL
permits to appear on that interface, so from this I can conclude it
all WorksAsConfigured(tm).
I think by default the OpenStack assumes the endpoint connected to VPP
is a host, which in your case it isn't. So you would need to relax the
whichever policies govern that. I am not sure where, since I deal with
the VPP side, all I see is the API calls :-) - maybe Naveen might give
a pointer or two here...
This all is fairly old code, so I think tweaking the config will work.
But, for curiosity/if my assumption in the beginning is wrong, or in
case some curious reader finds this mail later, I will go a bit
further and tell what the next steps would have been if we *did* find
that the MACIP ACL was configured correctly to permit the source
IP+MAC that we saw in frame 3 of the trace. For that we need to know
how MACIP ACL functions under the hood.
ACL plugin reuses the pre-existing core feature of VPP called "IPACL
with classifier tables".
A bit confusing, but that's why you see it in the trace.
Classifier tables are a pretty cool thing - you can match up to 5*u32
worth of contiguous bits with an arbitrary bitmask pretty much
anywhere in the packet, with a fixed offset. But the drawback is that
all the possible values matched need to be the *same* place and the
same bitmask. So to deal with that, the classifier tables can be
chained - and the lookup made sequentially.
If you do "show classify tables verbose" you will see the tables in
all the glory, including the contents. There one can verify at a lower
level that what the ACL plugin programs does make sense.
hope this helps!
--a
On 9/6/19, Eyle Brinkhuis <[email protected]> wrote:
> Okay.. so when I pull up the macip acl’s:
>
> vpp# show acl-plugin macip acl
> MACIP acl_index: 0, count: 2 (true len 2) tag {} is free pool slot: 0
> ip4_table_index 5, ip6_table_index 5, l2_table_index 5
> out_ip4_table_index -1, out_ip6_table_index -1, out_l2_table_index -1
> rule 0: ipv4 action 1 ip 0.0.0.0/32 mac fa:16:3e:79:87:05 mask
> ff:ff:ff:ff:ff:ff
> rule 1: ipv4 action 1 ip 145.144.1.5/32 mac fa:16:3e:79:87:05 mask
> ff:ff:ff:ff:ff:ff
> applied on sw_if_index(s):
> MACIP acl_index: 1, count: 2 (true len 2) tag {} is free pool slot: 0
> ip4_table_index 11, ip6_table_index 11, l2_table_index 11
> out_ip4_table_index -1, out_ip6_table_index -1, out_l2_table_index -1
> rule 0: ipv4 action 1 ip 0.0.0.0/32 mac fa:16:3e:79:87:05 mask
> ff:ff:ff:ff:ff:ff
> rule 1: ipv4 action 1 ip 145.144.1.5/32 mac fa:16:3e:79:87:05 mask
> ff:ff:ff:ff:ff:ff
> applied on sw_if_index(s): 4
> MACIP acl_index: 2, count: 2 (true len 2) tag {} is free pool slot: 0
> ip4_table_index 17, ip6_table_index 17, l2_table_index 17
> out_ip4_table_index -1, out_ip6_table_index -1, out_l2_table_index -1
> rule 0: ipv4 action 1 ip 0.0.0.0/32 mac fa:16:3e:26:3e:0e mask
> ff:ff:ff:ff:ff:ff
> rule 1: ipv4 action 1 ip 145.144.1.78/32 mac fa:16:3e:26:3e:0e mask
> ff:ff:ff:ff:ff:ff
> applied on sw_if_index(s): 9
> MACIP acl_index: 3, count: 2 (true len 2) tag {} is free pool slot: 0
> ip4_table_index 23, ip6_table_index 23, l2_table_index 23
> out_ip4_table_index -1, out_ip6_table_index -1, out_l2_table_index -1
> rule 0: ipv4 action 1 ip 0.0.0.0/32 mac fa:16:3e:10:04:3e mask
> ff:ff:ff:ff:ff:ff
> rule 1: ipv4 action 1 ip 145.144.1.29/32 mac fa:16:3e:10:04:3e mask
> ff:ff:ff:ff:ff:ff
> applied on sw_if_index(s): 5
> MACIP acl_index: 4, count: 2 (true len 2) tag {} is free pool slot: 0
> ip4_table_index 29, ip6_table_index 29, l2_table_index 29
> out_ip4_table_index -1, out_ip6_table_index -1, out_l2_table_index -1
> rule 0: ipv4 action 1 ip 0.0.0.0/32 mac fa:16:3e:7c:96:d0 mask
> ff:ff:ff:ff:ff:ff
> rule 1: ipv4 action 1 ip 145.144.1.53/32 mac fa:16:3e:7c:96:d0 mask
> ff:ff:ff:ff:ff:ff
> applied on sw_if_index(s): 7
> MACIP acl_index: 6, count: 2 (true len 2) tag {} is free pool slot: 0
> ip4_table_index 41, ip6_table_index 41, l2_table_index 41
> out_ip4_table_index -1, out_ip6_table_index -1, out_l2_table_index -1
> rule 0: ipv4 action 1 ip 0.0.0.0/32 mac fa:16:3e:93:0c:50 mask
> ff:ff:ff:ff:ff:ff
> rule 1: ipv4 action 1 ip 145.144.1.84/32 mac fa:16:3e:93:0c:50 mask
> ff:ff:ff:ff:ff:ff
> applied on sw_if_index(s): 10
>
>
>
> But then compare that to the show hardware:
> vpp# sh hard
> Name Idx Link Hardware
> VirtualEthernet0/0/0 3 up VirtualEthernet0/0/0
> Link speed: unknown
> Ethernet address fa:16:3c:c9:a8:50
> VirtualEthernet0/0/1 4 up VirtualEthernet0/0/1
> Link speed: unknown
> Ethernet address fa:16:3c:70:15:b3
> VirtualEthernet0/0/2 5 up VirtualEthernet0/0/2
> Link speed: unknown
> Ethernet address fa:16:3c:05:66:7c
> VirtualEthernet0/0/3 6 up VirtualEthernet0/0/3
> Link speed: unknown
> Ethernet address fa:16:3c:f0:21:0a
> VirtualEthernet0/0/4 7 up VirtualEthernet0/0/4
> Link speed: unknown
> Ethernet address fa:16:3c:0f:9d:5d
> local0 0 down local0
> Link speed: unknown
> local
> rdma0 1 up rdma0
> Link speed: 40 Gbps
> Ethernet address 02:fe:99:32:82:4f
> flags: admin-up promiscuous
> rdma1 2 up rdma1
> Link speed: 40 Gbps
> Ethernet address 02:fe:27:ea:09:82
> flags: admin-up
>
> It looks like there doesn’t even exist an acl for VirtualEthernet0/0/3? Is
> that why it is dropped?
>
> Eyle
>
> From: Andrew 👽 Yourtchenko <[email protected]>
> Date: Thursday, 5 September 2019 at 19:20
> To: "Naveen Joy (najoy)" <[email protected]>
> Cc: Eyle Brinkhuis <[email protected]>, "[email protected]"
> <[email protected]>
> Subject: Re: [vpp-dev] ACL drops while pinging another interface
>
> It hits the session, so it does pass the L3 acl. Just before ...
> --a
>
> On 5 Sep 2019, at 18:52, Naveen Joy (najoy)
> <[email protected]<mailto:[email protected]>> wrote:
> From the trace, it appears like ICMP Echo reply is not permitted by the
> security-group applied on neutron’s port corresponding to
> VirtualEthernet0/0/3.
> This could be causing the ICMP reply packet from the firewall to drop.
>
> lc_index 0 l3 ip4 145.144.1.53 -> 145.144.1.84 l4 lsb_of_sw_if_index 9
> proto 1 l4_is_input 1 l4_slow_path 1 l4_flags 0x03 port 0 -> 0 tcp flags
> (invalid) 00 rsvd 0
>
> 00:53:47:316359: l2-input-feat-arc-end
>
> IN-FEAT-ARC: head 0 feature_bitmap 100525 ethertype 0 sw_if_index -1,
> next_index 17
>
> 00:53:47:316360: l2-input-acl
>
> INACL: sw_if_index 9, next_index 0, table 12, offset -1
>
> 00:53:47:316361: error-drop
>
> rx:VirtualEthernet0/0/3
>
> -Naveen
>
> From: <[email protected]<mailto:[email protected]>> on behalf of Andrew
> Yourtchenko <[email protected]<mailto:[email protected]>>
> Date: Thursday, September 5, 2019 at 7:20 AM
> To: Eyle Brinkhuis
> <[email protected]<mailto:[email protected]>>
> Cc: "[email protected]<mailto:[email protected]>"
> <[email protected]<mailto:[email protected]>>
> Subject: Re: [vpp-dev] ACL drops while pinging another interface
>
> Thanks for the traces !
>
> MACIP acl uses the classifier-bases “ip-acl”; so it sounds like it is not
> programmed with the source Mac of your packets.
>
> “Show acl-plugin macip” will help to see what the acl plugin sees, and if it
> looks legit, then you can check the classifier tables applied as input acl
> to verify those are programmed correctly.
>
> Shout if you get stuck :)
>
> --a
>
> On 5 Sep 2019, at 14:18, Eyle Brinkhuis
> <[email protected]<mailto:[email protected]>> wrote:
> Hi guys,
>
> I’m using VPP 19.08 with networking-vpp in an openstack stein environment,
> where we are busy building an open environment that is specifically built
> for NFV applications. One of those functions is a firewall setup, where we
> firewall a customer’s traffic and provide said customer with a ‘clean and
> safe’ internet connection.
>
> As such, I am evaluating a VPP setup, which looks very promising. However:
> in the following scenario, I run into an issue:
>
> I have a compute host on which I have a firewall running ánd a guest (cirros
> for now). Setup is as follows:
>
> 145.144.1.53-fa:16:3e:7c:96:d0 – VirtualEthernet0/0/2 | firewall instance |
> VirtualEthernet0/03 145.144.1.78 - fa:16:3e:26:3e:0e <–> vlan 69 <–>
> 145.144.1.84 - fa:16:3e:93:0c:50- VirtualEthernet0/0/4 | cirros instance |
>
> From the cirros instance pingin the inside interface of the firewall (0/0/3)
> works like a charm, I wouldn’t have expected any different.
>
> When I try to ping the outside interface of the firewall (0/0/2), traces
> show the following:
>
>
> 0:53:47:316205: vhost-user-input
>
> VirtualEthernet0/0/4 queue 0
>
> virtio flags:
>
> INDIRECT Indirect descriptor
>
> virtio_net_hdr first_desc_len 12
>
> flags 0x00 gso_type 0
>
> num_buff 0
>
> 00:53:47:316208: ethernet-input
>
> frame: flags 0x1, hw-if-index 7, sw-if-index 10
>
> IP4: fa:16:3e:93:0c:50 -> fa:16:3e:26:3e:0e
>
> 00:53:47:316209: l2-input
>
> l2-input: sw_if_index 10 dst fa:16:3e:26:3e:0e src fa:16:3e:93:0c:50
>
> 00:53:47:316210: l2-input-feat-arc
>
> IN-FEAT-ARC: head 1 feature_bitmap 500525 ethertype 800 sw_if_index 10,
> next_index 22
>
> 00:53:47:316211: acl-plugin-in-ip4-l2
>
> acl-plugin: lc_index: -1, sw_if_index 10, next index 1, action: 3, match:
> acl -1 rule 44 trace_bits 80000000
>
> pkt info 0000000000000000 0000000000000000 0000000000000000
> 3501909154019091 000a030100000008 0200ffff00000000
>
> lc_index 0 l3 ip4 145.144.1.84 -> 145.144.1.53 l4 lsb_of_sw_if_index 10
> proto 1 l4_is_input 1 l4_slow_path 1 l4_flags 0x03 port 8 -> 0 tcp flags
> (invalid) 00 rsvd 0
>
> 00:53:47:316214: l2-input-feat-arc-end
>
> IN-FEAT-ARC: head 0 feature_bitmap 100525 ethertype 0 sw_if_index -1,
> next_index 17
>
> 00:53:47:316215: l2-input-acl
>
> INACL: sw_if_index 10, next_index 9, table 41, offset 1392
>
> 00:53:47:316216: l2-learn
>
> l2-learn: sw_if_index 10 dst fa:16:3e:26:3e:0e src fa:16:3e:93:0c:50
> bd_index 3
>
> 00:53:47:316218: l2-fwd
>
> l2-fwd: sw_if_index 10 dst fa:16:3e:26:3e:0e src fa:16:3e:93:0c:50
> bd_index 3 result [0x5d50000000009, 9] none
>
> 00:53:47:316219: l2-output
>
> l2-output: sw_if_index 9 dst fa:16:3e:26:3e:0e src fa:16:3e:93:0c:50 data
> 08 00 45 00 00 54 33 9c 40 00 40 01
>
> 00:53:47:316220: l2-output-feat-arc
>
> OUT-FEAT-ARC: head 1 feature_bitmap 4001 ethertype 800 sw_if_index 9,
> next_index 11
>
> 00:53:47:316220: acl-plugin-out-ip4-l2
>
> acl-plugin: lc_index: 6, sw_if_index 9, next index 1, action: 1, match:
> acl 4 rule 2 trace_bits 00000000
>
> pkt info 0000000000000000 0000000000000000 0000000000000000
> 3501909154019091 0009020100000008 0200ffff00000006
>
> lc_index 6 l3 ip4 145.144.1.84 -> 145.144.1.53 l4 lsb_of_sw_if_index 9
> proto 1 l4_is_input 0 l4_slow_path 1 l4_flags 0x02 port 8 -> 0 tcp flags
> (invalid) 00 rsvd 0
>
> 00:53:47:316223: l2-output-feat-arc-end
>
> OUT-FEAT-ARC: head 0 feature_bitmap 1 ethertype 0 sw_if_index -1,
> next_index 0
>
> 00:53:47:316224: VirtualEthernet0/0/3-output
>
> VirtualEthernet0/0/3 l2_hdr_offset_valid l3_hdr_offset_valid
>
> IP4: fa:16:3e:93:0c:50 -> fa:16:3e:26:3e:0e
>
> ICMP: 145.144.1.84 -> 145.144.1.53
>
> tos 0x00, ttl 64, length 84, checksum 0xe163
>
> fragment id 0x339c, flags DONT_FRAGMENT
>
> ICMP echo_request checksum 0x9914
>
> 00:53:47:316225: VirtualEthernet0/0/3-tx
>
> VirtualEthernet0/0/3 queue 0
>
> virtio flags:
>
> SINGLE_DESC Single descriptor packet
>
> virtio_net_hdr first_desc_len 4096
>
> flags 0x00 gso_type 0
>
> num_buff 1
>
>
>
> Packet 3
>
>
>
> 00:53:47:316357: vhost-user-input
>
> VirtualEthernet0/0/3 queue 0
>
> virtio flags:
>
> INDIRECT Indirect descriptor
>
> virtio_net_hdr first_desc_len 12
>
> flags 0x00 gso_type 0
>
> num_buff 0
>
> 00:53:47:316358: ethernet-input
>
> frame: flags 0x1, hw-if-index 6, sw-if-index 9
>
> IP4: fa:16:3e:26:3e:0e -> fa:16:3e:93:0c:50
>
> 00:53:47:316358: l2-input
>
> l2-input: sw_if_index 9 dst fa:16:3e:93:0c:50 src fa:16:3e:26:3e:0e
>
> 00:53:47:316359: l2-input-feat-arc
>
> IN-FEAT-ARC: head 1 feature_bitmap 500525 ethertype 800 sw_if_index 9,
> next_index 22
>
> 00:53:47:316359: acl-plugin-in-ip4-l2
>
> acl-plugin: lc_index: -1, sw_if_index 9, next index 1, action: 3, match:
> acl -1 rule 97 trace_bits 80000000
>
> pkt info 0000000000000000 0000000000000000 0000000000000000
> 5401909135019091 0009030100000000 0200ffff00000000
>
> lc_index 0 l3 ip4 145.144.1.53 -> 145.144.1.84 l4 lsb_of_sw_if_index 9
> proto 1 l4_is_input 1 l4_slow_path 1 l4_flags 0x03 port 0 -> 0 tcp flags
> (invalid) 00 rsvd 0
>
> 00:53:47:316359: l2-input-feat-arc-end
>
> IN-FEAT-ARC: head 0 feature_bitmap 100525 ethertype 0 sw_if_index -1,
> next_index 17
>
> 00:53:47:316360: l2-input-acl
>
> INACL: sw_if_index 9, next_index 0, table 12, offset -1
>
> 00:53:47:316361: error-drop
>
> rx:VirtualEthernet0/0/3
>
> 00:53:47:316362: drop
>
> l2-input-acl: input ACL table-miss drops
>
>
> The packets get dropped.. I have found all the acl’s in vpp, but I cannot
> really see what would be wrong here, or what causes the packets to be
> dropped. Has anyone else had any experiences like this, or a solution?
>
> Regards,
>
> Eyle
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
>
> View/Reply Online (#13909): https://lists.fd.io/g/vpp-dev/message/13909
> Mute This Topic: https://lists.fd.io/mt/33152601/675608
> Group Owner: [email protected]<mailto:[email protected]>
> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub
> [[email protected]<mailto:[email protected]>]
> -=-=-=-=-=-=-=-=-=-=-=-
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#13916): https://lists.fd.io/g/vpp-dev/message/13916
Mute This Topic: https://lists.fd.io/mt/33152601/21656
Group Owner: [email protected]
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
