Re: [patch net-next 12/15] ipv4: fib: Notify about nexthop status changes

[Ido Schimmel]

On Wed, Feb 08, 2017 at 11:05:54AM -0700, David Ahern wrote:
> On 2/8/17 8:32 AM, Ido Schimmel wrote:
> > In the case of multipath routes, if some of the nexthops can be
> > reflected, then we do so, but periodically ask the kernel to try and
> > resolve the others. Otherwise, these nexthops will never be resolved, as
> > the kernel doesn't see the packets hitting the multipath route and
> > therefore lacks the motivation to resolve its nexthops.
> 
> It should get the motivation once the neigh entry is cleaned up. That
> can take a long time based on gc settings, but it can also happen from
> the remote side if it sends an arp message.

And when the remote side does that and the neighbour becomes valid we no
longer try to actively resolve it.

Re: [patch net-next 12/15] ipv4: fib: Notify about nexthop status changes

[David Ahern]

On 2/8/17 8:32 AM, Ido Schimmel wrote:
> In the case of multipath routes, if some of the nexthops can be
> reflected, then we do so, but periodically ask the kernel to try and
> resolve the others. Otherwise, these nexthops will never be resolved, as
> the kernel doesn't see the packets hitting the multipath route and
> therefore lacks the motivation to resolve its nexthops.

It should get the motivation once the neigh entry is cleaned up. That
can take a long time based on gc settings, but it can also happen from
the remote side if it sends an arp message.

Re: [patch net-next 12/15] ipv4: fib: Notify about nexthop status changes

[Ido Schimmel]

On Wed, Feb 08, 2017 at 09:56:00AM -0500, Andy Gospodarek wrote:
> On Wed, Feb 08, 2017 at 11:16:39AM +0100, Jiri Pirko wrote:
> > From: Ido Schimmel 
> > 
> > When a multipath route is hit the kernel doesn't consider nexthops that
> > are DEAD or LINKDOWN when IN_DEV_IGNORE_ROUTES_WITH_LINKDOWN is set.
> > Devices that offload multipath routes need to be made aware of nexthop
> > status changes. Otherwise, the device will keep forwarding packets to
> > non-functional nexthops.
> > 
> > Add the FIB_EVENT_NH_{ADD,DEL} events to the fib notification chain,
> > which notify capable devices when they should add or delete a nexthop
> > from their tables.
> 
> This looks good -- thanks for doing this.
> 
> IIUC the hardware forwarding use case for your hardware covered by David
> Ahern's patch[1] to the ipv4 software path selection is already covered,
> so this is probably the last known link/neighbor forwarding issue for
> ipv4 that needs coverage.
> 
> 1. a6db449 net: ipv4: Consider failed nexthops in multipath routes

Yep, it aligns the kernel's datapath with what we already implemented in
mlxsw. In case the neighbour can't be resolve, then the nexthop isn't
reflected to the device (we need the MAC...).

In the case of multipath routes, if some of the nexthops can be
reflected, then we do so, but periodically ask the kernel to try and
resolve the others. Otherwise, these nexthops will never be resolved, as
the kernel doesn't see the packets hitting the multipath route and
therefore lacks the motivation to resolve its nexthops.

> > Cc: Roopa Prabhu 
> > Cc: David Ahern 
> > Cc: Andy Gospodarek 
> 
> Reviewed-by: Andy Gospodarek 

Thanks for reviewing!

Re: [patch net-next 12/15] ipv4: fib: Notify about nexthop status changes

[Andy Gospodarek]

On Wed, Feb 8, 2017 at 5:16 AM, Jiri Pirko  wrote:
> From: Ido Schimmel 
>
> When a multipath route is hit the kernel doesn't consider nexthops that
> are DEAD or LINKDOWN when IN_DEV_IGNORE_ROUTES_WITH_LINKDOWN is set.
> Devices that offload multipath routes need to be made aware of nexthop
> status changes. Otherwise, the device will keep forwarding packets to
> non-functional nexthops.
>
> Add the FIB_EVENT_NH_{ADD,DEL} events to the fib notification chain,
> which notify capable devices when they should add or delete a nexthop
> from their tables.

This looks good -- thanks for doing this.

IIUC the hardware forwarding use case for your hardware covered by David
Ahern's patch[1] to the ipv4 software path selection is already covered,
so this is probably the last known link/neighbor forwarding issue for
ipv4 that needs coverage.

1. a6db449 net: ipv4: Consider failed nexthops in multipath routes

> Cc: Roopa Prabhu 
> Cc: David Ahern 
> Cc: Andy Gospodarek 

Reviewed-by Andy Gospodarek 

> Signed-off-by: Ido Schimmel 
> Signed-off-by: Jiri Pirko 
> ---
>  include/net/ip_fib.h |  7 +++
>  net/ipv4/fib_semantics.c | 33 +
>  2 files changed, 40 insertions(+)
>
> diff --git a/include/net/ip_fib.h b/include/net/ip_fib.h
> index 57c2a86..45a184e 100644
> --- a/include/net/ip_fib.h
> +++ b/include/net/ip_fib.h
> @@ -214,11 +214,18 @@ struct fib_entry_notifier_info {
> u32 nlflags;
>  };
>
> +struct fib_nh_notifier_info {
> +   struct fib_notifier_info info; /* must be first */
> +   struct fib_nh *fib_nh;
> +};
> +
>  enum fib_event_type {
> FIB_EVENT_ENTRY_ADD,
> FIB_EVENT_ENTRY_DEL,
> FIB_EVENT_RULE_ADD,
> FIB_EVENT_RULE_DEL,
> +   FIB_EVENT_NH_ADD,
> +   FIB_EVENT_NH_DEL,
>  };
>
>  int register_fib_notifier(struct notifier_block *nb,
> diff --git a/net/ipv4/fib_semantics.c b/net/ipv4/fib_semantics.c
> index 6306a67..317026a 100644
> --- a/net/ipv4/fib_semantics.c
> +++ b/net/ipv4/fib_semantics.c
> @@ -1355,6 +1355,36 @@ int fib_sync_down_addr(struct net_device *dev, __be32 
> local)
> return ret;
>  }
>
> +static int call_fib_nh_notifiers(struct fib_nh *fib_nh,
> +enum fib_event_type event_type)
> +{
> +   struct in_device *in_dev = __in_dev_get_rtnl(fib_nh->nh_dev);
> +   struct fib_nh_notifier_info info = {
> +   .fib_nh = fib_nh,
> +   };
> +
> +   switch (event_type) {
> +   case FIB_EVENT_NH_ADD:
> +   if (fib_nh->nh_flags & RTNH_F_DEAD)
> +   break;
> +   if (IN_DEV_IGNORE_ROUTES_WITH_LINKDOWN(in_dev) &&
> +   fib_nh->nh_flags & RTNH_F_LINKDOWN)
> +   break;
> +   return call_fib_notifiers(dev_net(fib_nh->nh_dev), event_type,
> + );
> +   case FIB_EVENT_NH_DEL:
> +   if ((IN_DEV_IGNORE_ROUTES_WITH_LINKDOWN(in_dev) &&
> +fib_nh->nh_flags & RTNH_F_LINKDOWN) ||
> +   (fib_nh->nh_flags & RTNH_F_DEAD))
> +   return call_fib_notifiers(dev_net(fib_nh->nh_dev),
> + event_type, );
> +   default:
> +   break;
> +   }
> +
> +   return NOTIFY_DONE;
> +}
> +
>  /* Event  force Flags   Description
>   * NETDEV_CHANGE  0 LINKDOWNCarrier OFF, not for scope host
>   * NETDEV_DOWN0 LINKDOWN|DEAD   Link down, not for scope host
> @@ -1396,6 +1426,8 @@ int fib_sync_down_dev(struct net_device *dev, unsigned 
> long event, bool force)
> nexthop_nh->nh_flags |= 
> RTNH_F_LINKDOWN;
> break;
> }
> +   call_fib_nh_notifiers(nexthop_nh,
> + FIB_EVENT_NH_DEL);
> dead++;
> }
>  #ifdef CONFIG_IP_ROUTE_MULTIPATH
> @@ -1550,6 +1582,7 @@ int fib_sync_up(struct net_device *dev, unsigned int 
> nh_flags)
> continue;
> alive++;
> nexthop_nh->nh_flags &= ~nh_flags;
> +   call_fib_nh_notifiers(nexthop_nh, FIB_EVENT_NH_ADD);
> } endfor_nexthops(fi)
>
> if (alive > 0) {
> --
> 2.7.4
>

Re: [patch net-next 12/15] ipv4: fib: Notify about nexthop status changes

[Andy Gospodarek]

On Wed, Feb 08, 2017 at 11:16:39AM +0100, Jiri Pirko wrote:
> From: Ido Schimmel 
> 
> When a multipath route is hit the kernel doesn't consider nexthops that
> are DEAD or LINKDOWN when IN_DEV_IGNORE_ROUTES_WITH_LINKDOWN is set.
> Devices that offload multipath routes need to be made aware of nexthop
> status changes. Otherwise, the device will keep forwarding packets to
> non-functional nexthops.
> 
> Add the FIB_EVENT_NH_{ADD,DEL} events to the fib notification chain,
> which notify capable devices when they should add or delete a nexthop
> from their tables.

This looks good -- thanks for doing this.

IIUC the hardware forwarding use case for your hardware covered by David
Ahern's patch[1] to the ipv4 software path selection is already covered,
so this is probably the last known link/neighbor forwarding issue for
ipv4 that needs coverage.

1. a6db449 net: ipv4: Consider failed nexthops in multipath routes

> Cc: Roopa Prabhu 
> Cc: David Ahern 
> Cc: Andy Gospodarek 

Reviewed-by: Andy Gospodarek 

> Signed-off-by: Ido Schimmel 
> Signed-off-by: Jiri Pirko 
> ---
>  include/net/ip_fib.h |  7 +++
>  net/ipv4/fib_semantics.c | 33 +
>  2 files changed, 40 insertions(+)
> 
> diff --git a/include/net/ip_fib.h b/include/net/ip_fib.h
> index 57c2a86..45a184e 100644
> --- a/include/net/ip_fib.h
> +++ b/include/net/ip_fib.h
> @@ -214,11 +214,18 @@ struct fib_entry_notifier_info {
>   u32 nlflags;
>  };
>  
> +struct fib_nh_notifier_info {
> + struct fib_notifier_info info; /* must be first */
> + struct fib_nh *fib_nh;
> +};
> +
>  enum fib_event_type {
>   FIB_EVENT_ENTRY_ADD,
>   FIB_EVENT_ENTRY_DEL,
>   FIB_EVENT_RULE_ADD,
>   FIB_EVENT_RULE_DEL,
> + FIB_EVENT_NH_ADD,
> + FIB_EVENT_NH_DEL,
>  };
>  
>  int register_fib_notifier(struct notifier_block *nb,
> diff --git a/net/ipv4/fib_semantics.c b/net/ipv4/fib_semantics.c
> index 6306a67..317026a 100644
> --- a/net/ipv4/fib_semantics.c
> +++ b/net/ipv4/fib_semantics.c
> @@ -1355,6 +1355,36 @@ int fib_sync_down_addr(struct net_device *dev, __be32 
> local)
>   return ret;
>  }
>  
> +static int call_fib_nh_notifiers(struct fib_nh *fib_nh,
> +  enum fib_event_type event_type)
> +{
> + struct in_device *in_dev = __in_dev_get_rtnl(fib_nh->nh_dev);
> + struct fib_nh_notifier_info info = {
> + .fib_nh = fib_nh,
> + };
> +
> + switch (event_type) {
> + case FIB_EVENT_NH_ADD:
> + if (fib_nh->nh_flags & RTNH_F_DEAD)
> + break;
> + if (IN_DEV_IGNORE_ROUTES_WITH_LINKDOWN(in_dev) &&
> + fib_nh->nh_flags & RTNH_F_LINKDOWN)
> + break;
> + return call_fib_notifiers(dev_net(fib_nh->nh_dev), event_type,
> +   );
> + case FIB_EVENT_NH_DEL:
> + if ((IN_DEV_IGNORE_ROUTES_WITH_LINKDOWN(in_dev) &&
> +  fib_nh->nh_flags & RTNH_F_LINKDOWN) ||
> + (fib_nh->nh_flags & RTNH_F_DEAD))
> + return call_fib_notifiers(dev_net(fib_nh->nh_dev),
> +   event_type, );
> + default:
> + break;
> + }
> +
> + return NOTIFY_DONE;
> +}
> +
>  /* Event  force Flags   Description
>   * NETDEV_CHANGE  0 LINKDOWNCarrier OFF, not for scope host
>   * NETDEV_DOWN0 LINKDOWN|DEAD   Link down, not for scope host
> @@ -1396,6 +1426,8 @@ int fib_sync_down_dev(struct net_device *dev, unsigned 
> long event, bool force)
>   nexthop_nh->nh_flags |= RTNH_F_LINKDOWN;
>   break;
>   }
> + call_fib_nh_notifiers(nexthop_nh,
> +   FIB_EVENT_NH_DEL);
>   dead++;
>   }
>  #ifdef CONFIG_IP_ROUTE_MULTIPATH
> @@ -1550,6 +1582,7 @@ int fib_sync_up(struct net_device *dev, unsigned int 
> nh_flags)
>   continue;
>   alive++;
>   nexthop_nh->nh_flags &= ~nh_flags;
> + call_fib_nh_notifiers(nexthop_nh, FIB_EVENT_NH_ADD);
>   } endfor_nexthops(fi)
>  
>   if (alive > 0) {
> -- 
> 2.7.4
> 

[patch net-next 12/15] ipv4: fib: Notify about nexthop status changes

[Jiri Pirko]

From: Ido Schimmel 

When a multipath route is hit the kernel doesn't consider nexthops that
are DEAD or LINKDOWN when IN_DEV_IGNORE_ROUTES_WITH_LINKDOWN is set.
Devices that offload multipath routes need to be made aware of nexthop
status changes. Otherwise, the device will keep forwarding packets to
non-functional nexthops.

Add the FIB_EVENT_NH_{ADD,DEL} events to the fib notification chain,
which notify capable devices when they should add or delete a nexthop
from their tables.

Cc: Roopa Prabhu 
Cc: David Ahern 
Cc: Andy Gospodarek 
Signed-off-by: Ido Schimmel 
Signed-off-by: Jiri Pirko 
---
 include/net/ip_fib.h |  7 +++
 net/ipv4/fib_semantics.c | 33 +
 2 files changed, 40 insertions(+)

diff --git a/include/net/ip_fib.h b/include/net/ip_fib.h
index 57c2a86..45a184e 100644
--- a/include/net/ip_fib.h
+++ b/include/net/ip_fib.h
@@ -214,11 +214,18 @@ struct fib_entry_notifier_info {
u32 nlflags;
 };
 
+struct fib_nh_notifier_info {
+   struct fib_notifier_info info; /* must be first */
+   struct fib_nh *fib_nh;
+};
+
 enum fib_event_type {
FIB_EVENT_ENTRY_ADD,
FIB_EVENT_ENTRY_DEL,
FIB_EVENT_RULE_ADD,
FIB_EVENT_RULE_DEL,
+   FIB_EVENT_NH_ADD,
+   FIB_EVENT_NH_DEL,
 };
 
 int register_fib_notifier(struct notifier_block *nb,
diff --git a/net/ipv4/fib_semantics.c b/net/ipv4/fib_semantics.c
index 6306a67..317026a 100644
--- a/net/ipv4/fib_semantics.c
+++ b/net/ipv4/fib_semantics.c
@@ -1355,6 +1355,36 @@ int fib_sync_down_addr(struct net_device *dev, __be32 
local)
return ret;
 }
 
+static int call_fib_nh_notifiers(struct fib_nh *fib_nh,
+enum fib_event_type event_type)
+{
+   struct in_device *in_dev = __in_dev_get_rtnl(fib_nh->nh_dev);
+   struct fib_nh_notifier_info info = {
+   .fib_nh = fib_nh,
+   };
+
+   switch (event_type) {
+   case FIB_EVENT_NH_ADD:
+   if (fib_nh->nh_flags & RTNH_F_DEAD)
+   break;
+   if (IN_DEV_IGNORE_ROUTES_WITH_LINKDOWN(in_dev) &&
+   fib_nh->nh_flags & RTNH_F_LINKDOWN)
+   break;
+   return call_fib_notifiers(dev_net(fib_nh->nh_dev), event_type,
+ );
+   case FIB_EVENT_NH_DEL:
+   if ((IN_DEV_IGNORE_ROUTES_WITH_LINKDOWN(in_dev) &&
+fib_nh->nh_flags & RTNH_F_LINKDOWN) ||
+   (fib_nh->nh_flags & RTNH_F_DEAD))
+   return call_fib_notifiers(dev_net(fib_nh->nh_dev),
+ event_type, );
+   default:
+   break;
+   }
+
+   return NOTIFY_DONE;
+}
+
 /* Event  force Flags   Description
  * NETDEV_CHANGE  0 LINKDOWNCarrier OFF, not for scope host
  * NETDEV_DOWN0 LINKDOWN|DEAD   Link down, not for scope host
@@ -1396,6 +1426,8 @@ int fib_sync_down_dev(struct net_device *dev, unsigned 
long event, bool force)
nexthop_nh->nh_flags |= RTNH_F_LINKDOWN;
break;
}
+   call_fib_nh_notifiers(nexthop_nh,
+ FIB_EVENT_NH_DEL);
dead++;
}
 #ifdef CONFIG_IP_ROUTE_MULTIPATH
@@ -1550,6 +1582,7 @@ int fib_sync_up(struct net_device *dev, unsigned int 
nh_flags)
continue;
alive++;
nexthop_nh->nh_flags &= ~nh_flags;
+   call_fib_nh_notifiers(nexthop_nh, FIB_EVENT_NH_ADD);
} endfor_nexthops(fi)
 
if (alive > 0) {
-- 
2.7.4