Re: [PATCH RESEND nf] netfilter: avoid a race between nf_register_hook() and cleanup_net()

[Pablo Neira Ayuso]

Hi Eric,

On Sat, Jul 30, 2016 at 08:24:37AM -0500, Eric W. Biederman wrote:
> Michal Kubecek  writes:
> 
> > There is a race condition between nf_{,un}register_hook() and
> > cleanup_net() which can either trigger WARN check or cause a memory
> > leak. The scenario is like this (2a and 2b are alternatives):
> >
> > 1.  cleanup_net() removes one or more struct net from net_namespace_list
> > 2a. nf_register_hook() adds per-netns hooks to all netns (but not those
> > removed in step 1) and adds the hook to global nf_hook_list
> > 2b. nf_unregister_hook() deletes per-netns hooks from all netns (but not
> > those removed in step 1) and removes the hook from nf_hook_list
> > 3.  cleanup_net() calls pernet subsystem exit functions for netns being
> > removed; one of them is netfilter_net_exit() which (among others)
> > calls nf_unregister_net_hook() to unregister per-netns hooks for all
> > hooks in nf_hook_list.
> >
> > In case (a), per-netns hooks are never added as the namespace was
> > already invisible to for_each_net() in step 2a but an attempt to remove
> > them in step 3 (the hook is already in nf_hook_list) triggers a WARN
> > check in nf_unregister_net_hook() (no real harm done, however). In case
> > (b), the per-netns hook is removed neither in step 2b (netns is already
> > invisible to for_each_net()) nor in step 3 (the hook is already removed
> > from nf_hook_list), causing a memory leak.
> >
> > Prevent the race by protecting the for_each_net() loop in
> > nf_{,un}register_hook() (also) by net_mutex. There is already a
> > precendens for this in rtnl_link_unregister() which addresses similar
> > race.
> 
> So this analysis of a problem appears to be spot on.
> 
> Reviewed-by: "Eric W. Biederman" 
> 
> I really really want there to be a better way to do this, but it is
> really not ok for a hook to continue it's life past
> nf_unregister_net_hook as after that point the code may be removed
> from the kernel (sigh).
> 
> Although keeping with the precedent and minimizing net_mutex
> we could remove the WARN and keep nf_register_hook as it is.
> But that sounds entirely too clever for a fix that will
> probably be backported.
> 
> But that sounds entirely too clever for a fix that likely needs to be
> backported.

OK... I'm going to place this in the nf.git tree... but this is very ugly.

So Eric, I'd really appreciate if you can follow up once this has hit
nf-next.git and we get rid of the rtnl_lock and net_lock mutex by
propagating up to the the caller the hook registration from init_net()
and unregistering this from exit_net(). So we don't need to loop on
the existing netns but we use the existing netns init and exit
callbacks.

Let me know, thanks.

Re: [PATCH RESEND nf] netfilter: avoid a race between nf_register_hook() and cleanup_net()

[Pablo Neira Ayuso]

On Sat, Jul 30, 2016 at 08:24:37AM -0500, Eric W. Biederman wrote:
> Michal Kubecek  writes:
> 
> > There is a race condition between nf_{,un}register_hook() and
> > cleanup_net() which can either trigger WARN check or cause a memory
> > leak. The scenario is like this (2a and 2b are alternatives):
> >
> > 1.  cleanup_net() removes one or more struct net from net_namespace_list
> > 2a. nf_register_hook() adds per-netns hooks to all netns (but not those
> > removed in step 1) and adds the hook to global nf_hook_list
> > 2b. nf_unregister_hook() deletes per-netns hooks from all netns (but not
> > those removed in step 1) and removes the hook from nf_hook_list
> > 3.  cleanup_net() calls pernet subsystem exit functions for netns being
> > removed; one of them is netfilter_net_exit() which (among others)
> > calls nf_unregister_net_hook() to unregister per-netns hooks for all
> > hooks in nf_hook_list.
> >
> > In case (a), per-netns hooks are never added as the namespace was
> > already invisible to for_each_net() in step 2a but an attempt to remove
> > them in step 3 (the hook is already in nf_hook_list) triggers a WARN
> > check in nf_unregister_net_hook() (no real harm done, however). In case
> > (b), the per-netns hook is removed neither in step 2b (netns is already
> > invisible to for_each_net()) nor in step 3 (the hook is already removed
> > from nf_hook_list), causing a memory leak.
> >
> > Prevent the race by protecting the for_each_net() loop in
> > nf_{,un}register_hook() (also) by net_mutex. There is already a
> > precendens for this in rtnl_link_unregister() which addresses similar
> > race.
> 
> So this analysis of a problem appears to be spot on.
> 
> Reviewed-by: "Eric W. Biederman" 
> 
> 
> I really really want there to be a better way to do this, but it is
> really not ok for a hook to continue it's life past
> nf_unregister_net_hook as after that point the code may be removed
> from the kernel (sigh).
> 
> Although keeping with the precedent and minimizing net_mutex
> we could remove the WARN and keep nf_register_hook as it is.
> But that sounds entirely too clever for a fix that will
> probably be backported.
> 
> But that sounds entirely too clever for a fix that likely needs to be
> backported.

Please, propagate up to the caller to register and unregister the
hooks from init_net and exit_net instead as I suggested time ago.

I understand that this is not as small as this patch, and that this
will require a bit more boiler plate code in iptable_*.c and nftables
itself, but we'll avoid the dependencies with both rtnl_lock and
net_lock.

Thanks.

Re: [PATCH RESEND nf] netfilter: avoid a race between nf_register_hook() and cleanup_net()

[Eric W. Biederman]

Michal Kubecek  writes:

> There is a race condition between nf_{,un}register_hook() and
> cleanup_net() which can either trigger WARN check or cause a memory
> leak. The scenario is like this (2a and 2b are alternatives):
>
> 1.  cleanup_net() removes one or more struct net from net_namespace_list
> 2a. nf_register_hook() adds per-netns hooks to all netns (but not those
> removed in step 1) and adds the hook to global nf_hook_list
> 2b. nf_unregister_hook() deletes per-netns hooks from all netns (but not
> those removed in step 1) and removes the hook from nf_hook_list
> 3.  cleanup_net() calls pernet subsystem exit functions for netns being
> removed; one of them is netfilter_net_exit() which (among others)
> calls nf_unregister_net_hook() to unregister per-netns hooks for all
> hooks in nf_hook_list.
>
> In case (a), per-netns hooks are never added as the namespace was
> already invisible to for_each_net() in step 2a but an attempt to remove
> them in step 3 (the hook is already in nf_hook_list) triggers a WARN
> check in nf_unregister_net_hook() (no real harm done, however). In case
> (b), the per-netns hook is removed neither in step 2b (netns is already
> invisible to for_each_net()) nor in step 3 (the hook is already removed
> from nf_hook_list), causing a memory leak.
>
> Prevent the race by protecting the for_each_net() loop in
> nf_{,un}register_hook() (also) by net_mutex. There is already a
> precendens for this in rtnl_link_unregister() which addresses similar
> race.

So this analysis of a problem appears to be spot on.

Reviewed-by: "Eric W. Biederman" 


I really really want there to be a better way to do this, but it is
really not ok for a hook to continue it's life past
nf_unregister_net_hook as after that point the code may be removed
from the kernel (sigh).

Although keeping with the precedent and minimizing net_mutex
we could remove the WARN and keep nf_register_hook as it is.
But that sounds entirely too clever for a fix that will
probably be backported.

But that sounds entirely too clever for a fix that likely needs to be
backported.

Eric

> Fixes: 085db2c04557 ("netfilter: Per network namespace netfilter hooks.")
> Signed-off-by: Michal Kubecek 
> ---
>  net/netfilter/core.c | 7 +++
>  1 file changed, 7 insertions(+)
>
> diff --git a/net/netfilter/core.c b/net/netfilter/core.c
> index f39276d1c2d7..860978c9f82e 100644
> --- a/net/netfilter/core.c
> +++ b/net/netfilter/core.c
> @@ -193,6 +193,8 @@ int nf_register_hook(struct nf_hook_ops *reg)
>   struct net *net, *last;
>   int ret;
>  
> + /* prevent race with cleanup_net() */
> + mutex_lock(_mutex);
>   rtnl_lock();
>   for_each_net(net) {
>   ret = nf_register_net_hook(net, reg);
> @@ -201,6 +203,7 @@ int nf_register_hook(struct nf_hook_ops *reg)
>   }
>   list_add_tail(>list, _hook_list);
>   rtnl_unlock();
> + mutex_unlock(_mutex);
>  
>   return 0;
>  rollback:
> @@ -211,6 +214,7 @@ rollback:
>   nf_unregister_net_hook(net, reg);
>   }
>   rtnl_unlock();
> + mutex_unlock(_mutex);
>   return ret;
>  }
>  EXPORT_SYMBOL(nf_register_hook);
> @@ -219,11 +223,14 @@ void nf_unregister_hook(struct nf_hook_ops *reg)
>  {
>   struct net *net;
>  
> + /* prevent race with cleanup_net() */
> + mutex_lock(_mutex);
>   rtnl_lock();
>   list_del(>list);
>   for_each_net(net)
>   nf_unregister_net_hook(net, reg);
>   rtnl_unlock();
> + mutex_unlock(_mutex);
>  }
>  EXPORT_SYMBOL(nf_unregister_hook);

[PATCH RESEND nf] netfilter: avoid a race between nf_register_hook() and cleanup_net()

[Michal Kubecek]

There is a race condition between nf_{,un}register_hook() and
cleanup_net() which can either trigger WARN check or cause a memory
leak. The scenario is like this (2a and 2b are alternatives):

1.  cleanup_net() removes one or more struct net from net_namespace_list
2a. nf_register_hook() adds per-netns hooks to all netns (but not those
removed in step 1) and adds the hook to global nf_hook_list
2b. nf_unregister_hook() deletes per-netns hooks from all netns (but not
those removed in step 1) and removes the hook from nf_hook_list
3.  cleanup_net() calls pernet subsystem exit functions for netns being
removed; one of them is netfilter_net_exit() which (among others)
calls nf_unregister_net_hook() to unregister per-netns hooks for all
hooks in nf_hook_list.

In case (a), per-netns hooks are never added as the namespace was
already invisible to for_each_net() in step 2a but an attempt to remove
them in step 3 (the hook is already in nf_hook_list) triggers a WARN
check in nf_unregister_net_hook() (no real harm done, however). In case
(b), the per-netns hook is removed neither in step 2b (netns is already
invisible to for_each_net()) nor in step 3 (the hook is already removed
from nf_hook_list), causing a memory leak.

Prevent the race by protecting the for_each_net() loop in
nf_{,un}register_hook() (also) by net_mutex. There is already a
precendens for this in rtnl_link_unregister() which addresses similar
race.

Fixes: 085db2c04557 ("netfilter: Per network namespace netfilter hooks.")
Signed-off-by: Michal Kubecek 
---
 net/netfilter/core.c | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/net/netfilter/core.c b/net/netfilter/core.c
index f39276d1c2d7..860978c9f82e 100644
--- a/net/netfilter/core.c
+++ b/net/netfilter/core.c
@@ -193,6 +193,8 @@ int nf_register_hook(struct nf_hook_ops *reg)
struct net *net, *last;
int ret;
 
+   /* prevent race with cleanup_net() */
+   mutex_lock(_mutex);
rtnl_lock();
for_each_net(net) {
ret = nf_register_net_hook(net, reg);
@@ -201,6 +203,7 @@ int nf_register_hook(struct nf_hook_ops *reg)
}
list_add_tail(>list, _hook_list);
rtnl_unlock();
+   mutex_unlock(_mutex);
 
return 0;
 rollback:
@@ -211,6 +214,7 @@ rollback:
nf_unregister_net_hook(net, reg);
}
rtnl_unlock();
+   mutex_unlock(_mutex);
return ret;
 }
 EXPORT_SYMBOL(nf_register_hook);
@@ -219,11 +223,14 @@ void nf_unregister_hook(struct nf_hook_ops *reg)
 {
struct net *net;
 
+   /* prevent race with cleanup_net() */
+   mutex_lock(_mutex);
rtnl_lock();
list_del(>list);
for_each_net(net)
nf_unregister_net_hook(net, reg);
rtnl_unlock();
+   mutex_unlock(_mutex);
 }
 EXPORT_SYMBOL(nf_unregister_hook);
 
-- 
2.9.2