Re: [PATCH] fs: dcache: Avoid livelock between d_alloc_parallel and __d_add

[Will Deacon]

Hi Matthew,

On Tue, Feb 13, 2018 at 07:16:08AM -0800, Matthew Wilcox wrote:
> On Tue, Feb 13, 2018 at 12:58:51PM +, Will Deacon wrote:
> > This patch resolves the livelock by not taking hlist_bl_lock in
> > d_alloc_parallel if the sequence counter is odd, since any subsequent
> > masked comparison with i_dir_seq will fail anyway.
> > 
> > Cc: Peter Zijlstra 
> > Cc: Al Viro 
> > Signed-off-by: Will Deacon 
> 
> Reviewed-by: Matthew Wilcox 

Thanks!

> I wonder whether it makes sense to turn i_dir_seq into a seqcount_t,
> which would give us the lockdep checking as well.

I'm not sure it's quite as simple as that. start_dir_add looks very much
like it's intended to run concurrently, so we'd need a write_seqcount
implementation that provides the same atomicity guarantees.

Will

Re: [PATCH] fs: dcache: Avoid livelock between d_alloc_parallel and __d_add

[Will Deacon]

Hi Matthew,

On Tue, Feb 13, 2018 at 07:16:08AM -0800, Matthew Wilcox wrote:
> On Tue, Feb 13, 2018 at 12:58:51PM +, Will Deacon wrote:
> > This patch resolves the livelock by not taking hlist_bl_lock in
> > d_alloc_parallel if the sequence counter is odd, since any subsequent
> > masked comparison with i_dir_seq will fail anyway.
> > 
> > Cc: Peter Zijlstra 
> > Cc: Al Viro 
> > Signed-off-by: Will Deacon 
> 
> Reviewed-by: Matthew Wilcox 

Thanks!

> I wonder whether it makes sense to turn i_dir_seq into a seqcount_t,
> which would give us the lockdep checking as well.

I'm not sure it's quite as simple as that. start_dir_add looks very much
like it's intended to run concurrently, so we'd need a write_seqcount
implementation that provides the same atomicity guarantees.

Will

Re: [PATCH] fs: dcache: Avoid livelock between d_alloc_parallel and __d_add

[Matthew Wilcox]

On Tue, Feb 13, 2018 at 12:58:51PM +, Will Deacon wrote:
> This patch resolves the livelock by not taking hlist_bl_lock in
> d_alloc_parallel if the sequence counter is odd, since any subsequent
> masked comparison with i_dir_seq will fail anyway.
> 
> Cc: Peter Zijlstra 
> Cc: Al Viro 
> Signed-off-by: Will Deacon 

Reviewed-by: Matthew Wilcox 

I wonder whether it makes sense to turn i_dir_seq into a seqcount_t,
which would give us the lockdep checking as well.

Re: [PATCH] fs: dcache: Avoid livelock between d_alloc_parallel and __d_add

[Matthew Wilcox]

On Tue, Feb 13, 2018 at 12:58:51PM +, Will Deacon wrote:
> This patch resolves the livelock by not taking hlist_bl_lock in
> d_alloc_parallel if the sequence counter is odd, since any subsequent
> masked comparison with i_dir_seq will fail anyway.
> 
> Cc: Peter Zijlstra 
> Cc: Al Viro 
> Signed-off-by: Will Deacon 

Reviewed-by: Matthew Wilcox 

I wonder whether it makes sense to turn i_dir_seq into a seqcount_t,
which would give us the lockdep checking as well.

Re: [PATCH] fs: dcache: Avoid livelock between d_alloc_parallel and __d_add

[Peter Zijlstra]

On Tue, Feb 13, 2018 at 12:58:51PM +, Will Deacon wrote:
> If d_alloc_parallel runs concurrently with __d_add, it is possible for
> d_alloc_parallel to continuously retry whilst i_dir_seq has been
> incremented to an odd value by __d_add:
> 
> CPU0:
> __d_add
>   n = start_dir_add(dir);
>   cmpxchg(>i_dir_seq, n, n + 1) == n
> 
> CPU1:
> d_alloc_parallel
> retry:
>   seq = smp_load_acquire(>d_inode->i_dir_seq) & ~1;
>   hlist_bl_lock(b);
>   bit_spin_lock(0, (unsigned long *)b); // Always succeeds
> 
> CPU0:
>   __d_lookup_done(dentry)
>   hlist_bl_lock
>   bit_spin_lock(0, (unsigned long *)b); // Never succeeds
> 
> CPU1:
>   if (unlikely(parent->d_inode->i_dir_seq != seq)) {
>   hlist_bl_unlock(b);
>   goto retry;
>   }
> 
> Since the simple bit_spin_lock used to implement hlist_bl_lock does not

And cannot, a single bit is just not enough state.

> provide any fairness guarantees, then CPU1 can starve CPU0 of the lock
> and prevent it from reaching end_dir_add(dir), therefore CPU1 cannot
> exit its retry loop because the sequence number always has the bottom
> bit set.
> 
> This patch resolves the livelock by not taking hlist_bl_lock in
> d_alloc_parallel if the sequence counter is odd, since any subsequent
> masked comparison with i_dir_seq will fail anyway.
> 

Acked-by: Peter Zijlstra (Intel) 

> Cc: Al Viro 
> Signed-off-by: Will Deacon 
> ---
>  fs/dcache.c | 8 +++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/fs/dcache.c b/fs/dcache.c
> index 7c38f39958bc..b243deec298c 100644
> --- a/fs/dcache.c
> +++ b/fs/dcache.c
> @@ -2474,7 +2474,7 @@ struct dentry *d_alloc_parallel(struct dentry *parent,
>  
>  retry:
>   rcu_read_lock();
> - seq = smp_load_acquire(>d_inode->i_dir_seq) & ~1;
> + seq = smp_load_acquire(>d_inode->i_dir_seq);
>   r_seq = read_seqbegin(_lock);
>   dentry = __d_lookup_rcu(parent, name, _seq);
>   if (unlikely(dentry)) {
> @@ -2495,6 +2495,12 @@ struct dentry *d_alloc_parallel(struct dentry *parent,
>   rcu_read_unlock();
>   goto retry;
>   }
> +
> + if (unlikely(seq & 1)) {
> + rcu_read_unlock();
> + goto retry;
> + }
> +
>   hlist_bl_lock(b);
>   if (unlikely(parent->d_inode->i_dir_seq != seq)) {

Also, should that not read:

if (unlikely(READ_ONCE(parent->d_inode->i_dir_seq) != seq)) {

I mean, load-tearing can only result in additional failure, but still.

>   hlist_bl_unlock(b);

Re: [PATCH] fs: dcache: Avoid livelock between d_alloc_parallel and __d_add

[Peter Zijlstra]

On Tue, Feb 13, 2018 at 12:58:51PM +, Will Deacon wrote:
> If d_alloc_parallel runs concurrently with __d_add, it is possible for
> d_alloc_parallel to continuously retry whilst i_dir_seq has been
> incremented to an odd value by __d_add:
> 
> CPU0:
> __d_add
>   n = start_dir_add(dir);
>   cmpxchg(>i_dir_seq, n, n + 1) == n
> 
> CPU1:
> d_alloc_parallel
> retry:
>   seq = smp_load_acquire(>d_inode->i_dir_seq) & ~1;
>   hlist_bl_lock(b);
>   bit_spin_lock(0, (unsigned long *)b); // Always succeeds
> 
> CPU0:
>   __d_lookup_done(dentry)
>   hlist_bl_lock
>   bit_spin_lock(0, (unsigned long *)b); // Never succeeds
> 
> CPU1:
>   if (unlikely(parent->d_inode->i_dir_seq != seq)) {
>   hlist_bl_unlock(b);
>   goto retry;
>   }
> 
> Since the simple bit_spin_lock used to implement hlist_bl_lock does not

And cannot, a single bit is just not enough state.

> provide any fairness guarantees, then CPU1 can starve CPU0 of the lock
> and prevent it from reaching end_dir_add(dir), therefore CPU1 cannot
> exit its retry loop because the sequence number always has the bottom
> bit set.
> 
> This patch resolves the livelock by not taking hlist_bl_lock in
> d_alloc_parallel if the sequence counter is odd, since any subsequent
> masked comparison with i_dir_seq will fail anyway.
> 

Acked-by: Peter Zijlstra (Intel) 

> Cc: Al Viro 
> Signed-off-by: Will Deacon 
> ---
>  fs/dcache.c | 8 +++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/fs/dcache.c b/fs/dcache.c
> index 7c38f39958bc..b243deec298c 100644
> --- a/fs/dcache.c
> +++ b/fs/dcache.c
> @@ -2474,7 +2474,7 @@ struct dentry *d_alloc_parallel(struct dentry *parent,
>  
>  retry:
>   rcu_read_lock();
> - seq = smp_load_acquire(>d_inode->i_dir_seq) & ~1;
> + seq = smp_load_acquire(>d_inode->i_dir_seq);
>   r_seq = read_seqbegin(_lock);
>   dentry = __d_lookup_rcu(parent, name, _seq);
>   if (unlikely(dentry)) {
> @@ -2495,6 +2495,12 @@ struct dentry *d_alloc_parallel(struct dentry *parent,
>   rcu_read_unlock();
>   goto retry;
>   }
> +
> + if (unlikely(seq & 1)) {
> + rcu_read_unlock();
> + goto retry;
> + }
> +
>   hlist_bl_lock(b);
>   if (unlikely(parent->d_inode->i_dir_seq != seq)) {

Also, should that not read:

if (unlikely(READ_ONCE(parent->d_inode->i_dir_seq) != seq)) {

I mean, load-tearing can only result in additional failure, but still.

>   hlist_bl_unlock(b);