Re: page allocation stall in kernel 4.9 when copying files from one btrfs hdd to another

2016-12-15 Thread admin 
Hi,

The source is a software raid 5 (md) of 4x 4TB Western Digital RE4 disks. The 
destinations is a hardware raid 5 enclosure containing 4x 8TB Seagate Archival 
disks connected using e-sata. 

I am currently trying Duncans suggestions. With them, the page allocation stall 
doesn't seem to appear and overall system responsiveness is also much better 
during copying.

Thanks,
David Arendt


Xin Zhou – Thu., 15. December 2016 0:24
> Hi,
> 
> The dirty data is in large amount, probably unable to commit to disk.
> And this seems to happen when copying from 7200rpm to 5600rpm disks, 
> according to previous post.
> 
> Probably the I/Os are buffered and pending, unable to get finished in-time.
> It might be helpful to know if this only happens for specific types of 5600 
> rpm disks?
> 
> And are these disks on RAID groups? Thanks.
> Xin
>  
>  
> 
> Sent: Wednesday, December 14, 2016 at 3:38 AM
> From: admin <ad...@prnet.org>
> To: "Michal Hocko" <mho...@kernel.org>
> Cc: linux-btrfs@vger.kernel.org, linux-ker...@vger.kernel.org, "David Sterba" 
> <dste...@suse.cz>, "Chris Mason" <c...@fb.com>
> Subject: Re: page allocation stall in kernel 4.9 when copying files from one 
> btrfs hdd to another
> Hi,
> 
> I verified the log files and see no prior oom killer invocation. 
> Unfortunately the machine has been rebooted since. Next time it happens, I 
> will also look in dmesg.
> 
> Thanks,
> David Arendt
> 
> 
> Michal Hocko – Wed., 14. December 2016 11:31
> > Btw. the stall should be preceded by the OOM killer invocation. Could
> > you share the OOM report please. I am asking because such an OOM killer
> > would be clearly pre-mature as per your meminfo. I am trying to change
> > that code and seeing your numbers might help me.
> >
> > Thanks!
> >
> > On Wed 14-12-16 11:17:43, Michal Hocko wrote:
> > > On Tue 13-12-16 18:11:01, David Arendt wrote:
> > > > Hi,
> > > >
> > > > I receive the following page allocation stall while copying lots of
> > > > large files from one btrfs hdd to another.
> > > >
> > > > Dec 13 13:04:29 server kernel: kworker/u16:8: page allocation stalls 
> > > > for 12260ms, order:0, mode:0x2400840(GFP_NOFS|__GFP_NOFAIL)
> > > > Dec 13 13:04:29 server kernel: CPU: 0 PID: 24959 Comm: kworker/u16:8 
> > > > Tainted: P O 4.9.0 #1
> > > [...]
> > > > Dec 13 13:04:29 server kernel: Call Trace:
> > > > Dec 13 13:04:29 server kernel: [] ? 
> > > > dump_stack+0x46/0x5d
> > > > Dec 13 13:04:29 server kernel: [] ? 
> > > > warn_alloc+0x111/0x130
> > > > Dec 13 13:04:33 server kernel: [] ? 
> > > > __alloc_pages_nodemask+0xbe8/0xd30
> > > > Dec 13 13:04:33 server kernel: [] ? 
> > > > pagecache_get_page+0xe4/0x230
> > > > Dec 13 13:04:33 server kernel: [] ? 
> > > > alloc_extent_buffer+0x10b/0x400
> > > > Dec 13 13:04:33 server kernel: [] ? 
> > > > btrfs_alloc_tree_block+0x125/0x560
> > >
> > > OK, so this is
> > > find_or_create_page(mapping, index, GFP_NOFS|__GFP_NOFAIL)
> > >
> > > The main question is whether this really needs to be NOFS request...
> > >
> > > > Dec 13 13:04:33 server kernel: [] ? 
> > > > read_extent_buffer_pages+0x21f/0x280
> > > > Dec 13 13:04:33 server kernel: [] ? 
> > > > __btrfs_cow_block+0x141/0x580
> > > > Dec 13 13:04:33 server kernel: [] ? 
> > > > btrfs_cow_block+0x100/0x150
> > > > Dec 13 13:04:33 server kernel: [] ? 
> > > > btrfs_search_slot+0x1e9/0x9c0
> > > > Dec 13 13:04:33 server kernel: [] ? 
> > > > __set_extent_bit+0x512/0x550
> > > > Dec 13 13:04:33 server kernel: [] ? 
> > > > lookup_inline_extent_backref+0xf5/0x5e0
> > > > Dec 13 13:04:34 server kernel: [] ? 
> > > > set_extent_bit+0x24/0x30
> > > > Dec 13 13:04:34 server kernel: [] ? 
> > > > update_block_group.isra.34+0x114/0x380
> > > > Dec 13 13:04:34 server kernel: [] ? 
> > > > __btrfs_free_extent.isra.35+0xf4/0xd20
> > > > Dec 13 13:04:34 server kernel: [] ? 
> > > > btrfs_merge_delayed_refs+0x61/0x5d0
> > > > Dec 13 13:04:34 server kernel: [] ? 
> > > > __btrfs_run_delayed_refs+0x902/0x10a0
> > > > Dec 13 13:04:34 server kernel: [] ? 
> > > > btrfs_run_delayed_refs+0x90/0x2a0
> > > > Dec 13 13:04:34 server kernel: [] ? 
> > > > delayed_ref_async_start+0x84/0xa0

Re: page allocation stall in kernel 4.9 when copying files from one btrfs hdd to another

2016-12-14 Thread Xin Zhou 
Hi,

The dirty data is in large amount, probably unable to commit to disk.
And this seems to happen when copying from 7200rpm to 5600rpm disks, according 
to previous post.

Probably the I/Os are buffered and pending, unable to get finished in-time.
It might be helpful to know if this only happens for specific types of 5600 rpm 
disks?

And are these disks on RAID groups? Thanks.
Xin
 
 

Sent: Wednesday, December 14, 2016 at 3:38 AM
From: admin <ad...@prnet.org>
To: "Michal Hocko" <mho...@kernel.org>
Cc: linux-btrfs@vger.kernel.org, linux-ker...@vger.kernel.org, "David Sterba" 
<dste...@suse.cz>, "Chris Mason" <c...@fb.com>
Subject: Re: page allocation stall in kernel 4.9 when copying files from one 
btrfs hdd to another
Hi,

I verified the log files and see no prior oom killer invocation. Unfortunately 
the machine has been rebooted since. Next time it happens, I will also look in 
dmesg.

Thanks,
David Arendt


Michal Hocko – Wed., 14. December 2016 11:31
> Btw. the stall should be preceded by the OOM killer invocation. Could
> you share the OOM report please. I am asking because such an OOM killer
> would be clearly pre-mature as per your meminfo. I am trying to change
> that code and seeing your numbers might help me.
>
> Thanks!
>
> On Wed 14-12-16 11:17:43, Michal Hocko wrote:
> > On Tue 13-12-16 18:11:01, David Arendt wrote:
> > > Hi,
> > >
> > > I receive the following page allocation stall while copying lots of
> > > large files from one btrfs hdd to another.
> > >
> > > Dec 13 13:04:29 server kernel: kworker/u16:8: page allocation stalls for 
> > > 12260ms, order:0, mode:0x2400840(GFP_NOFS|__GFP_NOFAIL)
> > > Dec 13 13:04:29 server kernel: CPU: 0 PID: 24959 Comm: kworker/u16:8 
> > > Tainted: P O 4.9.0 #1
> > [...]
> > > Dec 13 13:04:29 server kernel: Call Trace:
> > > Dec 13 13:04:29 server kernel: [] ? dump_stack+0x46/0x5d
> > > Dec 13 13:04:29 server kernel: [] ? 
> > > warn_alloc+0x111/0x130
> > > Dec 13 13:04:33 server kernel: [] ? 
> > > __alloc_pages_nodemask+0xbe8/0xd30
> > > Dec 13 13:04:33 server kernel: [] ? 
> > > pagecache_get_page+0xe4/0x230
> > > Dec 13 13:04:33 server kernel: [] ? 
> > > alloc_extent_buffer+0x10b/0x400
> > > Dec 13 13:04:33 server kernel: [] ? 
> > > btrfs_alloc_tree_block+0x125/0x560
> >
> > OK, so this is
> > find_or_create_page(mapping, index, GFP_NOFS|__GFP_NOFAIL)
> >
> > The main question is whether this really needs to be NOFS request...
> >
> > > Dec 13 13:04:33 server kernel: [] ? 
> > > read_extent_buffer_pages+0x21f/0x280
> > > Dec 13 13:04:33 server kernel: [] ? 
> > > __btrfs_cow_block+0x141/0x580
> > > Dec 13 13:04:33 server kernel: [] ? 
> > > btrfs_cow_block+0x100/0x150
> > > Dec 13 13:04:33 server kernel: [] ? 
> > > btrfs_search_slot+0x1e9/0x9c0
> > > Dec 13 13:04:33 server kernel: [] ? 
> > > __set_extent_bit+0x512/0x550
> > > Dec 13 13:04:33 server kernel: [] ? 
> > > lookup_inline_extent_backref+0xf5/0x5e0
> > > Dec 13 13:04:34 server kernel: [] ? 
> > > set_extent_bit+0x24/0x30
> > > Dec 13 13:04:34 server kernel: [] ? 
> > > update_block_group.isra.34+0x114/0x380
> > > Dec 13 13:04:34 server kernel: [] ? 
> > > __btrfs_free_extent.isra.35+0xf4/0xd20
> > > Dec 13 13:04:34 server kernel: [] ? 
> > > btrfs_merge_delayed_refs+0x61/0x5d0
> > > Dec 13 13:04:34 server kernel: [] ? 
> > > __btrfs_run_delayed_refs+0x902/0x10a0
> > > Dec 13 13:04:34 server kernel: [] ? 
> > > btrfs_run_delayed_refs+0x90/0x2a0
> > > Dec 13 13:04:34 server kernel: [] ? 
> > > delayed_ref_async_start+0x84/0xa0
> >
> > What would cause the reclaim recursion?
> >
> > > Dec 13 13:04:34 server kernel: Mem-Info:
> > > Dec 13 13:04:34 server kernel: active_anon:20 inactive_anon:34
> > > isolated_anon:0\x0a active_file:7370032 inactive_file:450105
> > > isolated_file:320\x0a unevictable:0 dirty:522748 writeback:189
> > > unstable:0\x0a slab_reclaimable:178255 slab_unreclaimable:124617\x0a
> > > mapped:4236 shmem:0 pagetables:1163 bounce:0\x0a free:38224 free_pcp:241
> > > free_cma:0
> >
> > This speaks for itself. There is a lot of dirty data, basically no
> > anonymous memory and GFP_NOFS cannot do much to reclaim obviously. This
> > is either a configuraion bug as somebody noted down the thread (setting
> > the dirty_ratio) or suboptimality of the btrfs code which might request
> > NOFS even though it is not strictly necessary. This would be more for
> > btrfs developers.
> > --
> > Michal Hocko
> > SUSE Labs
>
> --
> Michal Hocko
> SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: page allocation stall in kernel 4.9 when copying files from one btrfs hdd to another

2016-12-14 Thread admin 
Hi,

I verified the log files and see no prior oom killer invocation. Unfortunately 
the machine has been rebooted since. Next time it happens, I will also look in 
dmesg.

Thanks,
David Arendt 


Michal Hocko – Wed., 14. December 2016 11:31
> Btw. the stall should be preceded by the OOM killer invocation. Could
> you share the OOM report please. I am asking because such an OOM killer
> would be clearly pre-mature as per your meminfo. I am trying to change
> that code and seeing your numbers might help me.
> 
> Thanks!
> 
> On Wed 14-12-16 11:17:43, Michal Hocko wrote:
> > On Tue 13-12-16 18:11:01, David Arendt wrote:
> > > Hi,
> > > 
> > > I receive the following page allocation stall while copying lots of
> > > large files from one btrfs hdd to another.
> > > 
> > > Dec 13 13:04:29 server kernel: kworker/u16:8: page allocation stalls for 
> > > 12260ms, order:0, mode:0x2400840(GFP_NOFS|__GFP_NOFAIL)
> > > Dec 13 13:04:29 server kernel: CPU: 0 PID: 24959 Comm: kworker/u16:8 
> > > Tainted: P   O4.9.0 #1
> > [...]
> > > Dec 13 13:04:29 server kernel: Call Trace:
> > > Dec 13 13:04:29 server kernel:  [] ? 
> > > dump_stack+0x46/0x5d
> > > Dec 13 13:04:29 server kernel:  [] ? 
> > > warn_alloc+0x111/0x130
> > > Dec 13 13:04:33 server kernel:  [] ? 
> > > __alloc_pages_nodemask+0xbe8/0xd30
> > > Dec 13 13:04:33 server kernel:  [] ? 
> > > pagecache_get_page+0xe4/0x230
> > > Dec 13 13:04:33 server kernel:  [] ? 
> > > alloc_extent_buffer+0x10b/0x400
> > > Dec 13 13:04:33 server kernel:  [] ? 
> > > btrfs_alloc_tree_block+0x125/0x560
> > 
> > OK, so this is
> > find_or_create_page(mapping, index, GFP_NOFS|__GFP_NOFAIL)
> > 
> > The main question is whether this really needs to be NOFS request...
> > 
> > > Dec 13 13:04:33 server kernel:  [] ? 
> > > read_extent_buffer_pages+0x21f/0x280
> > > Dec 13 13:04:33 server kernel:  [] ? 
> > > __btrfs_cow_block+0x141/0x580
> > > Dec 13 13:04:33 server kernel:  [] ? 
> > > btrfs_cow_block+0x100/0x150
> > > Dec 13 13:04:33 server kernel:  [] ?  
> > > btrfs_search_slot+0x1e9/0x9c0
> > > Dec 13 13:04:33 server kernel:  [] ? 
> > > __set_extent_bit+0x512/0x550
> > > Dec 13 13:04:33 server kernel:  [] ? 
> > > lookup_inline_extent_backref+0xf5/0x5e0
> > > Dec 13 13:04:34 server kernel:  [] ? 
> > > set_extent_bit+0x24/0x30
> > > Dec 13 13:04:34 server kernel:  [] ? 
> > > update_block_group.isra.34+0x114/0x380
> > > Dec 13 13:04:34 server kernel:  [] ? 
> > > __btrfs_free_extent.isra.35+0xf4/0xd20
> > > Dec 13 13:04:34 server kernel:  [] ? 
> > > btrfs_merge_delayed_refs+0x61/0x5d0
> > > Dec 13 13:04:34 server kernel:  [] ? 
> > > __btrfs_run_delayed_refs+0x902/0x10a0
> > > Dec 13 13:04:34 server kernel:  [] ? 
> > > btrfs_run_delayed_refs+0x90/0x2a0
> > > Dec 13 13:04:34 server kernel:  [] ? 
> > > delayed_ref_async_start+0x84/0xa0
> > 
> > What would cause the reclaim recursion?
> > 
> > > Dec 13 13:04:34 server kernel: Mem-Info:
> > > Dec 13 13:04:34 server kernel: active_anon:20 inactive_anon:34
> > > isolated_anon:0\x0a active_file:7370032 inactive_file:450105
> > > isolated_file:320\x0a unevictable:0 dirty:522748 writeback:189
> > > unstable:0\x0a slab_reclaimable:178255 slab_unreclaimable:124617\x0a
> > > mapped:4236 shmem:0 pagetables:1163 bounce:0\x0a free:38224 free_pcp:241
> > > free_cma:0
> > 
> > This speaks for itself. There is a lot of dirty data, basically no
> > anonymous memory and GFP_NOFS cannot do much to reclaim obviously. This
> > is either a configuraion bug as somebody noted down the thread (setting
> > the dirty_ratio) or suboptimality of the btrfs code which might request
> > NOFS even though it is not strictly necessary. This would be more for
> > btrfs developers.
> > -- 
> > Michal Hocko
> > SUSE Labs
> 
> -- 
> Michal Hocko
> SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: page allocation stall in kernel 4.9 when copying files from one btrfs hdd to another

2016-12-14 Thread Michal Hocko 
Btw. the stall should be preceded by the OOM killer invocation. Could
you share the OOM report please. I am asking because such an OOM killer
would be clearly pre-mature as per your meminfo. I am trying to change
that code and seeing your numbers might help me.

Thanks!

On Wed 14-12-16 11:17:43, Michal Hocko wrote:
> On Tue 13-12-16 18:11:01, David Arendt wrote:
> > Hi,
> > 
> > I receive the following page allocation stall while copying lots of
> > large files from one btrfs hdd to another.
> > 
> > Dec 13 13:04:29 server kernel: kworker/u16:8: page allocation stalls for 
> > 12260ms, order:0, mode:0x2400840(GFP_NOFS|__GFP_NOFAIL)
> > Dec 13 13:04:29 server kernel: CPU: 0 PID: 24959 Comm: kworker/u16:8 
> > Tainted: P   O4.9.0 #1
> [...]
> > Dec 13 13:04:29 server kernel: Call Trace:
> > Dec 13 13:04:29 server kernel:  [] ? dump_stack+0x46/0x5d
> > Dec 13 13:04:29 server kernel:  [] ? 
> > warn_alloc+0x111/0x130
> > Dec 13 13:04:33 server kernel:  [] ? 
> > __alloc_pages_nodemask+0xbe8/0xd30
> > Dec 13 13:04:33 server kernel:  [] ? 
> > pagecache_get_page+0xe4/0x230
> > Dec 13 13:04:33 server kernel:  [] ? 
> > alloc_extent_buffer+0x10b/0x400
> > Dec 13 13:04:33 server kernel:  [] ? 
> > btrfs_alloc_tree_block+0x125/0x560
> 
> OK, so this is
>   find_or_create_page(mapping, index, GFP_NOFS|__GFP_NOFAIL)
> 
> The main question is whether this really needs to be NOFS request...
> 
> > Dec 13 13:04:33 server kernel:  [] ? 
> > read_extent_buffer_pages+0x21f/0x280
> > Dec 13 13:04:33 server kernel:  [] ? 
> > __btrfs_cow_block+0x141/0x580
> > Dec 13 13:04:33 server kernel:  [] ? 
> > btrfs_cow_block+0x100/0x150
> > Dec 13 13:04:33 server kernel:  [] ?  
> > btrfs_search_slot+0x1e9/0x9c0
> > Dec 13 13:04:33 server kernel:  [] ? 
> > __set_extent_bit+0x512/0x550
> > Dec 13 13:04:33 server kernel:  [] ? 
> > lookup_inline_extent_backref+0xf5/0x5e0
> > Dec 13 13:04:34 server kernel:  [] ? 
> > set_extent_bit+0x24/0x30
> > Dec 13 13:04:34 server kernel:  [] ? 
> > update_block_group.isra.34+0x114/0x380
> > Dec 13 13:04:34 server kernel:  [] ? 
> > __btrfs_free_extent.isra.35+0xf4/0xd20
> > Dec 13 13:04:34 server kernel:  [] ? 
> > btrfs_merge_delayed_refs+0x61/0x5d0
> > Dec 13 13:04:34 server kernel:  [] ? 
> > __btrfs_run_delayed_refs+0x902/0x10a0
> > Dec 13 13:04:34 server kernel:  [] ? 
> > btrfs_run_delayed_refs+0x90/0x2a0
> > Dec 13 13:04:34 server kernel:  [] ? 
> > delayed_ref_async_start+0x84/0xa0
> 
> What would cause the reclaim recursion?
> 
> > Dec 13 13:04:34 server kernel: Mem-Info:
> > Dec 13 13:04:34 server kernel: active_anon:20 inactive_anon:34
> > isolated_anon:0\x0a active_file:7370032 inactive_file:450105
> > isolated_file:320\x0a unevictable:0 dirty:522748 writeback:189
> > unstable:0\x0a slab_reclaimable:178255 slab_unreclaimable:124617\x0a
> > mapped:4236 shmem:0 pagetables:1163 bounce:0\x0a free:38224 free_pcp:241
> > free_cma:0
> 
> This speaks for itself. There is a lot of dirty data, basically no
> anonymous memory and GFP_NOFS cannot do much to reclaim obviously. This
> is either a configuraion bug as somebody noted down the thread (setting
> the dirty_ratio) or suboptimality of the btrfs code which might request
> NOFS even though it is not strictly necessary. This would be more for
> btrfs developers.
> -- 
> Michal Hocko
> SUSE Labs

-- 
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: page allocation stall in kernel 4.9 when copying files from one btrfs hdd to another

2016-12-14 Thread Michal Hocko 
On Tue 13-12-16 18:11:01, David Arendt wrote:
> Hi,
> 
> I receive the following page allocation stall while copying lots of
> large files from one btrfs hdd to another.
> 
> Dec 13 13:04:29 server kernel: kworker/u16:8: page allocation stalls for 
> 12260ms, order:0, mode:0x2400840(GFP_NOFS|__GFP_NOFAIL)
> Dec 13 13:04:29 server kernel: CPU: 0 PID: 24959 Comm: kworker/u16:8 Tainted: 
> P   O4.9.0 #1
[...]
> Dec 13 13:04:29 server kernel: Call Trace:
> Dec 13 13:04:29 server kernel:  [] ? dump_stack+0x46/0x5d
> Dec 13 13:04:29 server kernel:  [] ? warn_alloc+0x111/0x130
> Dec 13 13:04:33 server kernel:  [] ? 
> __alloc_pages_nodemask+0xbe8/0xd30
> Dec 13 13:04:33 server kernel:  [] ? 
> pagecache_get_page+0xe4/0x230
> Dec 13 13:04:33 server kernel:  [] ? 
> alloc_extent_buffer+0x10b/0x400
> Dec 13 13:04:33 server kernel:  [] ? 
> btrfs_alloc_tree_block+0x125/0x560

OK, so this is
find_or_create_page(mapping, index, GFP_NOFS|__GFP_NOFAIL)

The main question is whether this really needs to be NOFS request...

> Dec 13 13:04:33 server kernel:  [] ? 
> read_extent_buffer_pages+0x21f/0x280
> Dec 13 13:04:33 server kernel:  [] ? 
> __btrfs_cow_block+0x141/0x580
> Dec 13 13:04:33 server kernel:  [] ? 
> btrfs_cow_block+0x100/0x150
> Dec 13 13:04:33 server kernel:  [] ?  
> btrfs_search_slot+0x1e9/0x9c0
> Dec 13 13:04:33 server kernel:  [] ? 
> __set_extent_bit+0x512/0x550
> Dec 13 13:04:33 server kernel:  [] ? 
> lookup_inline_extent_backref+0xf5/0x5e0
> Dec 13 13:04:34 server kernel:  [] ? 
> set_extent_bit+0x24/0x30
> Dec 13 13:04:34 server kernel:  [] ? 
> update_block_group.isra.34+0x114/0x380
> Dec 13 13:04:34 server kernel:  [] ? 
> __btrfs_free_extent.isra.35+0xf4/0xd20
> Dec 13 13:04:34 server kernel:  [] ? 
> btrfs_merge_delayed_refs+0x61/0x5d0
> Dec 13 13:04:34 server kernel:  [] ? 
> __btrfs_run_delayed_refs+0x902/0x10a0
> Dec 13 13:04:34 server kernel:  [] ? 
> btrfs_run_delayed_refs+0x90/0x2a0
> Dec 13 13:04:34 server kernel:  [] ? 
> delayed_ref_async_start+0x84/0xa0

What would cause the reclaim recursion?

> Dec 13 13:04:34 server kernel: Mem-Info:
> Dec 13 13:04:34 server kernel: active_anon:20 inactive_anon:34
> isolated_anon:0\x0a active_file:7370032 inactive_file:450105
> isolated_file:320\x0a unevictable:0 dirty:522748 writeback:189
> unstable:0\x0a slab_reclaimable:178255 slab_unreclaimable:124617\x0a
> mapped:4236 shmem:0 pagetables:1163 bounce:0\x0a free:38224 free_pcp:241
> free_cma:0

This speaks for itself. There is a lot of dirty data, basically no
anonymous memory and GFP_NOFS cannot do much to reclaim obviously. This
is either a configuraion bug as somebody noted down the thread (setting
the dirty_ratio) or suboptimality of the btrfs code which might request
NOFS even though it is not strictly necessary. This would be more for
btrfs developers.
-- 
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: page allocation stall in kernel 4.9 when copying files from one btrfs hdd to another

2016-12-13 Thread David Arendt 
Hi,

unfortunately I did not dump meminfo before the crash.

Here is the actual meminfo as of now with the copy running for about 3
hours.

MemTotal:   32806572 kB
MemFree:  197336 kB
MemAvailable:   31226888 kB
Buffers:  52 kB
Cached: 30603160 kB
SwapCached:11880 kB
Active: 29015008 kB
Inactive:2017292 kB
Active(anon): 162124 kB
Inactive(anon):   285104 kB
Active(file):   28852884 kB
Inactive(file):  1732188 kB
Unevictable:7092 kB
Mlocked:7092 kB
SwapTotal:  62522692 kB
SwapFree:   62460464 kB
Dirty:231944 kB
Writeback: 0 kB
AnonPages:425160 kB
Mapped:   227656 kB
Shmem: 12160 kB
Slab:1380280 kB
SReclaimable: 774584 kB
SUnreclaim:   605696 kB
KernelStack:7840 kB
PageTables:12800 kB
NFS_Unstable:  0 kB
Bounce:0 kB
WritebackTmp:  0 kB
CommitLimit:78925976 kB
Committed_AS:1883256 kB
VmallocTotal:   34359738367 kB
VmallocUsed:   0 kB
VmallocChunk:  0 kB
HugePages_Total:   0
HugePages_Free:0
HugePages_Rsvd:0
HugePages_Surp:0
Hugepagesize:   2048 kB
DirectMap4k:20220592 kB
DirectMap2M:13238272 kB
DirectMap1G: 1048576 kB

I will write a cronjob that dumps meminfo every 5 minutes to a file, so
I will have more info on the next crash.

The crash is not an isolated one as I already had this crash multiple
times with -rc7 and -rc8. It seems only to occur when copying from
7200rpm harddisks to 5600rpm ones, and never when copying between two
7200rpm or two 5400rpm.

Thanks,
David Arendt

On 12/13/2016 08:55 PM, Xin Zhou wrote:
> Hi David,
>
> It has GFP_NOFS flags, according to definition,
> the issue might have happened during initial DISK/IO.
>
> By the way, did you get a chance to dump the meminfo and run "top" before the 
> system hang?
> It seems more info about the system running state needed to know the issue. 
> Thanks.
>
> Xin
>
>  
>
> Sent: Tuesday, December 13, 2016 at 9:11 AM
> From: "David Arendt" 
> To: linux-btrfs@vger.kernel.org, linux-ker...@vger.kernel.org
> Subject: page allocation stall in kernel 4.9 when copying files from one 
> btrfs hdd to another
> Hi,
>
> I receive the following page allocation stall while copying lots of
> large files from one btrfs hdd to another.
>
> Dec 13 13:04:29 server kernel: kworker/u16:8: page allocation stalls for
> 12260ms, order:0, mode:0x2400840(GFP_NOFS|__GFP_NOFAIL)
> Dec 13 13:04:29 server kernel: CPU: 0 PID: 24959 Comm: kworker/u16:8
> Tainted: P O 4.9.0 #1
> Dec 13 13:04:29 server kernel: Hardware name: ASUS All Series/H87M-PRO,
> BIOS 2102 10/28/2014
> Dec 13 13:04:29 server kernel: Workqueue: btrfs-extent-refs
> btrfs_extent_refs_helper
> Dec 13 13:04:29 server kernel:  813f3a59
> 81976b28 c90011093750
> Dec 13 13:04:29 server kernel: 81114fc1 02400840f39b6bc0
> 81976b28 c900110936f8
> Dec 13 13:04:29 server kernel: 88070010 c90011093760
> c90011093710 02400840
> Dec 13 13:04:29 server kernel: Call Trace:
> Dec 13 13:04:29 server kernel: [] ? dump_stack+0x46/0x5d
> Dec 13 13:04:29 server kernel: [] ?
> warn_alloc+0x111/0x130
> Dec 13 13:04:33 server kernel: [] ?
> __alloc_pages_nodemask+0xbe8/0xd30
> Dec 13 13:04:33 server kernel: [] ?
> pagecache_get_page+0xe4/0x230
> Dec 13 13:04:33 server kernel: [] ?
> alloc_extent_buffer+0x10b/0x400
> Dec 13 13:04:33 server kernel: [] ?
> btrfs_alloc_tree_block+0x125/0x560
> Dec 13 13:04:33 server kernel: [] ?
> read_extent_buffer_pages+0x21f/0x280
> Dec 13 13:04:33 server kernel: [] ?
> __btrfs_cow_block+0x141/0x580
> Dec 13 13:04:33 server kernel: [] ?
> btrfs_cow_block+0x100/0x150
> Dec 13 13:04:33 server kernel: [] ?
> btrfs_search_slot+0x1e9/0x9c0
> Dec 13 13:04:33 server kernel: [] ?
> __set_extent_bit+0x512/0x550
> Dec 13 13:04:33 server kernel: [] ?
> lookup_inline_extent_backref+0xf5/0x5e0
> Dec 13 13:04:34 server kernel: [] ?
> set_extent_bit+0x24/0x30
> Dec 13 13:04:34 server kernel: [] ?
> update_block_group.isra.34+0x114/0x380
> Dec 13 13:04:34 server kernel: [] ?
> __btrfs_free_extent.isra.35+0xf4/0xd20
> Dec 13 13:04:34 server kernel: [] ?
> btrfs_merge_delayed_refs+0x61/0x5d0
> Dec 13 13:04:34 server kernel: [] ?
> __btrfs_run_delayed_refs+0x902/0x10a0
> Dec 13 13:04:34 server kernel: [] ?
> btrfs_run_delayed_refs+0x90/0x2a0
> Dec 13 13:04:34 server kernel: [] ?
> delayed_ref_async_start+0x84/0xa0
> Dec 13 13:04:34 server kernel: [] ?
> process_one_work+0x11d/0x3b0
> Dec 13 13:04:34 server kernel: [] ?
> worker_thread+0x42/0x4b0
> Dec 13 13:04:34 server kernel: [] ?
> process_one_work+0x3b0/0x3b0
> Dec 13 13:04:34 server kernel: [] ?
> process_one_work+0x3b0/0x3b0
> Dec 13 13:04:34 server kernel: [] ?
> do_group_exit+0x2e/0xa0
> Dec 13 13:04:34 server kernel: [] ? kthread+0xb9/0xd0
> Dec 13 13:04:34 server kernel: [] ?
>

Re: page allocation stall in kernel 4.9 when copying files from one btrfs hdd to another

2016-12-13 Thread Xin Zhou 
Hi David,

It has GFP_NOFS flags, according to definition,
the issue might have happened during initial DISK/IO.

By the way, did you get a chance to dump the meminfo and run "top" before the 
system hang?
It seems more info about the system running state needed to know the issue. 
Thanks.

Xin

 

Sent: Tuesday, December 13, 2016 at 9:11 AM
From: "David Arendt" 
To: linux-btrfs@vger.kernel.org, linux-ker...@vger.kernel.org
Subject: page allocation stall in kernel 4.9 when copying files from one btrfs 
hdd to another
Hi,

I receive the following page allocation stall while copying lots of
large files from one btrfs hdd to another.

Dec 13 13:04:29 server kernel: kworker/u16:8: page allocation stalls for
12260ms, order:0, mode:0x2400840(GFP_NOFS|__GFP_NOFAIL)
Dec 13 13:04:29 server kernel: CPU: 0 PID: 24959 Comm: kworker/u16:8
Tainted: P O 4.9.0 #1
Dec 13 13:04:29 server kernel: Hardware name: ASUS All Series/H87M-PRO,
BIOS 2102 10/28/2014
Dec 13 13:04:29 server kernel: Workqueue: btrfs-extent-refs
btrfs_extent_refs_helper
Dec 13 13:04:29 server kernel:  813f3a59
81976b28 c90011093750
Dec 13 13:04:29 server kernel: 81114fc1 02400840f39b6bc0
81976b28 c900110936f8
Dec 13 13:04:29 server kernel: 88070010 c90011093760
c90011093710 02400840
Dec 13 13:04:29 server kernel: Call Trace:
Dec 13 13:04:29 server kernel: [] ? dump_stack+0x46/0x5d
Dec 13 13:04:29 server kernel: [] ?
warn_alloc+0x111/0x130
Dec 13 13:04:33 server kernel: [] ?
__alloc_pages_nodemask+0xbe8/0xd30
Dec 13 13:04:33 server kernel: [] ?
pagecache_get_page+0xe4/0x230
Dec 13 13:04:33 server kernel: [] ?
alloc_extent_buffer+0x10b/0x400
Dec 13 13:04:33 server kernel: [] ?
btrfs_alloc_tree_block+0x125/0x560
Dec 13 13:04:33 server kernel: [] ?
read_extent_buffer_pages+0x21f/0x280
Dec 13 13:04:33 server kernel: [] ?
__btrfs_cow_block+0x141/0x580
Dec 13 13:04:33 server kernel: [] ?
btrfs_cow_block+0x100/0x150
Dec 13 13:04:33 server kernel: [] ?
btrfs_search_slot+0x1e9/0x9c0
Dec 13 13:04:33 server kernel: [] ?
__set_extent_bit+0x512/0x550
Dec 13 13:04:33 server kernel: [] ?
lookup_inline_extent_backref+0xf5/0x5e0
Dec 13 13:04:34 server kernel: [] ?
set_extent_bit+0x24/0x30
Dec 13 13:04:34 server kernel: [] ?
update_block_group.isra.34+0x114/0x380
Dec 13 13:04:34 server kernel: [] ?
__btrfs_free_extent.isra.35+0xf4/0xd20
Dec 13 13:04:34 server kernel: [] ?
btrfs_merge_delayed_refs+0x61/0x5d0
Dec 13 13:04:34 server kernel: [] ?
__btrfs_run_delayed_refs+0x902/0x10a0
Dec 13 13:04:34 server kernel: [] ?
btrfs_run_delayed_refs+0x90/0x2a0
Dec 13 13:04:34 server kernel: [] ?
delayed_ref_async_start+0x84/0xa0
Dec 13 13:04:34 server kernel: [] ?
process_one_work+0x11d/0x3b0
Dec 13 13:04:34 server kernel: [] ?
worker_thread+0x42/0x4b0
Dec 13 13:04:34 server kernel: [] ?
process_one_work+0x3b0/0x3b0
Dec 13 13:04:34 server kernel: [] ?
process_one_work+0x3b0/0x3b0
Dec 13 13:04:34 server kernel: [] ?
do_group_exit+0x2e/0xa0
Dec 13 13:04:34 server kernel: [] ? kthread+0xb9/0xd0
Dec 13 13:04:34 server kernel: [] ?
kthread_park+0x50/0x50
Dec 13 13:04:34 server kernel: [] ?
ret_from_fork+0x22/0x30
Dec 13 13:04:34 server kernel: Mem-Info:
Dec 13 13:04:34 server kernel: active_anon:20 inactive_anon:34
isolated_anon:0\x0a active_file:7370032 inactive_file:450105
isolated_file:320\x0a unevictable:0 dirty:522748 writeback:189
unstable:0\x0a slab_reclaimable:178255 slab_unreclaimable:124617\x0a
mapped:4236 shmem:0 pagetables:1163 bounce:0\x0a free:38224 free_pcp:241
free_cma:0
Dec 13 13:04:34 server kernel: Node 0 active_anon:80kB
inactive_anon:136kB active_file:29480128kB inactive_file:1800420kB
unevictable:0kB isolated(anon):0kB isolated(file):1280kB mapped:16944kB
dirty:2090992kB writeback:756kB shmem:0kB writeback_tmp:0kB unstable:0kB
pages_scanned:258821 all_unreclaimable? no
Dec 13 13:04:34 server kernel: DMA free:15868kB min:8kB low:20kB
high:32kB active_anon:0kB inactive_anon:0kB active_file:0kB
inactive_file:0kB unevictable:0kB writepending:0kB present:15976kB
managed:15892kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:24kB
kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB
free_cma:0kB
Dec 13 13:04:34 server kernel: lowmem_reserve[]: 0 3428 32019 32019
Dec 13 13:04:34 server kernel: DMA32 free:116800kB min:2448kB low:5956kB
high:9464kB active_anon:0kB inactive_anon:0kB active_file:3087928kB
inactive_file:191336kB unevictable:0kB writepending:221828kB
present:3590832kB managed:3513936kB mlocked:0kB slab_reclaimable:93252kB
slab_unreclaimable:20520kB kernel_stack:48kB pagetables:212kB bounce:0kB
free_pcp:4kB local_pcp:0kB free_cma:0kB
Dec 13 13:04:34 server kernel: lowmem_reserve[]: 0 0 0 0
Dec 13 13:04:34 server kernel: DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U)
1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U)
1*2048kB (M) 3*4096kB (M) = 15868kB
Dec 13 13:04:34 server kernel: DMA32: 940*4kB (UME) 4006*8kB (UME)
3308*16kB (UME) 791*32kB