Re: Delayed inode operations not doing the right thing with enospc

2011-07-14 Thread Christian Brunner
2011/7/13 Josef Bacik jo...@redhat.com:
 On 07/12/2011 11:20 AM, Christian Brunner wrote:
 2011/6/7 Josef Bacik jo...@redhat.com:
 On 06/06/2011 09:39 PM, Miao Xie wrote:
 On fri, 03 Jun 2011 14:46:10 -0400, Josef Bacik wrote:
 I got a lot of these when running stress.sh on my test box



 This is because use_block_rsv() is having to do a
 reserve_metadata_bytes(), which shouldn't happen as we should have
 reserved enough space for those operations to complete.  This is
 happening because use_block_rsv() will call get_block_rsv(), which if
 root-ref_cows is set (which is the case on all fs roots) we will use
 trans-block_rsv, which will only have what the current transaction
 starter had reserved.

 What needs to be done instead is we need to have a block reserve that
 any reservation that is done at create time for these inodes is migrated
 to this special reserve, and then when you run the delayed inode items
 stuff you set trans-block_rsv to the special block reserve so the
 accounting is all done properly.

 This is just off the top of my head, there may be a better way to do it,
 I've not actually looked that the delayed inode code at all.

 I would do this myself but I have a ever increasing list of shit to do
 so will somebody pick this up and fix it please?  Thanks,

 Sorry, it's my miss.
 I forgot to set trans-block_rsv to global_block_rsv, since we have 
 migrated
 the space from trans_block_rsv to global_block_rsv.

 I'll fix it soon.


 There is another problem, we're failing xfstest 204.  I tried making
 reserve_metadata_bytes commit the transaction regardless of whether or
 not there were pinned bytes but the test just hung there.  Usually it
 takes 7 seconds to run and I ctrl+c'ed it after a couple of minutes.
 204 just creates a crap ton of files, which is what is killing us.
 There needs to be a way to start flushing delayed inode items so we can
 reclaim the space they are holding onto so we don't get enospc, and it
 needs to be better than just committing the transaction because that is
 dog slow.  Thanks,

 Josef

 Is there a solution for this?

 I'm running a 2.6.38.8 kernel with all the btrfs patches from 3.0rc7
 (except the pluging). When starting a ceph rebuild on the btrfs
 volumes I get a lot of warnings from block_rsv_use_bytes in
 use_block_rsv:


 Ok I think I've got this nailed down.  Will you run with this patch and make 
 sure the warnings go away?  Thanks,

I'm sorry, I'm still getting a lot of warnings like the one below.

I've also noticed, that I'm not getting these messages when the
free_space_cache is disabled.

Christian

[  697.398097] [ cut here ]
[  697.398109] WARNING: at fs/btrfs/extent-tree.c:5693
btrfs_alloc_free_block+0x1f8/0x360 [btrfs]()
[  697.398111] Hardware name: ProLiant DL180 G6
[  697.398112] Modules linked in: btrfs zlib_deflate libcrc32c bonding
ipv6 serio_raw pcspkr ghes hed iTCO_wdt iTCO_vendor_support
i7core_edac edac_core ixgbe dca mdio iomemory_vsl(P) hpsa squashfs
usb_storage [last unloaded: scsi_wait_scan]
[  697.398122] Pid: 6591, comm: btrfs-freespace Tainted: PW
3.0.0-1.fits.1.el6.x86_64 #1
[  697.398124] Call Trace:
[  697.398128]  [810630af] warn_slowpath_common+0x7f/0xc0
[  697.398131]  [8106310a] warn_slowpath_null+0x1a/0x20
[  697.398142]  [a022cb88] btrfs_alloc_free_block+0x1f8/0x360 [btrfs]
[  697.398156]  [a025ae08] ? read_extent_buffer+0xd8/0x1d0 [btrfs]
[  697.398316]  [a021d112] split_leaf+0x142/0x8c0 [btrfs]
[  697.398325]  [a021629b] ? generic_bin_search+0x19b/0x210 [btrfs]
[  697.398334]  [a0218a1a] ? btrfs_leaf_free_space+0x8a/0xe0 [btrfs]
[  697.398344]  [a021df63] btrfs_search_slot+0x6d3/0x7a0 [btrfs]
[  697.398355]  [a0230942] btrfs_csum_file_blocks+0x632/0x830 [btrfs]
[  697.398369]  [a025c03a] ? clear_extent_bit+0x17a/0x440 [btrfs]
[  697.398382]  [a023c009] add_pending_csums+0x49/0x70 [btrfs]
[  697.398395]  [a023ef5d] btrfs_finish_ordered_io+0x22d/0x360 [btrfs]
[  697.398408]  [a023f0dc]
btrfs_writepage_end_io_hook+0x4c/0xa0 [btrfs]
[  697.398422]  [a025c4fb]
end_bio_extent_writepage+0x13b/0x180 [btrfs]
[  697.398425]  [81558b5b] ? schedule_timeout+0x17b/0x2e0
[  697.398436]  [a02336d9] ? end_workqueue_fn+0xe9/0x130 [btrfs]
[  697.398439]  [8118f24d] bio_endio+0x1d/0x40
[  697.398451]  [a02336e4] end_workqueue_fn+0xf4/0x130 [btrfs]
[  697.398464]  [a02671de] worker_loop+0x13e/0x540 [btrfs]
[  697.398477]  [a02670a0] ? btrfs_queue_worker+0x2d0/0x2d0 [btrfs]
[  697.398490]  [a02670a0] ? btrfs_queue_worker+0x2d0/0x2d0 [btrfs]
[  697.398493]  [81085896] kthread+0x96/0xa0
[  697.398496]  [81563844] kernel_thread_helper+0x4/0x10
[  697.398499]  [81085800] ? kthread_worker_fn+0x1a0/0x1a0
[  697.398502]  [81563840] ? gs_change+0x13/0x13
[  697.398503] ---[ end trace 8c77269b0de3f0fb ]---
[  

Re: Delayed inode operations not doing the right thing with enospc

2011-07-14 Thread Josef Bacik
On 07/14/2011 03:27 AM, Christian Brunner wrote:
 2011/7/13 Josef Bacik jo...@redhat.com:
 On 07/12/2011 11:20 AM, Christian Brunner wrote:
 2011/6/7 Josef Bacik jo...@redhat.com:
 On 06/06/2011 09:39 PM, Miao Xie wrote:
 On fri, 03 Jun 2011 14:46:10 -0400, Josef Bacik wrote:
 I got a lot of these when running stress.sh on my test box



 This is because use_block_rsv() is having to do a
 reserve_metadata_bytes(), which shouldn't happen as we should have
 reserved enough space for those operations to complete.  This is
 happening because use_block_rsv() will call get_block_rsv(), which if
 root-ref_cows is set (which is the case on all fs roots) we will use
 trans-block_rsv, which will only have what the current transaction
 starter had reserved.

 What needs to be done instead is we need to have a block reserve that
 any reservation that is done at create time for these inodes is migrated
 to this special reserve, and then when you run the delayed inode items
 stuff you set trans-block_rsv to the special block reserve so the
 accounting is all done properly.

 This is just off the top of my head, there may be a better way to do it,
 I've not actually looked that the delayed inode code at all.

 I would do this myself but I have a ever increasing list of shit to do
 so will somebody pick this up and fix it please?  Thanks,

 Sorry, it's my miss.
 I forgot to set trans-block_rsv to global_block_rsv, since we have 
 migrated
 the space from trans_block_rsv to global_block_rsv.

 I'll fix it soon.


 There is another problem, we're failing xfstest 204.  I tried making
 reserve_metadata_bytes commit the transaction regardless of whether or
 not there were pinned bytes but the test just hung there.  Usually it
 takes 7 seconds to run and I ctrl+c'ed it after a couple of minutes.
 204 just creates a crap ton of files, which is what is killing us.
 There needs to be a way to start flushing delayed inode items so we can
 reclaim the space they are holding onto so we don't get enospc, and it
 needs to be better than just committing the transaction because that is
 dog slow.  Thanks,

 Josef

 Is there a solution for this?

 I'm running a 2.6.38.8 kernel with all the btrfs patches from 3.0rc7
 (except the pluging). When starting a ceph rebuild on the btrfs
 volumes I get a lot of warnings from block_rsv_use_bytes in
 use_block_rsv:


 Ok I think I've got this nailed down.  Will you run with this patch and make 
 sure the warnings go away?  Thanks,
 
 I'm sorry, I'm still getting a lot of warnings like the one below.
 
 I've also noticed, that I'm not getting these messages when the
 free_space_cache is disabled.
 


Ok I see what's wrong, our checksum calculation is completely bogus.
I'm in the middle of something big so I can't give you a nice clean
patch, so if you can just go into extent-tree.c and replace
calc_csum_metadata_size with this you should be good to go

static u64 calc_csum_metadata_size(struct inode *inode, u64 num_bytes)
{
struct btrfs_root *root = BTRFS_I(inode)-root;
int num_leaves;
int num_csums;
u16 csum_size =
btrfs_super_csum_size(root-fs_info-super_copy);

num_csums = (int)div64_u64(num_bytes, root-sectorsize);
num_leaves = (int)((num_csums * csum_size) / root-leafsize);

return btrfs_calc_trans_metadata_size(root, num_leaves);
}


Thanks,

Josef
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: Delayed inode operations not doing the right thing with enospc

2011-07-14 Thread Josef Bacik
On 07/14/2011 03:27 AM, Christian Brunner wrote:
 2011/7/13 Josef Bacik jo...@redhat.com:
 On 07/12/2011 11:20 AM, Christian Brunner wrote:
 2011/6/7 Josef Bacik jo...@redhat.com:
 On 06/06/2011 09:39 PM, Miao Xie wrote:
 On fri, 03 Jun 2011 14:46:10 -0400, Josef Bacik wrote:
 I got a lot of these when running stress.sh on my test box



 This is because use_block_rsv() is having to do a
 reserve_metadata_bytes(), which shouldn't happen as we should have
 reserved enough space for those operations to complete.  This is
 happening because use_block_rsv() will call get_block_rsv(), which if
 root-ref_cows is set (which is the case on all fs roots) we will use
 trans-block_rsv, which will only have what the current transaction
 starter had reserved.

 What needs to be done instead is we need to have a block reserve that
 any reservation that is done at create time for these inodes is migrated
 to this special reserve, and then when you run the delayed inode items
 stuff you set trans-block_rsv to the special block reserve so the
 accounting is all done properly.

 This is just off the top of my head, there may be a better way to do it,
 I've not actually looked that the delayed inode code at all.

 I would do this myself but I have a ever increasing list of shit to do
 so will somebody pick this up and fix it please?  Thanks,

 Sorry, it's my miss.
 I forgot to set trans-block_rsv to global_block_rsv, since we have 
 migrated
 the space from trans_block_rsv to global_block_rsv.

 I'll fix it soon.


 There is another problem, we're failing xfstest 204.  I tried making
 reserve_metadata_bytes commit the transaction regardless of whether or
 not there were pinned bytes but the test just hung there.  Usually it
 takes 7 seconds to run and I ctrl+c'ed it after a couple of minutes.
 204 just creates a crap ton of files, which is what is killing us.
 There needs to be a way to start flushing delayed inode items so we can
 reclaim the space they are holding onto so we don't get enospc, and it
 needs to be better than just committing the transaction because that is
 dog slow.  Thanks,

 Josef

 Is there a solution for this?

 I'm running a 2.6.38.8 kernel with all the btrfs patches from 3.0rc7
 (except the pluging). When starting a ceph rebuild on the btrfs
 volumes I get a lot of warnings from block_rsv_use_bytes in
 use_block_rsv:


 Ok I think I've got this nailed down.  Will you run with this patch and make 
 sure the warnings go away?  Thanks,
 
 I'm sorry, I'm still getting a lot of warnings like the one below.
 
 I've also noticed, that I'm not getting these messages when the
 free_space_cache is disabled.
 

Ok ditch that previous patch and try this one, it should work.  Thanks,

Josef


diff --git a/fs/btrfs/btrfs_inode.h b/fs/btrfs/btrfs_inode.h
index 52d7eca..2263d29 100644
--- a/fs/btrfs/btrfs_inode.h
+++ b/fs/btrfs/btrfs_inode.h
@@ -112,9 +112,6 @@ struct btrfs_inode {
 */
u64 disk_i_size;

-   /* flags field from the on disk inode */
-   u32 flags;
-
/*
 * if this is a directory then index_cnt is the counter for the index
 * number for new files that are created
@@ -128,14 +125,8 @@ struct btrfs_inode {
 */
u64 last_unlink_trans;

-   /*
-* Counters to keep track of the number of extent item's we may use due
-* to delalloc and such.  outstanding_extents is the number of extent
-* items we think we'll end up using, and reserved_extents is the number
-* of extent items we've reserved metadata for.
-*/
-   atomic_t outstanding_extents;
-   atomic_t reserved_extents;
+   /* flags field from the on disk inode */
+   u32 flags;

/*
 * ordered_data_close is set by truncate when a file that used
@@ -151,12 +142,21 @@ struct btrfs_inode {
unsigned orphan_meta_reserved:1;
unsigned dummy_inode:1;
unsigned in_defrag:1;
-
/*
 * always compress this one file
 */
unsigned force_compress:4;

+   /*
+* Counters to keep track of the number of extent item's we may use due
+* to delalloc and such.  outstanding_extents is the number of extent
+* items we think we'll end up using, and reserved_extents is the number
+* of extent items we've reserved metadata for.
+*/
+   spinlock_t extents_count_lock;
+   unsigned outstanding_extents;
+   unsigned reserved_extents;
+
struct btrfs_delayed_node *delayed_node;

struct inode vfs_inode;
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index be02cae..3ba4d5f 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -2133,7 +2133,7 @@ static inline bool btrfs_mixed_space_info(struct
btrfs_space_info *space_info)

 /* extent-tree.c */
 static inline u64 btrfs_calc_trans_metadata_size(struct btrfs_root *root,
-int num_items)
+

Re: Delayed inode operations not doing the right thing with enospc

2011-07-13 Thread Josef Bacik
On 07/12/2011 11:20 AM, Christian Brunner wrote:
 2011/6/7 Josef Bacik jo...@redhat.com:
 On 06/06/2011 09:39 PM, Miao Xie wrote:
 On fri, 03 Jun 2011 14:46:10 -0400, Josef Bacik wrote:
 I got a lot of these when running stress.sh on my test box



 This is because use_block_rsv() is having to do a
 reserve_metadata_bytes(), which shouldn't happen as we should have
 reserved enough space for those operations to complete.  This is
 happening because use_block_rsv() will call get_block_rsv(), which if
 root-ref_cows is set (which is the case on all fs roots) we will use
 trans-block_rsv, which will only have what the current transaction
 starter had reserved.

 What needs to be done instead is we need to have a block reserve that
 any reservation that is done at create time for these inodes is migrated
 to this special reserve, and then when you run the delayed inode items
 stuff you set trans-block_rsv to the special block reserve so the
 accounting is all done properly.

 This is just off the top of my head, there may be a better way to do it,
 I've not actually looked that the delayed inode code at all.

 I would do this myself but I have a ever increasing list of shit to do
 so will somebody pick this up and fix it please?  Thanks,

 Sorry, it's my miss.
 I forgot to set trans-block_rsv to global_block_rsv, since we have migrated
 the space from trans_block_rsv to global_block_rsv.

 I'll fix it soon.


 There is another problem, we're failing xfstest 204.  I tried making
 reserve_metadata_bytes commit the transaction regardless of whether or
 not there were pinned bytes but the test just hung there.  Usually it
 takes 7 seconds to run and I ctrl+c'ed it after a couple of minutes.
 204 just creates a crap ton of files, which is what is killing us.
 There needs to be a way to start flushing delayed inode items so we can
 reclaim the space they are holding onto so we don't get enospc, and it
 needs to be better than just committing the transaction because that is
 dog slow.  Thanks,

 Josef
 
 Is there a solution for this?
 
 I'm running a 2.6.38.8 kernel with all the btrfs patches from 3.0rc7
 (except the pluging). When starting a ceph rebuild on the btrfs
 volumes I get a lot of warnings from block_rsv_use_bytes in
 use_block_rsv:
 

Ok I think I've got this nailed down.  Will you run with this patch and make 
sure the warnings go away?  Thanks,

Josef

diff --git a/fs/btrfs/btrfs_inode.h b/fs/btrfs/btrfs_inode.h
index 52d7eca..2263d29 100644
--- a/fs/btrfs/btrfs_inode.h
+++ b/fs/btrfs/btrfs_inode.h
@@ -112,9 +112,6 @@ struct btrfs_inode {
 */
u64 disk_i_size;
 
-   /* flags field from the on disk inode */
-   u32 flags;
-
/*
 * if this is a directory then index_cnt is the counter for the index
 * number for new files that are created
@@ -128,14 +125,8 @@ struct btrfs_inode {
 */
u64 last_unlink_trans;
 
-   /*
-* Counters to keep track of the number of extent item's we may use due
-* to delalloc and such.  outstanding_extents is the number of extent
-* items we think we'll end up using, and reserved_extents is the number
-* of extent items we've reserved metadata for.
-*/
-   atomic_t outstanding_extents;
-   atomic_t reserved_extents;
+   /* flags field from the on disk inode */
+   u32 flags;
 
/*
 * ordered_data_close is set by truncate when a file that used
@@ -151,12 +142,21 @@ struct btrfs_inode {
unsigned orphan_meta_reserved:1;
unsigned dummy_inode:1;
unsigned in_defrag:1;
-
/*
 * always compress this one file
 */
unsigned force_compress:4;
 
+   /*
+* Counters to keep track of the number of extent item's we may use due
+* to delalloc and such.  outstanding_extents is the number of extent
+* items we think we'll end up using, and reserved_extents is the number
+* of extent items we've reserved metadata for.
+*/
+   spinlock_t extents_count_lock;
+   unsigned outstanding_extents;
+   unsigned reserved_extents;
+
struct btrfs_delayed_node *delayed_node;
 
struct inode vfs_inode;
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index be02cae..3ba4d5f 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -2133,7 +2133,7 @@ static inline bool btrfs_mixed_space_info(struct 
btrfs_space_info *space_info)
 
 /* extent-tree.c */
 static inline u64 btrfs_calc_trans_metadata_size(struct btrfs_root *root,
-int num_items)
+unsigned num_items)
 {
return (root-leafsize + root-nodesize * (BTRFS_MAX_LEVEL - 1)) *
3 * num_items;
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 3e52b85..65a721c 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -3952,13 +3952,35 @@ static u64 

Re: Delayed inode operations not doing the right thing with enospc

2011-07-12 Thread Christian Brunner
2011/6/7 Josef Bacik jo...@redhat.com:
 On 06/06/2011 09:39 PM, Miao Xie wrote:
 On fri, 03 Jun 2011 14:46:10 -0400, Josef Bacik wrote:
 I got a lot of these when running stress.sh on my test box



 This is because use_block_rsv() is having to do a
 reserve_metadata_bytes(), which shouldn't happen as we should have
 reserved enough space for those operations to complete.  This is
 happening because use_block_rsv() will call get_block_rsv(), which if
 root-ref_cows is set (which is the case on all fs roots) we will use
 trans-block_rsv, which will only have what the current transaction
 starter had reserved.

 What needs to be done instead is we need to have a block reserve that
 any reservation that is done at create time for these inodes is migrated
 to this special reserve, and then when you run the delayed inode items
 stuff you set trans-block_rsv to the special block reserve so the
 accounting is all done properly.

 This is just off the top of my head, there may be a better way to do it,
 I've not actually looked that the delayed inode code at all.

 I would do this myself but I have a ever increasing list of shit to do
 so will somebody pick this up and fix it please?  Thanks,

 Sorry, it's my miss.
 I forgot to set trans-block_rsv to global_block_rsv, since we have migrated
 the space from trans_block_rsv to global_block_rsv.

 I'll fix it soon.


 There is another problem, we're failing xfstest 204.  I tried making
 reserve_metadata_bytes commit the transaction regardless of whether or
 not there were pinned bytes but the test just hung there.  Usually it
 takes 7 seconds to run and I ctrl+c'ed it after a couple of minutes.
 204 just creates a crap ton of files, which is what is killing us.
 There needs to be a way to start flushing delayed inode items so we can
 reclaim the space they are holding onto so we don't get enospc, and it
 needs to be better than just committing the transaction because that is
 dog slow.  Thanks,

 Josef

Is there a solution for this?

I'm running a 2.6.38.8 kernel with all the btrfs patches from 3.0rc7
(except the pluging). When starting a ceph rebuild on the btrfs
volumes I get a lot of warnings from block_rsv_use_bytes in
use_block_rsv:

[ 2157.922054] [ cut here ]
[ 2157.927270] WARNING: at fs/btrfs/extent-tree.c:5683
btrfs_alloc_free_block+0x1f8/0x360 [btrfs]()
[ 2157.937123] Hardware name: ProLiant DL180 G6
[ 2157.942132] Modules linked in: btrfs zlib_deflate libcrc32c bonding
ipv6 pcspkr serio_raw iTCO_wdt iTCO_vendor_support ghes hed
i7core_edac edac_core ixgbe dca mdio iomemory_vsl(P) hpsa squashfs
usb_storage [last unloaded: scsi_wait_scan]
[ 2157.967386] Pid: 10280, comm: btrfs-freespace Tainted: PW
2.6.38.8-1.fits.4.el6.x86_64 #1
[ 2157.977554] Call Trace:
[ 2157.980383]  [8106482f] ? warn_slowpath_common+0x7f/0xc0
[ 2157.987382]  [8106488a] ? warn_slowpath_null+0x1a/0x20
[ 2157.994192]  [a0240b88] ?
btrfs_alloc_free_block+0x1f8/0x360 [btrfs]
[ 2158.002354]  [a026eda8] ? read_extent_buffer+0xd8/0x1d0 [btrfs]
[ 2158.010014]  [a0231132] ? split_leaf+0x142/0x8c0 [btrfs]
[ 2158.016990]  [a022a29b] ? generic_bin_search+0x19b/0x210 [btrfs]
[ 2158.024784]  [a022ca1a] ? btrfs_leaf_free_space+0x8a/0xe0 [btrfs]
[ 2158.032627]  [a0231f83] ? btrfs_search_slot+0x6d3/0x7a0 [btrfs]
[ 2158.040325]  [a0244942] ?
btrfs_csum_file_blocks+0x632/0x830 [btrfs]
[ 2158.048477]  [a026ffda] ? clear_extent_bit+0x17a/0x440 [btrfs]
[ 2158.056026]  [a024ffc5] ? add_pending_csums+0x45/0x70 [btrfs]
[ 2158.063530]  [a0252dad] ?
btrfs_finish_ordered_io+0x22d/0x360 [btrfs]
[ 2158.071755]  [a0252f2c] ?
btrfs_writepage_end_io_hook+0x4c/0xa0 [btrfs]
[ 2158.080172]  [a027049b] ?
end_bio_extent_writepage+0x13b/0x180 [btrfs]
[ 2158.088505]  [815406fb] ? schedule_timeout+0x17b/0x2e0
[ 2158.095258]  [8118964d] ? bio_endio+0x1d/0x40
[ 2158.101171]  [a0247764] ? end_workqueue_fn+0xf4/0x130 [btrfs]
[ 2158.108621]  [a027b30e] ? worker_loop+0x13e/0x540 [btrfs]
[ 2158.115703]  [a027b1d0] ? worker_loop+0x0/0x540 [btrfs]
[ 2158.122563]  [a027b1d0] ? worker_loop+0x0/0x540 [btrfs]
[ 2158.129413]  [81086356] ? kthread+0x96/0xa0
[ 2158.135093]  [8100ce44] ? kernel_thread_helper+0x4/0x10
[ 2158.141913]  [810862c0] ? kthread+0x0/0xa0
[ 2158.147467]  [8100ce40] ? kernel_thread_helper+0x0/0x10
[ 2158.154287] ---[ end trace 55e53c726a04ecd7 ]---

Thanks,
Christian
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: Delayed inode operations not doing the right thing with enospc

2011-06-07 Thread Josef Bacik

On 06/06/2011 09:39 PM, Miao Xie wrote:

On fri, 03 Jun 2011 14:46:10 -0400, Josef Bacik wrote:

I got a lot of these when running stress.sh on my test box

[ 9792.654889] [ cut here ]
[ 9792.654898] WARNING: at fs/btrfs/extent-tree.c:5681
btrfs_alloc_free_block+0xca/0x27c [btrfs]()
[ 9792.654899] Hardware name: To Be Filled By O.E.M.
[ 9792.654900] Modules linked in: btrfs zlib_deflate libcrc32c
ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables
arc4 rt61pci rt2x00pci rt2x00lib snd_hda_codec_hdmi mac80211
snd_hda_codec_realtek cfg80211 snd_hda_intel edac_core snd_seq rfkill
pcspkr serio_raw snd_hda_codec eeprom_93cx6 edac_mce_amd sp5100_tco
i2c_piix4 k10temp snd_hwdep snd_seq_device snd_pcm floppy r8169 xhci_hcd
mii snd_timer snd soundcore snd_page_alloc ipv6 firewire_ohci pata_acpi
ata_generic firewire_core pata_via crc_itu_t radeon ttm drm_kms_helper
drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan]
[ 9792.654919] Pid: 2762, comm: rm Tainted: GW   2.6.39+ #1
[ 9792.654920] Call Trace:
[ 9792.654922]  [81053c4a] warn_slowpath_common+0x83/0x9b
[ 9792.654925]  [81053c7c] warn_slowpath_null+0x1a/0x1c
[ 9792.654933]  [a038e747] btrfs_alloc_free_block+0xca/0x27c
[btrfs]
[ 9792.654945]  [a03b8562] ? map_extent_buffer+0x6e/0xa8 [btrfs]
[ 9792.654953]  [a038189b] __btrfs_cow_block+0xfc/0x30c [btrfs]
[ 9792.654963]  [a0396aa6] ? btrfs_buffer_uptodate+0x47/0x58
[btrfs]
[ 9792.654970]  [a0382e48] ? read_block_for_search+0x94/0x368
[btrfs]
[ 9792.654978]  [a0381ba9] btrfs_cow_block+0xfe/0x146 [btrfs]
[ 9792.654986]  [a03848b0] btrfs_search_slot+0x14d/0x4b6 [btrfs]
[ 9792.654997]  [a03b8562] ? map_extent_buffer+0x6e/0xa8 [btrfs]
[ 9792.655022]  [a03938e8] btrfs_lookup_inode+0x2f/0x8f [btrfs]
[ 9792.655025]  [8147afac] ? _cond_resched+0xe/0x22
[ 9792.655027]  [8147b892] ? mutex_lock+0x29/0x50
[ 9792.655039]  [a03d41b1]
btrfs_update_delayed_inode+0x72/0x137 [btrfs]
[ 9792.655051]  [a03d4ea2] btrfs_run_delayed_items+0x90/0xdb
[btrfs]
[ 9792.655062]  [a039a69b]
btrfs_commit_transaction+0x228/0x654 [btrfs]
[ 9792.655064]  [8106e8da] ? remove_wait_queue+0x3a/0x3a
[ 9792.655075]  [a03a2fa5] btrfs_evict_inode+0x14d/0x202 [btrfs]
[ 9792.655077]  [81132bd6] evict+0x71/0x111
[ 9792.655079]  [81132de0] iput+0x12a/0x132
[ 9792.655081]  [8112aa3a] do_unlinkat+0x106/0x155
[ 9792.655083]  [81127b83] ? path_put+0x1f/0x23
[ 9792.655085]  [8109c53c] ? audit_syscall_entry+0x145/0x171
[ 9792.655087]  [81128410] ? putname+0x34/0x36
[ 9792.655090]  [8112b441] sys_unlinkat+0x29/0x2b
[ 9792.655092]  [81482c42] system_call_fastpath+0x16/0x1b
[ 9792.655093] ---[ end trace 02b696eb02b3f768 ]---


This is because use_block_rsv() is having to do a
reserve_metadata_bytes(), which shouldn't happen as we should have
reserved enough space for those operations to complete.  This is
happening because use_block_rsv() will call get_block_rsv(), which if
root-ref_cows is set (which is the case on all fs roots) we will use
trans-block_rsv, which will only have what the current transaction
starter had reserved.

What needs to be done instead is we need to have a block reserve that
any reservation that is done at create time for these inodes is migrated
to this special reserve, and then when you run the delayed inode items
stuff you set trans-block_rsv to the special block reserve so the
accounting is all done properly.

This is just off the top of my head, there may be a better way to do it,
I've not actually looked that the delayed inode code at all.

I would do this myself but I have a ever increasing list of shit to do
so will somebody pick this up and fix it please?  Thanks,


Sorry, it's my miss.
I forgot to set trans-block_rsv to global_block_rsv, since we have migrated
the space from trans_block_rsv to global_block_rsv.

I'll fix it soon.



Great thanks Miao,

Josef
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: Delayed inode operations not doing the right thing with enospc

2011-06-07 Thread Josef Bacik
On 06/06/2011 09:39 PM, Miao Xie wrote:
 On fri, 03 Jun 2011 14:46:10 -0400, Josef Bacik wrote:
 I got a lot of these when running stress.sh on my test box

 [ 9792.654889] [ cut here ]
 [ 9792.654898] WARNING: at fs/btrfs/extent-tree.c:5681
 btrfs_alloc_free_block+0xca/0x27c [btrfs]()
 [ 9792.654899] Hardware name: To Be Filled By O.E.M.
 [ 9792.654900] Modules linked in: btrfs zlib_deflate libcrc32c
 ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables
 arc4 rt61pci rt2x00pci rt2x00lib snd_hda_codec_hdmi mac80211
 snd_hda_codec_realtek cfg80211 snd_hda_intel edac_core snd_seq rfkill
 pcspkr serio_raw snd_hda_codec eeprom_93cx6 edac_mce_amd sp5100_tco
 i2c_piix4 k10temp snd_hwdep snd_seq_device snd_pcm floppy r8169 xhci_hcd
 mii snd_timer snd soundcore snd_page_alloc ipv6 firewire_ohci pata_acpi
 ata_generic firewire_core pata_via crc_itu_t radeon ttm drm_kms_helper
 drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan]
 [ 9792.654919] Pid: 2762, comm: rm Tainted: GW   2.6.39+ #1
 [ 9792.654920] Call Trace:
 [ 9792.654922]  [81053c4a] warn_slowpath_common+0x83/0x9b
 [ 9792.654925]  [81053c7c] warn_slowpath_null+0x1a/0x1c
 [ 9792.654933]  [a038e747] btrfs_alloc_free_block+0xca/0x27c
 [btrfs]
 [ 9792.654945]  [a03b8562] ? map_extent_buffer+0x6e/0xa8 [btrfs]
 [ 9792.654953]  [a038189b] __btrfs_cow_block+0xfc/0x30c [btrfs]
 [ 9792.654963]  [a0396aa6] ? btrfs_buffer_uptodate+0x47/0x58
 [btrfs]
 [ 9792.654970]  [a0382e48] ? read_block_for_search+0x94/0x368
 [btrfs]
 [ 9792.654978]  [a0381ba9] btrfs_cow_block+0xfe/0x146 [btrfs]
 [ 9792.654986]  [a03848b0] btrfs_search_slot+0x14d/0x4b6 [btrfs]
 [ 9792.654997]  [a03b8562] ? map_extent_buffer+0x6e/0xa8 [btrfs]
 [ 9792.655022]  [a03938e8] btrfs_lookup_inode+0x2f/0x8f [btrfs]
 [ 9792.655025]  [8147afac] ? _cond_resched+0xe/0x22
 [ 9792.655027]  [8147b892] ? mutex_lock+0x29/0x50
 [ 9792.655039]  [a03d41b1]
 btrfs_update_delayed_inode+0x72/0x137 [btrfs]
 [ 9792.655051]  [a03d4ea2] btrfs_run_delayed_items+0x90/0xdb
 [btrfs]
 [ 9792.655062]  [a039a69b]
 btrfs_commit_transaction+0x228/0x654 [btrfs]
 [ 9792.655064]  [8106e8da] ? remove_wait_queue+0x3a/0x3a
 [ 9792.655075]  [a03a2fa5] btrfs_evict_inode+0x14d/0x202 [btrfs]
 [ 9792.655077]  [81132bd6] evict+0x71/0x111
 [ 9792.655079]  [81132de0] iput+0x12a/0x132
 [ 9792.655081]  [8112aa3a] do_unlinkat+0x106/0x155
 [ 9792.655083]  [81127b83] ? path_put+0x1f/0x23
 [ 9792.655085]  [8109c53c] ? audit_syscall_entry+0x145/0x171
 [ 9792.655087]  [81128410] ? putname+0x34/0x36
 [ 9792.655090]  [8112b441] sys_unlinkat+0x29/0x2b
 [ 9792.655092]  [81482c42] system_call_fastpath+0x16/0x1b
 [ 9792.655093] ---[ end trace 02b696eb02b3f768 ]---


 This is because use_block_rsv() is having to do a
 reserve_metadata_bytes(), which shouldn't happen as we should have
 reserved enough space for those operations to complete.  This is
 happening because use_block_rsv() will call get_block_rsv(), which if
 root-ref_cows is set (which is the case on all fs roots) we will use
 trans-block_rsv, which will only have what the current transaction
 starter had reserved.

 What needs to be done instead is we need to have a block reserve that
 any reservation that is done at create time for these inodes is migrated
 to this special reserve, and then when you run the delayed inode items
 stuff you set trans-block_rsv to the special block reserve so the
 accounting is all done properly.

 This is just off the top of my head, there may be a better way to do it,
 I've not actually looked that the delayed inode code at all.

 I would do this myself but I have a ever increasing list of shit to do
 so will somebody pick this up and fix it please?  Thanks,
 
 Sorry, it's my miss.
 I forgot to set trans-block_rsv to global_block_rsv, since we have migrated
 the space from trans_block_rsv to global_block_rsv.
 
 I'll fix it soon.
 

There is another problem, we're failing xfstest 204.  I tried making
reserve_metadata_bytes commit the transaction regardless of whether or
not there were pinned bytes but the test just hung there.  Usually it
takes 7 seconds to run and I ctrl+c'ed it after a couple of minutes.
204 just creates a crap ton of files, which is what is killing us.
There needs to be a way to start flushing delayed inode items so we can
reclaim the space they are holding onto so we don't get enospc, and it
needs to be better than just committing the transaction because that is
dog slow.  Thanks,

Josef
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: Delayed inode operations not doing the right thing with enospc

2011-06-06 Thread Miao Xie
On fri, 03 Jun 2011 14:46:10 -0400, Josef Bacik wrote:
 I got a lot of these when running stress.sh on my test box
 
 [ 9792.654889] [ cut here ]
 [ 9792.654898] WARNING: at fs/btrfs/extent-tree.c:5681
 btrfs_alloc_free_block+0xca/0x27c [btrfs]()
 [ 9792.654899] Hardware name: To Be Filled By O.E.M.
 [ 9792.654900] Modules linked in: btrfs zlib_deflate libcrc32c
 ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables
 arc4 rt61pci rt2x00pci rt2x00lib snd_hda_codec_hdmi mac80211
 snd_hda_codec_realtek cfg80211 snd_hda_intel edac_core snd_seq rfkill
 pcspkr serio_raw snd_hda_codec eeprom_93cx6 edac_mce_amd sp5100_tco
 i2c_piix4 k10temp snd_hwdep snd_seq_device snd_pcm floppy r8169 xhci_hcd
 mii snd_timer snd soundcore snd_page_alloc ipv6 firewire_ohci pata_acpi
 ata_generic firewire_core pata_via crc_itu_t radeon ttm drm_kms_helper
 drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan]
 [ 9792.654919] Pid: 2762, comm: rm Tainted: GW   2.6.39+ #1
 [ 9792.654920] Call Trace:
 [ 9792.654922]  [81053c4a] warn_slowpath_common+0x83/0x9b
 [ 9792.654925]  [81053c7c] warn_slowpath_null+0x1a/0x1c
 [ 9792.654933]  [a038e747] btrfs_alloc_free_block+0xca/0x27c
 [btrfs]
 [ 9792.654945]  [a03b8562] ? map_extent_buffer+0x6e/0xa8 [btrfs]
 [ 9792.654953]  [a038189b] __btrfs_cow_block+0xfc/0x30c [btrfs]
 [ 9792.654963]  [a0396aa6] ? btrfs_buffer_uptodate+0x47/0x58
 [btrfs]
 [ 9792.654970]  [a0382e48] ? read_block_for_search+0x94/0x368
 [btrfs]
 [ 9792.654978]  [a0381ba9] btrfs_cow_block+0xfe/0x146 [btrfs]
 [ 9792.654986]  [a03848b0] btrfs_search_slot+0x14d/0x4b6 [btrfs]
 [ 9792.654997]  [a03b8562] ? map_extent_buffer+0x6e/0xa8 [btrfs]
 [ 9792.655022]  [a03938e8] btrfs_lookup_inode+0x2f/0x8f [btrfs]
 [ 9792.655025]  [8147afac] ? _cond_resched+0xe/0x22
 [ 9792.655027]  [8147b892] ? mutex_lock+0x29/0x50
 [ 9792.655039]  [a03d41b1]
 btrfs_update_delayed_inode+0x72/0x137 [btrfs]
 [ 9792.655051]  [a03d4ea2] btrfs_run_delayed_items+0x90/0xdb
 [btrfs]
 [ 9792.655062]  [a039a69b]
 btrfs_commit_transaction+0x228/0x654 [btrfs]
 [ 9792.655064]  [8106e8da] ? remove_wait_queue+0x3a/0x3a
 [ 9792.655075]  [a03a2fa5] btrfs_evict_inode+0x14d/0x202 [btrfs]
 [ 9792.655077]  [81132bd6] evict+0x71/0x111
 [ 9792.655079]  [81132de0] iput+0x12a/0x132
 [ 9792.655081]  [8112aa3a] do_unlinkat+0x106/0x155
 [ 9792.655083]  [81127b83] ? path_put+0x1f/0x23
 [ 9792.655085]  [8109c53c] ? audit_syscall_entry+0x145/0x171
 [ 9792.655087]  [81128410] ? putname+0x34/0x36
 [ 9792.655090]  [8112b441] sys_unlinkat+0x29/0x2b
 [ 9792.655092]  [81482c42] system_call_fastpath+0x16/0x1b
 [ 9792.655093] ---[ end trace 02b696eb02b3f768 ]---
 
 
 This is because use_block_rsv() is having to do a
 reserve_metadata_bytes(), which shouldn't happen as we should have
 reserved enough space for those operations to complete.  This is
 happening because use_block_rsv() will call get_block_rsv(), which if
 root-ref_cows is set (which is the case on all fs roots) we will use
 trans-block_rsv, which will only have what the current transaction
 starter had reserved.
 
 What needs to be done instead is we need to have a block reserve that
 any reservation that is done at create time for these inodes is migrated
 to this special reserve, and then when you run the delayed inode items
 stuff you set trans-block_rsv to the special block reserve so the
 accounting is all done properly.
 
 This is just off the top of my head, there may be a better way to do it,
 I've not actually looked that the delayed inode code at all.
 
 I would do this myself but I have a ever increasing list of shit to do
 so will somebody pick this up and fix it please?  Thanks,

Sorry, it's my miss.
I forgot to set trans-block_rsv to global_block_rsv, since we have migrated
the space from trans_block_rsv to global_block_rsv.

I'll fix it soon.

Thanks for your help.
Miao

 
 Josef
 --
 To unsubscribe from this list: send the line unsubscribe linux-btrfs in
 the body of a message to majord...@vger.kernel.org
 More majordomo info at  http://vger.kernel.org/majordomo-info.html
 

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html