Re: Delayed inode operations not doing the right thing with enospc
2011/7/13 Josef Bacik jo...@redhat.com: On 07/12/2011 11:20 AM, Christian Brunner wrote: 2011/6/7 Josef Bacik jo...@redhat.com: On 06/06/2011 09:39 PM, Miao Xie wrote: On fri, 03 Jun 2011 14:46:10 -0400, Josef Bacik wrote: I got a lot of these when running stress.sh on my test box This is because use_block_rsv() is having to do a reserve_metadata_bytes(), which shouldn't happen as we should have reserved enough space for those operations to complete. This is happening because use_block_rsv() will call get_block_rsv(), which if root-ref_cows is set (which is the case on all fs roots) we will use trans-block_rsv, which will only have what the current transaction starter had reserved. What needs to be done instead is we need to have a block reserve that any reservation that is done at create time for these inodes is migrated to this special reserve, and then when you run the delayed inode items stuff you set trans-block_rsv to the special block reserve so the accounting is all done properly. This is just off the top of my head, there may be a better way to do it, I've not actually looked that the delayed inode code at all. I would do this myself but I have a ever increasing list of shit to do so will somebody pick this up and fix it please? Thanks, Sorry, it's my miss. I forgot to set trans-block_rsv to global_block_rsv, since we have migrated the space from trans_block_rsv to global_block_rsv. I'll fix it soon. There is another problem, we're failing xfstest 204. I tried making reserve_metadata_bytes commit the transaction regardless of whether or not there were pinned bytes but the test just hung there. Usually it takes 7 seconds to run and I ctrl+c'ed it after a couple of minutes. 204 just creates a crap ton of files, which is what is killing us. There needs to be a way to start flushing delayed inode items so we can reclaim the space they are holding onto so we don't get enospc, and it needs to be better than just committing the transaction because that is dog slow. Thanks, Josef Is there a solution for this? I'm running a 2.6.38.8 kernel with all the btrfs patches from 3.0rc7 (except the pluging). When starting a ceph rebuild on the btrfs volumes I get a lot of warnings from block_rsv_use_bytes in use_block_rsv: Ok I think I've got this nailed down. Will you run with this patch and make sure the warnings go away? Thanks, I'm sorry, I'm still getting a lot of warnings like the one below. I've also noticed, that I'm not getting these messages when the free_space_cache is disabled. Christian [ 697.398097] [ cut here ] [ 697.398109] WARNING: at fs/btrfs/extent-tree.c:5693 btrfs_alloc_free_block+0x1f8/0x360 [btrfs]() [ 697.398111] Hardware name: ProLiant DL180 G6 [ 697.398112] Modules linked in: btrfs zlib_deflate libcrc32c bonding ipv6 serio_raw pcspkr ghes hed iTCO_wdt iTCO_vendor_support i7core_edac edac_core ixgbe dca mdio iomemory_vsl(P) hpsa squashfs usb_storage [last unloaded: scsi_wait_scan] [ 697.398122] Pid: 6591, comm: btrfs-freespace Tainted: PW 3.0.0-1.fits.1.el6.x86_64 #1 [ 697.398124] Call Trace: [ 697.398128] [810630af] warn_slowpath_common+0x7f/0xc0 [ 697.398131] [8106310a] warn_slowpath_null+0x1a/0x20 [ 697.398142] [a022cb88] btrfs_alloc_free_block+0x1f8/0x360 [btrfs] [ 697.398156] [a025ae08] ? read_extent_buffer+0xd8/0x1d0 [btrfs] [ 697.398316] [a021d112] split_leaf+0x142/0x8c0 [btrfs] [ 697.398325] [a021629b] ? generic_bin_search+0x19b/0x210 [btrfs] [ 697.398334] [a0218a1a] ? btrfs_leaf_free_space+0x8a/0xe0 [btrfs] [ 697.398344] [a021df63] btrfs_search_slot+0x6d3/0x7a0 [btrfs] [ 697.398355] [a0230942] btrfs_csum_file_blocks+0x632/0x830 [btrfs] [ 697.398369] [a025c03a] ? clear_extent_bit+0x17a/0x440 [btrfs] [ 697.398382] [a023c009] add_pending_csums+0x49/0x70 [btrfs] [ 697.398395] [a023ef5d] btrfs_finish_ordered_io+0x22d/0x360 [btrfs] [ 697.398408] [a023f0dc] btrfs_writepage_end_io_hook+0x4c/0xa0 [btrfs] [ 697.398422] [a025c4fb] end_bio_extent_writepage+0x13b/0x180 [btrfs] [ 697.398425] [81558b5b] ? schedule_timeout+0x17b/0x2e0 [ 697.398436] [a02336d9] ? end_workqueue_fn+0xe9/0x130 [btrfs] [ 697.398439] [8118f24d] bio_endio+0x1d/0x40 [ 697.398451] [a02336e4] end_workqueue_fn+0xf4/0x130 [btrfs] [ 697.398464] [a02671de] worker_loop+0x13e/0x540 [btrfs] [ 697.398477] [a02670a0] ? btrfs_queue_worker+0x2d0/0x2d0 [btrfs] [ 697.398490] [a02670a0] ? btrfs_queue_worker+0x2d0/0x2d0 [btrfs] [ 697.398493] [81085896] kthread+0x96/0xa0 [ 697.398496] [81563844] kernel_thread_helper+0x4/0x10 [ 697.398499] [81085800] ? kthread_worker_fn+0x1a0/0x1a0 [ 697.398502] [81563840] ? gs_change+0x13/0x13 [ 697.398503] ---[ end trace 8c77269b0de3f0fb ]--- [
Re: Delayed inode operations not doing the right thing with enospc
On 07/14/2011 03:27 AM, Christian Brunner wrote:
2011/7/13 Josef Bacik jo...@redhat.com:
On 07/12/2011 11:20 AM, Christian Brunner wrote:
2011/6/7 Josef Bacik jo...@redhat.com:
On 06/06/2011 09:39 PM, Miao Xie wrote:
On fri, 03 Jun 2011 14:46:10 -0400, Josef Bacik wrote:
I got a lot of these when running stress.sh on my test box
This is because use_block_rsv() is having to do a
reserve_metadata_bytes(), which shouldn't happen as we should have
reserved enough space for those operations to complete. This is
happening because use_block_rsv() will call get_block_rsv(), which if
root-ref_cows is set (which is the case on all fs roots) we will use
trans-block_rsv, which will only have what the current transaction
starter had reserved.
What needs to be done instead is we need to have a block reserve that
any reservation that is done at create time for these inodes is migrated
to this special reserve, and then when you run the delayed inode items
stuff you set trans-block_rsv to the special block reserve so the
accounting is all done properly.
This is just off the top of my head, there may be a better way to do it,
I've not actually looked that the delayed inode code at all.
I would do this myself but I have a ever increasing list of shit to do
so will somebody pick this up and fix it please? Thanks,
Sorry, it's my miss.
I forgot to set trans-block_rsv to global_block_rsv, since we have
migrated
the space from trans_block_rsv to global_block_rsv.
I'll fix it soon.
There is another problem, we're failing xfstest 204. I tried making
reserve_metadata_bytes commit the transaction regardless of whether or
not there were pinned bytes but the test just hung there. Usually it
takes 7 seconds to run and I ctrl+c'ed it after a couple of minutes.
204 just creates a crap ton of files, which is what is killing us.
There needs to be a way to start flushing delayed inode items so we can
reclaim the space they are holding onto so we don't get enospc, and it
needs to be better than just committing the transaction because that is
dog slow. Thanks,
Josef
Is there a solution for this?
I'm running a 2.6.38.8 kernel with all the btrfs patches from 3.0rc7
(except the pluging). When starting a ceph rebuild on the btrfs
volumes I get a lot of warnings from block_rsv_use_bytes in
use_block_rsv:
Ok I think I've got this nailed down. Will you run with this patch and make
sure the warnings go away? Thanks,
I'm sorry, I'm still getting a lot of warnings like the one below.
I've also noticed, that I'm not getting these messages when the
free_space_cache is disabled.
Ok I see what's wrong, our checksum calculation is completely bogus.
I'm in the middle of something big so I can't give you a nice clean
patch, so if you can just go into extent-tree.c and replace
calc_csum_metadata_size with this you should be good to go
static u64 calc_csum_metadata_size(struct inode *inode, u64 num_bytes)
{
struct btrfs_root *root = BTRFS_I(inode)-root;
int num_leaves;
int num_csums;
u16 csum_size =
btrfs_super_csum_size(root-fs_info-super_copy);
num_csums = (int)div64_u64(num_bytes, root-sectorsize);
num_leaves = (int)((num_csums * csum_size) / root-leafsize);
return btrfs_calc_trans_metadata_size(root, num_leaves);
}
Thanks,
Josef
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Delayed inode operations not doing the right thing with enospc
On 07/14/2011 03:27 AM, Christian Brunner wrote:
2011/7/13 Josef Bacik jo...@redhat.com:
On 07/12/2011 11:20 AM, Christian Brunner wrote:
2011/6/7 Josef Bacik jo...@redhat.com:
On 06/06/2011 09:39 PM, Miao Xie wrote:
On fri, 03 Jun 2011 14:46:10 -0400, Josef Bacik wrote:
I got a lot of these when running stress.sh on my test box
This is because use_block_rsv() is having to do a
reserve_metadata_bytes(), which shouldn't happen as we should have
reserved enough space for those operations to complete. This is
happening because use_block_rsv() will call get_block_rsv(), which if
root-ref_cows is set (which is the case on all fs roots) we will use
trans-block_rsv, which will only have what the current transaction
starter had reserved.
What needs to be done instead is we need to have a block reserve that
any reservation that is done at create time for these inodes is migrated
to this special reserve, and then when you run the delayed inode items
stuff you set trans-block_rsv to the special block reserve so the
accounting is all done properly.
This is just off the top of my head, there may be a better way to do it,
I've not actually looked that the delayed inode code at all.
I would do this myself but I have a ever increasing list of shit to do
so will somebody pick this up and fix it please? Thanks,
Sorry, it's my miss.
I forgot to set trans-block_rsv to global_block_rsv, since we have
migrated
the space from trans_block_rsv to global_block_rsv.
I'll fix it soon.
There is another problem, we're failing xfstest 204. I tried making
reserve_metadata_bytes commit the transaction regardless of whether or
not there were pinned bytes but the test just hung there. Usually it
takes 7 seconds to run and I ctrl+c'ed it after a couple of minutes.
204 just creates a crap ton of files, which is what is killing us.
There needs to be a way to start flushing delayed inode items so we can
reclaim the space they are holding onto so we don't get enospc, and it
needs to be better than just committing the transaction because that is
dog slow. Thanks,
Josef
Is there a solution for this?
I'm running a 2.6.38.8 kernel with all the btrfs patches from 3.0rc7
(except the pluging). When starting a ceph rebuild on the btrfs
volumes I get a lot of warnings from block_rsv_use_bytes in
use_block_rsv:
Ok I think I've got this nailed down. Will you run with this patch and make
sure the warnings go away? Thanks,
I'm sorry, I'm still getting a lot of warnings like the one below.
I've also noticed, that I'm not getting these messages when the
free_space_cache is disabled.
Ok ditch that previous patch and try this one, it should work. Thanks,
Josef
diff --git a/fs/btrfs/btrfs_inode.h b/fs/btrfs/btrfs_inode.h
index 52d7eca..2263d29 100644
--- a/fs/btrfs/btrfs_inode.h
+++ b/fs/btrfs/btrfs_inode.h
@@ -112,9 +112,6 @@ struct btrfs_inode {
*/
u64 disk_i_size;
- /* flags field from the on disk inode */
- u32 flags;
-
/*
* if this is a directory then index_cnt is the counter for the index
* number for new files that are created
@@ -128,14 +125,8 @@ struct btrfs_inode {
*/
u64 last_unlink_trans;
- /*
-* Counters to keep track of the number of extent item's we may use due
-* to delalloc and such. outstanding_extents is the number of extent
-* items we think we'll end up using, and reserved_extents is the number
-* of extent items we've reserved metadata for.
-*/
- atomic_t outstanding_extents;
- atomic_t reserved_extents;
+ /* flags field from the on disk inode */
+ u32 flags;
/*
* ordered_data_close is set by truncate when a file that used
@@ -151,12 +142,21 @@ struct btrfs_inode {
unsigned orphan_meta_reserved:1;
unsigned dummy_inode:1;
unsigned in_defrag:1;
-
/*
* always compress this one file
*/
unsigned force_compress:4;
+ /*
+* Counters to keep track of the number of extent item's we may use due
+* to delalloc and such. outstanding_extents is the number of extent
+* items we think we'll end up using, and reserved_extents is the number
+* of extent items we've reserved metadata for.
+*/
+ spinlock_t extents_count_lock;
+ unsigned outstanding_extents;
+ unsigned reserved_extents;
+
struct btrfs_delayed_node *delayed_node;
struct inode vfs_inode;
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index be02cae..3ba4d5f 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -2133,7 +2133,7 @@ static inline bool btrfs_mixed_space_info(struct
btrfs_space_info *space_info)
/* extent-tree.c */
static inline u64 btrfs_calc_trans_metadata_size(struct btrfs_root *root,
-int num_items)
+
Re: Delayed inode operations not doing the right thing with enospc
On 07/12/2011 11:20 AM, Christian Brunner wrote:
2011/6/7 Josef Bacik jo...@redhat.com:
On 06/06/2011 09:39 PM, Miao Xie wrote:
On fri, 03 Jun 2011 14:46:10 -0400, Josef Bacik wrote:
I got a lot of these when running stress.sh on my test box
This is because use_block_rsv() is having to do a
reserve_metadata_bytes(), which shouldn't happen as we should have
reserved enough space for those operations to complete. This is
happening because use_block_rsv() will call get_block_rsv(), which if
root-ref_cows is set (which is the case on all fs roots) we will use
trans-block_rsv, which will only have what the current transaction
starter had reserved.
What needs to be done instead is we need to have a block reserve that
any reservation that is done at create time for these inodes is migrated
to this special reserve, and then when you run the delayed inode items
stuff you set trans-block_rsv to the special block reserve so the
accounting is all done properly.
This is just off the top of my head, there may be a better way to do it,
I've not actually looked that the delayed inode code at all.
I would do this myself but I have a ever increasing list of shit to do
so will somebody pick this up and fix it please? Thanks,
Sorry, it's my miss.
I forgot to set trans-block_rsv to global_block_rsv, since we have migrated
the space from trans_block_rsv to global_block_rsv.
I'll fix it soon.
There is another problem, we're failing xfstest 204. I tried making
reserve_metadata_bytes commit the transaction regardless of whether or
not there were pinned bytes but the test just hung there. Usually it
takes 7 seconds to run and I ctrl+c'ed it after a couple of minutes.
204 just creates a crap ton of files, which is what is killing us.
There needs to be a way to start flushing delayed inode items so we can
reclaim the space they are holding onto so we don't get enospc, and it
needs to be better than just committing the transaction because that is
dog slow. Thanks,
Josef
Is there a solution for this?
I'm running a 2.6.38.8 kernel with all the btrfs patches from 3.0rc7
(except the pluging). When starting a ceph rebuild on the btrfs
volumes I get a lot of warnings from block_rsv_use_bytes in
use_block_rsv:
Ok I think I've got this nailed down. Will you run with this patch and make
sure the warnings go away? Thanks,
Josef
diff --git a/fs/btrfs/btrfs_inode.h b/fs/btrfs/btrfs_inode.h
index 52d7eca..2263d29 100644
--- a/fs/btrfs/btrfs_inode.h
+++ b/fs/btrfs/btrfs_inode.h
@@ -112,9 +112,6 @@ struct btrfs_inode {
*/
u64 disk_i_size;
- /* flags field from the on disk inode */
- u32 flags;
-
/*
* if this is a directory then index_cnt is the counter for the index
* number for new files that are created
@@ -128,14 +125,8 @@ struct btrfs_inode {
*/
u64 last_unlink_trans;
- /*
-* Counters to keep track of the number of extent item's we may use due
-* to delalloc and such. outstanding_extents is the number of extent
-* items we think we'll end up using, and reserved_extents is the number
-* of extent items we've reserved metadata for.
-*/
- atomic_t outstanding_extents;
- atomic_t reserved_extents;
+ /* flags field from the on disk inode */
+ u32 flags;
/*
* ordered_data_close is set by truncate when a file that used
@@ -151,12 +142,21 @@ struct btrfs_inode {
unsigned orphan_meta_reserved:1;
unsigned dummy_inode:1;
unsigned in_defrag:1;
-
/*
* always compress this one file
*/
unsigned force_compress:4;
+ /*
+* Counters to keep track of the number of extent item's we may use due
+* to delalloc and such. outstanding_extents is the number of extent
+* items we think we'll end up using, and reserved_extents is the number
+* of extent items we've reserved metadata for.
+*/
+ spinlock_t extents_count_lock;
+ unsigned outstanding_extents;
+ unsigned reserved_extents;
+
struct btrfs_delayed_node *delayed_node;
struct inode vfs_inode;
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index be02cae..3ba4d5f 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -2133,7 +2133,7 @@ static inline bool btrfs_mixed_space_info(struct
btrfs_space_info *space_info)
/* extent-tree.c */
static inline u64 btrfs_calc_trans_metadata_size(struct btrfs_root *root,
-int num_items)
+unsigned num_items)
{
return (root-leafsize + root-nodesize * (BTRFS_MAX_LEVEL - 1)) *
3 * num_items;
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 3e52b85..65a721c 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -3952,13 +3952,35 @@ static u64
Re: Delayed inode operations not doing the right thing with enospc
2011/6/7 Josef Bacik jo...@redhat.com: On 06/06/2011 09:39 PM, Miao Xie wrote: On fri, 03 Jun 2011 14:46:10 -0400, Josef Bacik wrote: I got a lot of these when running stress.sh on my test box This is because use_block_rsv() is having to do a reserve_metadata_bytes(), which shouldn't happen as we should have reserved enough space for those operations to complete. This is happening because use_block_rsv() will call get_block_rsv(), which if root-ref_cows is set (which is the case on all fs roots) we will use trans-block_rsv, which will only have what the current transaction starter had reserved. What needs to be done instead is we need to have a block reserve that any reservation that is done at create time for these inodes is migrated to this special reserve, and then when you run the delayed inode items stuff you set trans-block_rsv to the special block reserve so the accounting is all done properly. This is just off the top of my head, there may be a better way to do it, I've not actually looked that the delayed inode code at all. I would do this myself but I have a ever increasing list of shit to do so will somebody pick this up and fix it please? Thanks, Sorry, it's my miss. I forgot to set trans-block_rsv to global_block_rsv, since we have migrated the space from trans_block_rsv to global_block_rsv. I'll fix it soon. There is another problem, we're failing xfstest 204. I tried making reserve_metadata_bytes commit the transaction regardless of whether or not there were pinned bytes but the test just hung there. Usually it takes 7 seconds to run and I ctrl+c'ed it after a couple of minutes. 204 just creates a crap ton of files, which is what is killing us. There needs to be a way to start flushing delayed inode items so we can reclaim the space they are holding onto so we don't get enospc, and it needs to be better than just committing the transaction because that is dog slow. Thanks, Josef Is there a solution for this? I'm running a 2.6.38.8 kernel with all the btrfs patches from 3.0rc7 (except the pluging). When starting a ceph rebuild on the btrfs volumes I get a lot of warnings from block_rsv_use_bytes in use_block_rsv: [ 2157.922054] [ cut here ] [ 2157.927270] WARNING: at fs/btrfs/extent-tree.c:5683 btrfs_alloc_free_block+0x1f8/0x360 [btrfs]() [ 2157.937123] Hardware name: ProLiant DL180 G6 [ 2157.942132] Modules linked in: btrfs zlib_deflate libcrc32c bonding ipv6 pcspkr serio_raw iTCO_wdt iTCO_vendor_support ghes hed i7core_edac edac_core ixgbe dca mdio iomemory_vsl(P) hpsa squashfs usb_storage [last unloaded: scsi_wait_scan] [ 2157.967386] Pid: 10280, comm: btrfs-freespace Tainted: PW 2.6.38.8-1.fits.4.el6.x86_64 #1 [ 2157.977554] Call Trace: [ 2157.980383] [8106482f] ? warn_slowpath_common+0x7f/0xc0 [ 2157.987382] [8106488a] ? warn_slowpath_null+0x1a/0x20 [ 2157.994192] [a0240b88] ? btrfs_alloc_free_block+0x1f8/0x360 [btrfs] [ 2158.002354] [a026eda8] ? read_extent_buffer+0xd8/0x1d0 [btrfs] [ 2158.010014] [a0231132] ? split_leaf+0x142/0x8c0 [btrfs] [ 2158.016990] [a022a29b] ? generic_bin_search+0x19b/0x210 [btrfs] [ 2158.024784] [a022ca1a] ? btrfs_leaf_free_space+0x8a/0xe0 [btrfs] [ 2158.032627] [a0231f83] ? btrfs_search_slot+0x6d3/0x7a0 [btrfs] [ 2158.040325] [a0244942] ? btrfs_csum_file_blocks+0x632/0x830 [btrfs] [ 2158.048477] [a026ffda] ? clear_extent_bit+0x17a/0x440 [btrfs] [ 2158.056026] [a024ffc5] ? add_pending_csums+0x45/0x70 [btrfs] [ 2158.063530] [a0252dad] ? btrfs_finish_ordered_io+0x22d/0x360 [btrfs] [ 2158.071755] [a0252f2c] ? btrfs_writepage_end_io_hook+0x4c/0xa0 [btrfs] [ 2158.080172] [a027049b] ? end_bio_extent_writepage+0x13b/0x180 [btrfs] [ 2158.088505] [815406fb] ? schedule_timeout+0x17b/0x2e0 [ 2158.095258] [8118964d] ? bio_endio+0x1d/0x40 [ 2158.101171] [a0247764] ? end_workqueue_fn+0xf4/0x130 [btrfs] [ 2158.108621] [a027b30e] ? worker_loop+0x13e/0x540 [btrfs] [ 2158.115703] [a027b1d0] ? worker_loop+0x0/0x540 [btrfs] [ 2158.122563] [a027b1d0] ? worker_loop+0x0/0x540 [btrfs] [ 2158.129413] [81086356] ? kthread+0x96/0xa0 [ 2158.135093] [8100ce44] ? kernel_thread_helper+0x4/0x10 [ 2158.141913] [810862c0] ? kthread+0x0/0xa0 [ 2158.147467] [8100ce40] ? kernel_thread_helper+0x0/0x10 [ 2158.154287] ---[ end trace 55e53c726a04ecd7 ]--- Thanks, Christian -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Delayed inode operations not doing the right thing with enospc
On 06/06/2011 09:39 PM, Miao Xie wrote: On fri, 03 Jun 2011 14:46:10 -0400, Josef Bacik wrote: I got a lot of these when running stress.sh on my test box [ 9792.654889] [ cut here ] [ 9792.654898] WARNING: at fs/btrfs/extent-tree.c:5681 btrfs_alloc_free_block+0xca/0x27c [btrfs]() [ 9792.654899] Hardware name: To Be Filled By O.E.M. [ 9792.654900] Modules linked in: btrfs zlib_deflate libcrc32c ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables arc4 rt61pci rt2x00pci rt2x00lib snd_hda_codec_hdmi mac80211 snd_hda_codec_realtek cfg80211 snd_hda_intel edac_core snd_seq rfkill pcspkr serio_raw snd_hda_codec eeprom_93cx6 edac_mce_amd sp5100_tco i2c_piix4 k10temp snd_hwdep snd_seq_device snd_pcm floppy r8169 xhci_hcd mii snd_timer snd soundcore snd_page_alloc ipv6 firewire_ohci pata_acpi ata_generic firewire_core pata_via crc_itu_t radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan] [ 9792.654919] Pid: 2762, comm: rm Tainted: GW 2.6.39+ #1 [ 9792.654920] Call Trace: [ 9792.654922] [81053c4a] warn_slowpath_common+0x83/0x9b [ 9792.654925] [81053c7c] warn_slowpath_null+0x1a/0x1c [ 9792.654933] [a038e747] btrfs_alloc_free_block+0xca/0x27c [btrfs] [ 9792.654945] [a03b8562] ? map_extent_buffer+0x6e/0xa8 [btrfs] [ 9792.654953] [a038189b] __btrfs_cow_block+0xfc/0x30c [btrfs] [ 9792.654963] [a0396aa6] ? btrfs_buffer_uptodate+0x47/0x58 [btrfs] [ 9792.654970] [a0382e48] ? read_block_for_search+0x94/0x368 [btrfs] [ 9792.654978] [a0381ba9] btrfs_cow_block+0xfe/0x146 [btrfs] [ 9792.654986] [a03848b0] btrfs_search_slot+0x14d/0x4b6 [btrfs] [ 9792.654997] [a03b8562] ? map_extent_buffer+0x6e/0xa8 [btrfs] [ 9792.655022] [a03938e8] btrfs_lookup_inode+0x2f/0x8f [btrfs] [ 9792.655025] [8147afac] ? _cond_resched+0xe/0x22 [ 9792.655027] [8147b892] ? mutex_lock+0x29/0x50 [ 9792.655039] [a03d41b1] btrfs_update_delayed_inode+0x72/0x137 [btrfs] [ 9792.655051] [a03d4ea2] btrfs_run_delayed_items+0x90/0xdb [btrfs] [ 9792.655062] [a039a69b] btrfs_commit_transaction+0x228/0x654 [btrfs] [ 9792.655064] [8106e8da] ? remove_wait_queue+0x3a/0x3a [ 9792.655075] [a03a2fa5] btrfs_evict_inode+0x14d/0x202 [btrfs] [ 9792.655077] [81132bd6] evict+0x71/0x111 [ 9792.655079] [81132de0] iput+0x12a/0x132 [ 9792.655081] [8112aa3a] do_unlinkat+0x106/0x155 [ 9792.655083] [81127b83] ? path_put+0x1f/0x23 [ 9792.655085] [8109c53c] ? audit_syscall_entry+0x145/0x171 [ 9792.655087] [81128410] ? putname+0x34/0x36 [ 9792.655090] [8112b441] sys_unlinkat+0x29/0x2b [ 9792.655092] [81482c42] system_call_fastpath+0x16/0x1b [ 9792.655093] ---[ end trace 02b696eb02b3f768 ]--- This is because use_block_rsv() is having to do a reserve_metadata_bytes(), which shouldn't happen as we should have reserved enough space for those operations to complete. This is happening because use_block_rsv() will call get_block_rsv(), which if root-ref_cows is set (which is the case on all fs roots) we will use trans-block_rsv, which will only have what the current transaction starter had reserved. What needs to be done instead is we need to have a block reserve that any reservation that is done at create time for these inodes is migrated to this special reserve, and then when you run the delayed inode items stuff you set trans-block_rsv to the special block reserve so the accounting is all done properly. This is just off the top of my head, there may be a better way to do it, I've not actually looked that the delayed inode code at all. I would do this myself but I have a ever increasing list of shit to do so will somebody pick this up and fix it please? Thanks, Sorry, it's my miss. I forgot to set trans-block_rsv to global_block_rsv, since we have migrated the space from trans_block_rsv to global_block_rsv. I'll fix it soon. Great thanks Miao, Josef -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Delayed inode operations not doing the right thing with enospc
On 06/06/2011 09:39 PM, Miao Xie wrote: On fri, 03 Jun 2011 14:46:10 -0400, Josef Bacik wrote: I got a lot of these when running stress.sh on my test box [ 9792.654889] [ cut here ] [ 9792.654898] WARNING: at fs/btrfs/extent-tree.c:5681 btrfs_alloc_free_block+0xca/0x27c [btrfs]() [ 9792.654899] Hardware name: To Be Filled By O.E.M. [ 9792.654900] Modules linked in: btrfs zlib_deflate libcrc32c ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables arc4 rt61pci rt2x00pci rt2x00lib snd_hda_codec_hdmi mac80211 snd_hda_codec_realtek cfg80211 snd_hda_intel edac_core snd_seq rfkill pcspkr serio_raw snd_hda_codec eeprom_93cx6 edac_mce_amd sp5100_tco i2c_piix4 k10temp snd_hwdep snd_seq_device snd_pcm floppy r8169 xhci_hcd mii snd_timer snd soundcore snd_page_alloc ipv6 firewire_ohci pata_acpi ata_generic firewire_core pata_via crc_itu_t radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan] [ 9792.654919] Pid: 2762, comm: rm Tainted: GW 2.6.39+ #1 [ 9792.654920] Call Trace: [ 9792.654922] [81053c4a] warn_slowpath_common+0x83/0x9b [ 9792.654925] [81053c7c] warn_slowpath_null+0x1a/0x1c [ 9792.654933] [a038e747] btrfs_alloc_free_block+0xca/0x27c [btrfs] [ 9792.654945] [a03b8562] ? map_extent_buffer+0x6e/0xa8 [btrfs] [ 9792.654953] [a038189b] __btrfs_cow_block+0xfc/0x30c [btrfs] [ 9792.654963] [a0396aa6] ? btrfs_buffer_uptodate+0x47/0x58 [btrfs] [ 9792.654970] [a0382e48] ? read_block_for_search+0x94/0x368 [btrfs] [ 9792.654978] [a0381ba9] btrfs_cow_block+0xfe/0x146 [btrfs] [ 9792.654986] [a03848b0] btrfs_search_slot+0x14d/0x4b6 [btrfs] [ 9792.654997] [a03b8562] ? map_extent_buffer+0x6e/0xa8 [btrfs] [ 9792.655022] [a03938e8] btrfs_lookup_inode+0x2f/0x8f [btrfs] [ 9792.655025] [8147afac] ? _cond_resched+0xe/0x22 [ 9792.655027] [8147b892] ? mutex_lock+0x29/0x50 [ 9792.655039] [a03d41b1] btrfs_update_delayed_inode+0x72/0x137 [btrfs] [ 9792.655051] [a03d4ea2] btrfs_run_delayed_items+0x90/0xdb [btrfs] [ 9792.655062] [a039a69b] btrfs_commit_transaction+0x228/0x654 [btrfs] [ 9792.655064] [8106e8da] ? remove_wait_queue+0x3a/0x3a [ 9792.655075] [a03a2fa5] btrfs_evict_inode+0x14d/0x202 [btrfs] [ 9792.655077] [81132bd6] evict+0x71/0x111 [ 9792.655079] [81132de0] iput+0x12a/0x132 [ 9792.655081] [8112aa3a] do_unlinkat+0x106/0x155 [ 9792.655083] [81127b83] ? path_put+0x1f/0x23 [ 9792.655085] [8109c53c] ? audit_syscall_entry+0x145/0x171 [ 9792.655087] [81128410] ? putname+0x34/0x36 [ 9792.655090] [8112b441] sys_unlinkat+0x29/0x2b [ 9792.655092] [81482c42] system_call_fastpath+0x16/0x1b [ 9792.655093] ---[ end trace 02b696eb02b3f768 ]--- This is because use_block_rsv() is having to do a reserve_metadata_bytes(), which shouldn't happen as we should have reserved enough space for those operations to complete. This is happening because use_block_rsv() will call get_block_rsv(), which if root-ref_cows is set (which is the case on all fs roots) we will use trans-block_rsv, which will only have what the current transaction starter had reserved. What needs to be done instead is we need to have a block reserve that any reservation that is done at create time for these inodes is migrated to this special reserve, and then when you run the delayed inode items stuff you set trans-block_rsv to the special block reserve so the accounting is all done properly. This is just off the top of my head, there may be a better way to do it, I've not actually looked that the delayed inode code at all. I would do this myself but I have a ever increasing list of shit to do so will somebody pick this up and fix it please? Thanks, Sorry, it's my miss. I forgot to set trans-block_rsv to global_block_rsv, since we have migrated the space from trans_block_rsv to global_block_rsv. I'll fix it soon. There is another problem, we're failing xfstest 204. I tried making reserve_metadata_bytes commit the transaction regardless of whether or not there were pinned bytes but the test just hung there. Usually it takes 7 seconds to run and I ctrl+c'ed it after a couple of minutes. 204 just creates a crap ton of files, which is what is killing us. There needs to be a way to start flushing delayed inode items so we can reclaim the space they are holding onto so we don't get enospc, and it needs to be better than just committing the transaction because that is dog slow. Thanks, Josef -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Delayed inode operations not doing the right thing with enospc
On fri, 03 Jun 2011 14:46:10 -0400, Josef Bacik wrote: I got a lot of these when running stress.sh on my test box [ 9792.654889] [ cut here ] [ 9792.654898] WARNING: at fs/btrfs/extent-tree.c:5681 btrfs_alloc_free_block+0xca/0x27c [btrfs]() [ 9792.654899] Hardware name: To Be Filled By O.E.M. [ 9792.654900] Modules linked in: btrfs zlib_deflate libcrc32c ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables arc4 rt61pci rt2x00pci rt2x00lib snd_hda_codec_hdmi mac80211 snd_hda_codec_realtek cfg80211 snd_hda_intel edac_core snd_seq rfkill pcspkr serio_raw snd_hda_codec eeprom_93cx6 edac_mce_amd sp5100_tco i2c_piix4 k10temp snd_hwdep snd_seq_device snd_pcm floppy r8169 xhci_hcd mii snd_timer snd soundcore snd_page_alloc ipv6 firewire_ohci pata_acpi ata_generic firewire_core pata_via crc_itu_t radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan] [ 9792.654919] Pid: 2762, comm: rm Tainted: GW 2.6.39+ #1 [ 9792.654920] Call Trace: [ 9792.654922] [81053c4a] warn_slowpath_common+0x83/0x9b [ 9792.654925] [81053c7c] warn_slowpath_null+0x1a/0x1c [ 9792.654933] [a038e747] btrfs_alloc_free_block+0xca/0x27c [btrfs] [ 9792.654945] [a03b8562] ? map_extent_buffer+0x6e/0xa8 [btrfs] [ 9792.654953] [a038189b] __btrfs_cow_block+0xfc/0x30c [btrfs] [ 9792.654963] [a0396aa6] ? btrfs_buffer_uptodate+0x47/0x58 [btrfs] [ 9792.654970] [a0382e48] ? read_block_for_search+0x94/0x368 [btrfs] [ 9792.654978] [a0381ba9] btrfs_cow_block+0xfe/0x146 [btrfs] [ 9792.654986] [a03848b0] btrfs_search_slot+0x14d/0x4b6 [btrfs] [ 9792.654997] [a03b8562] ? map_extent_buffer+0x6e/0xa8 [btrfs] [ 9792.655022] [a03938e8] btrfs_lookup_inode+0x2f/0x8f [btrfs] [ 9792.655025] [8147afac] ? _cond_resched+0xe/0x22 [ 9792.655027] [8147b892] ? mutex_lock+0x29/0x50 [ 9792.655039] [a03d41b1] btrfs_update_delayed_inode+0x72/0x137 [btrfs] [ 9792.655051] [a03d4ea2] btrfs_run_delayed_items+0x90/0xdb [btrfs] [ 9792.655062] [a039a69b] btrfs_commit_transaction+0x228/0x654 [btrfs] [ 9792.655064] [8106e8da] ? remove_wait_queue+0x3a/0x3a [ 9792.655075] [a03a2fa5] btrfs_evict_inode+0x14d/0x202 [btrfs] [ 9792.655077] [81132bd6] evict+0x71/0x111 [ 9792.655079] [81132de0] iput+0x12a/0x132 [ 9792.655081] [8112aa3a] do_unlinkat+0x106/0x155 [ 9792.655083] [81127b83] ? path_put+0x1f/0x23 [ 9792.655085] [8109c53c] ? audit_syscall_entry+0x145/0x171 [ 9792.655087] [81128410] ? putname+0x34/0x36 [ 9792.655090] [8112b441] sys_unlinkat+0x29/0x2b [ 9792.655092] [81482c42] system_call_fastpath+0x16/0x1b [ 9792.655093] ---[ end trace 02b696eb02b3f768 ]--- This is because use_block_rsv() is having to do a reserve_metadata_bytes(), which shouldn't happen as we should have reserved enough space for those operations to complete. This is happening because use_block_rsv() will call get_block_rsv(), which if root-ref_cows is set (which is the case on all fs roots) we will use trans-block_rsv, which will only have what the current transaction starter had reserved. What needs to be done instead is we need to have a block reserve that any reservation that is done at create time for these inodes is migrated to this special reserve, and then when you run the delayed inode items stuff you set trans-block_rsv to the special block reserve so the accounting is all done properly. This is just off the top of my head, there may be a better way to do it, I've not actually looked that the delayed inode code at all. I would do this myself but I have a ever increasing list of shit to do so will somebody pick this up and fix it please? Thanks, Sorry, it's my miss. I forgot to set trans-block_rsv to global_block_rsv, since we have migrated the space from trans_block_rsv to global_block_rsv. I'll fix it soon. Thanks for your help. Miao Josef -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Delayed inode operations not doing the right thing with enospc
Hello, I got a lot of these when running stress.sh on my test box [ 9792.654889] [ cut here ] [ 9792.654898] WARNING: at fs/btrfs/extent-tree.c:5681 btrfs_alloc_free_block+0xca/0x27c [btrfs]() [ 9792.654899] Hardware name: To Be Filled By O.E.M. [ 9792.654900] Modules linked in: btrfs zlib_deflate libcrc32c ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables arc4 rt61pci rt2x00pci rt2x00lib snd_hda_codec_hdmi mac80211 snd_hda_codec_realtek cfg80211 snd_hda_intel edac_core snd_seq rfkill pcspkr serio_raw snd_hda_codec eeprom_93cx6 edac_mce_amd sp5100_tco i2c_piix4 k10temp snd_hwdep snd_seq_device snd_pcm floppy r8169 xhci_hcd mii snd_timer snd soundcore snd_page_alloc ipv6 firewire_ohci pata_acpi ata_generic firewire_core pata_via crc_itu_t radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan] [ 9792.654919] Pid: 2762, comm: rm Tainted: GW 2.6.39+ #1 [ 9792.654920] Call Trace: [ 9792.654922] [81053c4a] warn_slowpath_common+0x83/0x9b [ 9792.654925] [81053c7c] warn_slowpath_null+0x1a/0x1c [ 9792.654933] [a038e747] btrfs_alloc_free_block+0xca/0x27c [btrfs] [ 9792.654945] [a03b8562] ? map_extent_buffer+0x6e/0xa8 [btrfs] [ 9792.654953] [a038189b] __btrfs_cow_block+0xfc/0x30c [btrfs] [ 9792.654963] [a0396aa6] ? btrfs_buffer_uptodate+0x47/0x58 [btrfs] [ 9792.654970] [a0382e48] ? read_block_for_search+0x94/0x368 [btrfs] [ 9792.654978] [a0381ba9] btrfs_cow_block+0xfe/0x146 [btrfs] [ 9792.654986] [a03848b0] btrfs_search_slot+0x14d/0x4b6 [btrfs] [ 9792.654997] [a03b8562] ? map_extent_buffer+0x6e/0xa8 [btrfs] [ 9792.655022] [a03938e8] btrfs_lookup_inode+0x2f/0x8f [btrfs] [ 9792.655025] [8147afac] ? _cond_resched+0xe/0x22 [ 9792.655027] [8147b892] ? mutex_lock+0x29/0x50 [ 9792.655039] [a03d41b1] btrfs_update_delayed_inode+0x72/0x137 [btrfs] [ 9792.655051] [a03d4ea2] btrfs_run_delayed_items+0x90/0xdb [btrfs] [ 9792.655062] [a039a69b] btrfs_commit_transaction+0x228/0x654 [btrfs] [ 9792.655064] [8106e8da] ? remove_wait_queue+0x3a/0x3a [ 9792.655075] [a03a2fa5] btrfs_evict_inode+0x14d/0x202 [btrfs] [ 9792.655077] [81132bd6] evict+0x71/0x111 [ 9792.655079] [81132de0] iput+0x12a/0x132 [ 9792.655081] [8112aa3a] do_unlinkat+0x106/0x155 [ 9792.655083] [81127b83] ? path_put+0x1f/0x23 [ 9792.655085] [8109c53c] ? audit_syscall_entry+0x145/0x171 [ 9792.655087] [81128410] ? putname+0x34/0x36 [ 9792.655090] [8112b441] sys_unlinkat+0x29/0x2b [ 9792.655092] [81482c42] system_call_fastpath+0x16/0x1b [ 9792.655093] ---[ end trace 02b696eb02b3f768 ]--- This is because use_block_rsv() is having to do a reserve_metadata_bytes(), which shouldn't happen as we should have reserved enough space for those operations to complete. This is happening because use_block_rsv() will call get_block_rsv(), which if root-ref_cows is set (which is the case on all fs roots) we will use trans-block_rsv, which will only have what the current transaction starter had reserved. What needs to be done instead is we need to have a block reserve that any reservation that is done at create time for these inodes is migrated to this special reserve, and then when you run the delayed inode items stuff you set trans-block_rsv to the special block reserve so the accounting is all done properly. This is just off the top of my head, there may be a better way to do it, I've not actually looked that the delayed inode code at all. I would do this myself but I have a ever increasing list of shit to do so will somebody pick this up and fix it please? Thanks, Josef -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
