[GIT PULL] Btrfs
Hi Linus, My for-linus-4.11 branch: git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git for-linus-4.11 Has a series of fixes and cleanups that Dave Sterba has been collecting: There is a pretty big variety here, cleaning up internal APIs and fixing corner cases. David Sterba (46) commits (+235/-313): btrfs: remove unused parameter from btrfs_subvolume_release_metadata (+6/-11) btrfs: remove pointless rcu protection from btrfs_qgroup_inherit (+0/-2) btrfs: check quota status earlier and don't do unnecessary frees (+3/-2) btrfs: remove unused parameter from btrfs_prepare_extent_commit (+3/-5) btrfs: remove unnecessary mutex lock in qgroup_account_snapshot (+1/-5) btrfs: embed extent_changeset::range_changed to the structure (+11/-17) btrfs: remove unused parameter from cleanup_write_cache_enospc (+2/-3) btrfs: remove unused parameters from __btrfs_write_out_cache (+3/-8) btrfs: remove unused parameter from clone_copy_inline_extent (+2/-3) btrfs: remove unused parameter from extent_write_cache_pages (+2/-4) btrfs: remove unused parameter from tree_move_next_or_upnext (+2/-4) btrfs: remove unused parameter from btrfs_check_super_valid (+3/-5) btrfs: remove unused logic of limiting async delalloc pages (+0/-7) btrfs: fix over-80 lines introduced by previous cleanups (+74/-63) btrfs: remove unused parameter from read_block_for_search (+5/-5) btrfs: remove unused parameter from adjust_slots_upwards (+2/-3) btrfs: remove unused parameter from init_first_rw_device (+3/-5) btrfs: make space cache inode readahead failure nonfatal (+3/-7) btrfs: remove unused parameters from scrub_setup_wr_ctx (+3/-7) btrfs: remove unused parameter from __btrfs_alloc_chunk (+4/-6) btrfs: add wrapper for counting BTRFS_MAX_EXTENT_SIZE (+23/-31) btrfs: remove unused parameter from submit_extent_page (+3/-9) btrfs: remove unused parameter from clean_tree_block (+17/-19) btrfs: use GFP_KERNEL in btrfs_add/del_qgroup_relation (+2/-2) btrfs: remove unused parameter from __add_inline_refs (+2/-3) btrfs: remove unused parameter from add_pending_csums (+2/-4) btrfs: remove unused parameter from update_nr_written (+4/-4) btrfs: remove unused parameter from __push_leaf_right (+2/-3) btrfs: remove unused parameter from check_async_write (+2/-2) btrfs: remove unused parameter from btrfs_fill_super (+2/-3) btrfs: remove unused parameter from __push_leaf_left (+2/-3) btrfs: remove unused parameter from write_dev_supers (+3/-3) btrfs: remove unused parameter from __add_inode_ref (+1/-2) btrfs: remove unused parameters from btrfs_cmp_data (+2/-3) btrfs: remove unused parameter from create_snapshot (+2/-2) btrfs: ulist: make the finalization function public (+2/-1) btrfs: remove unused parameter from tree_move_down (+2/-2) btrfs: ulist: rename ulist_fini to ulist_release (+10/-10) btrfs: qgroups: make __del_qgroup_relation static (+1/-1) btrfs: use GFP_KERNEL in btrfs_read_qgroup_config (+1/-1) btrfs: remove unused parameter from split_item (+2/-3) btrfs: merge two superblock writing helpers (+4/-11) btrfs: qgroups: opencode qgroup_free helper (+9/-9) btrfs: use GFP_KERNEL in btrfs_quota_enable (+1/-1) btrfs: use GFP_KERNEL in create_snapshot (+2/-2) btrfs: remove unused ulist members (+0/-7) Nikolay Borisov (36) commits (+476/-480): btrfs: Make btrfs_delayed_inode_reserve_metadata take btrfs_inode (+8/-8) btrfs: Make btrfs_inode_delayed_dir_index_count take btrfs_inode (+5/-5) btrfs: Make btrfs_commit_inode_delayed_items take btrfs_inode (+4/-4) btrfs: Make btrfs_commit_inode_delayed_inode take btrfs_inode (+6/-6) btrfs: Make btrfs_get_or_create_delayed_node take btrfs_inode (+5/-6) btrfs: Make btrfs_kill_delayed_inode_items take btrfs_inode (+4/-4) btrfs: Make btrfs_delayed_delete_inode_ref take btrfs_inode (+5/-5) btrfs: Make btrfs_delete_delayed_dir_index take btrfs_inode (+6/-6) btrfs: Make btrfs_insert_delayed_dir_index take btrfs_inode (+5/-5) btrfs: Make btrfs_check_ref_name_override take btrfs_inode (+4/-5) btrfs: Make btrfs_record_snapshot_destroy take btrfs_inode (+6/-6) btrfs: Make btrfs_must_commit_transaction take btrfs_inode (+9/-9) btrfs: Make btrfs_del_dir_entries_in_log take btrfs_inode (+7/-7) btrfs: Make btrfs_log_changed_extents take btrfs_inode (+11/-11) btrfs: Make btrfs_record_unlink_dir take btrfs_inode (+14/-14) btrfs: Make btrfs_remove_delayed_node take btrfs_inode (+5/-5) btrfs: Make btrfs_get_logged_extents take btrfs_inode (+4/-4) btrfs: Make btrfs_log_trailing_hole take btrfs_inode (+4/-4) btrfs: Make btrfs_get_delayed_node take btrfs_inode (+8/-9) btrfs: Make btrfs_ino take a struct btrfs_inode (+151/-151) btrfs: Make log_directory_changes take btrfs_inode (+5/-6)
Re: [PATCH] Btrfs: fix deadlock between dedup on same file and starting writeback
On Tue, Feb 21, 2017 at 05:14:52PM +, fdman...@kernel.org wrote: > From: Filipe Manana > > If we are deduping two ranges of the same file we need to make sure that > we lock all pages in ascending order, that is, lock first the pages from > the range with lower offset and then the pages from the other range, as > otherwise we can deadlock with a concurrent task that is starting delalloc > (writeback). Example trace: > > [74073.052218] INFO: task kworker/u32:10:17997 blocked for more than 120 > seconds. > [74073.053889] Tainted: GW 4.9.0-rc7-btrfs-next-36+ #1 > [74073.055071] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables > this message. > [74073.056696] kworker/u32:10 D0 17997 2 0x > [74073.058606] Workqueue: writeback wb_workfn (flush-btrfs-53176) > [74073.061370] 880031e79858 8802159d2580 880237004580 > 880031e79240 > [74073.064784] 88023f4978c0 c9000817b638 814c15e1 > > [74073.068386] 88023f4978d8 88023f4978c0 0017b620 > 880031e79240 > [74073.071712] Call Trace: > [74073.072884] [] ? __schedule+0x48f/0x6f4 > [74073.075395] [] ? bit_wait+0x2f/0x2f > [74073.077511] [] schedule+0x8c/0xa0 > [74073.079440] [] schedule_timeout+0x43/0xff > [74073.081637] [] ? time_hardirqs_on+0x9/0x14 > [74073.083809] [] ? trace_hardirqs_on_caller+0x16/0x197 > [74073.086314] [] ? timekeeping_get_ns+0x1e/0x32 > [74073.100654] [] ? ktime_get+0x41/0x52 > [74073.102619] [] io_schedule_timeout+0xa0/0x102 > [74073.104771] [] ? io_schedule_timeout+0xa0/0x102 > [74073.106969] [] bit_wait_io+0x1b/0x39 > [74073.108954] [] __wait_on_bit_lock+0x4f/0x99 > [74073.110981] [] __lock_page+0x6b/0x6d > [74073.112833] [] ? autoremove_wake_function+0x3a/0x3a > [74073.115010] [] lock_page+0x2f/0x32 [btrfs] > [74073.116999] [] lock_delalloc_pages+0xc7/0x1a0 [btrfs] > [74073.119243] [] find_lock_delalloc_range+0xc3/0x1a4 > [btrfs] > [74073.121636] [] writepage_delalloc.isra.31+0x8b/0x134 > [btrfs] > [74073.124229] [] __extent_writepage+0x1c1/0x2bf [btrfs] > [74073.126372] [] > extent_write_cache_pages.isra.30.constprop.49+0x28b/0x36c [btrfs] > [74073.129371] [] extent_writepages+0x4b/0x5c [btrfs] > [74073.131440] [] ? > insert_reserved_file_extent.constprop.42+0x261/0x261 [btrfs] > [74073.134303] [] ? writeback_sb_inodes+0xe0/0x4a1 > [74073.136298] [] btrfs_writepages+0x28/0x2a [btrfs] > [74073.138248] [] do_writepages+0x23/0x2c > [74073.139910] [] __writeback_single_inode+0x105/0x6d2 > [74073.142003] [] writeback_sb_inodes+0x292/0x4a1 > [74073.136298] [] btrfs_writepages+0x28/0x2a [btrfs] > [74073.138248] [] do_writepages+0x23/0x2c > [74073.139910] [] __writeback_single_inode+0x105/0x6d2 > [74073.142003] [] writeback_sb_inodes+0x292/0x4a1 > [74073.143911] [] __writeback_inodes_wb+0x76/0xae > [74073.145787] [] wb_writeback+0x1cc/0x4d7 > [74073.147452] [] wb_workfn+0x194/0x37d > [74073.149084] [] ? wb_workfn+0x194/0x37d > [74073.150726] [] ? process_one_work+0x154/0x4e4 > [74073.152694] [] process_one_work+0x273/0x4e4 > [74073.154452] [] worker_thread+0x1eb/0x2ca > [74073.156138] [] ? rescuer_thread+0x2b6/0x2b6 > [74073.157837] [] kthread+0xd5/0xdd > [74073.159339] [] ? __kthread_unpark+0x5a/0x5a > [74073.161088] [] ret_from_fork+0x27/0x40 > [74073.162680] INFO: lockdep is turned off. > [74073.163855] INFO: task do-dedup:30264 blocked for more than 120 seconds. > [74073.181180] Tainted: GW 4.9.0-rc7-btrfs-next-36+ #1 > [74073.181180] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables > this message. > [74073.185296] fdm-stress D0 30264 29974 0x > [74073.186810] 880089595118 880211b8eac0 880237030380 > 880089594b00 > [74073.188998] 88023f2978c0 c900063abb68 814c15e1 > > [74073.191070] 88023f2978d8 88023f2978c0 003abb50 > 880089594b00 > [74073.193286] Call Trace: > [74073.193990] [] ? __schedule+0x48f/0x6f4 > [74073.195418] [] ? bit_wait+0x2f/0x2f > [74073.196796] [] schedule+0x8c/0xa0 > [74073.198163] [] schedule_timeout+0x43/0xff > [74073.199621] [] ? trace_hardirqs_on+0xd/0xf > [74073.201100] [] ? timekeeping_get_ns+0x1e/0x32 > [74073.202686] [] ? ktime_get+0x41/0x52 > [74073.204051] [] io_schedule_timeout+0xa0/0x102 > [74073.205585] [] ? io_schedule_timeout+0xa0/0x102 > [74073.207123] [] bit_wait_io+0x1b/0x39 > [74073.208238] [] __wait_on_bit_lock+0x4f/0x99 > [74073.208871] [] __lock_page+0x6b/0x6d > [74073.209430] [] ? autoremove_wake_function+0x3a/0x3a > [74073.210101] [] lock_page+0x2f/0x32 > [74073.210636] [] pagecache_get_page+0x5e/0x153 > [74073.211270] [] gather_extent_pages+0x4e/0x109 [btrfs] > [74073.212166] [] btrfs_dedupe_file_range+0x1e1/0x4dd > [btrfs] > [74073.213257] [] vfs_dedupe_file_range+0x1c1/0x221 > [74073.214086] [] do_vfs_ioctl+0x442/0x600 > [74073.214767] [] ? rcu_read_unlock+0x5b/0x5d > [74073.215619] []
Re: Downgrading kernel 4.9 to 4.4 with space_cache=v2 enabled?
On Thu, Feb 23, 2017 at 08:38:02AM -0500, Austin S. Hemmelgarn wrote: > On 2017-02-23 08:19, Christian Theune wrote: > > Hi, > > > > just for future reference if someone finds this thread: there is a bit of > > output I’m seeing with this crashing kernel (unclear whether related to > > btrfs or not): > > > > 31 | 02/23/2017 | 09:51:22 | OS Stop/Shutdown #0x4f | Run-time critical > > stop | Asserted > > 32 | Linux kernel panic: Out of memo > > 33 | Linux kernel panic: ry and no k > > 34 | Linux kernel panic: illable pro > > 35 | Linux kernel panic: cesses... > > > OK, an OOM condition is (probably) not BTRFS related FWIW. IIRC, there were > a couple of issues with the OOM killer in 4.9, but I don't remember > specifics. To be able to determine anything useful though, you'd need a lot > more context in the kernel logs than just that. Yeah, if there is a problem with the free space tree, I'd be happy to take a look, but it's impossible to tell from this. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Downgrading kernel 4.9 to 4.4 with space_cache=v2 enabled?
On Fri, Feb 24, 2017 at 07:23:42AM -0500, Austin S. Hemmelgarn wrote: > On 2017-02-23 19:54, Qu Wenruo wrote: > > > > > > At 02/23/2017 06:51 PM, Christian Theune wrote: > > > Hi, > > > > > > not sure whether it’s possible, but we tried space_cache=v2 and > > > obviously after working fine in staging it broke in production. Or > > > rather: we upgraded from 4.4 to 4.9 and enabled the space_cache. Our > > > production volume is around 50TiB usable (underlying HW Raid 6). > > > > > > The machine crashes silently every 15 hours or so and takes _ages_ to > > > reboot. It current is stuck trying to mount the local filesystems and > > > I guess btrfs is doing something, but I don’t have shell access yet. > > > > > > I’m wondering whether we can downgrade by booting back into 4.4 or > > > will this break things even further? (We’ve had some unpleasant > > > surprises with FS’ in the last months, so I thought I’d rather ask.) > > > > You could use "btrfs check --clear-space-cache" to completely cleanup > > space cache. Both v1(free space cache, file based) or v2(free space > > tree) are supported. > > > > And specially for v2 space cache (space cache tree), "btrfs check > > --clear-space-cache" will also clear the ro_compat flag, so older kernel > > should mount the fs without problem. > That's really good to know. I hadn't remembered that using mount options > didn't clear the flag. Mounting with -oclear_cache,nospace_cache will clear the free space tree and the ro_compat flag. Mounting with just -oclear_cache rebuilds the free space tree (and keeps the ro_compat flag). -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: List of corruption cases for scrub
Hi, On 2017-02-24 03:32, Lakshmipathi.G wrote: > Hi. > > I tried to a create list of corruption test scenarios for scrubbing process > with RAID5. > Here's the list: > https://btrfs.wiki.kernel.org/index.php/Scrub_corruption_cases > > If similar list already exists in btrfs wiki, let me know, I'll move this data > to there. Feel free to add/update/delete entries or suggest new corrupt ideas! Thanks for doing that. About the test "Device-corruption", I suppose that you want to test a scenario where one disk fails. If so you do not have to always corrupt "D0" block, but the first block of the file which is stored in the failing disk (which could be different from the D0). Another scenario which should be tested, is the reshaping of the raid: add a drive (or remove a drive) then re-balance the system (with or without some corruption). IIRC this use the same scrub code. > > Cheers. > Lakshmipathi.G > -- BR G.Baroncelli -- gpg @keyserver.linux.it: Goffredo Baroncelli Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] Btrfs bug fixes and improvements for 4.11 (2nd retry)
On Fri, Feb 24, 2017 at 03:25:09AM +, fdman...@kernel.org wrote: From: Filipe Manana Hi Chris, since my previous pull request (sent timely) was either missed or not pulled for some reason I'm not aware of, here I send it again (with one more patch included). The following is taken from the former pull request: Hi Filipe, I've got this queued up and will send Monday/Tuesday. Thanks! -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[GIT PULL] Btrfs bug fixes and improvements for 4.11 (2nd retry)
From: Filipe Manana Hi Chris, since my previous pull request (sent timely) was either missed or not pulled for some reason I'm not aware of, here I send it again (with one more patch included). The following is taken from the former pull request: "Please consider the following changes for the 4.11 merge window. This time there is nothing particularly outstanding when compared to the usual set of bug fixes. These are mostly fixes for send and the no-holes feature introduced in 3.14. Test cases for fstests were sent for half of these changes, with some already merged and two not yet merged." This was rebased against your current for-linus-4.11 branch and re-tested. Thanks. The following changes since commit 6288d6eabc7505f42dda34a2c2962f91914be3a4: Btrfs: use the correct type when creating cow dio extent (2017-02-22 15:55:03 -0800) are available in the git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/fdmanana/linux.git for-chris-4.11 for you to fetch changes up to 263d3995c93c6020576f6c93506412a0b9d1e932: Btrfs: try harder to migrate items to left sibling before splitting a leaf (2017-02-24 00:39:44 +) Filipe Manana (8): Btrfs: incremental send, do not delay rename when parent inode is new Btrfs: bulk delete checksum items in the same leaf Btrfs: do not create explicit holes when replaying log tree if NO_HOLES enabled Btrfs: fix assertion failure when freeing block groups at close_ctree() Btrfs: fix use-after-free due to wrong order of destroying work queues Btrfs: incremental send, fix unnecessary hole writes for sparse files Btrfs: fix data loss after truncate when using the no-holes feature Btrfs: try harder to migrate items to left sibling before splitting a leaf Robbie Ko (3): Btrfs: send, fix failure to rename top level inode due to name collision Btrfs: incremental send, do not issue invalid rmdir operations Btrfs: fix leak of subvolume writers counter fs/btrfs/ctree.c | 7 +++ fs/btrfs/disk-io.c | 15 ++- fs/btrfs/extent-tree.c | 9 ++--- fs/btrfs/file-item.c | 28 +++- fs/btrfs/inode.c | 29 ++--- fs/btrfs/send.c| 125 +++-- fs/btrfs/tree-log.c| 5 + 7 files changed, 188 insertions(+), 30 deletions(-) -- 2.7.0.rc3 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Downgrading kernel 4.9 to 4.4 with space_cache=v2 enabled?
On 2017-02-23 19:54, Qu Wenruo wrote: At 02/23/2017 06:51 PM, Christian Theune wrote: Hi, not sure whether it’s possible, but we tried space_cache=v2 and obviously after working fine in staging it broke in production. Or rather: we upgraded from 4.4 to 4.9 and enabled the space_cache. Our production volume is around 50TiB usable (underlying HW Raid 6). The machine crashes silently every 15 hours or so and takes _ages_ to reboot. It current is stuck trying to mount the local filesystems and I guess btrfs is doing something, but I don’t have shell access yet. I’m wondering whether we can downgrade by booting back into 4.4 or will this break things even further? (We’ve had some unpleasant surprises with FS’ in the last months, so I thought I’d rather ask.) You could use "btrfs check --clear-space-cache" to completely cleanup space cache. Both v1(free space cache, file based) or v2(free space tree) are supported. And specially for v2 space cache (space cache tree), "btrfs check --clear-space-cache" will also clear the ro_compat flag, so older kernel should mount the fs without problem. That's really good to know. I hadn't remembered that using mount options didn't clear the flag. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
