[PATCH v3 4/5] generic: add test for truncate/fpunch of an active swapfile

2018-05-22 Thread Omar Sandoval
From: Omar Sandoval These should not be allowed. Signed-off-by: Omar Sandoval --- tests/generic/493 | 73 +++ tests/generic/493.out | 8 + tests/generic/group | 1 + 3 files changed, 82 insertions(+) create mode 100755 tests/generic/493

[PATCH v3 2/5] generic: enable swapfile tests on Btrfs

2018-05-22 Thread Omar Sandoval
From: Omar Sandoval Commit 8c96cfbfe530 ("generic/35[67]: disable swapfile tests on Btrfs") disabled the swapfile tests on Btrfs because it did not support swapfiles at the time. Now that we're adding support, we want these tests to run, but they don't. _require_scratch_swapf

Re: [PATCH v2 2/5] generic: enable swapfile tests on Btrfs

2018-05-22 Thread Omar Sandoval
On Tue, May 22, 2018 at 02:41:45PM +0800, Eryu Guan wrote: > On Wed, May 16, 2018 at 01:38:46PM -0700, Omar Sandoval wrote: > > From: Omar Sandoval > > > > Commit 8c96cfbfe530 ("generic/35[67]: disable swapfile tests on Btrfs") > > disabled the swapfile test

[PATCH] Btrfs: fix memory and mount leak in btrfs_ioctl_rm_dev_v2()

2018-05-22 Thread Omar Sandoval
From: Omar Sandoval If we have invalid flags set, when we error out we must drop our writer counter and free the buffer we allocated for the arguments. This bug is trivially reproduced with the following program: #include #include #include #include

[PATCH 2/2] Btrfs: fix dedupe vs chattr NODATASUM race

2018-05-22 Thread Omar Sandoval
From: Omar Sandoval In btrfs_extent_same(), we must check the NODATASUM flag while the inodes are locked. Otherwise, it's possible that btrfs_ioctl_setflags() will change the flags after we check. This was correct until a recent change. Fixes: 0a38effca37c ("Btrfs: split btrfs_e

[PATCH 1/2] Btrfs: fix clone vs chattr NODATASUM race

2018-05-22 Thread Omar Sandoval
From: Omar Sandoval In btrfs_clone_files(), we must check the NODATASUM flag while the inodes are locked. Otherwise, it's possible that btrfs_ioctl_setflags() will change the flags after we check and we can end up with a party checksummed file. Fixes: 0e7b824c4ef9 ("Btrfs: don

[PATCH 0/2] Btrfs: fix partly checksummed file races

2018-05-22 Thread Omar Sandoval
From: Omar Sandoval Clone and dedupe both have checks to make sure that we're not mixing checksummed and not checksummed extents in a single file. However, both checks are racy; we need to have the inodes locked or else the flags can change after we check. The clone check has always been

Re: [PATCH 1/2] btrfs: kill btrfs_write_inode

2018-05-22 Thread Omar Sandoval
o reason, and it could deadlock with delayed iput's. Reviewed-by: Omar Sandoval > Signed-off-by: Josef Bacik > --- > fs/btrfs/inode.c | 26 -- > fs/btrfs/super.c | 1 - > 2 files changed, 27 deletions(-) > > diff --git a/fs/btrfs/inode.c b

Re: [PATCH v2] Btrfs: fix error handling in btrfs_truncate()

2018-05-22 Thread Omar Sandoval
On Tue, May 22, 2018 at 10:41:11AM -0700, Omar Sandoval wrote: > On Tue, May 22, 2018 at 10:37:14AM -0700, Omar Sandoval wrote: > > On Tue, May 22, 2018 at 07:17:48PM +0200, David Sterba wrote: > > > On Tue, May 22, 2018 at 09:47:58AM -0700, Omar Sandoval wrote: > >

Re: [PATCH v2] Btrfs: fix error handling in btrfs_truncate()

2018-05-22 Thread Omar Sandoval
On Tue, May 22, 2018 at 10:37:14AM -0700, Omar Sandoval wrote: > On Tue, May 22, 2018 at 07:17:48PM +0200, David Sterba wrote: > > On Tue, May 22, 2018 at 09:47:58AM -0700, Omar Sandoval wrote: > > > From: Omar Sandoval > > > > > > Jun Wu at Facebook reported

Re: [PATCH v2] Btrfs: fix error handling in btrfs_truncate()

2018-05-22 Thread Omar Sandoval
On Tue, May 22, 2018 at 07:17:48PM +0200, David Sterba wrote: > On Tue, May 22, 2018 at 09:47:58AM -0700, Omar Sandoval wrote: > > From: Omar Sandoval > > > > Jun Wu at Facebook reported that an internal service was seeing a return > > value of 1 from ftruncate() on B

Re: [PATCH] Btrfs: implement unlocked buffered write

2018-05-22 Thread Omar Sandoval
On Wed, May 16, 2018 at 11:52:37AM +0800, robbieko wrote: > From: Robbie Ko > > This idea is from direct io. By this patch, we can make the buffered > write parallel, and improve the performance and latency. But because we > can not update isize without i_mutex, the unlocked buffered write just >

[PATCH] Btrfs: clean up error handling in btrfs_truncate()

2018-05-22 Thread Omar Sandoval
From: Omar Sandoval btrfs_truncate() uses two variables for error handling, ret and err (if this sounds familiar, it's because btrfs_truncate_inode_items() did something similar). This is error prone, as was made evident by "Btrfs: fix error handling in btrfs_truncate()". W

[PATCH v2] Btrfs: fix error handling in btrfs_truncate()

2018-05-22 Thread Omar Sandoval
From: Omar Sandoval Jun Wu at Facebook reported that an internal service was seeing a return value of 1 from ftruncate() on Btrfs in some cases. This is coming from the NEED_TRUNCATE_BLOCK return value from btrfs_truncate_inode_items(). btrfs_truncate() uses two variables for error handling

Re: [PATCH] Btrfs: allow empty subvol= again

2018-05-21 Thread Omar Sandoval
On Mon, May 21, 2018 at 05:07:19PM -0700, Omar Sandoval wrote: > From: Omar Sandoval > > I got a report that after upgrading to 4.16, someone's filesystems > weren't mounting: > > [ 23.845852] BTRFS info (device loop0): unrecognized mount option 'subvol=&#x

[PATCH] Btrfs: allow empty subvol= again

2018-05-21 Thread Omar Sandoval
From: Omar Sandoval I got a report that after upgrading to 4.16, someone's filesystems weren't mounting: [ 23.845852] BTRFS info (device loop0): unrecognized mount option 'subvol=' Before 4.16, this mounted the default subvolume. It turns out that this empty &quo

Re: Any chance to get snapshot-aware defragmentation?

2018-05-18 Thread Omar Sandoval
On Fri, May 18, 2018 at 04:26:16PM -0600, Chris Murphy wrote: > On Fri, May 18, 2018 at 12:33 PM, Austin S. Hemmelgarn > wrote: > > On 2018-05-18 13:18, Niccolò Belli wrote: > >> > >> On venerdì 18 maggio 2018 19:10:02 CEST, Austin S. Hemmelgarn wrote: > >>> > >>> and also forces the people who ha

Re: [PATCH] Btrfs: fix error handling in btrfs_truncate()

2018-05-18 Thread Omar Sandoval
On Fri, May 18, 2018 at 02:43:02PM -0700, Omar Sandoval wrote: > From: Omar Sandoval > > Jun Wu at Facebook reported that an internal service was seeing a return > value of 1 from ftruncate() on Btrfs when compression is enabled. This > is coming from the NEED_TRUNCATE_BLOCK re

[PATCH] Btrfs: fix error handling in btrfs_truncate()

2018-05-18 Thread Omar Sandoval
From: Omar Sandoval Jun Wu at Facebook reported that an internal service was seeing a return value of 1 from ftruncate() on Btrfs when compression is enabled. This is coming from the NEED_TRUNCATE_BLOCK return value from btrfs_truncate_inode_items(). btrfs_truncate() uses two variables for

Re: [PATCH v4 00/12] Btrfs: orphan and truncate fixes

2018-05-18 Thread Omar Sandoval
On Fri, May 18, 2018 at 05:14:35PM +0200, David Sterba wrote: > On Fri, May 11, 2018 at 01:13:28PM -0700, Omar Sandoval wrote: > > This is the fourth (and hopefully final) version of the orphan item > > early ENOSPC and related fixes. > > > > Changes since v3: >

[PATCH v2 2/5] generic: enable swapfile tests on Btrfs

2018-05-16 Thread Omar Sandoval
From: Omar Sandoval Commit 8c96cfbfe530 ("generic/35[67]: disable swapfile tests on Btrfs") disabled the swapfile tests on Btrfs because it did not support swapfiles at the time. Now that we're adding support, we want these tests to run, but they don't. _require_scratch_swapf

[PATCH v2 1/5] xfstests: create swap group

2018-05-16 Thread Omar Sandoval
From: Omar Sandoval I'm going to add a bunch of tests for swap files, so create a group for them and add the existing tests. Signed-off-by: Omar Sandoval --- tests/generic/group | 4 ++-- tests/xfs/group | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/ge

[PATCH v2 0/5] xfstests: generic swap file tests

2018-05-16 Thread Omar Sandoval
From: Omar Sandoval Changes since v1: - Add patch 1 to create a group for swap - Add a helper for formatting a swap file instead of open coding everywhere - Use $CHATTR_PROG instead of chattr in a few places that I forgot Thanks! Omar Sandoval (5): xfstests: create swap group generic

[PATCH v2 4/5] generic: add test for truncate/fpunch of an active swapfile

2018-05-16 Thread Omar Sandoval
From: Omar Sandoval These should not be allowed. Signed-off-by: Omar Sandoval --- tests/generic/489 | 73 +++ tests/generic/489.out | 8 + tests/generic/group | 1 + 3 files changed, 82 insertions(+) create mode 100755 tests/generic/489

[PATCH v2 5/5] generic: test invalid swap file activation

2018-05-16 Thread Omar Sandoval
From: Omar Sandoval Swap files cannot have holes, and they must at least two pages. swapon(8) and mkswap(8) have stricter restrictions, so add versions of those commands without any restrictions. Signed-off-by: Omar Sandoval --- .gitignore| 2 ++ src/Makefile | 2

[PATCH v2 3/5] generic: add test for dedupe on an active swapfile

2018-05-16 Thread Omar Sandoval
From: Omar Sandoval Similar to generic/356 that makes sure we can't reflink an active swapfile. Signed-off-by: Omar Sandoval --- tests/generic/488 | 76 +++ tests/generic/488.out | 7 tests/generic/group | 1 + 3 files change

Re: [RFC PATCH 5/6] btrfs: add send_stream_version attribute to sysfs

2018-05-16 Thread Omar Sandoval
w); > + > +static struct attribute *btrfs_send_attrs[] = { > + BTRFS_ATTR_PTR(, stream_version), > + NULL > +}; > + > +static const struct attribute_group btrfs_send_attr_group = { > + .name = "send", > + .attrs = btrfs_send_attrs, > +}; >

Re: [RFC PATCH 6/6] btrfs: add chattr support for send/receive

2018-05-16 Thread Omar Sandoval
On Tue, May 08, 2018 at 10:06:51PM -0400, Howard McLauchlan wrote: > From: Howard McLauchlan > > Presently btrfs send/receive does not propagate inode attribute flags; > all chattr operations are effectively discarded upon transmission. > > This patch adds kernel support for inode attribute flag

Re: [RFC PATCH 1/6] btrfs: send, bump stream version

2018-05-16 Thread Omar Sandoval
IMES, but it includes OTIME too. */ I've had feature requests for the ability to send compressed data without decompressing it on the send side or the receive side. Could you add a placeholder BTRFS_SEND_C_WRITE_COMPRESSED here? This way when I get around to implementing it I won't have to do a v3. Other than that, Reviewed-by: Omar Sandoval -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs: test ENOSPC caused by many orphan items

2018-05-15 Thread Omar Sandoval
On Wed, May 16, 2018 at 09:48:58AM +0800, Eryu Guan wrote: > On Wed, May 09, 2018 at 11:21:55PM -0700, Omar Sandoval wrote: > > From: Omar Sandoval > > > > Btrfs has a bug where we can prematurely ENOSPC if we have lots of > > orphaned files, i.e., deleted files which

Re: Clarification needed about libbtrfs & libbtrfsutil

2018-05-14 Thread Omar Sandoval
On Mon, May 14, 2018 at 09:40:19AM +0100, Dimitri John Ledkov wrote: > Are both of these meant to be public libraries, installed on the user > systems, and available in .so variant as well for 3rd party > development and public dynamic linking? > > Or are these private internal libraries, which ar

Re: [PATCH 1/3] fs: add initial bh_result->b_private value to __blockdev_direct_IO()

2018-05-11 Thread Omar Sandoval
On Fri, May 11, 2018 at 09:32:28PM +0100, Al Viro wrote: > On Fri, May 11, 2018 at 01:30:01PM -0700, Omar Sandoval wrote: > > On Fri, May 11, 2018 at 09:05:38PM +0100, Al Viro wrote: > > > On Thu, May 10, 2018 at 11:30:10PM -0700, Omar Sandoval wrote: > > > > do_b

Re: [PATCH 1/3] fs: add initial bh_result->b_private value to __blockdev_direct_IO()

2018-05-11 Thread Omar Sandoval
On Fri, May 11, 2018 at 09:05:38PM +0100, Al Viro wrote: > On Thu, May 10, 2018 at 11:30:10PM -0700, Omar Sandoval wrote: > > do_blockdev_direct_IO(struct kiocb *iocb, struct inode *inode, > > struct block_device *bdev, struct iov_iter *iter, > >

[PATCH v4 06/12] Btrfs: delete dead code in btrfs_orphan_commit_root()

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval btrfs_orphan_commit_root() tries to delete an orphan item for a subvolume in the tree root, but we don't actually insert that item in the first place. See commit 0a0d4415e338 ("Btrfs: delete dead code in btrfs_orphan_add()"). We can get rid of it. Reviewed-

[PATCH v4 09/12] Btrfs: fix ENOSPC caused by orphan items reservations

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval Currently, we keep space reserved for all inode orphan items until the inode is evicted (i.e., all references to it are dropped). We hit an issue where an application would keep a bunch of deleted files open (by design) and thus keep a large amount of space reserved, causing

[PATCH v4 12/12] Btrfs: reserve space for O_TMPFILE orphan item deletion

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval btrfs_link() calls btrfs_orphan_del() if it's linking an O_TMPFILE but it doesn't reserve space to do so. Even before the removal of the orphan_block_rsv it wasn't using it. Fixes: ef3b9af50bfa ("Btrfs: implement inode_operations callback tmpfile") R

[PATCH v4 10/12] Btrfs: get rid of unused orphan infrastructure

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval Now that we don't keep long-standing reservations for orphan items, root->orphan_block_rsv isn't used. We can git rid of it, along with: - root->orphan_lock, which was used to protect root->orphan_block_rsv - root->orphan_inodes, which was used

[PATCH v4 07/12] Btrfs: don't return ino to ino cache if inode item removal fails

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval In btrfs_evict_inode(), if btrfs_truncate_inode_items() fails, the inode item will still be in the tree but we still return the ino to the ino cache. That will blow up later when someone tries to allocate that ino, so don't return it to the cache. Fixes: 581bb050941b (&

[PATCH v4 05/12] Btrfs: get rid of BTRFS_INODE_HAS_ORPHAN_ITEM

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval Now that we don't add orphan items for truncate, there can't be races on adding or deleting an orphan item, so this bit is unnecessary. Reviewed-by: Nikolay Borisov Signed-off-by: Omar Sandoval --- fs/btrfs/btrfs_inode.h | 1 - fs/btrfs/inode.c

[PATCH v4 03/12] Btrfs: don't BUG_ON() in btrfs_truncate_inode_items()

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval btrfs_free_extent() can fail because of ENOMEM. There's no reason to panic here, we can just abort the transaction. Fixes: f4b9aa8d3b87 ("btrfs_truncate") Reviewed-by: Nikolay Borisov Signed-off-by: Omar Sandoval --- fs/btrfs/inode.c | 5 - 1

[PATCH v4 00/12] Btrfs: orphan and truncate fixes

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval Hi, This is the fourth (and hopefully final) version of the orphan item early ENOSPC and related fixes. Changes since v3: - Changed another stale comment in patch 1 - Moved BTRFS_INODE_ORPHAN_META_RESERVED flag removal to patch 10 instead of patch 9 - Moved inode runtime

[PATCH v4 01/12] Btrfs: update stale comments referencing vmtruncate()

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval Commit a41ad394a03b ("Btrfs: convert to the new truncate sequence") changed btrfs_setsize() to call truncate_setsize() instead of vmtruncate() but didn't update the comment above it. truncate_setsize() never fails (the IS_SWAPFILE() check happens elsewhere)

[PATCH v4 11/12] Btrfs: renumber BTRFS_INODE_ runtime flags

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval We got rid of BTRFS_INODE_HAS_ORPHAN_ITEM and BTRFS_INODE_ORPHAN_META_RESERVED, so we can renumber the flags to make them consecutive again. Signed-off-by: Omar Sandoval --- fs/btrfs/btrfs_inode.h | 16 1 file changed, 8 insertions(+), 8 deletions(-) diff

[PATCH v4 08/12] Btrfs: refactor btrfs_evict_inode() reserve refill dance

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval The truncate loop in btrfs_evict_inode() does two things at once: - It refills the temporary block reserve, potentially stealing from the global reserve or committing - It calls btrfs_truncate_inode_items() The tangle of continues hides the fact that these two steps are

[PATCH v4 02/12] Btrfs: fix error handling in btrfs_truncate_inode_items()

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval btrfs_truncate_inode_items() uses two variables for error handling, ret and err. These are not handled consistently, leading to a couple of bugs. - Errors from btrfs_del_items() are handled but not propagated to the caller - If btrfs_run_delayed_refs() fails and aborts the

[PATCH v4 04/12] Btrfs: stop creating orphan items for truncate

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval Currently, we insert an orphan item during a truncate so that if there's a crash, we don't leak extents past the on-disk i_size. However, since commit 7f4f6e0a3f6d ("Btrfs: only update disk_i_size as we remove extents"), we keep disk_i_size in sync with th

Re: [PATCH] btrfs: use kvzalloc for EXTENT_SAME temporary data

2018-05-11 Thread Omar Sandoval
On Fri, May 11, 2018 at 06:49:16PM +0200, David Sterba wrote: > On Fri, May 11, 2018 at 05:25:50PM +0100, Filipe Manana wrote: > > On Fri, May 11, 2018 at 4:57 PM, David Sterba wrote: > > > The dedupe range is 16 MiB, with 4KiB pages and 8 byte pointers, the > > > arrays can be 32KiB large. To avo

Re: [PATCH 0/3] Btrfs: stop abusing current->journal_info for direct I/O

2018-05-11 Thread Omar Sandoval
On Fri, May 11, 2018 at 12:53:36PM +0300, Nikolay Borisov wrote: > > > On 11.05.2018 09:30, Omar Sandoval wrote: > > From: Omar Sandoval > > > > Hi, everyone, > > > > Btrfs currently abuses current->journal_info in btrfs_direct_IO() in > > or

Re: [PATCH v3 01/11] Btrfs: remove stale comment referencing vmtruncate()

2018-05-11 Thread Omar Sandoval
On Fri, May 11, 2018 at 12:19:43PM +0200, David Sterba wrote: > On Fri, May 11, 2018 at 12:56:06AM -0700, Omar Sandoval wrote: > > From: Omar Sandoval > > > > Commit a41ad394a03b ("Btrfs: convert to the new truncate sequence") > > changed vmtruncate() to tru

Re: [PATCH v3 05/11] Btrfs: get rid of BTRFS_INODE_HAS_ORPHAN_ITEM

2018-05-11 Thread Omar Sandoval
On Fri, May 11, 2018 at 06:51:30PM +0200, David Sterba wrote: > On Fri, May 11, 2018 at 12:10:38PM -0400, Josef Bacik wrote: > > I told him to do this, these flags aren't exposed anywhere are they? > > They are in-kernel specific stuff, please tell me we aren't exposing > > these via sysfs? > > No

[PATCH v3 09/11] Btrfs: fix ENOSPC caused by orphan items reservations

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval Currently, we keep space reserved for all inode orphan items until the inode is evicted (i.e., all references to it are dropped). We hit an issue where an application would keep a bunch of deleted files open (by design) and thus keep a large amount of space reserved, causing

[PATCH v3 04/11] Btrfs: stop creating orphan items for truncate

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval Currently, we insert an orphan item during a truncate so that if there's a crash, we don't leak extents past the on-disk i_size. However, since commit 7f4f6e0a3f6d ("Btrfs: only update disk_i_size as we remove extents"), we keep disk_i_size in sync with th

[PATCH v3 06/11] Btrfs: delete dead code in btrfs_orphan_commit_root()

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval btrfs_orphan_commit_root() tries to delete an orphan item for a subvolume in the tree root, but we don't actually insert that item in the first place. See commit 0a0d4415e338 ("Btrfs: delete dead code in btrfs_orphan_add()"). We can get rid of it. Sign

[PATCH v3 10/11] Btrfs: get rid of unused orphan infrastructure

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval Now that we don't keep long-standing reservations for orphan items, root->orphan_block_rsv isn't used. We can git rid of it, along with root->orphan_lock, which was used to protect it, root->orphan_inodes, which was used as a refcount for it, and btrfs

[PATCH v3 07/11] Btrfs: don't return ino to ino cache if inode item removal fails

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval In btrfs_evict_inode(), if btrfs_truncate_inode_items() fails, the inode item will still be in the tree but we still return the ino to the ino cache. That will blow up later when someone tries to allocate that ino, so don't return it to the cache. Fixes: 581bb050941b (&

[PATCH v3 08/11] Btrfs: refactor btrfs_evict_inode() reserve refill dance

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval The truncate loop in btrfs_evict_inode() does two things at once: - It refills the temporary block reserve, potentially stealing from the global reserve or committing - It calls btrfs_truncate_inode_items() The tangle of continues hides the fact that these two steps are

[PATCH v3 11/11] Btrfs: reserve space for O_TMPFILE orphan item deletion

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval btrfs_link() calls btrfs_orphan_del() if it's linking an O_TMPFILE but it doesn't reserve space to do so. Even before the removal of the orphan_block_rsv it wasn't using it. Fixes: ef3b9af50bfa ("Btrfs: implement inode_operations callback tmpfile") R

[PATCH v3 05/11] Btrfs: get rid of BTRFS_INODE_HAS_ORPHAN_ITEM

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval Now that we don't add orphan items for truncate, there can't be races on adding or deleting an orphan item, so this bit is unnecessary. Signed-off-by: Omar Sandoval --- fs/btrfs/btrfs_inode.h | 13 fs/btrfs/inode.c

[PATCH v3 03/11] Btrfs: don't BUG_ON() in btrfs_truncate_inode_items()

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval btrfs_free_extent() can fail because of ENOMEM. There's no reason to panic here, we can just abort the transaction. Fixes: f4b9aa8d3b87 ("btrfs_truncate") Reviewed-by: Nikolay Borisov Signed-off-by: Omar Sandoval --- fs/btrfs/inode.c | 5 - 1

[PATCH v3 02/11] Btrfs: fix error handling in btrfs_truncate_inode_items()

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval btrfs_truncate_inode_items() uses two variables for error handling, ret and err. These are not handled consistently, leading to a couple of bugs. - Errors from btrfs_del_items() are handled but not propagated to the caller - If btrfs_run_delayed_refs() fails and aborts the

[PATCH v3 01/11] Btrfs: remove stale comment referencing vmtruncate()

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval Commit a41ad394a03b ("Btrfs: convert to the new truncate sequence") changed vmtruncate() to truncate_setsize() but didn't update the comment above it. truncate_setsize() never fails (the IS_SWAPFILE() check happens elsewhere), so remove the comment. Sign

[PATCH v3 00/11] Btrfs: orphan and truncate fixes

2018-05-11 Thread Omar Sandoval
From: Omar Sandoval Hi, This is v3 of the fixes for the orphan item early ENOSPC issue we hit at Facebook. The big change is that I now also got rid of BTRFS_INODE_HAS_ORPHAN_ITEM (thanks, Nikolay) and shuffled the patches around so there is less churn. Changes since v2: - Add patch 5 to get

Re: [PATCH v2 09/12] Btrfs: get rid of root->orphan_block_rsv and root->orphan_lock

2018-05-11 Thread Omar Sandoval
On Thu, May 10, 2018 at 11:48:28PM -0700, Omar Sandoval wrote: > On Fri, May 11, 2018 at 09:44:36AM +0300, Nikolay Borisov wrote: > > > > > > On 11.05.2018 03:11, Omar Sandoval wrote: > > > From: Omar Sandoval > > > > > > Now that we don&#x

Re: [PATCH v2 10/12] Btrfs: get rid of btrfs_orphan_commit_root() and root->orphan_inodes

2018-05-11 Thread Omar Sandoval
On Fri, May 11, 2018 at 10:01:34AM +0300, Nikolay Borisov wrote: > > > On 11.05.2018 03:11, Omar Sandoval wrote: > > From: Omar Sandoval > > > > btrfs_orphan_commit_root() tries to delete an orphan item for a > > subvolume in the tree root, but we don'

Re: [PATCH v2 08/12] Btrfs: fix ENOSPC caused by orphan items reservations

2018-05-10 Thread Omar Sandoval
On Fri, May 11, 2018 at 09:38:15AM +0300, Nikolay Borisov wrote: > > > On 11.05.2018 03:11, Omar Sandoval wrote: > > From: Omar Sandoval > > > > Currently, we keep space reserved for all inode orphan items until the > > inode is evicted (i.e., all referen

Re: [PATCH v2 09/12] Btrfs: get rid of root->orphan_block_rsv and root->orphan_lock

2018-05-10 Thread Omar Sandoval
On Fri, May 11, 2018 at 09:44:36AM +0300, Nikolay Borisov wrote: > > > On 11.05.2018 03:11, Omar Sandoval wrote: > > From: Omar Sandoval > > > > Now that we don't keep long-standing reservations for orphan items, > > root->orphan_block_rsv isn&

[PATCH 1/3] fs: add initial bh_result->b_private value to __blockdev_direct_IO()

2018-05-10 Thread Omar Sandoval
From: Omar Sandoval Btrfs abuses current->journal_info in btrfs_direct_IO() in order to pass around some state to get_block() and submit_io(). The generic DIO code already provides bh_result->b_private as a way to pass data between calls to get_block() and end_io(), but it is always initi

[PATCH 0/3] Btrfs: stop abusing current->journal_info for direct I/O

2018-05-10 Thread Omar Sandoval
From: Omar Sandoval Hi, everyone, Btrfs currently abuses current->journal_info in btrfs_direct_IO() in order to pass around some state to get_block() and submit_io(). This hack is ugly and unnecessary because the data we pass around is only used in one call frame. Robbie Ko also pointed out

[PATCH 3/3] Btrfs: stop abusing current->journal_info in btrfs_direct_IO()

2018-05-10 Thread Omar Sandoval
From: Omar Sandoval Now that we can pass around the struct btrfs_dio_data through the different callbacks generically, we don't need to shove it in current->journal_info. Signed-off-by: Omar Sandoval --- fs/btrfs/inode.c | 33 +++-- 1 file changed, 7 in

[PATCH 2/3] fs: add private argument to dio_submit_t

2018-05-10 Thread Omar Sandoval
From: Omar Sandoval get_block() already has access to this through bh_result->b_private, and it gets passed to end_io() as private, so do the same for submit_io(). Along with the previous change, this will allow us to get rid Btrfs' current->journal_info abuse in btrfs_direct_IO().

[PATCH v2 02/12] Btrfs: fix error handling in btrfs_truncate_inode_items()

2018-05-10 Thread Omar Sandoval
From: Omar Sandoval btrfs_truncate_inode_items() uses two variables for error handling, ret and err. These are not handled consistently, leading to a couple of bugs. - Errors from btrfs_del_items() are handled but not propagated to the caller - If btrfs_run_delayed_refs() fails and aborts the

[PATCH v2 04/12] Btrfs: stop creating orphan items for truncate

2018-05-10 Thread Omar Sandoval
From: Omar Sandoval Currently, we insert an orphan item during a truncate so that if there's a crash, we don't leak extents past the on-disk i_size. However, since commit 7f4f6e0a3f6d ("Btrfs: only update disk_i_size as we remove extents"), we keep disk_i_size in sync with th

[PATCH v2 06/12] Btrfs: don't return ino to ino cache if inode item removal fails

2018-05-10 Thread Omar Sandoval
From: Omar Sandoval In btrfs_evict_inode(), if btrfs_truncate_inode_items() fails, the inode item will still be in the tree but we still return the ino to the ino cache. That will blow up later when someone tries to allocate that ino, so don't return it to the cache. Fixes: 581bb050941b (&

[PATCH v2 09/12] Btrfs: get rid of root->orphan_block_rsv and root->orphan_lock

2018-05-10 Thread Omar Sandoval
From: Omar Sandoval Now that we don't keep long-standing reservations for orphan items, root->orphan_block_rsv isn't used. We can git rid of it and root->orphan_lock, which was used to protect it. Signed-off-by: Omar Sandoval --- fs/btrfs/ctree.h | 5 - fs/btrfs/di

[PATCH v2 11/12] Btrfs: simplify error handling in btrfs_evict_inode()

2018-05-10 Thread Omar Sandoval
From: Omar Sandoval We have the same error handling code duplicated all over the place. Put it all in one place, and while we're here, get rid of the weird btrfs_orphan_del() trans == NULL case and just clear the orphan item bit directly since that's all it does anymore. Signed-of

[PATCH v2 10/12] Btrfs: get rid of btrfs_orphan_commit_root() and root->orphan_inodes

2018-05-10 Thread Omar Sandoval
From: Omar Sandoval btrfs_orphan_commit_root() tries to delete an orphan item for a subvolume in the tree root, but we don't actually insert that item in the first place. See commit 0a0d4415e338 ("Btrfs: delete dead code in btrfs_orphan_add()"). We can get rid of it. root->

[PATCH v2 05/12] Btrfs: don't release reserve or decrement orphan count if orphan item already existed

2018-05-10 Thread Omar Sandoval
From: Omar Sandoval Currently, if btrfs_insert_orphan_item() fails, we release the reserved space and decrement root->orphan_inodes. However, we also ignore -EEXIST errors, so we still want the space reservation for when we delete the item, and we still need to count the orphan because we

[PATCH v2 12/12] Btrfs: reserve space for O_TMPFILE orphan item deletion

2018-05-10 Thread Omar Sandoval
From: Omar Sandoval btrfs_link() calls btrfs_orphan_del() if it's linking an O_TMPFILE but it doesn't reserve space to do so. Even before the removal of the orphan_block_rsv it wasn't using it. Fixes: ef3b9af50bfa ("Btrfs: implement inode_operations callback tmpfile") R

[PATCH v2 08/12] Btrfs: fix ENOSPC caused by orphan items reservations

2018-05-10 Thread Omar Sandoval
From: Omar Sandoval Currently, we keep space reserved for all inode orphan items until the inode is evicted (i.e., all references to it are dropped). We hit an issue where an application would keep a bunch of deleted files open (by design) and thus keep a large amount of space reserved, causing

[PATCH v2 00/12] Btrfs: orphan and truncate fixes

2018-05-10 Thread Omar Sandoval
From: Omar Sandoval Hi, This is v2 of the fixes for the orphan item early ENOSPC issue we hit at Facebook. Changes since v1: - Added two extra cleanups, patches 10 and 11 - Added a forgotten clear of the orphan bit in patch 8 - Reworded titles of patches 6 and 9 - Added people's reviewe

[PATCH v2 03/12] Btrfs: don't BUG_ON() in btrfs_truncate_inode_items()

2018-05-10 Thread Omar Sandoval
From: Omar Sandoval btrfs_free_extent() can fail because of ENOMEM. There's no reason to panic here, we can just abort the transaction. Fixes: f4b9aa8d3b87 ("btrfs_truncate") Reviewed-by: Nikolay Borisov Signed-off-by: Omar Sandoval --- fs/btrfs/inode.c | 5 - 1

[PATCH v2 01/12] Btrfs: remove stale comment referencing vmtruncate()

2018-05-10 Thread Omar Sandoval
From: Omar Sandoval Commit a41ad394a03b ("Btrfs: convert to the new truncate sequence") changed vmtruncate() to truncate_setsize() but didn't update the comment above it. truncate_setsize() never fails (the IS_SWAPFILE() check happens elsewhere), so remove the comment. Sign

[PATCH v2 07/12] Btrfs: refactor btrfs_evict_inode() reserve refill dance

2018-05-10 Thread Omar Sandoval
From: Omar Sandoval The truncate loop in btrfs_evict_inode() does two things at once: - It refills the temporary block reserve, potentially stealing from the global reserve or committing - It calls btrfs_truncate_inode_items() The tangle of continues hides the fact that these two steps are

Re: [PATCH 08/10] Btrfs: fix ENOSPC caused by orphan items reservations

2018-05-10 Thread Omar Sandoval
On Thu, May 10, 2018 at 11:50:11AM +0300, Nikolay Borisov wrote: > > > On 10.05.2018 09:21, Omar Sandoval wrote: > > From: Omar Sandoval > > > > Currently, we keep space reserved for all inode orphan items until the > > inode is evicted (i.e., all referen

Re: [PATCH 06/10] Btrfs: don't return ino if inode item removal fails

2018-05-10 Thread Omar Sandoval
On Thu, May 10, 2018 at 11:29:18AM +0300, Nikolay Borisov wrote: > > > On 10.05.2018 09:21, Omar Sandoval wrote: > > From: Omar Sandoval > > > > In btrfs_evict_inode(), if btrfs_truncate_inode_items() fails, the inode > > item will still be in the tree but w

[PATCH 07/10] Btrfs: refactor btrfs_evict_inode() reserve refill dance

2018-05-09 Thread Omar Sandoval
From: Omar Sandoval The truncate loop in btrfs_evict_inode() does two things at once: - It refills the temporary block reserve, potentially stealing from the global reserve or committing - It calls btrfs_truncate_inode_items() The tangle of continues hides the fact that these two steps are

[PATCH] btrfs: test ENOSPC caused by many orphan items

2018-05-09 Thread Omar Sandoval
From: Omar Sandoval Btrfs has a bug where we can prematurely ENOSPC if we have lots of orphaned files, i.e., deleted files which are still open. Add a test which repeatedly creates and deletes a file while keeping all of the file descriptors open. This should succeed but doesn't on Btrfs wi

[PATCH 04/10] Btrfs: stop creating orphan items for truncate

2018-05-09 Thread Omar Sandoval
From: Omar Sandoval Currently, we insert an orphan item during a truncate so that if there's a crash, we don't leak extents past the on-disk i_size. However, since commit 7f4f6e0a3f6d ("Btrfs: only update disk_i_size as we remove extents"), we keep disk_i_size in sync with th

[PATCH 05/10] Btrfs: don't release reserve or decrement orphan count if orphan item already existed

2018-05-09 Thread Omar Sandoval
From: Omar Sandoval Currently, if btrfs_insert_orphan_item() fails, we release the reserved space and decrement root->orphan_inodes. However, we also ignore -EEXIST errors, so we still want the space reservation for when we delete the item, and we still need to count the orphan because we

[PATCH 03/10] Btrfs: don't BUG_ON() in btrfs_truncate_inode_items()

2018-05-09 Thread Omar Sandoval
From: Omar Sandoval btrfs_free_extent() can fail because of ENOMEM. There's no reason to panic here, we can just abort the transaction. Fixes: f4b9aa8d3b87 ("btrfs_truncate") Signed-off-by: Omar Sandoval --- fs/btrfs/inode.c | 5 - 1 file changed, 4 insertions(+), 1 de

[PATCH 06/10] Btrfs: don't return ino if inode item removal fails

2018-05-09 Thread Omar Sandoval
From: Omar Sandoval In btrfs_evict_inode(), if btrfs_truncate_inode_items() fails, the inode item will still be in the tree but we still return the ino to the ino cache. That will blow up later when someone tries to allocate that ino, so don't return it to the cache. Fixes: 581bb050941b (&

[PATCH 09/10] Btrfs: get rid of unused orphan infrastructure

2018-05-09 Thread Omar Sandoval
From: Omar Sandoval Now that we don't keep long-standing reservations for orphan items, we can get rid of root->orphan_lock and root->orphan_block_rsv, as well as some related functions. Signed-off-by: Omar Sandoval --- fs/btrfs/ctree.h | 5 - fs/btrfs/disk-io

[PATCH 10/10] Btrfs: reserve space for O_TMPFILE orphan item deletion

2018-05-09 Thread Omar Sandoval
From: Omar Sandoval btrfs_link() calls btrfs_orphan_del() if it's linking an O_TMPFILE but it doesn't reserve space to do so. Even before the removal of the orphan_block_rsv it wasn't using it. Fixes: ef3b9af50bfa ("Btrfs: implement inode_operations callback tmpfile&q

[PATCH 08/10] Btrfs: fix ENOSPC caused by orphan items reservations

2018-05-09 Thread Omar Sandoval
From: Omar Sandoval Currently, we keep space reserved for all inode orphan items until the inode is evicted (i.e., all references to it are dropped). We hit an issue where an application would keep a bunch of deleted files open (by design) and thus keep a large amount of space reserved, causing

[PATCH 02/10] Btrfs: fix error handling in btrfs_truncate_inode_items()

2018-05-09 Thread Omar Sandoval
From: Omar Sandoval btrfs_truncate_inode_items() uses two variables for error handling, ret and err. These are not handled consistently, leading to a couple of bugs. - Errors from btrfs_del_items() are handled but not propagated to the caller - If btrfs_run_delayed_refs() fails and aborts the

[PATCH 01/10] Btrfs: remove stale comment referencing vmtruncate()

2018-05-09 Thread Omar Sandoval
From: Omar Sandoval Commit a41ad394a03b ("Btrfs: convert to the new truncate sequence") changed vmtruncate() to truncate_setsize() but didn't update the comment above it. truncate_setsize() never fails (the IS_SWAPFILE() check happens elsewhere), so remove the comment. Sign

[PATCH 00/10] Btrfs: orphan and truncate fixes

2018-05-09 Thread Omar Sandoval
From: Omar Sandoval Hi, At Facebook we hit an early ENOSPC issue which we tracked down to the reservations for orphan items of deleted-but-still-open files. The primary function of this series is to fix that bug, but I ended up uncovering a pile of other issues in the process, most notably that

Re: [PATCH] btrfs: fix invalid memory access with journal_info

2018-05-09 Thread Omar Sandoval
On Wed, May 09, 2018 at 06:35:25PM +0800, robbieko wrote: > From: Robbie Ko > > When send process requires memory allocation, shrinker may be triggered > due to insufficient memory. > Then evict_inode gets called when inode is freed, and this function > may need to start transaction. > However, t

[PATCH v2 2/2] common/rc: raise mixed mode threshold to 1GB

2018-04-11 Thread Omar Sandoval
From: Omar Sandoval generic/427 creates a 256 MB filesystem and then writes a 200 MB file, which fails on Btrfs if mixed mode is not enabled. Raise the threshold to 1GB, which is where we typically recommend mixed mode. Signed-off-by: Omar Sandoval --- common/rc | 2 +- 1 file changed, 1

[PATCH v2 1/2] aio-dio-eof-race: handle aio pwrite errors and short reads

2018-04-11 Thread Omar Sandoval
From: Omar Sandoval generic/427 fails on Btrfs with a cryptic "pread: Success" message. This is because an aio pwrite fails with ENOSPC, so the file isn't as long as we expect it to be. Make sure we check the result of the aio writes and also print a more explicit message for sho

<    1   2   3   4   5   6   7   8   9   10   >