Re: [PATCH 06/17] xfs: run xfs_repair at the end of each test

2016-08-01 Thread Darrick J. Wong
On Sun, Jul 31, 2016 at 11:27:19PM -0700, Christoph Hellwig wrote: > On Thu, Jul 21, 2016 at 04:46:54PM -0700, Darrick J. Wong wrote: > > Run xfs_repair twice at the end of each test -- once to rebuild > > the btree indices, and again with -n to check the rebuild work. > >

Re: FIDEDUPERANGE with src_length == 0

2016-08-01 Thread Darrick J. Wong
On Thu, Jul 14, 2016 at 11:16:47AM -0700, Omar Sandoval wrote: > On Thu, Jul 14, 2016 at 02:12:58PM -0400, Chris Mason wrote: > > > > > > On 07/14/2016 02:06 PM, Darrick J. Wong wrote: > > > On Wed, Jul 13, 2016 at 03:19:38PM +0200, David Sterba wrote: > > &g

[PATCH] xfs: test attr_list_by_handle cursor iteration

2016-08-02 Thread Darrick J. Wong
Apparently the XFS attr_list_by_handle ioctl has never actually copied the cursor contents back to user space, which means that iteration has never worked. Add a test case for this and see the patch "xfs: in _attrlist_by_handle, copy the cursor back to userspace". Signed-off-by: Darri

Re: [PATCH 06/17] xfs: run xfs_repair at the end of each test

2016-08-03 Thread Darrick J. Wong
On Wed, Aug 03, 2016 at 05:15:42PM +0800, Eryu Guan wrote: > On Thu, Jul 21, 2016 at 04:46:54PM -0700, Darrick J. Wong wrote: > > Run xfs_repair twice at the end of each test -- once to rebuild > > the btree indices, and again with -n to check the rebuild work. > > See

Re: [PATCH 07/17] xfs/128: cycle_mount the scratch device, not the test device

2016-08-03 Thread Darrick J. Wong
On Wed, Aug 03, 2016 at 05:37:55PM +0800, Eryu Guan wrote: > On Thu, Jul 21, 2016 at 04:47:01PM -0700, Darrick J. Wong wrote: > > This test uses the scratch device, so cycle that, not the test dev. > > This is also a xfs_fsr test, so put it in the fsr group. > > > > Sig

Re: [PATCH 09/17] common/dmerror: fix mount option issues

2016-08-03 Thread Darrick J. Wong
On Wed, Aug 03, 2016 at 06:38:00PM +0800, Eryu Guan wrote: > On Thu, Jul 21, 2016 at 04:47:13PM -0700, Darrick J. Wong wrote: > > Calling _mount doesn't work when we want to add mount options > > such as realtime devices. Since it's just a normal scratch device > &

[PATCH v2 06/17] xfs: optionally test xfs_repair index rebuilding at the end of each test

2016-08-03 Thread Darrick J. Wong
Run xfs_repair twice more at the end of each test -- once to rebuild the btree indices, and again with -n to check the rebuild work. This is in addition to the regular dry-run spot check. v2: Reorder the repair invocations and label them better in the output. Signed-off-by: Darrick J. Wong

[PATCH v2 07/17] xfs/128: cycle_mount the scratch device, not the test device

2016-08-03 Thread Darrick J. Wong
This test uses the scratch device, so cycle that, not the test dev. This is also a xfs_fsr test, so put it in the fsr group. Signed-off-by: Darrick J. Wong --- tests/xfs/128 |6 +++--- tests/xfs/group |2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/xfs/128

[PATCH v2 09/17] common/dmerror: fix mount option issues

2016-08-03 Thread Darrick J. Wong
Signed-off-by: Darrick J. Wong --- common/dmerror |8 ++-- 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/common/dmerror b/common/dmerror index 5ad9994..d05c4c8 100644 --- a/common/dmerror +++ b/common/dmerror @@ -41,14 +41,10 @@ _dmerror_init() DMERROR_TABLE="

[PATCH v2] xfs: test attr_list_by_handle cursor iteration

2016-08-03 Thread Darrick J. Wong
E for less confusion, fix build errors on RHEL6. Signed-off-by: Darrick J. Wong --- .gitignore|1 src/Makefile |3 - src/attr-list-by-handle-cursor-test.c | 185 + tests/xfs/700

Re: [PATCH] xfs: test attr_list_by_handle cursor iteration

2016-08-03 Thread Darrick J. Wong
On Wed, Aug 03, 2016 at 12:47:22PM +0800, Eryu Guan wrote: > On Tue, Aug 02, 2016 at 04:52:51PM -0700, Darrick J. Wong wrote: > > Apparently the XFS attr_list_by_handle ioctl has never actually copied > > the cursor contents back to user space, which means that iteration has > &g

Re: [PATCH v2 09/17] common/dmerror: fix mount option issues

2016-08-03 Thread Darrick J. Wong
On Wed, Aug 03, 2016 at 03:50:04PM -0700, Darrick J. Wong wrote: > Calling _mount doesn't work when we want to add mount options > such as realtime devices. Since it's just a normal scratch device > mount except for the source device, just call _scratch_mount with > SCRATCH_

[PATCH v3 09/17] common/dmerror: fix mount option issues

2016-08-03 Thread Darrick J. Wong
Signed-off-by: Darrick J. Wong --- common/dmerror |9 +++-- 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/common/dmerror b/common/dmerror index 5ad9994..d46c5d0 100644 --- a/common/dmerror +++ b/common/dmerror @@ -41,14 +41,11 @@ _dmerror_init() DMERROR_TABLE="

[PATCH v3] xfs: test attr_list_by_handle cursor iteration

2016-08-04 Thread Darrick J. Wong
E for less confusion, fix build errors on RHEL6. v3: Use path_to_handle instead of bulkstat. Signed-off-by: Darrick J. Wong --- .gitignore|1 src/Makefile |3 + src/attr-list-by-handle-cursor-test.c | 118 + tes

[PATCH] generic: test accurate shared extent reporting

2016-08-05 Thread Darrick J. Wong
Ensure that we can create a file with a single extent, reflink two blocks out of the middle of that extent, and the resulting fiemap reports two shared extents, instead of lazily reporting the entire huge extent as shared. Signed-off-by: Darrick J. Wong --- tests/generic/929 | 89

Re: [PATCH] generic: test accurate shared extent reporting

2016-08-05 Thread Darrick J. Wong
On Fri, Aug 05, 2016 at 03:46:07PM +0800, Eryu Guan wrote: > On Fri, Aug 05, 2016 at 12:21:47AM -0700, Darrick J. Wong wrote: > > Ensure that we can create a file with a single extent, reflink two > > blocks out of the middle of that extent, and the resulting fiemap > > repor

[PATCH v2] generic: test accurate shared extent reporting

2016-08-05 Thread Darrick J. Wong
Ensure that we can create a file with a single extent, reflink two blocks out of the middle of that extent, and the resulting fiemap reports two shared extents, instead of lazily reporting the entire huge extent as shared. v2: add _supported_fs Signed-off-by: Darrick J. Wong --- tests/generic

Re: [PATCH 12/17] reflink: test cross-mountpoint reflink and dedupe

2016-08-08 Thread Darrick J. Wong
On Mon, Aug 08, 2016 at 03:56:01PM +0800, Eryu Guan wrote: > On Thu, Jul 21, 2016 at 04:47:32PM -0700, Darrick J. Wong wrote: > > Test sharing blocks via reflink and dedupe between two different > > mountpoints of the same filesystem. This shouldn't work, since > > we do

Re: [PATCH 13/17] xfs: test swapext with reflink

2016-08-08 Thread Darrick J. Wong
On Mon, Aug 08, 2016 at 04:13:59PM +0800, Eryu Guan wrote: > On Thu, Jul 21, 2016 at 04:47:38PM -0700, Darrick J. Wong wrote: > > Add a few tests to stress the new swapext code for reflink and rmap. > > > > Signed-off-by: Darrick J. Wong > > --- &g

Re: [PATCH 14/17] xfs: more rmapbt tests

2016-08-08 Thread Darrick J. Wong
On Mon, Aug 08, 2016 at 04:48:14PM +0800, Eryu Guan wrote: > On Thu, Jul 21, 2016 at 04:47:45PM -0700, Darrick J. Wong wrote: > > More tests for the reverse mapping functionality. > > > > Signed-off-by: Darrick J. Wong > > --- &g

Re: [PATCH 15/17] xfs: scrub fs (if still mounted) at the end of the test

2016-08-08 Thread Darrick J. Wong
On Mon, Aug 08, 2016 at 04:52:39PM +0800, Eryu Guan wrote: > On Thu, Jul 21, 2016 at 04:47:51PM -0700, Darrick J. Wong wrote: > > Teach _check_xfs_filesystem to scrub mounted filesystems before > > unmounting and fscking them. This is mostly to test the online > > scrub to

Re: [PATCH 17/17] xfs: test realtime rmapbt code

2016-08-08 Thread Darrick J. Wong
On Mon, Aug 08, 2016 at 05:08:00PM +0800, Eryu Guan wrote: > On Thu, Jul 21, 2016 at 04:48:07PM -0700, Darrick J. Wong wrote: > > Test the realtime rmap btree code by exercising various IO patterns > > on realtime files. > > > > Signed-off-by: Darrick J. Wong

Re: [PATCH 13/17] xfs: test swapext with reflink

2016-08-19 Thread Darrick J. Wong
On Tue, Aug 09, 2016 at 04:29:43PM +1000, Dave Chinner wrote: > On Mon, Aug 08, 2016 at 10:41:32AM -0700, Darrick J. Wong wrote: > > On Mon, Aug 08, 2016 at 04:13:59PM +0800, Eryu Guan wrote: > > > On Thu, Jul 21, 2016 at 04:47:38PM -0700, Darrick J. Wong wrote: > > >

Re: Will Btrfs have an official command to "uncow" existing files?

2016-08-22 Thread Darrick J. Wong
[add Dave and Christoph to cc] On Mon, Aug 22, 2016 at 04:14:19PM -0400, Jeff Mahoney wrote: > On 8/21/16 2:59 PM, Tomokhov Alexander wrote: > > Btrfs wiki FAQ gives a link to example Python script: > > https://github.com/stsquad/scripts/blob/master/uncow.py > > > > But such a crucial and fundam

Re: Will Btrfs have an official command to "uncow" existing files?

2016-08-23 Thread Darrick J. Wong
On Tue, Aug 23, 2016 at 08:48:20PM -0400, Jeff Mahoney wrote: > On 8/22/16 7:06 PM, Darrick J. Wong wrote: > > [add Dave and Christoph to cc] > > > > On Mon, Aug 22, 2016 at 04:14:19PM -0400, Jeff Mahoney wrote: > >> On 8/21/16 2:59 PM, Tomokhov Alexander wrote: >

Re: Will Btrfs have an official command to "uncow" existing files?

2016-08-24 Thread Darrick J. Wong
On Wed, Aug 24, 2016 at 11:34:27AM -0700, Omar Sandoval wrote: > On Mon, Aug 22, 2016 at 08:43:18PM -0600, Chris Murphy wrote: > > On Mon, Aug 22, 2016 at 5:06 PM, Darrick J. Wong > > wrote: > > > [add Dave and Christoph to cc] > > > > > > On Mon, Aug

Re: [PATCH] generic/159: record error value correctly

2016-08-29 Thread Darrick J. Wong
On Tue, Aug 30, 2016 at 12:00:44PM +0800, Wang Xiaoguang wrote: > Hi, > > On 08/30/2016 11:59 AM, Eryu Guan wrote: > >On Tue, Aug 30, 2016 at 11:51:14AM +0800, Wang Xiaoguang wrote: > >>This case failed in btrfs, it's because when trying to open an immutable > >>file, vfs returns EPERM, not EACCES

Re: [PATCH 3/3] ioctl_xfs_ioc_getfsmap.2: document XFS_IOC_GETFSMAP ioctl

2016-08-30 Thread Darrick J. Wong
[add a few more relevant lists to cc] On Mon, Aug 29, 2016 at 03:34:11PM -0600, Andreas Dilger wrote: > On Aug 25, 2016, at 5:26 PM, Darrick J. Wong wrote: > > > > Document the new XFS_IOC_GETFSMAP ioctl that returns the physical > > layout of a (disk-based) filesystem. &

[PATCH v2 3/3] ioctl_getfsmap.2: document the GETFSMAP ioctl

2016-09-03 Thread Darrick J. Wong
Document the new GETFSMAP ioctl that returns the physical layout of a (disk-based) filesystem. Signed-off-by: Darrick J. Wong --- man2/ioctl_getfsmap.2 | 313 + 1 file changed, 313 insertions(+) create mode 100644 man2/ioctl_getfsmap.2 diff

Re: [PATCH] btrfs: fix false enospc error when truncating heavily reflinked file

2016-09-07 Thread Darrick J. Wong
On Wed, Sep 07, 2016 at 08:17:38PM +0800, Wang Xiaoguang wrote: > Below test script can reveal this bug: > dd if=/dev/zero of=fs.img bs=$((1024*1024)) count=100 > dev=$(losetup --show -f fs.img) > mkdir -p /mnt/mntpoint > mkfs.btrfs -f $dev > mount $dev /mnt/mntpoint > cd /

Re: [PATCH 3/3] ioctl_xfs_ioc_getfsmap.2: document XFS_IOC_GETFSMAP ioctl

2016-09-08 Thread Darrick J. Wong
On Fri, Sep 09, 2016 at 09:38:06AM +1000, Dave Chinner wrote: > On Tue, Aug 30, 2016 at 12:09:49PM -0700, Darrick J. Wong wrote: > > > I recall for FIEMAP that some filesystems may not have files aligned > > > to sector offsets, and we just used byte offsets. Storage l

[PATCH v3 3/3] ioctl_getfsmap.2: document the GETFSMAP ioctl

2016-09-08 Thread Darrick J. Wong
Document the new GETFSMAP ioctl that returns the physical layout of a (disk-based) filesystem. Signed-off-by: Darrick J. Wong --- man2/ioctl_getfsmap.2 | 313 + 1 file changed, 313 insertions(+) create mode 100644 man2/ioctl_getfsmap.2 diff

Re: [PATCH 3/3] ioctl_xfs_ioc_getfsmap.2: document XFS_IOC_GETFSMAP ioctl

2016-09-11 Thread Darrick J. Wong
On Sat, Sep 10, 2016 at 10:00:29AM +1000, Dave Chinner wrote: > On Thu, Sep 08, 2016 at 11:07:16PM -0700, Darrick J. Wong wrote: > > On Fri, Sep 09, 2016 at 09:38:06AM +1000, Dave Chinner wrote: > > > On Tue, Aug 30, 2016 at 12:09:49PM -0700, Darrick J. Wong wrote: > > &

Re: [PATCH v2] generic: check whether we can truncate heavily reflinked file

2016-09-21 Thread Darrick J. Wong
On Thu, Sep 22, 2016 at 10:13:08AM +0800, Wang Xiaoguang wrote: > In btrfs, there is a bug about btrfs' truncate codes, it'll leak > some fs space as the truncate operation proceeds. If this truncate > operation is very large, later metadata request in this truncate > operation may fail for enospc

Re: [PATCH] generic: make 17[1-4] work well when btrfs compression is enabled

2016-10-09 Thread Darrick J. Wong
On Sat, Oct 08, 2016 at 01:36:10AM +1100, Dave Chinner wrote: > On Fri, Oct 07, 2016 at 03:00:42PM +0800, Wang Xiaoguang wrote: > > When enabling btrfs compression, original codes can not fill fs > > correctly, fix this. > > > > Signed-off-by: Wang Xiaoguang > > --- > > tests/generic/171 | 4 +--

Re: [PATCH] generic: make 17[1-4] work well when btrfs compression is enabled

2016-10-09 Thread Darrick J. Wong
On Mon, Oct 10, 2016 at 11:49:03AM +0800, Wang Xiaoguang wrote: > hi, > > On 10/10/2016 05:04 AM, Darrick J. Wong wrote: > >On Sat, Oct 08, 2016 at 01:36:10AM +1100, Dave Chinner wrote: > >>On Fri, Oct 07, 2016 at 03:00:42PM +0800, Wang Xiaoguang wrote: > >>

Re: [PATCH] generic/175: disable inline data feature for btrfs

2016-10-10 Thread Darrick J. Wong
On Mon, Oct 10, 2016 at 05:03:49PM +0800, Anand Jain wrote: > > > On 10/10/16 13:06, Wang Xiaoguang wrote: > >For btrfs, if compression is enabled, it may generate inline data for a > >blocksize data range, this inline data is stored in fs tree, will not have > >a individual extent, try to reflin

Re: [PATCH 2/3] misc: fix fallocate commands that need the unshare switch

2016-10-17 Thread Darrick J. Wong
On Sat, Oct 15, 2016 at 11:18:37PM -0700, Christoph Hellwig wrote: > On Sat, Oct 15, 2016 at 10:03:03AM -0700, Christoph Hellwig wrote: > > The poster child would be btrfs, and I would have added some output > > here if btrfs support in xfstests wasn't completely broken at this > > point. > > > >

[PATCH 2/2] man2/ioctl_fideduperange.2: clarify operation some more

2016-10-17 Thread Darrick J. Wong
Clarify the behavior of the dedupe ioctl. Signed-off-by: Darrick J. Wong --- man2/ioctl_ficlonerange.2 |2 +- man2/ioctl_fideduperange.2 | 26 ++ 2 files changed, 23 insertions(+), 5 deletions(-) diff --git a/man2/ioctl_ficlonerange.2 b/man2/ioctl_ficlonerange.2

[PATCH 1/2] man2/fallocate.2: document behavior with shared blocks

2016-10-17 Thread Darrick J. Wong
Add a blurb to the fallocate manpage explaining that the fallocate command with the UNSHARE mode flag may use CoW to unshare blocks to guarantee that a disk write won't fail with ENOSPC. Signed-off-by: Darrick J. Wong --- man2/fallocate.2 | 10 ++ 1 file changed, 10 inser

Re: About reflink len = 0 behavior

2016-10-20 Thread Darrick J. Wong
On Thu, Oct 20, 2016 at 08:47:33AM -0700, Omar Sandoval wrote: > On Thu, Oct 20, 2016 at 03:50:46PM +0800, Qu Wenruo wrote: > > Add cc to new xfs mail list. > > > > > > At 10/20/2016 03:46 PM, Qu Wenruo wrote: > > > Hi Darrick, xfs guys and btrfs guys. > > > > > > Although such question is quite

[PATCH 01/15] xfs: add a per-xfs trace_printk macro

2018-10-04 Thread Darrick J. Wong
From: Darrick J. Wong Add a "xfs_tprintk" macro so that developers can use trace_printk to print out arbitrary debugging information with the XFS device name attached to the trace output. Signed-off-by: Darrick J. Wong --- fs/xfs/xfs_error.h |5 + 1 file changed, 5

[PATCH 00/15] fs: fixes for serious clone/dedupe problems

2018-10-04 Thread Darrick J. Wong
Hi all, Dave, Eric, and I have been chasing a stale data exposure bug in the XFS reflink implementation, and tracked it down to reflink forgetting to do some of the file-extending activities that must happen for regular writes. We then started auditing the clone, dedupe, and copyfile code and rea

[PATCH 02/15] xfs: refactor clonerange preparation into a separate helper

2018-10-04 Thread Darrick J. Wong
From: Darrick J. Wong Refactor all the reflink preparation steps into a separate helper that we'll use to land all the upcoming fixes for insufficient input checks. Signed-off-by: Darrick J. Wong --- fs/xfs/xfs_reflink.c | 96 +- 1

[PATCH 07/15] vfs: skip zero-length dedupe requests

2018-10-04 Thread Darrick J. Wong
From: Darrick J. Wong Don't bother calling the filesystem for a zero-length dedupe request; we can return zero and exit. Signed-off-by: Darrick J. Wong --- fs/read_write.c |5 + 1 file changed, 5 insertions(+) diff --git a/fs/read_write.c b/fs/read_write.c index 973d3da

[PATCH 03/15] xfs: zero posteof blocks when cloning above eof

2018-10-04 Thread Darrick J. Wong
From: Darrick J. Wong When we're reflinking between two files and the destination file range is well beyond the destination file's EOF marker, zero any posteof speculative preallocations in the destination file so that we don't expose stale disk contents. The previous strate

[PATCH 06/15] vfs: strengthen checking of file range inputs to clone/dedupe range

2018-10-04 Thread Darrick J. Wong
From: Darrick J. Wong Clone range is an optimization on a regular file write. File writes that extend the file length are subject to various constraints which are not checked by clonerange. This is a correctness problem, because we're never allowed to touch ranges that the page cache

[PATCH 04/15] xfs: update ctime and remove suid before cloning files

2018-10-04 Thread Darrick J. Wong
From: Darrick J. Wong Before cloning into a file, update the ctime and remove sensitive attributes like suid, just like we'd do for a regular file write. Signed-off-by: Darrick J. Wong --- fs/xfs/xfs_reflink.c | 25 + 1 file changed, 25 insertions(+) diff --

[PATCH 09/15] vfs: pass operation flags to {clone, dedupe}_file_range implementations

2018-10-04 Thread Darrick J. Wong
From: Darrick J. Wong Pass operational flags to the per-filesystem clone and dedupe implementations. This enables the vfs to signal when it can deal with short clone and short dedupe operations. Signed-off-by: Darrick J. Wong --- fs/btrfs/ctree.h|3 ++- fs/btrfs/ioctl.c

[PATCH 05/15] vfs: check file ranges before cloning files

2018-10-04 Thread Darrick J. Wong
From: Darrick J. Wong Move the file range checks from vfs_clone_file_prep into a separate generic_clone_checks function so that all the checks are collected in a central location. This forms the basis for adding more checks from generic_write_checks that will make cloning's input checking

[PATCH 10/15] vfs: make cloning to source file eof more explicit

2018-10-04 Thread Darrick J. Wong
From: Darrick J. Wong Use the new CLONE_RANGE_EOF flag to explicitly declare that the caller wants the clone implementation to set *len appropriately once the files are locked. Signed-off-by: Darrick J. Wong --- fs/read_write.c | 14 +- 1 file changed, 9 insertions(+), 5

[PATCH 08/15] vfs: change clone and dedupe range function pointers to return bytes completed

2018-10-04 Thread Darrick J. Wong
From: Darrick J. Wong Change the clone_file_range and dedupe_file_range functions to return the number of bytes they operated on. This is the precursor to allowing fs implementations to return short clone/dedupe results to the user, which will enable us to obey resource limits in a graceful

[PATCH 11/15] vfs: allow short clone and dedupe operations

2018-10-04 Thread Darrick J. Wong
From: Darrick J. Wong Allow the clone and dedupe prep function to shorten the request if the caller can handle it. Signed-off-by: Darrick J. Wong --- fs/read_write.c|8 ++-- include/linux/fs.h |2 +- mm/filemap.c | 19 +++ 3 files changed, 18 insertions

[PATCH 14/15] ocfs2: support partial clone range and dedupe range

2018-10-04 Thread Darrick J. Wong
From: Darrick J. Wong Change the ocfs2 remap code to allow for returning partial results. Signed-off-by: Darrick J. Wong --- fs/ocfs2/file.c | 14 -- fs/ocfs2/refcounttree.c | 35 --- fs/ocfs2/refcounttree.h |4 ++-- 3 files changed

[PATCH 13/15] ocfs2: truncate page cache for clone destination file before remapping

2018-10-04 Thread Darrick J. Wong
From: Darrick J. Wong When cloning blocks into another file, truncate the page cache before we start remapping blocks so that concurrent reads wait for us to finish. Signed-off-by: Darrick J. Wong --- fs/ocfs2/refcounttree.c | 10 -- 1 file changed, 4 insertions(+), 6 deletions

[PATCH 12/15] vfs: implement opportunistic short dedupe

2018-10-04 Thread Darrick J. Wong
From: Darrick J. Wong For a given dedupe request, the bytes_deduped field in the control structure tells userspace if we managed to deduplicate some, but not all of, the requested regions starting from the file offsets supplied. However, due to sloppy coding, the current dedupe code returns

[PATCH 15/15] xfs: support returning partial reflink results

2018-10-04 Thread Darrick J. Wong
From: Darrick J. Wong Back when the XFS reflink code only supported clone_file_range, we were only able to return zero or negative error codes to userspace. However, now that copy_file_range (which returns bytes copied) can use XFS' clone_file_range, we have the opportunity to return pa

Re: [PATCH 00/15] fs: fixes for serious clone/dedupe problems

2018-10-04 Thread Darrick J. Wong
On Fri, Oct 05, 2018 at 11:17:18AM +1000, Dave Chinner wrote: > On Thu, Oct 04, 2018 at 05:44:34PM -0700, Darrick J. Wong wrote: > > Hi all, > > > > Dave, Eric, and I have been chasing a stale data exposure bug in the XFS > > reflink implementation, and tracked it down

Re: [PATCH 02/15] xfs: refactor clonerange preparation into a separate helper

2018-10-05 Thread Darrick J. Wong
On Fri, Oct 05, 2018 at 03:28:09PM +1000, Dave Chinner wrote: > On Thu, Oct 04, 2018 at 05:44:47PM -0700, Darrick J. Wong wrote: > > From: Darrick J. Wong > > > > Refactor all the reflink preparation steps into a separate helper that > > we'll use to land all the

Re: [PATCH 02/15] xfs: refactor clonerange preparation into a separate helper

2018-10-05 Thread Darrick J. Wong
On Fri, Oct 05, 2018 at 07:02:42PM +1000, Dave Chinner wrote: > On Fri, Oct 05, 2018 at 05:02:28PM +1000, Dave Chinner wrote: > > On Thu, Oct 04, 2018 at 05:44:47PM -0700, Darrick J. Wong wrote: > > > From: Darrick J. Wong > > > > > > Refactor all the refli

Re: [PATCH 06/15] vfs: strengthen checking of file range inputs to clone/dedupe range

2018-10-05 Thread Darrick J. Wong
On Fri, Oct 05, 2018 at 09:10:12AM +0300, Amir Goldstein wrote: > On Fri, Oct 5, 2018 at 3:46 AM Darrick J. Wong > wrote: > > > > From: Darrick J. Wong > > > > Clone range is an optimization on a regular file write. File writes > > that extend the file lengt

Re: [PATCH 12/15] vfs: implement opportunistic short dedupe

2018-10-05 Thread Darrick J. Wong
On Fri, Oct 05, 2018 at 09:40:44AM +0300, Amir Goldstein wrote: > On Fri, Oct 5, 2018 at 3:46 AM Darrick J. Wong > wrote: > > > > From: Darrick J. Wong > > > > For a given dedupe request, the bytes_deduped field in the control > > structure tells userspace if

Re: [PATCH 09/15] vfs: pass operation flags to {clone, dedupe}_file_range implementations

2018-10-05 Thread Darrick J. Wong
On Fri, Oct 05, 2018 at 10:07:43AM +0300, Amir Goldstein wrote: > On Fri, Oct 5, 2018 at 3:46 AM Darrick J. Wong > wrote: > > > > From: Darrick J. Wong > > > > Pass operational flags to the per-filesystem clone and dedupe > > implementations. This enable

Re: [PATCH 08/15] vfs: change clone and dedupe range function pointers to return bytes completed

2018-10-05 Thread Darrick J. Wong
On Fri, Oct 05, 2018 at 11:06:54AM +0300, Amir Goldstein wrote: > On Fri, Oct 5, 2018 at 3:46 AM Darrick J. Wong > wrote: > > > > From: Darrick J. Wong > > > > Change the clone_file_range and dedupe_file_range functions to return > > the number of bytes they

Re: [PATCH 08/15] vfs: change clone and dedupe range function pointers to return bytes completed

2018-10-08 Thread Darrick J. Wong
On Sat, Oct 06, 2018 at 03:41:34AM -0700, Christoph Hellwig wrote: > On Thu, Oct 04, 2018 at 05:45:35PM -0700, Darrick J. Wong wrote: > > From: Darrick J. Wong > > > > Change the clone_file_range and dedupe_file_range functions to return > > the number of bytes the

[PATCH 02/25] xfs: refactor clonerange preparation into a separate helper

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong Refactor all the reflink preparation steps into a separate helper that we'll use to land all the upcoming fixes for insufficient input checks. This rework also moves the invalidation of the destination range to the prep function so that it is done before the ran

[PATCH 04/25] xfs: update ctime and remove suid before cloning files

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong Before cloning into a file, update the ctime and remove sensitive attributes like suid, just like we'd do for a regular file write. Signed-off-by: Darrick J. Wong Reviewed-by: Dave Chinner --- fs/xfs/xfs_reflink.c | 25 + 1 file change

[PATCH 01/25] xfs: add a per-xfs trace_printk macro

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong Add a "xfs_tprintk" macro so that developers can use trace_printk to print out arbitrary debugging information with the XFS device name attached to the trace output. Signed-off-by: Darrick J. Wong --- fs/xfs/xfs_error.h |5 + 1 file changed, 5

[PATCH v2 00/25] fs: fixes for serious clone/dedupe problems

2018-10-09 Thread Darrick J. Wong
Hi all, Dave, Eric, and I have been chasing a stale data exposure bug in the XFS reflink implementation, and tracked it down to reflink forgetting to do some of the file-extending activities that must happen for regular writes. We then started auditing the clone, dedupe, and copyfile code and rea

[PATCH 03/25] xfs: zero posteof blocks when cloning above eof

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong When we're reflinking between two files and the destination file range is well beyond the destination file's EOF marker, zero any posteof speculative preallocations in the destination file so that we don't expose stale disk contents. The previous strate

[PATCH 07/25] vfs: skip zero-length dedupe requests

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong Don't bother calling the filesystem for a zero-length dedupe request; we can return zero and exit. Signed-off-by: Darrick J. Wong Reviewed-by: Christoph Hellwig --- fs/read_write.c |5 + 1 file changed, 5 insertions(+) diff --git a/fs/read_write.c

[PATCH 06/25] vfs: strengthen checking of file range inputs to generic_remap_checks

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong File range remapping, if allowed to run past the destination file's EOF, is an optimization on a regular file write. Regular file writes that extend the file length are subject to various constraints which are not checked by range cloning. This is a correctness pr

[PATCH 05/25] vfs: check file ranges before cloning files

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong Move the file range checks from vfs_clone_file_prep into a separate generic_remap_checks function so that all the checks are collected in a central location. This forms the basis for adding more checks from generic_write_checks that will make cloning's input checking

[PATCH 09/25] vfs: rename vfs_clone_file_prep to be more descriptive

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong The vfs_clone_file_prep is a generic function to be called by filesystem implementations only. Rename the prefix to generic_ and make it more clear that it applies to remap operations, not just clones. Signed-off-by: Darrick J. Wong --- fs/ocfs2/refcounttree.c |2

[PATCH 08/25] vfs: combine the clone and dedupe into a single remap_file_range

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong Combine the clone_file_range and dedupe_file_range operations into a single remap_file_range file operation dispatch since they're fundamentally the same operation. The differences between the two can be made in the prep functions. Signed-off-by: Darrick J.

[PATCH 14/25] vfs: make remap_file_range functions take and return bytes completed

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong Change the remap_file_range functions to take a number of bytes to operate upon and return the number of bytes they operated on. This is a requirement for allowing fs implementations to return short clone/dedupe results to the user, which will enable us to obey resource

[PATCH 11/25] vfs: create generic_remap_file_range_touch to update inode metadata

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong Create a new VFS helper to handle inode metadata updates when remapping into a file. If the operation can possibly alter the file contents, we must update the ctime and mtime and remove security privileges, just like we do for regular file writes. Wire up ocfs2 to ensure

[PATCH 18/25] vfs: enable remap callers that can handle short operations

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong Plumb in a remap flag that enables the filesystem remap handler to shorten remapping requests for callers that can handle it. Now copy_file_range can report partial success (in case we run up against alignment problems, resource limits, etc.). Signed-off-by: Darrick J

[PATCH 13/25] vfs: pass remap flags to generic_remap_checks

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong Pass the same remap flags to generic_remap_checks for consistency. Signed-off-by: Darrick J. Wong --- fs/read_write.c|2 +- include/linux/fs.h |2 +- mm/filemap.c |4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/fs

[PATCH 16/25] vfs: plumb RFR_* remap flags through the vfs dedupe functions

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong Plumb a remap_flags argument through the vfs_dedupe_file_range_one functions so that dedupe can take advantage of it. Signed-off-by: Darrick J. Wong --- fs/overlayfs/file.c |3 ++- fs/read_write.c |9 ++--- include/linux/fs.h |2 +- 3 files changed

[PATCH 12/25] vfs: pass remap flags to generic_remap_file_range_prep

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong Plumb the remap flags through the filesystem from the vfs function dispatcher all the way to the prep function to prepare for behavior changes in subsequent patches. Signed-off-by: Darrick J. Wong --- fs/ocfs2/file.c |2 +- fs/ocfs2/refcounttree.c |6

[PATCH 17/25] vfs: make remapping to source file eof more explicit

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong Create a RFR_TO_SRC_EOF flag to explicitly declare that the caller wants the remap implementation to remap to the end of the source file, once the files are locked. Signed-off-by: Darrick J. Wong --- fs/ioctl.c |3 ++- fs/nfsd/vfs.c |3 ++- fs

[PATCH 21/25] ocfs2: truncate page cache for clone destination file before remapping

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong When cloning blocks into another file, truncate the page cache before we start remapping blocks so that concurrent reads wait for us to finish. Signed-off-by: Darrick J. Wong --- fs/ocfs2/refcounttree.c | 10 -- 1 file changed, 4 insertions(+), 6 deletions

[PATCH 22/25] ocfs2: fix pagecache truncation prior to reflink

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong Prior to remapping blocks, it is necessary to remove pages from the destination file's page cache. Unfortunately, the truncation is not aggressive enough -- if page size > block size, we'll end up zeroing subpage blocks instead of removing them. So, round the

[PATCH 15/25] vfs: plumb RFR_* remap flags through the vfs clone functions

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong Plumb a remap_flags argument through the {do,vfs}_clone_file_range functions so that clone can take advantage of it. Signed-off-by: Darrick J. Wong --- fs/ioctl.c |2 +- fs/nfsd/vfs.c |2 +- fs/overlayfs/copy_up.c |2 +- fs/overlayfs

[PATCH 10/25] vfs: rename clone_verify_area to remap_verify_area

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong Since we use clone_verify_area for both clone and dedupe range checks, rename the function to make it clear that it's for both. Signed-off-by: Darrick J. Wong --- fs/read_write.c | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git

[PATCH 23/25] ocfs2: support partial clone range and dedupe range

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong Change the ocfs2 remap code to allow for returning partial results. Signed-off-by: Darrick J. Wong --- fs/ocfs2/file.c |7 + fs/ocfs2/refcounttree.c | 73 ++- fs/ocfs2/refcounttree.h | 12 3 files

[PATCH 19/25] vfs: hide file range comparison function

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong There are no callers of vfs_dedupe_file_range_compare, so we might as well make it a static helper and remove the export. Signed-off-by: Darrick J. Wong --- fs/read_write.c| 191 ++-- include/linux/fs.h |3 - 2

[PATCH 20/25] vfs: implement opportunistic short dedupe

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong For a given dedupe request, the bytes_deduped field in the control structure tells userspace if we managed to deduplicate some, but not all of, the requested regions starting from the file offsets supplied. However, due to sloppy coding, the current dedupe code returns

[PATCH 24/25] xfs: fix pagecache truncation prior to reflink

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong Prior to remapping blocks, it is necessary to remove pages from the destination file's page cache. Unfortunately, the truncation is not aggressive enough -- if page size > block size, we'll end up zeroing subpage blocks instead of removing them. So, round the

[PATCH 25/25] xfs: support returning partial reflink results

2018-10-09 Thread Darrick J. Wong
From: Darrick J. Wong Back when the XFS reflink code only supported clone_file_range, we were only able to return zero or negative error codes to userspace. However, now that copy_file_range (which returns bytes copied) can use XFS' clone_file_range, we have the opportunity to return pa

Re: [PATCH v2 00/25] fs: fixes for serious clone/dedupe problems

2018-10-09 Thread Darrick J. Wong
On Wed, Oct 10, 2018 at 12:02:08PM +1100, Dave Chinner wrote: > On Tue, Oct 09, 2018 at 05:10:38PM -0700, Darrick J. Wong wrote: > > Hi all, > > > > Dave, Eric, and I have been chasing a stale data exposure bug in the XFS > > reflink implementation, and tracked it down

[PATCH v2 01/25] xfs: add a per-xfs trace_printk macro

2018-10-10 Thread Darrick J. Wong
From: Darrick J. Wong Add a "xfs_tprintk" macro so that developers can use trace_printk to print out arbitrary debugging information with the XFS device name attached to the trace output. Signed-off-by: Darrick J. Wong --- fs/xfs/xfs_error.h |6 ++ 1 file changed, 6

Re: [PATCH 08/25] vfs: combine the clone and dedupe into a single remap_file_range

2018-10-10 Thread Darrick J. Wong
On Wed, Oct 10, 2018 at 08:54:44AM +0300, Amir Goldstein wrote: > On Wed, Oct 10, 2018 at 3:12 AM Darrick J. Wong > wrote: > > > > From: Darrick J. Wong > > > > Combine the clone_file_range and dedupe_file_range operations into a > > single remap_file_range f

Re: [PATCH 14/25] vfs: make remap_file_range functions take and return bytes completed

2018-10-10 Thread Darrick J. Wong
On Wed, Oct 10, 2018 at 09:47:00AM +0300, Amir Goldstein wrote: > On Wed, Oct 10, 2018 at 3:14 AM Darrick J. Wong > wrote: > > > > From: Darrick J. Wong > > > > Change the remap_file_range functions to take a number of bytes to > > operate upon and return

Re: [PATCH 17/25] vfs: make remapping to source file eof more explicit

2018-10-10 Thread Darrick J. Wong
On Wed, Oct 10, 2018 at 03:29:06PM +0300, Amir Goldstein wrote: > On Wed, Oct 10, 2018 at 3:14 AM Darrick J. Wong > wrote: > > > > From: Darrick J. Wong > > > > Create a RFR_TO_SRC_EOF flag to explicitly declare that the caller wants > > the remap imple

Re: [PATCH 06/25] vfs: strengthen checking of file range inputs to generic_remap_checks

2018-10-10 Thread Darrick J. Wong
On Wed, Oct 10, 2018 at 08:23:27AM +0300, Amir Goldstein wrote: > On Wed, Oct 10, 2018 at 3:11 AM Darrick J. Wong > wrote: > > > > From: Darrick J. Wong > > > > File range remapping, if allowed to run past the destination file's EOF, > > is an optimizati

Re: [PATCH 14/25] vfs: make remap_file_range functions take and return bytes completed

2018-10-10 Thread Darrick J. Wong
On Wed, Oct 10, 2018 at 09:28:34PM +0300, Amir Goldstein wrote: > On Wed, Oct 10, 2018 at 6:51 PM Darrick J. Wong > wrote: > > > > On Wed, Oct 10, 2018 at 09:47:00AM +0300, Amir Goldstein wrote: > > > On Wed, Oct 10, 2018 at 3:14 AM Darrick J. Wong > > > wr

Re: [PATCH 05/25] vfs: check file ranges before cloning files

2018-10-10 Thread Darrick J. Wong
On Thu, Oct 11, 2018 at 10:06:39AM +1100, Dave Chinner wrote: > On Tue, Oct 09, 2018 at 05:11:13PM -0700, Darrick J. Wong wrote: > > From: Darrick J. Wong > > > > Move the file range checks from vfs_clone_file_prep into a separate > > generic_remap_checks function

Re: [PATCH v3 00/25] fs: fixes for serious clone/dedupe problems

2018-10-11 Thread Darrick J. Wong
On Thu, Oct 11, 2018 at 11:33:57AM +0300, Amir Goldstein wrote: > On Thu, Oct 11, 2018 at 7:12 AM Darrick J. Wong > wrote: > > > > Hi all, > > > > Dave, Eric, and I have been chasing a stale data exposure bug in the XFS > > reflink implementation, and tracked

  1   2   3   4   5   6   7   8   9   >