Need help with incremental backup strategy (snapshots, defragmentingt & performance)

2017-10-31 Thread Dave
Our use case requires snapshots. btrfs snapshots are best solution we have found for our requirements, and over the last year snapshots have proven their value to us. (For this discussion I am considering both the "root" volume and the "home" volume on a typical desktop workstation. Also, all btfs

Re: defragmenting best practice?

2017-10-31 Thread Dave
On Tue, Oct 31, 2017 at 7:06 PM, Peter Grandi wrote: > > Also nothing forces you to defragment a whole filesystem, you > can just defragment individual files or directories by using > 'find' with it. Thanks for that info. When defragmenting individual files on a BTRFS filesystem with COW, I assu

Re: defragmenting best practice?

2017-10-31 Thread Dave
or example. We found too many unexpected changes when using sync, so we do not use it. On Thu, Sep 21, 2017 at 7:09 AM, Duncan <1i5t5.dun...@cox.net> wrote: > Dave posted on Wed, 20 Sep 2017 02:38:13 -0400 as excerpted: > >> Here's my scenario. Some months ago I built an ov

Re: Problem with file system

2017-10-29 Thread Dave
This is a very helpful thread. I want to share an interesting related story. We have a machine with 4 btrfs volumes and 4 Snapper configs. I recently discovered that Snapper timeline cleanup been turned off for 3 of those volumes. In the Snapper configs I found this setting: TIMELINE_CLEANUP="no"

Re: [ANNOUNCE] fsperf: a simple fs/block performance testing framework

2017-10-09 Thread Dave Chinner
On Mon, Oct 09, 2017 at 09:00:51AM -0400, Josef Bacik wrote: > On Mon, Oct 09, 2017 at 04:17:31PM +1100, Dave Chinner wrote: > > On Sun, Oct 08, 2017 at 10:25:10PM -0400, Josef Bacik wrote: > > > > Integrating into fstests means it will be immediately available to > > &

Re: [ANNOUNCE] fsperf: a simple fs/block performance testing framework

2017-10-08 Thread Dave Chinner
On Sun, Oct 08, 2017 at 10:25:10PM -0400, Josef Bacik wrote: > On Mon, Oct 09, 2017 at 11:51:37AM +1100, Dave Chinner wrote: > > On Fri, Oct 06, 2017 at 05:09:57PM -0400, Josef Bacik wrote: > > > Hello, > > > > > > One thing that comes up a lot every LSF is the

Re: [ANNOUNCE] fsperf: a simple fs/block performance testing framework

2017-10-08 Thread Dave Chinner
; adding some more comparison options so you can compare against averages of all > previous runs and such. Yup, that fits exactly into what fstests is for... :P Integrating into fstests means it will be immediately available to all fs developers, it'll run on everything that everyone alrea

Re: Something like ZFS Channel Programs for BTRFS & probably XFS or even VFS?

2017-10-03 Thread Dave Chinner
On Tue, Oct 03, 2017 at 01:40:51PM -0700, Matthew Wilcox wrote: > On Wed, Oct 04, 2017 at 07:10:35AM +1100, Dave Chinner wrote: > > On Tue, Oct 03, 2017 at 03:19:18PM +0200, Martin Steigerwald wrote: > > > [repost. I didn´t notice autocompletion gave me wrong address

Re: Something like ZFS Channel Programs for BTRFS & probably XFS or even VFS?

2017-10-03 Thread Dave Chinner
On Tue, Oct 03, 2017 at 03:19:18PM +0200, Martin Steigerwald wrote: > [repost. I didn´t notice autocompletion gave me wrong address for fsdevel, > blacklisted now] > > Hello. > > What do you think of > > http://open-zfs.org/wiki/Projects/ZFS_Channel_Programs Domain not

Re: defragmenting best practice?

2017-09-21 Thread Dave
These are great suggestions. I will test several of them (or all of them) and report back with my results once I have done the testing. Thank you! This is a fantastic mailing list. P.S. I'm inclined to stay with Firefox, but I will definitely test Chromium vs Firefox after making a series of chang

Re: [PATCH] fstests: btrfs/150 regression test for reading compressed data

2017-09-20 Thread Dave Chinner
ailure > + # invalidate the page cache > + $XFS_IO_PROG -f -c "fadvise -d 0 128K" $SCRATCH_MNT/foobar | > _filter_xfs_io > + > + enable_io_failure > + od -x $SCRATCH_MNT/foobar > /dev/null & why are you using od to read the data when the outp

Re: ERROR: parent determination failed (btrfs send-receive)

2017-09-20 Thread Dave
On Tue, Sep 19, 2017 at 3:37 PM, Andrei Borzenkov wrote: > 18.09.2017 09:10, Dave пишет: >> I use snap-sync to create and send snapshots. >> >> GitHub - wesbarnett/snap-sync: Use snapper snapshots to backup to external >> drive >> https://github.com/wesbarnett/s

Re: defragmenting best practice?

2017-09-19 Thread Dave
>On Thu 2017-08-31 (09:05), Ulli Horlacher wrote: >> When I do a >> btrfs filesystem defragment -r /directory >> does it defragment really all files in this directory tree, even if it >> contains subvolumes? >> The man page does not mention subvolumes on this topic. > >No answer so far :-( > >But I

Re: snapshots of encrypted directories?

2017-09-19 Thread Dave
On Fri, Sep 15, 2017 at 6:01 AM, Ulli Horlacher wrote: > > On Fri 2017-09-15 (06:45), Andrei Borzenkov wrote: > > > The actual question is - do you need to mount each individual btrfs > > subvolume when using encfs? > > And even worse it goes with ecryptfs: I do not know at all how to mount a > sn

Re: ERROR: parent determination failed (btrfs send-receive)

2017-09-18 Thread Dave
On Mon, Sep 18, 2017 at 12:23 AM, Andrei Borzenkov wrote: > > 18.09.2017 05:31, Dave пишет: > > Sometimes when using btrfs send-receive, I get errors like this: > > > > ERROR: parent determination failed for > > > > When this happens, btrfs send-rece

difference between -c and -p for send-receive?

2017-09-18 Thread Dave
new subject for new question On Mon, Sep 18, 2017 at 1:37 PM, Andrei Borzenkov wrote: > >> What scenarios can lead to "ERROR: parent determination failed"? > > > > The man page for btrfs-send is reasonably clear on the requirements > > btrfs imposes. If you want to use incremental sends (i.e. th

Re: ERROR: parent determination failed (btrfs send-receive)

2017-09-17 Thread Dave
On Mon, Sep 18, 2017 at 12:23 AM, Andrei Borzenkov wrote: > 18.09.2017 05:31, Dave пишет: >> Sometimes when using btrfs send-receive, I get errors like this: >> >> ERROR: parent determination failed for >> >> When this happens, btrfs send-receive backups fail. A

ERROR: parent determination failed (btrfs send-receive)

2017-09-17 Thread Dave
Sometimes when using btrfs send-receive, I get errors like this: ERROR: parent determination failed for When this happens, btrfs send-receive backups fail. And all subsequent backups fail too. The issue seems to stem from the fact that an automated cleanup process removes certain earlier subvol

Re: send | receive: received snapshot is missing recent files

2017-09-13 Thread Dave
On Mon, Sep 11, 2017 at 11:19 PM, Andrei Borzenkov wrote: > 11.09.2017 20:53, Axel Burri пишет: >> On 2017-09-08 06:44, Dave wrote: >>> I'm referring to the link below. Using "btrfs subvolume snapshot -r" >>> copies the Received UUID from the source into

Re: send | receive: received snapshot is missing recent files

2017-09-07 Thread Dave
d to use "btrfs subvolume snapshot ". > > There is a FAQ entry on btrbk on how to fix this: > > https://github.com/digint/btrbk/blob/master/doc/FAQ.md#im-getting-an-error-aborted-received-uuid-is-set > > > On 2017-09-07 15:34, Dave wrote: > > I just ran a test.

Re: send | receive: received snapshot is missing recent files

2017-09-07 Thread Dave
l. Thanks for any further feedback, including answers to my questions and comments about whether this is a known issue. On Thu, Sep 7, 2017 at 8:39 AM, Dave wrote: > > Hello. Can anyone further explain this issue ("you have a Received UUID on > the source volume")? > >

Re: send | receive: received snapshot is missing recent files

2017-09-07 Thread Dave
ve a Received UUID on the source volume. This > breaks send-receive. > > From: Dave -- Sent: 2017-09-07 - 06:43 > >> Here is more info and a possible (shocking) explanation. This >> aggregates my prior messages and it provides an almost complete set of >> s

Re: send | receive: received snapshot is missing recent files

2017-09-06 Thread Dave
Here is more info and a possible (shocking) explanation. This aggregates my prior messages and it provides an almost complete set of steps to reproduce this problem. Linux srv 4.9.41-1-lts #1 SMP Mon Aug 7 17:32:35 CEST 2017 x86_64 GNU/Linux btrfs-progs v4.12 My steps: [root@srv]# sync [root@srv

Re: send | receive: received snapshot is missing recent files

2017-09-06 Thread Dave
me/test2/home/user2/Documents/ NOTE: all recent files are MISSING Any ideas what could be causing this problem with incremental backups? On Wed, Sep 6, 2017 at 3:23 PM, Dave wrote: > > Here is more info on this problem. I can reproduce this without using my > script. Simple btrfs command

send | receive: received snapshot is missing recent files

2017-09-05 Thread Dave
I'm running Arch Linux on BTRFS. I use Snapper to take hourly snapshots and it works without any issues. I have a bash script that uses send | receive to transfer snapshots to a couple external HDD's. The script runs daily on a systemd timer. I set all this up recently and I first noticed that it

Re: btrfs_wait_ordered_roots warning triggered

2017-06-21 Thread Dave Jones
On Wed, Jun 21, 2017 at 11:52:36AM -0400, Chris Mason wrote: > On 06/21/2017 11:16 AM, Dave Jones wrote: > > WARNING: CPU: 2 PID: 7153 at fs/btrfs/ordered-data.c:753 > > btrfs_wait_ordered_roots+0x1a3/0x220 > > CPU: 2 PID: 7153 Comm: kworker/u8:7 Not tainted

btrfs_wait_ordered_roots warning triggered

2017-06-21 Thread Dave Jones
WARNING: CPU: 2 PID: 7153 at fs/btrfs/ordered-data.c:753 btrfs_wait_ordered_roots+0x1a3/0x220 CPU: 2 PID: 7153 Comm: kworker/u8:7 Not tainted 4.12.0-rc6-think+ #4 Workqueue: events_unbound btrfs_async_reclaim_metadata_space task: 8804f08d5380 task.stack: c9000895c000 RIP: 0010:btrfs_wait_

Re: [RFC PATCH v1 00/30] fs: inode->i_version rework and optimization

2017-04-04 Thread Dave Chinner
On Mon, Apr 03, 2017 at 04:00:55PM +0200, Jan Kara wrote: > On Sun 02-04-17 09:05:26, Dave Chinner wrote: > > On Thu, Mar 30, 2017 at 12:12:31PM -0400, J. Bruce Fields wrote: > > > On Thu, Mar 30, 2017 at 07:11:48AM -0400, Jeff Layton wrote: > > > > On Thu, 2017-

Re: [RFC PATCH v1 00/30] fs: inode->i_version rework and optimization

2017-04-01 Thread Dave Chinner
ven know there was a crash at mount time because their architecture always leaves a consistent filesystem on disk (e.g. COW filesystems) > I wonder if repeated crashes can lead to any odd corner cases. WIthout defined, locked down behavour of the superblock counter, the almost certainly cor

Re: [RFC PATCH v1 00/30] fs: inode->i_version rework and optimization

2017-03-29 Thread Dave Chinner
before __mark_inode_dirty returns. > > > > > > > > So you think the filesystem can provide the atomicity? In more detail: > > > > > > > > > > Sorry, I hit send too quickly. That should have read: > > > > > > "Yeah, t

Re: [RFC PATCH v1 00/30] fs: inode->i_version rework and optimization

2017-03-21 Thread Dave Chinner
as the NFS clients are accessing and requiring synchronisation. > Not sure how big a problem that really is. This coherency problem has always existed on the server side... Cheers, Dave. -- Dave Chinner da...@fromorbit.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: [PATCH 5/8] nowait aio: return on congested block device

2017-03-16 Thread Dave Chinner
;t the congestion check at a higher layer like we do for page cache readahead? i.e. using the bdi*congested() API at the time we are doing all the other filesystem blocking checks. -Dave. -- Dave Chinner da...@fromorbit.com -- To unsubscribe from this list: send the line "unsubscribe linux-b

assertion failed: page_ops & PAGE_LOCK

2017-03-05 Thread Dave Jones
After commenting out the assertion that Liu bo pointed out was bogus, my trinity runs last a little longer.. This is a new one I think.. assertion failed: page_ops & PAGE_LOCK, file: fs/btrfs/extent_io.c, line: 1716 [ cut here ] kernel BUG at fs/btrfs/ctree.h:3423! invalid

Re: assertion failed: last_size == new_size, file: fs/btrfs/inode.c

2017-03-03 Thread Dave Jones
On Thu, Mar 02, 2017 at 06:04:33PM -0800, Liu Bo wrote: > On Thu, Mar 02, 2017 at 07:58:01AM -0800, Liu Bo wrote: > > On Wed, Mar 01, 2017 at 03:03:19PM -0500, Dave Jones wrote: > > > On Tue, Feb 28, 2017 at 05:12:01PM -0800, Liu Bo wrote: > > > > On Mon, Fe

Re: assertion failed: last_size == new_size, file: fs/btrfs/inode.c

2017-03-01 Thread Dave Jones
On Tue, Feb 28, 2017 at 05:12:01PM -0800, Liu Bo wrote: > On Mon, Feb 27, 2017 at 11:23:42AM -0500, Dave Jones wrote: > > On Mon, Feb 27, 2017 at 07:53:48AM -0800, Liu Bo wrote: > > > On Sun, Feb 26, 2017 at 07:18:42PM -0500, Dave Jones wrote: > > > > Hitting th

Re: assertion failed: last_size == new_size, file: fs/btrfs/inode.c

2017-02-27 Thread Dave Jones
On Mon, Feb 27, 2017 at 07:53:48AM -0800, Liu Bo wrote: > On Sun, Feb 26, 2017 at 07:18:42PM -0500, Dave Jones wrote: > > Hitting this fairly frequently.. I'm not sure if this is the same bug I've > > been hitting occasionally since 4.9. The assertion looks

assertion failed: last_size == new_size, file: fs/btrfs/inode.c

2017-02-26 Thread Dave Jones
Hitting this fairly frequently.. I'm not sure if this is the same bug I've been hitting occasionally since 4.9. The assertion looks new to me at least. Dave assertion failed: last_size == new_size, file: fs/btrfs/inode.c, line: 4619 [ cut here ] kernel

Re: [PATCH 4/6] xfs: use memalloc_nofs_{save,restore} instead of memalloc_noio*

2017-02-06 Thread Dave Chinner
dit of the caller paths is done and we're 100% certain that there are no lurking deadlocks. For example, I'm pretty sure we can call into _xfs_buf_map_pages() outside of a transaction context but with an inode ILOCK held exclusively. If we then recurse into memory reclaim and try to run a

4.10-rc btrfs gets 'stuck'.

2017-01-13 Thread Dave Jones
g child is trying to do is pretty simple.. fallocate(fd=7, mode=0x3, offset=114, len=9440) perhaps interesting, is that the file backing that fd has been unlinked. # ll /proc/16772/fd/7 lrwx-- 1 root root 64 Jan 13 11:21 /proc/16772/fd/7 -> /mnt/ssd/trinity/tmp/trinity.9kTIQN/tmp/trinit

Re: [PATCH 2/9] xfs: introduce and use KM_NOLOCKDEP to silence reclaim lockdep false positives

2016-12-20 Thread Dave Chinner
On Mon, Dec 19, 2016 at 02:06:19PM -0800, Darrick J. Wong wrote: > On Tue, Dec 20, 2016 at 08:24:13AM +1100, Dave Chinner wrote: > > On Thu, Dec 15, 2016 at 03:07:08PM +0100, Michal Hocko wrote: > > > From: Michal Hocko > > > > > > Now that the page al

Re: [PATCH 2/9] xfs: introduce and use KM_NOLOCKDEP to silence reclaim lockdep false positives

2016-12-19 Thread Dave Chinner
the unnecessary KM_NOFS allocations in one go. I've never liked whack-a-mole style changes like this - do it once, do it properly Cheers, Dave. -- Dave Chinner da...@fromorbit.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to

Re: [PATCH 1/6] Btrfs: add mount option for dax

2016-12-09 Thread Dave Chinner
On Fri, Dec 09, 2016 at 10:41:34AM -0800, Liu Bo wrote: > On Fri, Dec 09, 2016 at 03:47:20PM +1100, Dave Chinner wrote: > > On Wed, Dec 07, 2016 at 01:45:05PM -0800, Liu Bo wrote: > > > Signed-off-by: Liu Bo > > > --- > > > fs/btrfs/ctree.h

Re: [PATCH 4/6] Btrfs: add DAX support for nocow btrfs

2016-12-08 Thread Dave Chinner
ompatible if the architecture is correctly structured. i.e DAX should be able to work even with most of btrfs's special magic still enabled. Cheers, Dave. -- Dave Chinner da...@fromorbit.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a

Re: [PATCH 1/6] Btrfs: add mount option for dax

2016-12-08 Thread Dave Chinner
hing. The permanent functionality for DAX is supposed to be per-inode inheritable DAX flags - not mount options - so that applications can choose on a per file basis to enable/disable DAX access as they see fit. This also enables the filesystem to reject the attempt to turn on DAX if the set

Re: btrfs_destroy_inode warn (outstanding extents)

2016-12-07 Thread Dave Jones
On Sat, Dec 03, 2016 at 11:48:33AM -0500, Dave Jones wrote: > The interesting process here seems to be kworker/u8:17, and the trace > captures some of what that was doing before that bad page was hit. I'm travelling next week, so I'm trying to braindump the stuff I

Re: bio linked list corruption.

2016-12-05 Thread Dave Jones
ps://paste.fedoraproject.org/499794/14809582/ You may need to add some more. (I'll get around to making all this scripting go away, and have trinity just set this stuff up itself eventually) Oh, and if you're not running as root, you might need a diff like below so that trinity can sto

Re: btrfs_destroy_inode warn (outstanding extents)

2016-12-03 Thread Dave Jones
On Thu, Dec 01, 2016 at 10:32:09AM -0500, Dave Jones wrote: > http://codemonkey.org.uk/junk/btrfs-destroy-inode-outstanding-extents.txt > > Also same bug, different run, but a different traceview > http://codemonkey.org.uk/junk/btrfs-destroy-inode-outstanding-extents-functi

Re: btrfs_destroy_inode warn (outstanding extents)

2016-12-01 Thread Dave Jones
On Wed, Nov 23, 2016 at 02:58:45PM -0500, Dave Jones wrote: > On Wed, Nov 23, 2016 at 02:34:19PM -0500, Dave Jones wrote: > > > [ 317.689216] BUG: Bad page state in process kworker/u8:8 pfn:4d8fd4 > > trace from just before this happened. Does this shed any light ?

Re: [PATCH 01/10] fstests: common: Introduce function to check qgroup correctness

2016-11-29 Thread Dave Chinner
On Wed, Nov 30, 2016 at 08:56:03AM +0800, Qu Wenruo wrote: > > > At 11/30/2016 05:01 AM, Dave Chinner wrote: > >On Tue, Nov 29, 2016 at 03:32:54PM +0800, Qu Wenruo wrote: > >>Old btrfs qgroup test cases uses fix golden output numbers, which limits > >>the cover

Re: [PATCH 01/10] fstests: common: Introduce function to check qgroup correctness

2016-11-29 Thread Dave Chinner
ogs combinations... > + > +_btrfs_check_scratch_qgroup() > +{ > + _require_btrfs_check_qgroup This needs to go in the test itself before the test is run, not get hidden in a function call at the end of the test. Cheers, Dave. -- Dave Chinner da...@fromorbit.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 0/2] Btrfs: make a source length of 0 imply EOF for dedupe

2016-11-23 Thread Dave Chinner
On Wed, Nov 23, 2016 at 06:14:47PM -0500, Zygo Blaxell wrote: > On Thu, Nov 24, 2016 at 09:13:28AM +1100, Dave Chinner wrote: > > On Wed, Nov 23, 2016 at 08:55:59AM -0500, Zygo Blaxell wrote: > > > On Wed, Nov 23, 2016 at 03:26:32PM +1100, Dave Chinner wrote: > > > >

Re: [RFC PATCH 0/2] Btrfs: make a source length of 0 imply EOF for dedupe

2016-11-23 Thread Dave Chinner
On Wed, Nov 23, 2016 at 08:55:59AM -0500, Zygo Blaxell wrote: > On Wed, Nov 23, 2016 at 03:26:32PM +1100, Dave Chinner wrote: > > On Tue, Nov 22, 2016 at 09:02:10PM -0500, Zygo Blaxell wrote: > > > On Thu, Nov 17, 2016 at 04:07:48PM -0800, Omar Sandoval wrote: > > >

Re: bio linked list corruption.

2016-11-23 Thread Dave Jones
On Wed, Nov 23, 2016 at 02:34:19PM -0500, Dave Jones wrote: > [ 317.689216] BUG: Bad page state in process kworker/u8:8 pfn:4d8fd4 > trace from just before this happened. Does this shed any light ? > > https://codemonkey.org.uk/junk/trace.txt crap, I just noticed the times

Re: bio linked list corruption.

2016-11-23 Thread Dave Jones
On Mon, Oct 31, 2016 at 01:44:55PM -0600, Chris Mason wrote: > On Mon, Oct 31, 2016 at 12:35:16PM -0700, Linus Torvalds wrote: > >On Mon, Oct 31, 2016 at 11:55 AM, Dave Jones > >wrote: > >> > >> BUG: Bad page state in process kworker/u8:12 pfn:4e0e39 &

Re: [RFC PATCH 0/2] Btrfs: make a source length of 0 imply EOF for dedupe

2016-11-22 Thread Dave Chinner
to find out what the limit is, other than by trial > and error. It's not even in a header file, userspace just has to *know*. So add a define to the API to make it visible to applications and document it in the man page. Cheers, Dave. -- Dave Chinner da...@fromorbit.com -- To unsubscribe

Re: [PATCH] fstests: Block btrfs from test case generic/372

2016-11-16 Thread Dave Chinner
nk to use a _within_tolerance range would mean the test would validate file1 on all reflink supporting filesystems and we don't need to exclude btrfs at all... Cheers, Dave. -- Dave Chinner da...@fromorbit.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: btrfs btree_ctree_super fault

2016-11-10 Thread Dave Jones
ay_log:2491: errno=-17 Object already exists (Failed to recover log tree) BTRFS error (device sda3): cleaner transaction attach returned -30 BTRFS error (device sda3): open_ctree failed Guess I'll hit it with btrfsck and hope for the best.. Dave -- To unsubscribe from this list: s

Re: [PATCH] fstests: btrfs: Check false ENOSPC bug caused by incorrect metadata reserve

2016-11-09 Thread Dave Chinner
On Thu, Nov 10, 2016 at 11:24:34AM +0800, Eryu Guan wrote: > On Thu, Nov 10, 2016 at 10:42:36AM +0800, Qu Wenruo wrote: > > > > > > At 11/10/2016 10:19 AM, Dave Chinner wrote: > > > On Thu, Nov 10, 2016 at 08:34:20AM +0800, Qu Wenruo wrote: > > > &g

Re: [PATCH] fstests: btrfs: Check false ENOSPC bug caused by incorrect metadata reserve

2016-11-09 Thread Dave Chinner
n that exposes the problem. You wouldn't be suggesting this specific set of mount options if users weren't using that configuration. Hence you really need to run the entire test suite with that configuration to make sure you haven't broken those user's systems And, yes, I tes

Re: btrfs btree_ctree_super fault

2016-11-08 Thread Dave Jones
On Sun, Nov 06, 2016 at 11:55:39AM -0500, Dave Jones wrote: > > > On Mon, Oct 31, 2016 at 01:44:55PM -0600, Chris Mason wrote: > > On Mon, Oct 31, 2016 at 12:35:16PM -0700, Linus Torvalds wrote: > > >On Mon, Oct 31, 2016 at 11:55 AM, Dave Jones > wrote: >

Re: btrfs btree_ctree_super fault

2016-11-06 Thread Dave Jones
On Mon, Oct 31, 2016 at 01:44:55PM -0600, Chris Mason wrote: > On Mon, Oct 31, 2016 at 12:35:16PM -0700, Linus Torvalds wrote: > >On Mon, Oct 31, 2016 at 11:55 AM, Dave Jones > >wrote: > >> > >> BUG: Bad page state in process kworker/u8:12 pfn:4e0e39 &

Re: [PATCH] generic: create and delete files repeatedly to exercise ENOSPC behaviour

2016-11-01 Thread Dave Chinner
+ rm -f $testfile1 $testfile2 > + done > +} What are you trying to test here that other ENOSPC tests don't exercise? Why cant you just use preallocation to trigger ENOSPC repeatedly instead of writing data? That would allow multiple iterations per second, not one every few

Re: [PATCH v4] generic: make 17[1-4] work well when btrfs compression is enabled

2016-11-01 Thread Dave Chinner
ore it's included from common/populate. So there is no reason to put it in common/rc. Cheers, Dave. -- Dave Chinner da...@fromorbit.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo

Re: bio linked list corruption.

2016-10-31 Thread Dave Jones
On Wed, Oct 26, 2016 at 07:47:51PM -0400, Dave Jones wrote: > On Wed, Oct 26, 2016 at 07:38:08PM -0400, Chris Mason wrote: > > > >-hctx->queued++; > > >-data->hctx = hctx; > > >-data->ctx = ctx; > > >+

Re: [PATCH 5/5] fs: don't set *REFERENCED unless we are on the lru list

2016-10-27 Thread Dave Chinner
On Thu, Oct 27, 2016 at 09:13:44AM -0400, Josef Bacik wrote: > On 10/26/2016 08:30 PM, Dave Chinner wrote: > >On Wed, Oct 26, 2016 at 11:11:35AM -0400, Josef Bacik wrote: > >>On 10/25/2016 06:01 PM, Dave Chinner wrote: > >>>On Tue, Oct 25, 2016 at 02:41:44PM -0400,

Re: bio linked list corruption.

2016-10-27 Thread Dave Jones
On Thu, Oct 27, 2016 at 04:41:33PM +1100, Dave Chinner wrote: > And that's indicative of a delalloc metadata reservation being > being too small and so we're allocating unreserved blocks. > > Different symptoms, same underlying cause, I think. > > I see th

Re: bio linked list corruption.

2016-10-26 Thread Dave Chinner
On Tue, Oct 25, 2016 at 08:27:52PM -0400, Dave Jones wrote: > DaveC: Do these look like real problems, or is this more "looks like > random memory corruption" ? It's been a while since I did some stress > testing on XFS, so these might not be new.. > > XFS: Ass

Re: [PATCH 5/5] fs: don't set *REFERENCED unless we are on the lru list

2016-10-26 Thread Dave Chinner
On Wed, Oct 26, 2016 at 11:11:35AM -0400, Josef Bacik wrote: > On 10/25/2016 06:01 PM, Dave Chinner wrote: > >On Tue, Oct 25, 2016 at 02:41:44PM -0400, Josef Bacik wrote: > >>With anything that populates the inode/dentry cache with a lot of one time > >>use > >&

Re: bio linked list corruption.

2016-10-26 Thread Dave Jones
++; > >return rq; > > } > > This made it through an entire dbench 2048 run on btrfs. My script has > it running in a loop, but this is farther than I've gotten before. > Looking great so far. Fixed the splat during boot for me too. Now the fun part, let&#

Re: bio linked list corruption.

2016-10-26 Thread Dave Jones
On Wed, Oct 26, 2016 at 05:03:45PM -0600, Jens Axboe wrote: > On 10/26/2016 04:58 PM, Linus Torvalds wrote: > > On Wed, Oct 26, 2016 at 3:51 PM, Linus Torvalds > > wrote: > >> > >> Dave: it might be a good idea to split that "WARN_ON_ONCE()"

Re: bio linked list corruption.

2016-10-26 Thread Dave Jones
On Wed, Oct 26, 2016 at 03:51:01PM -0700, Linus Torvalds wrote: > Dave: it might be a good idea to split that "WARN_ON_ONCE()" in > blk_mq_merge_queue_io() into two, since right now it can trigger both > for the > > blk_mq_bio_to_request(rq, bio);

Re: bio linked list corruption.

2016-10-26 Thread Dave Jones
On Wed, Oct 26, 2016 at 03:21:53PM -0700, Linus Torvalds wrote: > Could you try the attached patch? It adds a couple of sanity tests: > > - a number of tests to verify that 'rq->queuelist' isn't already on > some queue when it is added to a queue > > - one test to verify that rq->mq_ctx

Re: [PATCH 5/5] fs: don't set *REFERENCED unless we are on the lru list

2016-10-26 Thread Dave Chinner
On Wed, Oct 26, 2016 at 04:03:54PM -0400, Josef Bacik wrote: > On 10/25/2016 07:36 PM, Dave Chinner wrote: > >So, 2-way has not improved. If changing referenced behaviour was an > >obvious win for btrfs, we'd expect to see that here as well. > >however, because 4-way im

Re: bio linked list corruption.

2016-10-26 Thread Dave Jones
On Wed, Oct 26, 2016 at 09:48:39AM -0700, Linus Torvalds wrote: > I know you already had this in some email, but I lost it. I think you > narrowed it down to a specific set of system calls that seems to > trigger this best. fallocate and xattrs or something? So I was about to give that a shot

Re: bio linked list corruption.

2016-10-26 Thread Dave Jones
On Wed, Oct 26, 2016 at 09:48:39AM -0700, Linus Torvalds wrote: > On Wed, Oct 26, 2016 at 9:30 AM, Dave Jones wrote: > > > > I gave this a go last thing last night. It crashed within 5 minutes, > > but it was one we've already seen (the bad page map trace) with no

Re: bio linked list corruption.

2016-10-26 Thread Dave Jones
d seems to work for me, but doesn't show anything. > > Dave, mind just trying that oneliner? I gave this a go last thing last night. It crashed within 5 minutes, but it was one we've already seen (the bad page map trace) with nothing additional that looked interesting. I re

Re: bio linked list corruption.

2016-10-25 Thread Dave Jones
On Mon, Oct 24, 2016 at 09:42:39AM -0400, Chris Mason wrote: > > > Well crud, we're back to wondering if this is Btrfs or the stack > > > corruption. Since the pagevecs are on the stack and this is a new > > > crash, my guess is you'll be able to trigger it on xfs/ext4 too. But we > > >

Re: [PATCH 5/5] fs: don't set *REFERENCED unless we are on the lru list

2016-10-25 Thread Dave Chinner
On Wed, Oct 26, 2016 at 09:01:13AM +1100, Dave Chinner wrote: > On Tue, Oct 25, 2016 at 02:41:44PM -0400, Josef Bacik wrote: > > With anything that populates the inode/dentry cache with a lot of one time > > use > > inodes we can really put a lot of pressure on the syste

Re: [PATCH 5/5] fs: don't set *REFERENCED unless we are on the lru list

2016-10-25 Thread Dave Chinner
he file systems. Less than 1% for XFS and ~1.5% for ext4 is well within the run-to-run variation of fsmark. It looks like it might be slightly faster, but it's not a cut-and-dried win for anything other than btrfs. Cheers, Dave. -- Dave Chinner da...@fromorbit.com -- To unsubscribe

Re: bio linked list corruption.

2016-10-23 Thread Dave Jones
On Sun, Oct 23, 2016 at 05:32:21PM -0400, Chris Mason wrote: > > > On 10/22/2016 11:20 AM, Dave Jones wrote: > > On Fri, Oct 21, 2016 at 04:02:45PM -0400, Dave Jones wrote: > > > > > > It could be worth trying this, too: > > > > >

Re: bio linked list corruption.

2016-10-22 Thread Dave Jones
On Fri, Oct 21, 2016 at 04:02:45PM -0400, Dave Jones wrote: > > It could be worth trying this, too: > > > > > https://git.kernel.org/cgit/linux/kernel/git/luto/linux.git/commit/?h=x86/vmap_stack&id=174531fef4e8 > > > > It occurred to me that the

Re: bio linked list corruption.

2016-10-21 Thread Dave Jones
t; If so then that would explain it. If not we should probably dig into it. > Thanks, Yeah, that's definitely possible. And it wasn't that long ago I added some code to always open testfiles multiple times with different modes, so the likely of O_DIRECT went up. That would explain why

Re: bio linked list corruption.

2016-10-21 Thread Dave Jones
> a block of all zeros in multiple files, > > but it struck me as surprising. > > > > btrfs people: is there an easy way to map those inodes to a filename ? I'm > > betting those are the > > test files that trinity generates. If so, it might point to a r

Re: bio linked list corruption.

2016-10-21 Thread Dave Jones
On Thu, Oct 20, 2016 at 04:23:32PM -0700, Andy Lutomirski wrote: > On Thu, Oct 20, 2016 at 4:03 PM, Dave Jones wrote: > > On Thu, Oct 20, 2016 at 04:01:12PM -0700, Andy Lutomirski wrote: > > > On Thu, Oct 20, 2016 at 3:50 PM, Dave Jones > > wrote: > > >

Re: bio linked list corruption.

2016-10-20 Thread Dave Jones
On Thu, Oct 20, 2016 at 04:01:12PM -0700, Andy Lutomirski wrote: > On Thu, Oct 20, 2016 at 3:50 PM, Dave Jones wrote: > > On Tue, Oct 18, 2016 at 06:05:57PM -0700, Andy Lutomirski wrote: > > > > > One possible debugging approach would be to change: > > >

Re: bio linked list corruption.

2016-10-20 Thread Dave Jones
RTUAL=y can be quite helpful for debugging stack > issues. I'm tempted to do something equivalent to hardwiring that > option on for a while if CONFIG_VMAP_STACK=y. This one I had on. Nothing interesting jumped out. Dave -- To unsubscribe from this list: send the line &

Re: bio linked list corruption.

2016-10-20 Thread Dave Jones
it within a few minutes using just writev, fsync, lsetxattr and lremovexattr. Then a day later, I found I could run for a day before seeing it. Position of the moon or something. Or it could have been entirely unrelated to the actual syscalls being run, and based just on how contended the cpu/mem

Re: bio linked list corruption.

2016-10-18 Thread Dave Jones
On Tue, Oct 11, 2016 at 10:45:07AM -0400, Dave Jones wrote: > WARNING: CPU: 1 PID: 3673 at lib/list_debug.c:33 __list_add+0x89/0xb0 > list_add corruption. prev->next should be next (e8806648), but was > c967fcd8. (prev=880503878b80). > CPU: 1 PID: 3673 C

Re: btrfs bio linked list corruption.

2016-10-15 Thread Dave Jones
On Thu, Oct 13, 2016 at 05:18:46PM -0400, Chris Mason wrote: > > > > .. and of course the first thing that happens is a completely > > different > > > > btrfs trace.. > > > > > > > > > > > > WARNING: CPU: 1 PID: 21706 at fs/btrfs/transaction.c:489 > > start_transaction+0x40a/0x440 [b

Re: btrfs bio linked list corruption.

2016-10-13 Thread Dave Jones
> > > Hasn't triggered here yet. I'll leave it running though. With that combo of params I triggered it 3-4 times in a row within minutes.. Then as soon as I posted, it stopped being so easy to repro. There's some other variable I haven't figured out yet (

Re: btrfs bio linked list corruption.

2016-10-13 Thread Dave Jones
On Wed, Oct 12, 2016 at 10:42:46AM -0400, Chris Mason wrote: > On 10/12/2016 10:40 AM, Dave Jones wrote: > > On Wed, Oct 12, 2016 at 09:47:17AM -0400, Dave Jones wrote: > > > On Tue, Oct 11, 2016 at 11:54:09AM -0400, Chris Mason wrote: > > > > > > &g

Re: btrfs bio linked list corruption.

2016-10-12 Thread Dave Jones
On Wed, Oct 12, 2016 at 09:47:17AM -0400, Dave Jones wrote: > On Tue, Oct 11, 2016 at 11:54:09AM -0400, Chris Mason wrote: > > > > > > On 10/11/2016 10:45 AM, Dave Jones wrote: > > > This is from Linus' current tree, with Al's iovec fixups on top.

Re: btrfs bio linked list corruption.

2016-10-12 Thread Dave Jones
On Tue, Oct 11, 2016 at 11:54:09AM -0400, Chris Mason wrote: > > > On 10/11/2016 10:45 AM, Dave Jones wrote: > > This is from Linus' current tree, with Al's iovec fixups on top. > > > > [ cut here ] > > WARNING: CPU: 1

Re: btrfs bio linked list corruption.

2016-10-11 Thread Dave Jones
On Tue, Oct 11, 2016 at 11:54:09AM -0400, Chris Mason wrote: > > > On 10/11/2016 10:45 AM, Dave Jones wrote: > > This is from Linus' current tree, with Al's iovec fixups on top. > > > > [ cut here ] > > WARNING: CPU: 1

Re: btrfs bio linked list corruption.

2016-10-11 Thread Dave Jones
On Tue, Oct 11, 2016 at 11:20:41AM -0400, Chris Mason wrote: > > > On 10/11/2016 11:19 AM, Dave Jones wrote: > > On Tue, Oct 11, 2016 at 04:11:39PM +0100, Al Viro wrote: > > > On Tue, Oct 11, 2016 at 10:45:08AM -0400, Dave Jones wrote: > > > > This is

btrfs bio linked list corruption.

2016-10-11 Thread Dave Jones
This is from Linus' current tree, with Al's iovec fixups on top. [ cut here ] WARNING: CPU: 1 PID: 3673 at lib/list_debug.c:33 __list_add+0x89/0xb0 list_add corruption. prev->next should be next (e8806648), but was c967fcd8. (prev=880503878b80). CPU: 1

Re: btrfs bio linked list corruption.

2016-10-11 Thread Dave Jones
On Tue, Oct 11, 2016 at 04:11:39PM +0100, Al Viro wrote: > On Tue, Oct 11, 2016 at 10:45:08AM -0400, Dave Jones wrote: > > This is from Linus' current tree, with Al's iovec fixups on top. > > Those iovec fixups are in the current tree... ah yeah, git quietly dro

Re: [PATCH v2] generic: make 17[1-4] work well when btrfs compression is enabled

2016-10-10 Thread Dave Chinner
ant size, especially as the data will compress down to nearly nothing. Trying to hack around compression artifacts by inflating the size of the file just doesn't work reliably. The way to fix this is to either use one of the "fill filesystem" functions that isn't affected by co

Re: [PATCH] generic/175: disable inline data feature for btrfs

2016-10-10 Thread Dave Chinner
needs to be avoided, then add an option to filter them out. e.g. something like this: _scratch_options_filter btrfs compress so that it removes any compression option from the btrfs mount/mkfs that is run for that test. Cheers, Dave. -- Dave Chinner da...@fromorbit.com -- To unsubscribe from t

Re: btrfs_direct_IO oops

2016-10-10 Thread Dave Jones
gers? Been running for an hour without incident so far, so this looks promising. I'll follow-up if anything jumps out, but normally I'd have seen this by now. Dave -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html

<    1   2   3   4   5   6   7   8   >