Re: Announcing btrfs-dedupe

Perhaps the complexity of doing this efficiently makes it inappropriate for inclusion in the tool itself, whereas I believe the core implementation's focus is on in-band deduplication, automatic and behind the scenes. On 08/11/16 03:40, Christoph Anton Mitterer wrote: On Mon, 2016-11-07 at

Re: Announcing btrfs-dedupe

On Mon, 2016-11-07 at 15:02 +0100, David Sterba wrote: > I think adding a whole-file dedup mode to duperemove would be better > (from user's POV) than writing a whole new tool What would IMO be really good from a user's POV was, if one of the tools, deemed to be the "best", would be added to the

[PATCH v3 2/3] btrfs: Add trace point for qgroup reserved space

Introduce the following trace points: qgroup_update_reserve qgroup_meta_reserve These trace points are handy to trace qgroup reserve space related problems. Signed-off-by: Qu Wenruo --- Although I hope these trace points will not be used again. v2: None v3:

[PATCH v3 3/3] btrfs: qgroup: Re-arrange tracepoint timing to co-operate with reserved space tracepoint

Newly introduced qgroup reserved space trace points are normally nested into several common qgroup operations. While some other trace points are not well placed to co-operate with them, causing confusing output. This patch re-arrange trace_btrfs_qgroup_release_data() and

[PATCH v3 1/3] btrfs: Add WARN_ON for qgroup reserved underflow

Goldwyn Rodrigues has exposed and fixed a bug which underflows btrfs qgroup reserved space, and leads to non-writable fs. This reminds us that we don't have enough underflow check for qgroup reserved space. For underflow case, we should not really underflow the numbers but warn and keeps qgroup

Re: Announcing btrfs-dedupe

On Mon, Nov 07, 2016 at 09:54:09PM +0100, Adam Borowski wrote: > On Mon, Nov 07, 2016 at 09:48:41AM -0800, Mark Fasheh wrote: > > also on XFS with the dedupe ioctl (I believe this should be out with > > Linux-4.9). > > It's already there in 4.9-rc1, although you need a special version of >

Re: btrfs support for filesystems >8TB on 32bit architectures

On Tue, Nov 08, 2016 at 08:43:34AM +0800, Qu Wenruo wrote: > That's strange, balance is done completely in kernel space. > > Unless we're calling vfs_* function we won't go through the extra check. > > What's the error reported? See below. Note however that is may be because btrfs received

Re: btrfs support for filesystems >8TB on 32bit architectures

At 11/08/2016 09:06 AM, Marc MERLIN wrote: On Tue, Nov 08, 2016 at 08:43:34AM +0800, Qu Wenruo wrote: That's strange, balance is done completely in kernel space. Unless we're calling vfs_* function we won't go through the extra check. What's the error reported? See below. Note however

Re: [PATCH v4 1/4] btrfs-progs: utils: Introduce function to escape characters

At 11/07/2016 06:20 PM, David Sterba wrote: On Thu, Nov 03, 2016 at 12:07:31PM +0800, Qu Wenruo wrote: Introduce new function, escape_string_inplace(), to escape specified characters in place. Signed-off-by: Qu Wenruo --- utils.c | 24

Re: btrfs support for filesystems >8TB on 32bit architectures

At 11/08/2016 08:39 AM, Marc MERLIN wrote: On Tue, Nov 08, 2016 at 08:35:54AM +0800, Qu Wenruo wrote: Understood. One big thing (for me) I forgot to confirm: 1) btrfs receive Unfortunately, receive is completely done in userspace. Only send works inside kernel. right, I've confirmed that

Re: btrfs support for filesystems >8TB on 32bit architectures

On Tue, Nov 08, 2016 at 08:35:54AM +0800, Qu Wenruo wrote: > >Understood. One big thing (for me) I forgot to confirm: > >1) btrfs receive > > Unfortunately, receive is completely done in userspace. > Only send works inside kernel. right, I've confirmed that btrfs receive fails. It looks like

Re: [PATCH v4 1/4] btrfs-progs: utils: Introduce function to escape characters

At 11/07/2016 06:20 PM, David Sterba wrote: On Thu, Nov 03, 2016 at 12:07:31PM +0800, Qu Wenruo wrote: Introduce new function, escape_string_inplace(), to escape specified characters in place. Signed-off-by: Qu Wenruo --- utils.c | 24

Re: btrfs support for filesystems >8TB on 32bit architectures

At 11/07/2016 10:55 PM, Marc MERLIN wrote: On Mon, Nov 07, 2016 at 02:16:37PM +0800, Qu Wenruo wrote: At 11/07/2016 01:36 PM, Marc MERLIN wrote: (sorry for the bad subject line from the mdadm list on the previous mail) On Mon, Nov 07, 2016 at 12:18:10PM +0800, Qu Wenruo wrote: I'm

Re: [PATCH] btrfs-progs: send-test: add checking of clone-src option

On 2016/11/08 0:16, David Sterba wrote: > On Fri, Nov 04, 2016 at 05:35:18PM +0900, Tsutomu Itoh wrote: >> +before_size=`ls -l "$here"/send.stream.before | awk '{print $5}'` >> +after_size=`ls -l "$here"/send.stream.after | awk '{print $5}'` > > Thanks for the test. Minor fixes: "stat

[PATCH v2] btrfs: make block group flags in balance printks human-readable

They're not even documented anywhere, letting users with no recourse but to RTFS. It's no big burden to output the bitfield as words. Also, display unknown flags as hex. Signed-off-by: Adam Borowski --- fs/btrfs/relocation.c | 53

Re: [PATCH] btrfs: make block group flags in balance printks human-readable

On Mon, Nov 07, 2016 at 05:58:41PM +0100, David Sterba wrote: > On Fri, Nov 04, 2016 at 08:26:54AM +0100, Adam Borowski wrote: > > They're not even documented anywhere, letting users with no recourse but > > to RTFS. It's no big burden to output the bitfield as words. > > Ok. Below are some

Re: Announcing btrfs-dedupe

On Mon, Nov 07, 2016 at 09:48:41AM -0800, Mark Fasheh wrote: > also on XFS with the dedupe ioctl (I believe this should be out with > Linux-4.9). It's already there in 4.9-rc1, although you need a special version of xfsprogs (possibly already released, I didn't check). It's an experimental

kernel BUG at fs/btrfs/delayed-inode.c

Hi, today I got this bug after a power failure. One file that was being written during the power failure, appeared after reboot as: $ ls -la data/0/3833 ?-. 1 root root 0 ene 1 1970 data/0/3833 I decided to delete it, but I got this bug after a few seconds and the system halted, I had

Re: Announcing btrfs-dedupe

FWIW I have updated my comments about duperemove and also the "caveat" section you mentioned in your other mail in the readme. http://btrfs-dedupe.com James On 07/11/16 19:49, James Pharaoh wrote: Annoyingly I can't find this now, but I definitely remember reading someone, apparently someone

Re: Announcing btrfs-dedupe

Annoyingly I can't find this now, but I definitely remember reading someone, apparently someone knowledgable, claim that the latest version of the kernel which I was using at the time, still suffered from issues regarding the dedupe code. This was a while ago, and I would be very pleased to

Re: [PATCH v14 02/15] btrfs: fix false enospc for compression

On Fri, Nov 04, 2016 at 09:32:51AM +0800, Qu Wenruo wrote: > int btrfs_set_extent_delalloc(struct inode *inode, u64 start, u64 end, > - struct extent_state **cached_state, int dedupe); > + struct extent_state **cached_state, int flag); During a

Re: [PATCH 0/4] Qgroup comment enhance and balance fix

On Tue, Oct 18, 2016 at 09:31:25AM +0800, Qu Wenruo wrote: > The patchset does the following things: > 1) Enhance comment for qgroup, rename 2 functions >Explain the how qgroup works, so new developers won't waste too much >time digging into the boring codes. > >The qgroup work flow

Re: Announcing btrfs-dedupe

Hi James, Re the following text on your project page: "IMPORTANT CAVEAT — I have read that there are race and/or error conditions which can cause filesystem corruption in the kernel implementation of the deduplication ioctl." Can you expound on that? I'm not aware of any bugs right now but if

Re: [PATCH v2 1/2] btrfs: Add WARN_ON for qgroup reserved underflow

On Thu, Oct 20, 2016 at 10:28:41AM +0800, Qu Wenruo wrote: > Goldwyn Rodrigues has exposed and fixed a bug which underflows btrfs > qgroup reserved space, and leads to non-writable fs. > > This reminds us that we don't have enough underflow check for qgroup > reserved space. > > For underflow

Re: [PATCH 2/2] btrfs: Add trace point for qgroup reserved space

On Thu, Oct 20, 2016 at 10:28:42AM +0800, Qu Wenruo wrote: > Introduce the following trace points: > qgroup_update_reserve > qgroup_meta_reserve > > And modify the timing of btrfs_qgroup_free_delayed_ref() and > btrfs_qgroup_release_data() events, to work with qgroup_update_reserve() > event. I

Re: Announcing btrfs-dedupe

Hi David and James, On Mon, Nov 7, 2016 at 6:02 AM, David Sterba wrote: > On Sun, Nov 06, 2016 at 02:30:52PM +0100, James Pharaoh wrote: >> I'm pleased to announce my btrfs deduplication utility, written in Rust. >> This operates on whole files, is fast, and I believe

Re: [PATCH v2 02/14] btrfs-progs: check: introduce function to find dir_item

On Thu, Nov 03, 2016 at 09:58:21AM +0800, Qu Wenruo wrote: > > > At 11/02/2016 11:21 PM, David Sterba wrote: > > On Wed, Sep 21, 2016 at 11:15:52AM +0800, Qu Wenruo wrote: > >> From: Lu Fengqi > >> > >> Introduce a new function find_dir_item() to find DIR_ITEM for the

Re: [PATCH] btrfs: make block group flags in balance printks human-readable

On Fri, Nov 04, 2016 at 08:26:54AM +0100, Adam Borowski wrote: > They're not even documented anywhere, letting users with no recourse but > to RTFS. It's no big burden to output the bitfield as words. Ok. Below are some comments (and style nitpicks). > Also, display unknown flags as hex. > >

Re: [PATCH V2] btrfs: Remove some dead code

On Fri, Nov 04, 2016 at 08:02:35AM +0100, Christophe JAILLET wrote: > 'btrfs_iget()' can not return NULL, so this test can be removed. > > Signed-off-by: Christophe JAILLET Patch replaced, I've edited the subject to be a bit more specific. -- To unsubscribe from

Re: [PATCH v2 3/3] btrfs-progs: send: fix handling of -c option

On Fri, Nov 04, 2016 at 05:33:58PM +0900, Tsutomu Itoh wrote: > When two or more -c options are specified, cannot find a suitable > parent. So, output stream is bigger than correct one. > > [before] > # btrfs send -f /tmp/data1 -c Snap0 -c ../SnapX Snap[12] ../SnapY > At subvol Snap1 > At subvol

Re: [PATCH] btrfs-progs: send-test: add checking of clone-src option

On Fri, Nov 04, 2016 at 05:35:18PM +0900, Tsutomu Itoh wrote: > +before_size=`ls -l "$here"/send.stream.before | awk '{print $5}'` > +after_size=`ls -l "$here"/send.stream.after | awk '{print $5}'` Thanks for the test. Minor fixes: "stat --format=%s" instead of the above, and /dev/zero instead of

Re: btrfs support for filesystems >8TB on 32bit architectures

On Mon, Nov 07, 2016 at 02:16:37PM +0800, Qu Wenruo wrote: > > > At 11/07/2016 01:36 PM, Marc MERLIN wrote: > > (sorry for the bad subject line from the mdadm list on the previous mail) > > > > On Mon, Nov 07, 2016 at 12:18:10PM +0800, Qu Wenruo wrote: > > > I'm totally wrong here. > > > > > >

Re: [PATCH] btrfs: make block group flags in balance printks human-readable

On 11/04/16 08:26, Adam Borowski wrote: > They're not even documented anywhere, letting users with no recourse but > to RTFS. It's no big burden to output the bitfield as words. > > Also, display unknown flags as hex. > > Signed-off-by: Adam Borowski Very helpful and

Re: Announcing btrfs-dedupe

On Sun, Nov 06, 2016 at 02:30:52PM +0100, James Pharaoh wrote: > I'm pleased to announce my btrfs deduplication utility, written in Rust. > This operates on whole files, is fast, and I believe complements the > existing utilities (duperemove, bedup), which exist currently. Mark can correct me

Re: [PATCH v4 2/4] btrfs-progs: introduce new send-dump object

On Thu, Nov 03, 2016 at 12:07:32PM +0800, Qu Wenruo wrote: > +/* > + * Disable format zero length warning since we use zero length format > + * for unified function parameters. > + */ > +#pragma GCC diagnostic ignored "-Wformat-zero-length" No such tricks should be necessary, NULL instead of ""

Re: [PATCH v4 1/4] btrfs-progs: utils: Introduce function to escape characters

On Thu, Nov 03, 2016 at 12:07:31PM +0800, Qu Wenruo wrote: > Introduce new function, escape_string_inplace(), to escape specified > characters in place. > > Signed-off-by: Qu Wenruo > --- > utils.c | 24 > utils.h | 14 ++ > 2 files

[PATCH 2/2] btrfs: increase tickets_id even for failed metadata request

Not functional change, it just makes codes logic more reasonable, then at least tickets_id can reflect the number of metadata requests we already handled. Signed-off-by: Wang Xiaoguang --- fs/btrfs/extent-tree.c | 6 -- 1 file changed, 4 insertions(+), 2

[PATCH 1/2] btrfs: add necessary comments about tickets_id

Tickets_id's name may result in some misunderstandings, it just indicates the next ticket will be handled and is not stored per ticket. Fixes: ce12965 ("btrfs: introduce tickets_id to determine whether asynchronous metadata reclaim work makes progress") Signed-off-by: Wang Xiaoguang