Re: no activity in kernel.org btrfs-progs git repo?

2012-12-12 Thread Nirbheek Chauhan
On Thu, Dec 13, 2012 at 4:21 AM, Hugo Mills h...@carfax.org.uk wrote:
Two things:

 1) That tree is generally only ever updated for pull requests to Linus
during the merge window. Would that it were otherwise, but I
suspect that...

 2) ... you may want to know about josef's btrfs-next tree:

 https://btrfs.wiki.kernel.org/index.php/Btrfs_source_repositories#Integration_repository_.28btrfs-next.29

It's regarded as fairly unstable, so only run it on systems for
 testing purposes (or if josef tells you it's OK and will fix one of
 your problems).


I think Zach was asking about the userspace tools, not the kernel
module. That link points to the kernel module integration branches,
and the link above that on the wiki page about btrfs-progs integration
hasn't had commits in the last 6 months, AFAICT.

I've been concerned about the status btrfs-progs maintenance myself.
I've seen quite a few obvious bugs about btrfs-progs on this list in
the past months—most of them with patches. However, I don't see them
enter any upstream git repositories.

Cheers,

--
~Nirbheek Chauhan
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[PATCH] Btrfs-progs: loopback files cannot be used if the path is too long

2012-10-12 Thread nirbheek . chauhan
From: Nirbheek Chauhan nirbheek.chau...@collabora.co.uk

If the path to a given loopback file is longer than 64 characters, none of the
Btrfs-progs tools can use it. This is because the size of loopinfo.lo_name
returned by the LOOP_GET_STATUS ioctl is 64.

The attached patch fixes this by fetching the backing file for a loopback device
from /sys/block; which is how `losetup` from util-linux does it as well.

Nirbheek Chauhan (1):
  Btrfs-progs: fix resolving of loop devices

 utils.c |   26 ++
 1 files changed, 14 insertions(+), 12 deletions(-)

-- 
1.7.8.6

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[PATCH] Btrfs-progs: fix resolving of loop devices

2012-10-12 Thread nirbheek . chauhan
From: Nirbheek Chauhan nirbheek.chau...@collabora.co.uk

The LOOP_GET_STATUS ioctl truncates filenames to 64 characters. We should get
the backing file for a given loop device from /sys/. This is how losetup does it
as well.
---
 utils.c |   26 ++
 1 files changed, 14 insertions(+), 12 deletions(-)

diff --git a/utils.c b/utils.c
index 205e667..cdd6f7d 100644
--- a/utils.c
+++ b/utils.c
@@ -20,6 +20,7 @@
 #define __USE_XOPEN2K
 #include stdio.h
 #include stdlib.h
+#include string.h
 #ifndef __CHECKER__
 #include sys/ioctl.h
 #include sys/mount.h
@@ -651,21 +652,22 @@ int is_loop_device (const char* device) {
  * the associated file (e.g. /images/my_btrfs.img) */
 int resolve_loop_device(const char* loop_dev, char* loop_file, int max_len)
 {
-   int loop_fd;
-   int ret_ioctl;
-   struct loop_info loopinfo;
+   int ret;
+   FILE *f;
+   char fmt[20];
+   char p[PATH_MAX];
+   char real_loop_dev[PATH_MAX];
 
-   if ((loop_fd = open(loop_dev, O_RDONLY))  0)
+   if (!realpath(loop_dev, real_loop_dev))
+   return -errno;
+   snprintf(p, PATH_MAX, /sys/block/%s/loop/backing_file, 
strrchr(real_loop_dev, '/'));
+   if (!(f = fopen(p, r)))
return -errno;
 
-   ret_ioctl = ioctl(loop_fd, LOOP_GET_STATUS, loopinfo);
-   close(loop_fd);
-
-   if (ret_ioctl == 0) {
-   strncpy(loop_file, loopinfo.lo_name, max_len);
-   if (max_len  0)
-   loop_file[max_len-1] = 0;
-   } else
+   snprintf(fmt, 20, %%%i[^\n], max_len-1);
+   ret = fscanf(f, fmt, loop_file);
+   fclose(f);
+   if (ret == EOF)
return -errno;
 
return 0;
-- 
1.7.8.6

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: Applications using fsync cause hangs for several seconds every few minutes

2011-07-20 Thread Nirbheek Chauhan
On Mon, Jul 18, 2011 at 11:47 PM, Josef Bacik jo...@redhat.com wrote:
 On 06/06/2011 06:58 PM, Nirbheek Chauhan wrote:
 What can I do to debug this issue? What other information should I
 supply? Could someone guide me on how to figure out why my machine is
 unusable now?

 I've been looking into this and I have a suspicion.  Would you run with this
 patch and see if the problem goes away?  If so I'm on the right track
 and I'll
 have more test patches for you to try :).  Thanks,


Hello!

Thanks for taking a look! I'm currently travelling and as soon as I
get a bit closer to my backups, I'll try the patch out and report back
for further testing :)

Cheers,

-- 
~Nirbheek Chauhan

Gentoo GNOME+Mozilla Team
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Applications using fsync cause hangs for several seconds every few minutes

2011-06-06 Thread Nirbheek Chauhan
Hello list,

I've been using btrfs on my personal machines for about two years now,
and on this machine for about a year with absolutely no problems.
Infact, it has held up better than ext4 with regards to reliability.

However, recently, perhaps with 2.6.39, or after I quickly started
filling up my disk again, it has become impossible for me to work for
long periods on my machine.

Every few minutes, (I guess) when applications do fsync (firefox,
xchat, vim, etc), all applications that use fsync() hang for several
seconds, and applications that use general IO suffer extreme
slowdowns. iotop shows various combinations of the processes listed
below doing writes, and the total write as 2-3MB/s.

[btrfs-dealloc-]
[btrfs-submit-0]
[btrfs-transacti]
[btrfs-endio-wri]
[flush-btrfs-1]

In some extreme cases, I've had hangs for 5 whole minutes. I'm really
beginning to appreciate how little I/O GNOME Shell does since it
remains completely responsive throughout this. I have a feeling that
the cause for this is extreme fragmentation.

My hard disk is a 500GB SATA hdd, my btrfs partition details are:

# btrfs filesystem show
Label: 'gentoo'  uuid: 6f539d7f-f70f-4216-a4a9-6f7a2117a04a
Total devices 1 FS bytes used 246.37GB
devid1 size 345.13GB used 345.13GB path /dev/sda7

Btrfs v0.19-35-g1b444cd-dirty

What can I do to debug this issue? What other information should I
supply? Could someone guide me on how to figure out why my machine is
unusable now?

Thanks in advance,

-- 
~Nirbheek Chauhan

Gentoo GNOME+Mozilla Team
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: Appending data to the middle of a file using btrfs-specific features

2010-12-07 Thread Nirbheek Chauhan
[I think the mail was sent to just me due to a reply-accident, I've
re-added the mailing list for this reply]

On Tue, Dec 7, 2010 at 3:50 PM, David Pottage da...@electric-spoon.com wrote:
 On 06/12/10 12:41, Nirbheek Chauhan wrote:

 I'd like to know if there has been any discussion about adding a new
 feature to write (add) data at an offset, but without overwriting
 existing data, or re-writing the existing data. Essentially, in-place
 addition/removal of data to a file at a place other than the end of
 the file.

 Some possible use-cases of such a feature would be:

 (a) Databases (currently hack around this by allocating sparse files)
 (b) Delta-patching (rsync, patch, xdelta, etc)
 (c) Video editors (especially if combined with reflink copies)

 Besides I/O savings, it would also have significant space savings if
 the current subvolume being written to has been snapshotted (a common
 use-case for incremental backups).


 This idea was discussed back in June. (Search the archives for Complex
 filesystem operations: split and join

 Back then the idea was to achieve insertion and removal of data by splitting
 and joining existing files, so to insert data in the middle of a file, you
 would cut it in two, append data to the first file and then re-join it.


Aha, I searched the archives and I found the thread in question[1],
thanks! The original thread seems to have gone for a split/join
implementation that would work with vfat along with a new syscall.

 I think that direct insertion and removal of data is a cleaner idea, though
 it may result in a more complex API. You could still achieve cutting files
 into two by creating a COW copy of the file and truncating one, and removing
 a block of bytes from the start of the other.


I agree, being able to manipulate file stream in a way similar to
inserting/deleting in linked lists would introduce new possibilities
(and challenges, I'm sure). As you mentioned in the original thread,
it's quite strange that there's no way to do this with current file
API.

 I still think it would be a good idea to be able to join files together with
 a file system API call, so the equivalent of:

    cat track1.mp3 track2.mp3 track3.mp3  mix_tape.mp3

 Could be done as a filesystem call to create mix_tape.mp3 as a de-duplicated
 copy of the contents of the three source files, without many megabytes of
 I/O.


Ah, this is relatively straightforward with the clone_range ioctl.
There was some talk about a reflink() or clone() syscall a while
ago[2], perhaps that could be extended as reflink_range() so that it
could be used with other filesystems which support reflinks as well.

1. http://thread.gmane.org/gmane.linux.kernel/996835
2. http://lwn.net/Articles/333783/

-- 
~Nirbheek Chauhan

Gentoo GNOME+Mozilla Team
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Appending data to the middle of a file using btrfs-specific features

2010-12-06 Thread Nirbheek Chauhan
Hello,

I'd like to know if there has been any discussion about adding a new
feature to write (add) data at an offset, but without overwriting
existing data, or re-writing the existing data. Essentially, in-place
addition/removal of data to a file at a place other than the end of
the file.

Some possible use-cases of such a feature would be:

(a) Databases (currently hack around this by allocating sparse files)
(b) Delta-patching (rsync, patch, xdelta, etc)
(c) Video editors (especially if combined with reflink copies)

Besides I/O savings, it would also have significant space savings if
the current subvolume being written to has been snapshotted (a common
use-case for incremental backups).

I've been told that the problem is somewhat difficult to solve
properly under block-based representation of data, but I was hoping
that btrfs' reflink mechanism and its space-efficient packing of small
files might make it doable.

A hack I can think of is to do a BTRFS_IOC_CLONE_RANGE into a new file
(upto the offset), writing whatever data is required, and then doing
another BTRFS_IOC_CLONE_RANGE with an offset for the rest of the
original file. This can be followed by a rename() over the original
file. Similarly for removing data from the middle of a file. Would
this work? Would it be cleaner to implement something equivalent
internally?

Thanks!

-- 
~Nirbheek Chauhan

Gentoo GNOME+Mozilla Team
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: Appending data to the middle of a file using btrfs-specific features

2010-12-06 Thread Nirbheek Chauhan
On Mon, Dec 6, 2010 at 9:35 PM, Chris Mason chris.ma...@oracle.com wrote:
 Excerpts from Nirbheek Chauhan's message of 2010-12-06 07:41:16 -0500:
[snip]
 Some possible use-cases of such a feature would be:

 (a) Databases (currently hack around this by allocating sparse files)
 (b) Delta-patching (rsync, patch, xdelta, etc)
 (c) Video editors (especially if combined with reflink copies)

 Besides I/O savings, it would also have significant space savings if
 the current subvolume being written to has been snapshotted (a common
 use-case for incremental backups).

[snip]
 A hack I can think of is to do a BTRFS_IOC_CLONE_RANGE into a new file
 (upto the offset), writing whatever data is required, and then doing
 another BTRFS_IOC_CLONE_RANGE with an offset for the rest of the
 original file. This can be followed by a rename() over the original
 file. Similarly for removing data from the middle of a file. Would
 this work? Would it be cleaner to implement something equivalent
 internally?

 It would work yes.  The operation has three cases:

 1) file size doesn't change
 2) extend the file with new bytes in the middle
 3) make the file smaller removing bytes in the middle

 #1 is the easiest case, you can just use the clone range ioctl directly

 For #2 and #3, all of the file pointers past the bytes you want to add
 or remove need to be updated with a new file offset.  I'd say for an
 initial implementation to use the IOC_CLONE_RANGE code, and after
 everything is working we can look at optimizing it with a shift ioctl if
 it makes sense.


Alrighty, I'll try this and report back any bugs and/or suggestions.

 Of the use cases you list, video editors seems the most useful.
 Databases already have things pretty much under control, and delta
 patching wants to go to a new file anyway.  Video editing software has
 long been looking for ways to do this.


As an aside, my primary motivation for this was that doing an
incremental backup of things like git bare repositories and databases
using btrfs subvolume snapshots is expensive w.r.t. disk space. Even
though rsync calculates a binary delta before transferring data, it
has to write everything out (except if just appending). So in that
case, each incremental backup is hardly so.

Thanks for your help! :)

-- 
~Nirbheek Chauhan

Gentoo GNOME+Mozilla Team
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: Appending data to the middle of a file using btrfs-specific features

2010-12-06 Thread Nirbheek Chauhan
On Tue, Dec 7, 2010 at 1:05 AM, Freddie Cash fjwc...@gmail.com wrote:
 On Mon, Dec 6, 2010 at 11:14 AM, Nirbheek Chauhan
 nirbheek.chau...@gmail.com wrote:
 As an aside, my primary motivation for this was that doing an
 incremental backup of things like git bare repositories and databases
 using btrfs subvolume snapshots is expensive w.r.t. disk space. Even
 though rsync calculates a binary delta before transferring data, it
 has to write everything out (except if just appending). So in that
 case, each incremental backup is hardly so.

 Since btrfs is Copy-on-Write, have you experimented with --inplace on
 the rsync command-line?  That way, rsync writes the changes over-top
 of the existing file, thus allowing btrfs to only write out the blocks
 that have changed, via CoW?

 We do this with our ZFS rsync backups, and found disk usage to go way
 down over the default write out new data to new file, rename overtop
 method that rsync uses.

 There's also the --no-whole-file option which causes rsync to only
 send delta changes for existing files, another useful feature with CoW
 filesystems.


I had tried the --inplace option, but it didn't seem to do anything
for me, so I didn't explore that further. However, after following
your suggestion and retrying with --no-whole-file, I see that the
behaviour is quite different! It seems that --whole-file is enabled by
default for local file transfers, and so --inplace had no effect.

But the behaviour of --inplace is not entirely to write out *only* the
blocks that have changed. From what I could make out, it does the
following:

(1) Calculate a delta b/w the src and trg files
(2) Seek to the first difference in the target file
(3) Start writing data

I'm glossing over the final step because I didn't look deeper, but I
think you can safely assume that after the first difference, all data
is rewritten. So this is halfway between rewrite the whole file and
write only the changed bits into the file. It doesn't actually use
any CoW features from what I can see. There is lots of room for btrfs
reflinking magic. :)

Note that I tested this behaviour on a btrfs partition with a vanilla
rsync-3.0.7 tarball; the copy you use with ZFS might be doing some CoW
magic.

Thanks for the tip!

-- 
~Nirbheek Chauhan

Gentoo GNOME+Mozilla Team
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: Appending data to the middle of a file using btrfs-specific features

2010-12-06 Thread Nirbheek Chauhan
On Tue, Dec 7, 2010 at 2:12 AM, Freddie Cash fjwc...@gmail.com wrote:
 On Mon, Dec 6, 2010 at 12:30 PM, Nirbheek Chauhan
 nirbheek.chau...@gmail.com wrote:
 But the behaviour of --inplace is not entirely to write out *only* the
 blocks that have changed. From what I could make out, it does the
 following:

 (1) Calculate a delta b/w the src and trg files
 (2) Seek to the first difference in the target file
 (3) Start writing data

 That may be true, I've never looked into the actual algorithm(s) that
 rsync uses.  Just played around with CLI options until we found the
 set that works best in our situation (--inplace --delete-during
 --no-whole-file --numeric-ids --hard-links --archive, over SSH with
 HPN patches).

 I'm glossing over the final step because I didn't look deeper, but I
 think you can safely assume that after the first difference, all data
 is rewritten. So this is halfway between rewrite the whole file and
 write only the changed bits into the file. It doesn't actually use
 any CoW features from what I can see. There is lots of room for btrfs
 reflinking magic. :)

 Note that I tested this behaviour on a btrfs partition with a vanilla
 rsync-3.0.7 tarball; the copy you use with ZFS might be doing some CoW
 magic.

 All the CoW magic is handled by the filesystem, and not the tools on
 top.  If the tool only updates X bytes, which fit into 1 block on the
 fs, then only that 1 block gets updated via CoW.


I'm quite sure that's what happens in btrfs too, but the thing about
updating in-place is that if you have

ABCDXXXEFGH

which needs to change to

ABCDZZZEFGH

You're all good. Only the blocks corresponding to XXX will be updated.
But if the change is

ABCDEFGH

You'll need to start rewriting EFGH since there's no way to insert
data in the middle (afaik) of a file with standard syscalls. Maybe
later you get a set of changes which sync you up with the file's
contents again, but the chances of that happening in a large file are
quite remote. That's why I said that it can be safely assumed that
after the first difference, all data is rewritten.

The only way to get around this on the filesystem level that I can
think of is data de-duplication; the filesystem doesn't let go of the
blocks for a while, and does reflinking if the same data is written
again. Perhaps that's what ZFS is doing, I have no idea :)

-- 
~Nirbheek Chauhan

Gentoo GNOME+Mozilla Team
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html