Re: no activity in kernel.org btrfs-progs git repo?
On Thu, Dec 13, 2012 at 4:21 AM, Hugo Mills wrote: >Two things: > > 1) That tree is generally only ever updated for pull requests to Linus >during the merge window. Would that it were otherwise, but I >suspect that... > > 2) ... you may want to know about josef's btrfs-next tree: > > https://btrfs.wiki.kernel.org/index.php/Btrfs_source_repositories#Integration_repository_.28btrfs-next.29 > >It's regarded as fairly unstable, so only run it on systems for > testing purposes (or if josef tells you it's OK and will fix one of > your problems). > I think Zach was asking about the userspace tools, not the kernel module. That link points to the kernel module integration branches, and the link above that on the wiki page about btrfs-progs integration hasn't had commits in the last 6 months, AFAICT. I've been concerned about the status btrfs-progs maintenance myself. I've seen quite a few obvious bugs about btrfs-progs on this list in the past months—most of them with patches. However, I don't see them enter any upstream git repositories. Cheers, -- ~Nirbheek Chauhan -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] Btrfs-progs: fix resolving of loop devices
From: Nirbheek Chauhan
The LOOP_GET_STATUS ioctl truncates filenames to 64 characters. We should get
the backing file for a given loop device from /sys/. This is how losetup does it
as well.
---
utils.c | 26 ++
1 files changed, 14 insertions(+), 12 deletions(-)
diff --git a/utils.c b/utils.c
index 205e667..cdd6f7d 100644
--- a/utils.c
+++ b/utils.c
@@ -20,6 +20,7 @@
#define __USE_XOPEN2K
#include
#include
+#include
#ifndef __CHECKER__
#include
#include
@@ -651,21 +652,22 @@ int is_loop_device (const char* device) {
* the associated file (e.g. /images/my_btrfs.img) */
int resolve_loop_device(const char* loop_dev, char* loop_file, int max_len)
{
- int loop_fd;
- int ret_ioctl;
- struct loop_info loopinfo;
+ int ret;
+ FILE *f;
+ char fmt[20];
+ char p[PATH_MAX];
+ char real_loop_dev[PATH_MAX];
- if ((loop_fd = open(loop_dev, O_RDONLY)) < 0)
+ if (!realpath(loop_dev, real_loop_dev))
+ return -errno;
+ snprintf(p, PATH_MAX, "/sys/block/%s/loop/backing_file",
strrchr(real_loop_dev, '/'));
+ if (!(f = fopen(p, "r")))
return -errno;
- ret_ioctl = ioctl(loop_fd, LOOP_GET_STATUS, &loopinfo);
- close(loop_fd);
-
- if (ret_ioctl == 0) {
- strncpy(loop_file, loopinfo.lo_name, max_len);
- if (max_len > 0)
- loop_file[max_len-1] = 0;
- } else
+ snprintf(fmt, 20, "%%%i[^\n]", max_len-1);
+ ret = fscanf(f, fmt, loop_file);
+ fclose(f);
+ if (ret == EOF)
return -errno;
return 0;
--
1.7.8.6
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] Btrfs-progs: loopback files cannot be used if the path is too long
From: Nirbheek Chauhan If the path to a given loopback file is longer than 64 characters, none of the Btrfs-progs tools can use it. This is because the size of loopinfo.lo_name returned by the LOOP_GET_STATUS ioctl is 64. The attached patch fixes this by fetching the backing file for a loopback device from /sys/block; which is how `losetup` from util-linux does it as well. Nirbheek Chauhan (1): Btrfs-progs: fix resolving of loop devices utils.c | 26 ++ 1 files changed, 14 insertions(+), 12 deletions(-) -- 1.7.8.6 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Applications using fsync cause hangs for several seconds every few minutes
On Mon, Jul 18, 2011 at 11:47 PM, Josef Bacik wrote: > On 06/06/2011 06:58 PM, Nirbheek Chauhan wrote: >> What can I do to debug this issue? What other information should I >> supply? Could someone guide me on how to figure out why my machine is >> unusable now? > > I've been looking into this and I have a suspicion. Would you run with this > patch and see if the problem goes away? If so I'm on the right track > and I'll > have more test patches for you to try :). Thanks, > Hello! Thanks for taking a look! I'm currently travelling and as soon as I get a bit closer to my backups, I'll try the patch out and report back for further testing :) Cheers, -- ~Nirbheek Chauhan Gentoo GNOME+Mozilla Team -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Applications using fsync cause hangs for several seconds every few minutes
Hello list, I've been using btrfs on my personal machines for about two years now, and on this machine for about a year with absolutely no problems. Infact, it has held up better than ext4 with regards to reliability. However, recently, perhaps with 2.6.39, or after I quickly started filling up my disk again, it has become impossible for me to work for long periods on my machine. Every few minutes, (I guess) when applications do fsync (firefox, xchat, vim, etc), all applications that use fsync() hang for several seconds, and applications that use general IO suffer extreme slowdowns. iotop shows various combinations of the processes listed below doing writes, and the total write as 2-3MB/s. [btrfs-dealloc-] [btrfs-submit-0] [btrfs-transacti] [btrfs-endio-wri] [flush-btrfs-1] In some extreme cases, I've had hangs for 5 whole minutes. I'm really beginning to appreciate how little I/O GNOME Shell does since it remains completely responsive throughout this. I have a feeling that the cause for this is extreme fragmentation. My hard disk is a 500GB SATA hdd, my btrfs partition details are: # btrfs filesystem show Label: 'gentoo' uuid: 6f539d7f-f70f-4216-a4a9-6f7a2117a04a Total devices 1 FS bytes used 246.37GB devid1 size 345.13GB used 345.13GB path /dev/sda7 Btrfs v0.19-35-g1b444cd-dirty What can I do to debug this issue? What other information should I supply? Could someone guide me on how to figure out why my machine is unusable now? Thanks in advance, -- ~Nirbheek Chauhan Gentoo GNOME+Mozilla Team -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: "Appending" data to the middle of a file using btrfs-specific features
[I think the mail was sent to just me due to a reply-accident, I've re-added the mailing list for this reply] On Tue, Dec 7, 2010 at 3:50 PM, David Pottage wrote: > On 06/12/10 12:41, Nirbheek Chauhan wrote: >> >> I'd like to know if there has been any discussion about adding a new >> feature to write (add) data at an offset, but without overwriting >> existing data, or re-writing the existing data. Essentially, in-place >> addition/removal of data to a file at a place other than the end of >> the file. >> >> Some possible use-cases of such a feature would be: >> >> (a) Databases (currently hack around this by allocating sparse files) >> (b) Delta-patching (rsync, patch, xdelta, etc) >> (c) Video editors (especially if combined with reflink copies) >> >> Besides I/O savings, it would also have significant space savings if >> the current subvolume being written to has been snapshotted (a common >> use-case for incremental backups). >> > > This idea was discussed back in June. (Search the archives for "Complex > filesystem operations: split and join" > > Back then the idea was to achieve insertion and removal of data by splitting > and joining existing files, so to insert data in the middle of a file, you > would cut it in two, append data to the first file and then re-join it. > Aha, I searched the archives and I found the thread in question[1], thanks! The original thread seems to have gone for a split/join implementation that would work with vfat along with a new syscall. > I think that direct insertion and removal of data is a cleaner idea, though > it may result in a more complex API. You could still achieve cutting files > into two by creating a COW copy of the file and truncating one, and removing > a block of bytes from the start of the other. > I agree, being able to manipulate file stream in a way similar to inserting/deleting in linked lists would introduce new possibilities (and challenges, I'm sure). As you mentioned in the original thread, it's quite strange that there's no way to do this with current file API. > I still think it would be a good idea to be able to join files together with > a file system API call, so the equivalent of: > > cat track1.mp3 track2.mp3 track3.mp3 > mix_tape.mp3 > > Could be done as a filesystem call to create mix_tape.mp3 as a de-duplicated > copy of the contents of the three source files, without many megabytes of > I/O. > Ah, this is relatively straightforward with the clone_range ioctl. There was some talk about a reflink() or clone() syscall a while ago[2], perhaps that could be extended as reflink_range() so that it could be used with other filesystems which support reflinks as well. 1. http://thread.gmane.org/gmane.linux.kernel/996835 2. http://lwn.net/Articles/333783/ -- ~Nirbheek Chauhan Gentoo GNOME+Mozilla Team -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: "Appending" data to the middle of a file using btrfs-specific features
On Tue, Dec 7, 2010 at 2:12 AM, Freddie Cash wrote: > On Mon, Dec 6, 2010 at 12:30 PM, Nirbheek Chauhan > wrote: >> But the behaviour of --inplace is not entirely to write out *only* the >> blocks that have changed. From what I could make out, it does the >> following: >> >> (1) Calculate a delta b/w the src and trg files >> (2) Seek to the first difference in the target file >> (3) Start writing data > > That may be true, I've never looked into the actual algorithm(s) that > rsync uses. Just played around with CLI options until we found the > set that works best in our situation (--inplace --delete-during > --no-whole-file --numeric-ids --hard-links --archive, over SSH with > HPN patches). > >> I'm glossing over the final step because I didn't look deeper, but I >> think you can safely assume that after the first difference, all data >> is rewritten. So this is halfway between "rewrite the whole file" and >> "write only the changed bits into the file". It doesn't actually use >> any CoW features from what I can see. There is lots of room for btrfs >> reflinking magic. :) >> >> Note that I tested this behaviour on a btrfs partition with a vanilla >> rsync-3.0.7 tarball; the copy you use with ZFS might be doing some CoW >> magic. > > All the CoW "magic" is handled by the filesystem, and not the tools on > top. If the tool only updates X bytes, which fit into 1 block on the > fs, then only that 1 block gets updated via CoW. > I'm quite sure that's what happens in btrfs too, but the thing about updating in-place is that if you have ABCDXXXEFGH which needs to change to ABCDZZZEFGH You're all good. Only the blocks corresponding to XXX will be updated. But if the change is ABCDEFGH You'll need to start rewriting EFGH since there's no way to insert data in the middle (afaik) of a file with standard syscalls. Maybe later you get a set of changes which sync you up with the file's contents again, but the chances of that happening in a large file are quite remote. That's why I said that it can be safely assumed that after the first difference, all data is rewritten. The only way to get around this on the filesystem level that I can think of is data de-duplication; the filesystem doesn't let go of the blocks for a while, and does reflinking if the same data is written again. Perhaps that's what ZFS is doing, I have no idea :) -- ~Nirbheek Chauhan Gentoo GNOME+Mozilla Team -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: "Appending" data to the middle of a file using btrfs-specific features
On Tue, Dec 7, 2010 at 1:05 AM, Freddie Cash wrote: > On Mon, Dec 6, 2010 at 11:14 AM, Nirbheek Chauhan > wrote: >> As an aside, my primary motivation for this was that doing an >> incremental backup of things like git bare repositories and databases >> using btrfs subvolume snapshots is expensive w.r.t. disk space. Even >> though rsync calculates a binary delta before transferring data, it >> has to write everything out (except if just appending). So in that >> case, each "incremental" backup is hardly so. > > Since btrfs is Copy-on-Write, have you experimented with --inplace on > the rsync command-line? That way, rsync writes the changes "over-top" > of the existing file, thus allowing btrfs to only write out the blocks > that have changed, via CoW? > > We do this with our ZFS rsync backups, and found disk usage to go way > down over the default "write out new data to new file, rename overtop" > method that rsync uses. > > There's also the --no-whole-file option which causes rsync to only > send delta changes for existing files, another useful feature with CoW > filesystems. > I had tried the --inplace option, but it didn't seem to do anything for me, so I didn't explore that further. However, after following your suggestion and retrying with --no-whole-file, I see that the behaviour is quite different! It seems that --whole-file is enabled by default for local file transfers, and so --inplace had no effect. But the behaviour of --inplace is not entirely to write out *only* the blocks that have changed. From what I could make out, it does the following: (1) Calculate a delta b/w the src and trg files (2) Seek to the first difference in the target file (3) Start writing data I'm glossing over the final step because I didn't look deeper, but I think you can safely assume that after the first difference, all data is rewritten. So this is halfway between "rewrite the whole file" and "write only the changed bits into the file". It doesn't actually use any CoW features from what I can see. There is lots of room for btrfs reflinking magic. :) Note that I tested this behaviour on a btrfs partition with a vanilla rsync-3.0.7 tarball; the copy you use with ZFS might be doing some CoW magic. Thanks for the tip! -- ~Nirbheek Chauhan Gentoo GNOME+Mozilla Team -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: "Appending" data to the middle of a file using btrfs-specific features
On Mon, Dec 6, 2010 at 9:35 PM, Chris Mason wrote: > Excerpts from Nirbheek Chauhan's message of 2010-12-06 07:41:16 -0500: [snip] >> Some possible use-cases of such a feature would be: >> >> (a) Databases (currently hack around this by allocating sparse files) >> (b) Delta-patching (rsync, patch, xdelta, etc) >> (c) Video editors (especially if combined with reflink copies) >> >> Besides I/O savings, it would also have significant space savings if >> the current subvolume being written to has been snapshotted (a common >> use-case for incremental backups). >> [snip] >> A hack I can think of is to do a BTRFS_IOC_CLONE_RANGE into a new file >> (upto the offset), writing whatever data is required, and then doing >> another BTRFS_IOC_CLONE_RANGE with an offset for the rest of the >> original file. This can be followed by a rename() over the original >> file. Similarly for removing data from the middle of a file. Would >> this work? Would it be cleaner to implement something equivalent >> internally? > > It would work yes. The operation has three cases: > > 1) file size doesn't change > 2) extend the file with new bytes in the middle > 3) make the file smaller removing bytes in the middle > > #1 is the easiest case, you can just use the clone range ioctl directly > > For #2 and #3, all of the file pointers past the bytes you want to add > or remove need to be updated with a new file offset. I'd say for an > initial implementation to use the IOC_CLONE_RANGE code, and after > everything is working we can look at optimizing it with a shift ioctl if > it makes sense. > Alrighty, I'll try this and report back any bugs and/or suggestions. > Of the use cases you list, video editors seems the most useful. > Databases already have things pretty much under control, and delta > patching wants to go to a new file anyway. Video editing software has > long been looking for ways to do this. > As an aside, my primary motivation for this was that doing an incremental backup of things like git bare repositories and databases using btrfs subvolume snapshots is expensive w.r.t. disk space. Even though rsync calculates a binary delta before transferring data, it has to write everything out (except if just appending). So in that case, each "incremental" backup is hardly so. Thanks for your help! :) -- ~Nirbheek Chauhan Gentoo GNOME+Mozilla Team -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
"Appending" data to the middle of a file using btrfs-specific features
Hello, I'd like to know if there has been any discussion about adding a new feature to write (add) data at an offset, but without overwriting existing data, or re-writing the existing data. Essentially, in-place addition/removal of data to a file at a place other than the end of the file. Some possible use-cases of such a feature would be: (a) Databases (currently hack around this by allocating sparse files) (b) Delta-patching (rsync, patch, xdelta, etc) (c) Video editors (especially if combined with reflink copies) Besides I/O savings, it would also have significant space savings if the current subvolume being written to has been snapshotted (a common use-case for incremental backups). I've been told that the problem is somewhat difficult to solve properly under block-based representation of data, but I was hoping that btrfs' reflink mechanism and its space-efficient packing of small files might make it doable. A hack I can think of is to do a BTRFS_IOC_CLONE_RANGE into a new file (upto the offset), writing whatever data is required, and then doing another BTRFS_IOC_CLONE_RANGE with an offset for the rest of the original file. This can be followed by a rename() over the original file. Similarly for removing data from the middle of a file. Would this work? Would it be cleaner to implement something equivalent internally? Thanks! -- ~Nirbheek Chauhan Gentoo GNOME+Mozilla Team -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
