Re: Many questions from a potential btrfs user

[Hugo Mills]

On Sun, Oct 13, 2013 at 11:54:42PM -0300, Rogério Brito wrote:
 Hi.
 
 I am seriously considering employing btrfs on my systems, particularly due
 to some space-saving features that it has (namely, deduplication and
 compression).
 
 In fact, I was (a few moments ago) trying to back up some of my systems to a
 2TB HD that has an ext4 filesystem and, in the middle of the last one, I got
 the error message that the backup HD was full.
 
 Given that what I backup there are systems where I have some of the data
 present multiple times (e.g., my mailbox that is sync'ed via offlineimap, or
 videos that I download from online learning sites) and that such data
 consists of many small files that are highly compressible (the e-mails) or
 large files (the videos), I would like to employ btrfs.
 
 So, after reading the documentation on https://btrfs.wiki.kernel.org/, I am
 still unsure of some points and I would like to have some clarifications
 and/or expectations set straight.
 
 
 * I understand that I can convert an ext4 filesystem to btrfs. Will such
   conversion work with an almost full ext4 filesystem? How much overhead
   will be needed to perform the conversion? I can (temporarily) remove some
   files that already are on this backup.

   I don't think we've ever explored the bounds of exactly how much
space you need for conversion. It'll be an absolute minimum of 0.1% of
the data used, probably quite a bit more, for the metadata.

 * Is it possible to deduplicate the files that are already in it? As
   mentioned before, there are likely to be many, and some of them are on the
   order of 1 to 2GBs.

   Yes, there's an out-of-band deduplicator. I'll have to go and look
it up to work out exactly what tools you need to make it work. :)

 * Doing a defragmentation with the filesystem mounted with compression will
   recompress the files (if they are deemed compressible by the
   filesystem). Is that understanding correct?  Will compressed blocks among
   many files also be deduplicated?

   You'll probably need to add -c to the defrag command, but yes, you
can persuade the FS to recompress files. I'm not sure how this affects
deduplication.

 * How exactly do the recently merged offline deduplication features in the
   kernel interfere with what was (in my limited understanding) already
   possible with userspace tools like https://github.com/g2p/bedup?  Are
   such third-party tools likely to be integrated into btrfs-progs? Are they
   supposed to be kept separate?

   The out-of-band (rather than offline) dedup kernel features simply
give a more reliable API call for merging identical extents, as it
allows them to be locked during the process -- without that API call,
there's a race condition that could potentially lead to data loss.

 * Does this change the on-disk format? Putting it another way, will it be
   safe to possibly go back to a previous kernel, if there is some problem
   with the current kernels? (Not that I necessarily want to go back to a
   previous kernel, but, sometimes, one would need to, say, git bisect the
   kernel).

   No, that feature doesn't change the on-disk format.

 * I most likely *don't* want to use online deduplication (given my bad
   experiences with ZFS).  With that in mind, is the current userspace
   deduplicaton intended to be run as a cron job? Is the offline
   deduplication too memory intensive?  How much RAM would it be needed for a
   2TB filesystem? Are 2GB enough? How about 4GB?

   Out-of-band dedup is indeed the kind of thing you'd run as a cron
job. However, there's a couple of better approaches you can use. I
don't know about RAM usage, I'm afraid.

   If you use rsync for backups, then you can keep one subvolume as
the current version of the backups, and use the --in-place option of
rsync. Then, immediately after finishing a backup run, you can
snapshot that subvolume to give yourself a read-only historical
record. This will ensure that the maximum quantity of data is shared
between the individual backups without having to use OOB dedup.

   If your source FS is btrfs as well, you can do pretty much the same
thing (it's a little more complicated to set up) with btrfs
send/receive, which uses the inherent knowledge of the FS to work out
the differences more efficiently than rsync.

 * Will further runs of the offline deduplication be incremental in some
   imprecise sense of the word? That is, if I run the deduplication once and
   immediately run it again (supposing nothing changes), will the 2nd time be
   faster than the first?  (If the disk caches are dropped?)

   I don't know, but probably (since it should be able to tell that
the extents are already CoW copies).

 * Will I be able to add further HDs to my btrfs filesystem, once I get some
   more money to run something like a RAID0 configuration? If I get more HDs
   later, will I be able to change the configuration to, say, RAID5 or RAID6?
   I don't intend to use lvm, unless I have to.

   Yes, you 

Re: [PATCH] xfstests btrfs/309: test btrfs-convert

[Rich Johnston]

This has been committed.

Thanks
--Rich

commit 65d4646af1f84883857a60ab00629145b5972b04
Author: Eric Sandeen sand...@redhat.com
Date:   Sun Aug 4 20:12:31 2013 +

xfstests btrfs/012: test btrfs-convert

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] Btrfs: btrfs_add_ordered_operation: Fix last modified transaction comparison.

[chandan]

Comparison of an inode's last modified transaction with the last committed
transaction is incorrect. Fix it.

Signed-off-by: chandan chan...@linux.vnet.ibm.com
---
 fs/btrfs/ordered-data.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/btrfs/ordered-data.c b/fs/btrfs/ordered-data.c
index c702cb6..1a36a0c 100644
--- a/fs/btrfs/ordered-data.c
+++ b/fs/btrfs/ordered-data.c
@@ -1076,7 +1076,7 @@ void btrfs_add_ordered_operation(struct 
btrfs_trans_handle *trans,
 * if this file hasn't been changed since the last transaction
 * commit, we can safely return without doing anything
 */
-   if (last_mod  root-fs_info-last_trans_committed)
+   if (last_mod = root-fs_info-last_trans_committed)
return;
 
spin_lock(root-fs_info-ordered_root_lock);
-- 
1.8.3.1


--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v6] btrfs: Fix memory leakage in the tree-log.c

[Josef Bacik]

On Sun, Oct 13, 2013 at 07:03:12PM -0300, Geyslan Gregório Bem wrote:
 2013/10/11 Stefan Behrens sbehr...@giantdisaster.de:
  On 10/11/2013 20:35, Geyslan G. Bem wrote:
 
  In add_inode_ref() function:
 
  Initializes local pointers.
 
  Reduces the logical condition with the __add_inode_ref() return
  value by using only one 'goto out'.
 
  Centralizes the exiting, ensuring the freeing of all used memory.
 
  Signed-off-by: Geyslan G. Bem geys...@gmail.com
  ---
fs/btrfs/tree-log.c | 33 +++--
1 file changed, 19 insertions(+), 14 deletions(-)
 
  diff --git a/fs/btrfs/tree-log.c b/fs/btrfs/tree-log.c
  index 79f057c..61bb051 100644
  --- a/fs/btrfs/tree-log.c
  +++ b/fs/btrfs/tree-log.c
  @@ -1113,11 +1113,11 @@ static noinline int add_inode_ref(struct
  btrfs_trans_handle *trans,
struct extent_buffer *eb, int slot,
struct btrfs_key *key)
{
  -   struct inode *dir;
  -   struct inode *inode;
  +   struct inode *dir = NULL;
  +   struct inode *inode = NULL;
  unsigned long ref_ptr;
  unsigned long ref_end;
  -   char *name;
  +   char *name = NULL;
  int namelen;
  int ret;
  int search_done = 0;
  @@ -1150,13 +1150,15 @@ static noinline int add_inode_ref(struct
  btrfs_trans_handle *trans,
   * care of the rest
   */
  dir = read_one_inode(root, parent_objectid);
  -   if (!dir)
  -   return -ENOENT;
  +   if (!dir) {
  +   ret = -ENOENT;
  +   goto out;
  +   }
 
  inode = read_one_inode(root, inode_objectid);
  if (!inode) {
  -   iput(dir);
  -   return -EIO;
  +   ret = -EIO;
  +   goto out;
  }
 
  while (ref_ptr  ref_end) {
  @@ -1169,14 +1171,16 @@ static noinline int add_inode_ref(struct
  btrfs_trans_handle *trans,
   */
  if (!dir)
  dir = read_one_inode(root,
  parent_objectid);
  -   if (!dir)
  -   return -ENOENT;
  +   if (!dir) {
  +   ret = -ENOENT;
  +   goto out;
  +   }
  } else {
  ret = ref_get_fields(eb, ref_ptr, namelen, name,
   ref_index);
  }
  if (ret)
  -   return ret;
  +   goto out;
 
  /* if we already have a perfect match, we're done */
  if (!inode_in_dir(root, path, btrfs_ino(dir),
  btrfs_ino(inode),
  @@ -1196,12 +1200,11 @@ static noinline int add_inode_ref(struct
  btrfs_trans_handle *trans,
parent_objectid,
ref_index, name,
  namelen,
search_done);
  -   if (ret == 1) {
  -   ret = 0;
  +   if (ret) {
  +   if (ret == 1)
  +   ret = 0;
  goto out;
  }
  -   if (ret)
  -   goto out;
  }
 
  /* insert our name */
  @@ -1215,6 +1218,7 @@ static noinline int add_inode_ref(struct
  btrfs_trans_handle *trans,
 
  ref_ptr = (unsigned long)(ref_ptr + ref_struct_size) +
  namelen;
  kfree(name);
  +   name = NULL;
  if (log_ref_ver) {
  iput(dir);
  dir = NULL;
  @@ -1225,6 +1229,7 @@ static noinline int add_inode_ref(struct
  btrfs_trans_handle *trans,
  ret = overwrite_item(trans, root, path, eb, slot, key);
out:
  btrfs_release_path(path);
  +   kfree(name);
  iput(dir);
  iput(inode);
  return ret;
 
 
  The patch looks really good now. Thanks!
 
  Reviewed-by: Stefan Behrens sbehr...@giantdisaster.de
 
 
 
 Will it to be committed to the next?

I will pull it into btrfs-next the next time I scrape the list for patches and
then it will go into 3.13.  Thanks,

Josef
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] xfstests: btrfs/011 improvement for compressed filesystems

[Rich Johnston]

This has been committed.

Thanks
--Rich

commit e15d36e67c9bd5f6cafe48f6fc3cef9bb8685e80
Author: Stefan Behrens sbehr...@giantdisaster.de
Date:   Fri Sep 13 10:27:21 2013 +

xfstests: btrfs/011 improvement for compressed filesystems

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v6] btrfs: Fix memory leakage in the tree-log.c

[Geyslan Gregório Bem]

2013/10/14 Josef Bacik jba...@fusionio.com:
 On Sun, Oct 13, 2013 at 07:03:12PM -0300, Geyslan Gregório Bem wrote:
 2013/10/11 Stefan Behrens sbehr...@giantdisaster.de:
  On 10/11/2013 20:35, Geyslan G. Bem wrote:
 
  In add_inode_ref() function:
 
  Initializes local pointers.
 
  Reduces the logical condition with the __add_inode_ref() return
  value by using only one 'goto out'.
 
  Centralizes the exiting, ensuring the freeing of all used memory.
 
  Signed-off-by: Geyslan G. Bem geys...@gmail.com
  ---
fs/btrfs/tree-log.c | 33 +++--
1 file changed, 19 insertions(+), 14 deletions(-)
 
  diff --git a/fs/btrfs/tree-log.c b/fs/btrfs/tree-log.c
  index 79f057c..61bb051 100644
  --- a/fs/btrfs/tree-log.c
  +++ b/fs/btrfs/tree-log.c
  @@ -1113,11 +1113,11 @@ static noinline int add_inode_ref(struct
  btrfs_trans_handle *trans,
struct extent_buffer *eb, int slot,
struct btrfs_key *key)
{
  -   struct inode *dir;
  -   struct inode *inode;
  +   struct inode *dir = NULL;
  +   struct inode *inode = NULL;
  unsigned long ref_ptr;
  unsigned long ref_end;
  -   char *name;
  +   char *name = NULL;
  int namelen;
  int ret;
  int search_done = 0;
  @@ -1150,13 +1150,15 @@ static noinline int add_inode_ref(struct
  btrfs_trans_handle *trans,
   * care of the rest
   */
  dir = read_one_inode(root, parent_objectid);
  -   if (!dir)
  -   return -ENOENT;
  +   if (!dir) {
  +   ret = -ENOENT;
  +   goto out;
  +   }
 
  inode = read_one_inode(root, inode_objectid);
  if (!inode) {
  -   iput(dir);
  -   return -EIO;
  +   ret = -EIO;
  +   goto out;
  }
 
  while (ref_ptr  ref_end) {
  @@ -1169,14 +1171,16 @@ static noinline int add_inode_ref(struct
  btrfs_trans_handle *trans,
   */
  if (!dir)
  dir = read_one_inode(root,
  parent_objectid);
  -   if (!dir)
  -   return -ENOENT;
  +   if (!dir) {
  +   ret = -ENOENT;
  +   goto out;
  +   }
  } else {
  ret = ref_get_fields(eb, ref_ptr, namelen, name,
   ref_index);
  }
  if (ret)
  -   return ret;
  +   goto out;
 
  /* if we already have a perfect match, we're done */
  if (!inode_in_dir(root, path, btrfs_ino(dir),
  btrfs_ino(inode),
  @@ -1196,12 +1200,11 @@ static noinline int add_inode_ref(struct
  btrfs_trans_handle *trans,
parent_objectid,
ref_index, name,
  namelen,
search_done);
  -   if (ret == 1) {
  -   ret = 0;
  +   if (ret) {
  +   if (ret == 1)
  +   ret = 0;
  goto out;
  }
  -   if (ret)
  -   goto out;
  }
 
  /* insert our name */
  @@ -1215,6 +1218,7 @@ static noinline int add_inode_ref(struct
  btrfs_trans_handle *trans,
 
  ref_ptr = (unsigned long)(ref_ptr + ref_struct_size) +
  namelen;
  kfree(name);
  +   name = NULL;
  if (log_ref_ver) {
  iput(dir);
  dir = NULL;
  @@ -1225,6 +1229,7 @@ static noinline int add_inode_ref(struct
  btrfs_trans_handle *trans,
  ret = overwrite_item(trans, root, path, eb, slot, key);
out:
  btrfs_release_path(path);
  +   kfree(name);
  iput(dir);
  iput(inode);
  return ret;
 
 
  The patch looks really good now. Thanks!
 
  Reviewed-by: Stefan Behrens sbehr...@giantdisaster.de
 
 

 Will it to be committed to the next?

 I will pull it into btrfs-next the next time I scrape the list for patches and
 then it will go into 3.13.  Thanks,

 Josef

Thanks to you.
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] btrfs: simplify kmalloc+copy_from_user to memdup_user

[Geyslan G. Bem]

Use memdup_user rather than duplicating its implementation
This is a little bit restricted to reduce false positives

The semantic patch that makes this report is available
in scripts/coccinelle/api/memdup_user.cocci.

More information about semantic patching is available at
http://coccinelle.lip6.fr/

Signed-off-by: Geyslan G. Bem geys...@gmail.com
---
 fs/btrfs/ioctl.c | 11 +++
 1 file changed, 3 insertions(+), 8 deletions(-)

diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
index 9d46f60..f0e3517 100644
--- a/fs/btrfs/ioctl.c
+++ b/fs/btrfs/ioctl.c
@@ -2727,15 +2727,10 @@ static long btrfs_ioctl_file_extent_same(struct file 
*file,
size = sizeof(tmp) +
tmp.dest_count * sizeof(struct btrfs_ioctl_same_extent_info);
 
-   same = kmalloc(size, GFP_NOFS);
-   if (!same) {
-   ret = -EFAULT;
-   goto out;
-   }
+   same = memdup_user((struct btrfs_ioctl_same_args __user *)argp, size);
 
-   if (copy_from_user(same,
-  (struct btrfs_ioctl_same_args __user *)argp, size)) {
-   ret = -EFAULT;
+   if (IS_ERR(same)) {
+   ret = PTR_ERR(same);
goto out;
}
 
-- 
1.8.4

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] btrfs: simplify kmalloc+copy_from_user to memdup_user

[Geyslan G. Bem]

Use memdup_user rather than duplicating its implementation
This is a little bit restricted to reduce false positives

The semantic patch that makes this report is available
in scripts/coccinelle/api/memdup_user.cocci.

More information about semantic patching is available at
http://coccinelle.lip6.fr/

Signed-off-by: Geyslan G. Bem geys...@gmail.com
---
 fs/btrfs/ioctl.c | 11 +++
 1 file changed, 3 insertions(+), 8 deletions(-)

diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
index 9d46f60..f0e3517 100644
--- a/fs/btrfs/ioctl.c
+++ b/fs/btrfs/ioctl.c
@@ -2727,15 +2727,10 @@ static long btrfs_ioctl_file_extent_same(struct file 
*file,
size = sizeof(tmp) +
tmp.dest_count * sizeof(struct btrfs_ioctl_same_extent_info);
 
-   same = kmalloc(size, GFP_NOFS);
-   if (!same) {
-   ret = -EFAULT;
-   goto out;
-   }
+   same = memdup_user((struct btrfs_ioctl_same_args __user *)argp, size);
 
-   if (copy_from_user(same,
-  (struct btrfs_ioctl_same_args __user *)argp, size)) {
-   ret = -EFAULT;
+   if (IS_ERR(same)) {
+   ret = PTR_ERR(same);
goto out;
}
 
-- 
1.8.4

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Issue building a file based rootfs image with mkfs.btrfs

[Saul Wold]

On 10/01/2013 06:18 PM, Chris Mason wrote:

Hi Saul,

The patch ended up a little bigger than I expected because it is sharing 
infrastructure with btfs-convert.  Travel added a little more delay, but I'm 
almost there.



Any news on this patch?

Sau!


-chris

From: Saul Wold [s...@linux.intel.com]
Sent: Tuesday, October 01, 2013 4:50 PM
To: Chris Mason
Cc: linux-btrfs@vger.kernel.org
Subject: Re: Issue building a file based rootfs image with mkfs.btrfs

On 09/28/2013 05:29 AM, Chris Mason wrote:

Quoting Saul Wold (2013-09-19 14:19:34)

Hi there,

I am attempting to build a rootfs image from an existing rootfs
directory tree.  I am using the 0.20 @ 194aa4a of Chris's git repo.

The couple problem I saw was that the target image file needed to exist,
although I think I can patch that then the FS size was much larger than
the actual size, I tracked this to the usage of ftw not accounting for
symlinks, I have a patch for that which I will send once I finish
getting the other issues resolved.

Next issue I hit was an assertion failure after getting not enough free
space message:

not enough free space
add_file_items failed
unable to traverse_directory
Making image is aborted.
mkfs.btrfs: mkfs.c:1542: main: Assertion `!(ret)' failed.

I am kind of stuck on this one, took it as far as I can right now.
Would I be better off dropping back to 0.19 or can we move forward
fixing this?


Hi Saul,

Update on my end, the problem is the image code expects every file to
fit inside a single chunk.  It's only creating 8MB chunks, so any file
over 8MB in size is causing problems.

I'm fixing it up here, I should have a patch for you on Monday.


Any update on this?  Just curious.

Thanks
 Sau!


Thanks!

-chris

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Issue building a file based rootfs image with mkfs.btrfs

[Chris Mason]

Hi Saul,

Sorry for the delay, I'll push the patch out this afternoon.

-chris

Quoting Saul Wold (2013-10-14 11:25:28)
 On 10/01/2013 06:18 PM, Chris Mason wrote:
  Hi Saul,
 
  The patch ended up a little bigger than I expected because it is sharing 
  infrastructure with btfs-convert.  Travel added a little more delay, but 
  I'm almost there.
 
 
 Any news on this patch?
 
 Sau!
 
  -chris
  
  From: Saul Wold [s...@linux.intel.com]
  Sent: Tuesday, October 01, 2013 4:50 PM
  To: Chris Mason
  Cc: linux-btrfs@vger.kernel.org
  Subject: Re: Issue building a file based rootfs image with mkfs.btrfs
 
  On 09/28/2013 05:29 AM, Chris Mason wrote:
  Quoting Saul Wold (2013-09-19 14:19:34)
  Hi there,
 
  I am attempting to build a rootfs image from an existing rootfs
  directory tree.  I am using the 0.20 @ 194aa4a of Chris's git repo.
 
  The couple problem I saw was that the target image file needed to exist,
  although I think I can patch that then the FS size was much larger than
  the actual size, I tracked this to the usage of ftw not accounting for
  symlinks, I have a patch for that which I will send once I finish
  getting the other issues resolved.
 
  Next issue I hit was an assertion failure after getting not enough free
  space message:
 
  not enough free space
  add_file_items failed
  unable to traverse_directory
  Making image is aborted.
  mkfs.btrfs: mkfs.c:1542: main: Assertion `!(ret)' failed.
 
  I am kind of stuck on this one, took it as far as I can right now.
  Would I be better off dropping back to 0.19 or can we move forward
  fixing this?
 
  Hi Saul,
 
  Update on my end, the problem is the image code expects every file to
  fit inside a single chunk.  It's only creating 8MB chunks, so any file
  over 8MB in size is causing problems.
 
  I'm fixing it up here, I should have a patch for you on Monday.
 
  Any update on this?  Just curious.
 
  Thanks
   Sau!
 
  Thanks!
 
  -chris
 
  --
  To unsubscribe from this list: send the line unsubscribe linux-btrfs in
  the body of a message to majord...@vger.kernel.org
  More majordomo info at  http://vger.kernel.org/majordomo-info.html
 
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Conversion from ext4 fails

[John Goerzen]

Hello,

I have been seeing this behavior (on the unmounted filesystem, of course!):

date  time btrfs-convert /dev/mapper/bakfs
Sat Oct 12 23:13:21 CDT 2013
creating btrfs metadata.
error during copy_inodes -1
conversion aborted.

real43m1.076s
user15m25.256s
sys 0m21.508s

This is 100% reproducible on this volume.  It passes fsck.ext4 -f with 
no errors, and I can detect nothing wrong.  This is on a Debian backport 
of kernel 3.10 on i686 32-bit.  The filesystem in question is 2.7T in 
size, with 908G free.  It is a target for BackupPC, and as such has 
many, many hardlinks and inodes in use due to the BackupPC hardlink 
pool.  Some files may be hardlinked to hundreds of others as well.  (A 
quick spot-guess revealed one with 418 hardlinks.  There may be some 
with more.)  I didn't capture df -ih output when I should have, but the 
number of inodes was between 30M and 40M.  The underlying device is an 
LVM RAID0 across two USB drives - so a slow device, but it has always 
worked perfectly. There are no kernel errors involved here.


The btrfs-tools is 0.19+20130705 plus the patch from 
https://patchwork.kernel.org/patch/2828820/ to address the No valid 
Btrfs found bug I encountered, which was reported at 
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=724265 and various 
other places.


What can I do to help with the debugging of this?

Thanks,

John
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] Btrfs: handle a missing extent for the first file extent

[Josef Bacik]

While trying to kill our hole extents I noticed I was seeing problems where we
seek into a file and then start writing and then try to fiemap that file later.
This is because we search for offset 0, don't find anything and so back up one
slot, which puts us at the inode ref or something like that, which means we goto
not_found and create an extent map for our entire search area.  This isn't quite
what we want, we want to move forward one slot and see if there is an extent
there so we can limit our hole extent.  This patch fixes this problem, I will
add a testcase for this as well.  Thanks,

Signed-off-by: Josef Bacik jba...@fusionio.com
---
 fs/btrfs/inode.c | 11 +--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 09c950b..974323a 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -5982,7 +5982,14 @@ again:
found_type = btrfs_key_type(found_key);
if (found_key.objectid != objectid ||
found_type != BTRFS_EXTENT_DATA_KEY) {
-   goto not_found;
+   /*
+* If we backup past the first extent we want to move forward
+* and see if there is an extent in front of us, otherwise we'll
+* say there is a hole for our whole search range which can
+* cause problems.
+*/
+   extent_end = start;
+   goto next;
}
 
found_type = btrfs_file_extent_type(leaf, item);
@@ -5997,7 +6004,7 @@ again:
size = btrfs_file_extent_inline_len(leaf, item);
extent_end = ALIGN(extent_start + size, root-sectorsize);
}
-
+next:
if (start = extent_end) {
path-slots[0]++;
if (path-slots[0] = btrfs_header_nritems(leaf)) {
-- 
1.8.3.1

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] Btrfs: add a sanity test for a vacant extent at the front of a file

[Josef Bacik]

Btrfs_get_extent was not handling this case properly, add a test to make sure we
don't regress.  Thanks,

Signed-off-by: Josef Bacik jba...@fusionio.com
---
 fs/btrfs/tests/inode-tests.c | 133 +--
 1 file changed, 128 insertions(+), 5 deletions(-)

diff --git a/fs/btrfs/tests/inode-tests.c b/fs/btrfs/tests/inode-tests.c
index 92416b3..ce9f9bb 100644
--- a/fs/btrfs/tests/inode-tests.c
+++ b/fs/btrfs/tests/inode-tests.c
@@ -86,6 +86,26 @@ static void insert_extent(struct btrfs_root *root, u64 
start, u64 len,
btrfs_set_file_extent_other_encoding(leaf, fi, 0);
 }
 
+static void insert_inode_item_key(struct btrfs_root *root)
+{
+   struct btrfs_path path;
+   struct extent_buffer *leaf = root-node;
+   struct btrfs_key key;
+   u32 value_len = 0;
+
+   memset(path, 0, sizeof(path));
+
+   path.nodes[0] = leaf;
+   path.slots[0] = 0;
+
+   key.objectid = BTRFS_INODE_ITEM_KEY;
+   key.type = BTRFS_INODE_ITEM_KEY;
+   key.offset = 0;
+
+   setup_items_for_insert(root, path, key, value_len, value_len,
+  value_len + sizeof(struct btrfs_item), 1);
+}
+
 /*
  * Build the most complicated map of extents the earth has ever seen.  We want
  * this so we can test all of the corner cases of btrfs_get_extent.  Here is a
@@ -236,10 +256,6 @@ static noinline int test_btrfs_get_extent(void)
u64 offset;
int ret = -ENOMEM;
 
-   set_bit(EXTENT_FLAG_COMPRESSED, compressed_only);
-   set_bit(EXTENT_FLAG_VACANCY, vacancy_only);
-   set_bit(EXTENT_FLAG_PREALLOC, prealloc_only);
-
inode = btrfs_new_test_inode();
if (!inode) {
test_msg(Couldn't allocate inode\n);
@@ -825,8 +841,115 @@ out:
return ret;
 }
 
+static int test_hole_first(void)
+{
+   struct inode *inode = NULL;
+   struct btrfs_root *root = NULL;
+   struct extent_map *em = NULL;
+   int ret = -ENOMEM;
+
+   inode = btrfs_new_test_inode();
+   if (!inode) {
+   test_msg(Couldn't allocate inode\n);
+   return ret;
+   }
+
+   BTRFS_I(inode)-location.type = BTRFS_INODE_ITEM_KEY;
+   BTRFS_I(inode)-location.objectid = BTRFS_FIRST_FREE_OBJECTID;
+   BTRFS_I(inode)-location.offset = 0;
+
+   root = btrfs_alloc_dummy_root();
+   if (IS_ERR(root)) {
+   test_msg(Couldn't allocate root\n);
+   goto out;
+   }
+
+   root-fs_info = alloc_dummy_fs_info();
+   if (!root-fs_info) {
+   test_msg(Couldn't allocate dummy fs info\n);
+   goto out;
+   }
+
+   root-node = alloc_dummy_extent_buffer(0, 4096);
+   if (!root-node) {
+   test_msg(Couldn't allocate dummy buffer\n);
+   goto out;
+   }
+
+   extent_buffer_get(root-node);
+   btrfs_set_header_nritems(root-node, 0);
+   btrfs_set_header_level(root-node, 0);
+   BTRFS_I(inode)-root = root;
+   ret = -EINVAL;
+
+   /*
+* Need a blank inode item here just so we don't confuse
+* btrfs_get_extent.
+*/
+   insert_inode_item_key(root);
+   insert_extent(root, 4096, 4096, 4096, 0, 4096, 4096,
+ BTRFS_FILE_EXTENT_REG, 0, 1);
+   em = btrfs_get_extent(inode, NULL, 0, 0, 8192, 0);
+   if (IS_ERR(em)) {
+   test_msg(Got an error when we shouldn't have\n);
+   goto out;
+   }
+   if (em-block_start != EXTENT_MAP_HOLE) {
+   test_msg(Expected a hole, got %Lu\n, em-block_start);
+   goto out;
+   }
+   if (em-start != 0 || em-len != 4096) {
+   test_msg(Unexpected extent wanted start 0 len 4096, got start 
+%Lu len %Lu\n, em-start, em-len);
+   goto out;
+   }
+   if (em-flags != vacancy_only) {
+   test_msg(Wrong flags, wanted %lu, have %lu\n, vacancy_only,
+em-flags);
+   goto out;
+   }
+   free_extent_map(em);
+
+   em = btrfs_get_extent(inode, NULL, 0, 4096, 8192, 0);
+   if (IS_ERR(em)) {
+   test_msg(Got an error when we shouldn't have\n);
+   goto out;
+   }
+   if (em-block_start != 4096) {
+   test_msg(Expected a real extent, got %Lu\n, em-block_start);
+   goto out;
+   }
+   if (em-start != 4096 || em-len != 4096) {
+   test_msg(Unexpected extent wanted start 4096 len 4096, got 
+start %Lu len %Lu\n, em-start, em-len);
+   goto out;
+   }
+   if (em-flags != 0) {
+   test_msg(Unexpected flags set, wanted 0 got %lu\n,
+em-flags);
+   goto out;
+   }
+   ret = 0;
+out:
+   if (!IS_ERR(em))
+   free_extent_map(em);
+   iput(inode);
+   free_dummy_root(root);
+   return ret;
+}
+
 int btrfs_test_inodes(void)
 {
+   

boot on btrfs, GRUB usage of subvolume set-default breaks boot

[Chris Murphy]

When /boot is on btrfs, and the default subvolume is changed, it makes the 
system unbootable. The basic configuration is subvolumes: boot, root, home, on 
one btrfs volume single device. The installer doesn't change the default 
subvolume, it remains ID 5.

If I change the default subvolume and reboot, I get a grub rescue prompt. This 
means core.img in the MBR gap was found, but /boot/grub/i386-pc/normal.mod 
could not be found. What's interesting is, at the grub rescue prompt, 'set' 
returns:

prefix=(hd0,msdos1)/boot/grub2
root=hd0,msdos1

This appears to be a full path name, so it should always start from top level 
regardless of the set-default subvolume, correct? Yet it's behaving relative to 
the default subvolume. If I use 'ls (hd0,msdos1)' I'm returned the contents of 
the subvolume I've set as default, which explains why bootability is broken.


Chris Murphy--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: boot on btrfs, GRUB usage of subvolume set-default breaks boot

[Chris Murphy]

On Oct 14, 2013, at 1:21 PM, Chris Murphy li...@colorremedies.com wrote:

 When /boot is on btrfs, and the default subvolume is changed, it makes the 
 system unbootable. The basic configuration is subvolumes: boot, root, home, 
 on one btrfs volume single device. The installer doesn't change the default 
 subvolume, it remains ID 5.
 
 If I change the default subvolume and reboot, I get a grub rescue prompt. 
 This means core.img in the MBR gap was found, but 
 /boot/grub/i386-pc/normal.mod could not be found. What's interesting is, at 
 the grub rescue prompt, 'set' returns:
 
 prefix=(hd0,msdos1)/boot/grub2
 root=hd0,msdos1
 
 This appears to be a full path name, so it should always start from top level 
 regardless of the set-default subvolume, correct? Yet it's behaving relative 
 to the default subvolume. If I use 'ls (hd0,msdos1)' I'm returned the 
 contents of the subvolume I've set as default, which explains why bootability 
 is broken.

Looks like what I'm experiencing is unintended. If a developer or someone more 
knowledgable than me can respond to the questions on grub-devel@ it would be 
very helpful so this can get fixed.

http://lists.gnu.org/archive/html/grub-devel/2013-10/msg00038.html


Chris Murphy--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: boot on btrfs, GRUB usage of subvolume set-default breaks boot

[Josef Bacik]

On Mon, Oct 14, 2013 at 02:23:58PM -0600, Chris Murphy wrote:
 
 On Oct 14, 2013, at 1:21 PM, Chris Murphy li...@colorremedies.com wrote:
 
  When /boot is on btrfs, and the default subvolume is changed, it makes the 
  system unbootable. The basic configuration is subvolumes: boot, root, home, 
  on one btrfs volume single device. The installer doesn't change the default 
  subvolume, it remains ID 5.
  
  If I change the default subvolume and reboot, I get a grub rescue prompt. 
  This means core.img in the MBR gap was found, but 
  /boot/grub/i386-pc/normal.mod could not be found. What's interesting is, at 
  the grub rescue prompt, 'set' returns:
  
  prefix=(hd0,msdos1)/boot/grub2
  root=hd0,msdos1
  
  This appears to be a full path name, so it should always start from top 
  level regardless of the set-default subvolume, correct? Yet it's behaving 
  relative to the default subvolume. If I use 'ls (hd0,msdos1)' I'm returned 
  the contents of the subvolume I've set as default, which explains why 
  bootability is broken.
 
 Looks like what I'm experiencing is unintended. If a developer or someone 
 more knowledgable than me can respond to the questions on grub-devel@ it 
 would be very helpful so this can get fixed.
 
 http://lists.gnu.org/archive/html/grub-devel/2013-10/msg00038.html
 

All grub has to do is read the default dir item in the tree_root and then get
the location out of that dir item and then use that root as the default root.
The kernel does it's own path walk to that root just for dentry sake, but I
imagine grub doesn't have to do that, they can just say ok this is our root
and carry on.  Thanks,

Josef
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Issue building a file based rootfs image with mkfs.btrfs

[Chris Mason]

Hi Saul,

Well, a little bigger than I expected turned into a lot bigger than I
expected.  I've pushed the current code to my integration branch, but
the next step is to re-integrate it with Dave's current tree and push
out rc2.

I need to retest convert as well.

-chris

Quoting Chris Mason (2013-10-14 11:26:55)
 Hi Saul,
 
 Sorry for the delay, I'll push the patch out this afternoon.
 
 -chris
 
 Quoting Saul Wold (2013-10-14 11:25:28)
  On 10/01/2013 06:18 PM, Chris Mason wrote:
   Hi Saul,
  
   The patch ended up a little bigger than I expected because it is sharing 
   infrastructure with btfs-convert.  Travel added a little more delay, but 
   I'm almost there.
  
  
  Any news on this patch?
  
  Sau!
  
   -chris
   
   From: Saul Wold [s...@linux.intel.com]
   Sent: Tuesday, October 01, 2013 4:50 PM
   To: Chris Mason
   Cc: linux-btrfs@vger.kernel.org
   Subject: Re: Issue building a file based rootfs image with mkfs.btrfs
  
   On 09/28/2013 05:29 AM, Chris Mason wrote:
   Quoting Saul Wold (2013-09-19 14:19:34)
   Hi there,
  
   I am attempting to build a rootfs image from an existing rootfs
   directory tree.  I am using the 0.20 @ 194aa4a of Chris's git repo.
  
   The couple problem I saw was that the target image file needed to exist,
   although I think I can patch that then the FS size was much larger than
   the actual size, I tracked this to the usage of ftw not accounting for
   symlinks, I have a patch for that which I will send once I finish
   getting the other issues resolved.
  
   Next issue I hit was an assertion failure after getting not enough free
   space message:
  
   not enough free space
   add_file_items failed
   unable to traverse_directory
   Making image is aborted.
   mkfs.btrfs: mkfs.c:1542: main: Assertion `!(ret)' failed.
  
   I am kind of stuck on this one, took it as far as I can right now.
   Would I be better off dropping back to 0.19 or can we move forward
   fixing this?
  
   Hi Saul,
  
   Update on my end, the problem is the image code expects every file to
   fit inside a single chunk.  It's only creating 8MB chunks, so any file
   over 8MB in size is causing problems.
  
   I'm fixing it up here, I should have a patch for you on Monday.
  
   Any update on this?  Just curious.
  
   Thanks
Sau!
  
   Thanks!
  
   -chris
  
   --
   To unsubscribe from this list: send the line unsubscribe linux-btrfs in
   the body of a message to majord...@vger.kernel.org
   More majordomo info at  http://vger.kernel.org/majordomo-info.html
  
 --
 To unsubscribe from this list: send the line unsubscribe linux-btrfs in
 the body of a message to majord...@vger.kernel.org
 More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: boot on btrfs, GRUB usage of subvolume set-default breaks boot

[Chris Murphy]

On Oct 14, 2013, at 2:29 PM, Josef Bacik jba...@fusionio.com wrote:

 On Mon, Oct 14, 2013 at 02:23:58PM -0600, Chris Murphy wrote:
 
 On Oct 14, 2013, at 1:21 PM, Chris Murphy li...@colorremedies.com wrote:
 
 When /boot is on btrfs, and the default subvolume is changed, it makes the 
 system unbootable. The basic configuration is subvolumes: boot, root, home, 
 on one btrfs volume single device. The installer doesn't change the default 
 subvolume, it remains ID 5.
 
 If I change the default subvolume and reboot, I get a grub rescue prompt. 
 This means core.img in the MBR gap was found, but 
 /boot/grub/i386-pc/normal.mod could not be found. What's interesting is, at 
 the grub rescue prompt, 'set' returns:
 
 prefix=(hd0,msdos1)/boot/grub2
 root=hd0,msdos1
 
 This appears to be a full path name, so it should always start from top 
 level regardless of the set-default subvolume, correct? Yet it's behaving 
 relative to the default subvolume. If I use 'ls (hd0,msdos1)' I'm returned 
 the contents of the subvolume I've set as default, which explains why 
 bootability is broken.
 
 Looks like what I'm experiencing is unintended. If a developer or someone 
 more knowledgable than me can respond to the questions on grub-devel@ it 
 would be very helpful so this can get fixed.
 
 http://lists.gnu.org/archive/html/grub-devel/2013-10/msg00038.html
 
 
 All grub has to do is read the default dir item in the tree_root and then 
 get
 the location out of that dir item and then use that root as the default root.
 The kernel does it's own path walk to that root just for dentry sake, but I
 imagine grub doesn't have to do that, they can just say ok this is our root
 and carry on. 

Thanks. I might be reading the inquiry on grub-devel@ wrong but it seems that's 
what they're doing now. The idea is that the GRUB prefix path should be 
immutable, regardless of the user using 'btrfs subvol set-default' to some 
nested subvolume, so that the system is always bootable.


Chris Murphy--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] Btrfs: fix hole check in log_one_extent

[Josef Bacik]

I added an assert to make sure we were looking up aligned offsets for csums and
I tripped it when running xfstests.  This is because log_one_extent was checking
if block_start == 0 for a hole instead of EXTENT_MAP_HOLE.  This worked out fine
in practice it seems, but it adds a lot of extra work that is uneeded.  With
this fix I'm no longer tripping my assert.  Thanks,

Signed-off-by: Josef Bacik jba...@fusionio.com
---
 fs/btrfs/tree-log.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/btrfs/tree-log.c b/fs/btrfs/tree-log.c
index 849b729..1134aa4 100644
--- a/fs/btrfs/tree-log.c
+++ b/fs/btrfs/tree-log.c
@@ -3373,7 +3373,7 @@ static int log_one_extent(struct btrfs_trans_handle 
*trans,
btrfs_set_token_file_extent_type(leaf, fi,
 BTRFS_FILE_EXTENT_REG,
 token);
-   if (em-block_start == 0)
+   if (em-block_start == EXTENT_MAP_HOLE)
skip_csum = true;
}
 
-- 
1.8.3.1

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

No apparent effect for btrfs device delete missing

[Martin]

Trying:

btrfs device delete missing /

appears not to do anything for a / mount for where I have swapped out
a HDD:


# btrfs filesystem show
Label: 'test_btrfs_misc_5'  uuid: 7d29d4e6-efdc-41dc-9aa8-e74dfbe13cc9
Total devices 2 FS bytes used 28.00KB
devid1 size 59.74GB used 2.03GB path /dev/sdd5
*** Some devices missing

Label: 'test_btrfs_root_4'  uuid: 269e142c-e561-4227-b2b0-fe2f9fb99391
Total devices 3 FS bytes used 10.55GB
devid4 size 56.00GB used 12.03GB path /dev/sde4
devid1 size 56.00GB used 12.05GB path /dev/sdd4
*** Some devices missing

Btrfs v0.20-rc1-358-g194aa4a
# btrfs device delete missing /
# btrfs filesystem show
Label: 'test_btrfs_misc_5'  uuid: 7d29d4e6-efdc-41dc-9aa8-e74dfbe13cc9
Total devices 2 FS bytes used 28.00KB
devid1 size 59.74GB used 2.03GB path /dev/sdd5
*** Some devices missing

Label: 'test_btrfs_root_4'  uuid: 269e142c-e561-4227-b2b0-fe2f9fb99391
Total devices 3 FS bytes used 10.55GB
devid4 size 56.00GB used 12.03GB path /dev/sde4
devid1 size 56.00GB used 12.05GB path /dev/sdd4
*** Some devices missing

Btrfs v0.20-rc1-358-g194aa4a


All on the latest Linux 3.11.5-gentoo.

# df -h | egrep '/$'
rootfs  112G   22G   89G  20% /
/dev/sdd4   112G   22G   89G  20% /



Aside: Adding the /dev/sde4 device caused no balance action until I
deleted a device to reduce the raid1 mirror (data and metadata) down to
the two devices.

The missing device was an old HDD that had physically failed. No data
was lost for that example failure.


Hope of interest,
Martin


--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] Btrfs: optimize tree-log.c:count_inode_refs()

[Filipe David Borba Manana]

Avoid repeated tree searches by processing all inode ref items in
a leaf at once instead of processing one at a time, followed by a
path release and a tree search for a key with a decremented offset.

Signed-off-by: Filipe David Borba Manana fdman...@gmail.com
---
 fs/btrfs/tree-log.c |5 +
 1 file changed, 5 insertions(+)

diff --git a/fs/btrfs/tree-log.c b/fs/btrfs/tree-log.c
index 849b729..7b077a8 100644
--- a/fs/btrfs/tree-log.c
+++ b/fs/btrfs/tree-log.c
@@ -1307,6 +1307,7 @@ static int count_inode_refs(struct btrfs_root *root,
break;
path-slots[0]--;
}
+process_slot:
btrfs_item_key_to_cpu(path-nodes[0], key,
  path-slots[0]);
if (key.objectid != ino ||
@@ -1327,6 +1328,10 @@ static int count_inode_refs(struct btrfs_root *root,
 
if (key.offset == 0)
break;
+   if (path-slots[0]  0) {
+   path-slots[0]--;
+   goto process_slot;
+   }
key.offset--;
btrfs_release_path(path);
}
-- 
1.7.9.5

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] Btrfs: fix btrfs_prev_leaf() previous key computation

[Filipe David Borba Manana]

If we decrement the key type, we must reset its offset to the largest
possible offset (u64)-1. If we decrement the key's objectid, then we
must reset the key's type and offset to their largest possible values,
(u8)-1 and (u64)-1 respectively. Not doing so can make us miss an
items in the tree.

Signed-off-by: Filipe David Borba Manana fdman...@gmail.com
---
 fs/btrfs/ctree.c |   12 
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/fs/btrfs/ctree.c b/fs/btrfs/ctree.c
index 33e9dbd..6d426b2 100644
--- a/fs/btrfs/ctree.c
+++ b/fs/btrfs/ctree.c
@@ -4827,14 +4827,18 @@ static int btrfs_prev_leaf(struct btrfs_root *root, 
struct btrfs_path *path)
 
btrfs_item_key_to_cpu(path-nodes[0], key, 0);
 
-   if (key.offset  0)
+   if (key.offset  0) {
key.offset--;
-   else if (key.type  0)
+   } else if (key.type  0) {
key.type--;
-   else if (key.objectid  0)
+   key.offset = (u64)-1;
+   } else if (key.objectid  0) {
key.objectid--;
-   else
+   key.type = (u8)-1;
+   key.offset = (u64)-1;
+   } else {
return 1;
+   }
 
btrfs_release_path(path);
ret = btrfs_search_slot(NULL, root, key, path, 0, 0);
-- 
1.7.9.5

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OK to take hourly snapshots, then cull older ones?

[Marc MERLIN]

On Mon, Oct 14, 2013 at 09:05:33PM -0700, David Madden wrote:
 I'd like to use BTRFS to do something like the old NetApp snapshot
 system: every hour or so, there'd be a snapshot, then the 23 of the
 snapshots during a day would be deleted, leaving just a day snapshot,
 then after a month, 6 of 7 snapshots would be deleted, leaving just a
 week snapshot, and so on.
 
 Is this a reasonable thing to do in a cron job with a BTRFS filesystem?
  Apart from running out of space, are there any resources that might get
 used up?  Has anybody done this for a year or two in an active
 filesystem, and encountered success or weirdness?

Way ahead of you there, been doing this for a year, no problems:

gandalfthegreat:~$ cat /etc/cron.d/btrfs_backup 
0 * * * * root test -d /mnt/btrfs_pool1  grep -q rootflags=subvol=root 
/proc/cmdline  /var/local/scr/btrfs_snaps hourly 3 | egrep -v '(Create a 
snapshot of|Will delete the oldest|Delete subvolume|Making snapshot of )'
1 0 * * * root test -d /mnt/btrfs_pool1  grep -q rootflags=subvol=root 
/proc/cmdline  /var/local/scr/btrfs_snaps daily  4 | egrep -v '(Create a 
snapshot of|Will delete the oldest|Delete subvolume|Making snapshot of )'
2 0 * * 0 root test -d /mnt/btrfs_pool1  grep -q rootflags=subvol=root 
/proc/cmdline  /var/local/scr/btrfs_snaps weekly 4 | egrep -v '(Create a 
snapshot of|Will delete the oldest|Delete subvolume|Making snapshot of )'

/var/local/scr/btrfs_snaps:
---
#!/bin/bash

: ${BTRFSROOT:=/mnt/btrfs_pool1}
DATE=$(date '+%Y%m%d_%H:%M:%S')

type=${1:-hourly}
keep=${2:-3}

cd $BTRFSROOT

for i in *
do
# Skip snapshot names.
[ ${i/_/-} != $i ]  continue
# Skip duplicate dirs once a year on DST 1h rewind.
test -d $BTRFSROOT/${i}_${type}_$DATE  continue
echo Making snapshot of $type
/sbin/btrfs subvolume snapshot $BTRFSROOT/$i 
$BTRFSROOT/${i}_${type}_$DATE
count=$(ls -d ${i}_${type}_* | wc -l)
echo Will delete the oldest $clip snapshots for $type
clip=$(( $count - $keep ))
if [ $clip -gt 0 ]; then
for sub in $(ls -d ${i}_${type}_* | head -n $clip)
do
#echo Will delete $sub
/sbin/btrfs subvolume delete $sub
done
fi
done
---

Marc
-- 
A mouse is a device used to point at the xterm you want to type in - A.S.R.
Microsoft is to operating systems 
   what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OK to take hourly snapshots, then cull older ones?

[Duncan]

David Madden posted on Mon, 14 Oct 2013 21:05:33 -0700 as excerpted:

 I'd like to use BTRFS to do something like the old NetApp snapshot
 system:
 every hour or so, there'd be a snapshot, then the 23 of the snapshots
 during a day would be deleted, leaving just a day snapshot, then after a
 month, 6 of 7 snapshots would be deleted, leaving just a week snapshot,
 and so on.
 
 Is this a reasonable thing to do in a cron job with a BTRFS filesystem?
 Apart from running out of space, are there any resources that might get
 used up?  Has anybody done this for a year or two in an active
 filesystem, and encountered success or weirdness?

There's discussion of this idea along with links to existing tools/
scripts for it, on the wiki:

https://btrfs.wiki.kernel.org

In particular, see documentation, guides and usage, use cases,
2. snapshots and subvolumes, 2.2. backups/time-machine.  However, that 
you didn't already know that was covered indicates that you either 
weren't aware of the wiki, or haven't read much on it recently, so 
there's likely a lot more information there that you'll find useful if 
you spend some time looking around and reading.

(I haven't done a whole lot with snapshotting myself as it doesn't fit my 
use case very well, but I knew about it from reading the wiki and had 
tagged it in my mind to look up again later should I need the 
information, so it was a matter of just a few seconds to find it again 
and type the path above so you could find it too.  Since I /haven't/ done 
much with snapshotting myself, I can't help much in saying which of the 
listed tools will be easiest, but that script link points at a list post 
with a pre-made script and crontab entries that look like they do just 
about exactly what you outline. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
Every nonfree program has a lord, a master --
and if you use the program, he is your master.  Richard Stallman

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OK to take hourly snapshots, then cull older ones?

[Roman Mamedov]

On Mon, 14 Oct 2013 21:05:33 -0700
David Madden d...@mersenne.com wrote:

 I'd like to use BTRFS to do something like the old NetApp snapshot
 system: every hour or so, there'd be a snapshot, then the 23 of the
 snapshots during a day would be deleted, leaving just a day snapshot,
 then after a month, 6 of 7 snapshots would be deleted, leaving just a
 week snapshot, and so on.
 
 Is this a reasonable thing to do in a cron job with a BTRFS filesystem?
  Apart from running out of space, are there any resources that might get
 used up?  Has anybody done this for a year or two in an active
 filesystem, and encountered success or weirdness?

Sure, that's one of the more awesome uses of btrfs.

But keep in mind that old snapshots on the same FS are not to be used instead
of a proper backup to external media/servers. If a block happened not to
change for a year, and it gets damaged on disk, it will become damaged in all
the snapshots all the way back to a year ago, and you lose that data.

-- 
With respect,
Roman


signature.asc
Description: PGP signature

Re: OK to take hourly snapshots, then cull older ones?

[David Madden]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 14-Oct-2013 21:53 , Roman Mamedov wrote:
 Sure, that's one of the more awesome uses of btrfs.
 
 But keep in mind that old snapshots on the same FS are not to be
 used instead of a proper backup to external media/servers. If a
 block happened not to change for a year, and it gets damaged on
 disk, it will become damaged in all the snapshots all the way back
 to a year ago, and you lose that data.

I haven't looked at the wiki carefully enough to understand this, but
could one reasonably back up the snapshots of one BTRFS filesystem to
an independent BTRFS filesystem, in a more efficient way than just
dump/restore or cpio or something?

It sounds like you could rely on on-the-fly deduping to save space on
the backup filesystem, but you'd still be reading the whole source
snapshot tree and sending it to be deduped/written.  Seems like you
could improve things by just pulling the changed blocks from the
snapshot to the previous snapshot, which would (I guess) be the only
blocks actually _in_ the snapshot.

I guess what would really help is a Best Practices Guide for BTRFS
Snapshotting and Backup.  Does anything like that exist?

Thanks,
- -- 
Mersenne Law LLC  ·  www.mersenne.com  ·  +1-503-679-1671
- - Small Business, Startup and Intellectual Property Law -
1500 SW First Ave. · Suite 1170 · Portland, Oregon  97201
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.20 (Darwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJSXM0RAAoJEIQakGgVoKPCbNUP/RppplehqwASZXuIHcVGZBW/
4Q0sKcA19O4VVW8/Byy5Wa1cH3DHf24X4S8IHdZIvdn30EfYNI7q489ldmwQk9h8
To2EdsbpG9VlRWQ6W3zSv1iwqp1i/1W1miB6dkk4hHIEZ64BilaFyKInuCJCdS1T
anQFmFBUMGbMGdMXbIxrLgHhwYr7VSyp4RXxtCspeQHENT1zUiy6tyzEDqoP5vf+
ZDScBXK8G/MFijQTr6BWfCyiuS3OFostCp9dITmwNtLc194Ae/TDF7+ZNo/CzpOy
uP4pEZDPBYraEEcZBeRm5Jefa6VPkr16aoiJKjEH0StHimuHDNX17epgLTqSJx+J
JN0AJoxLxHQa7A8ny3aOzIjibKGP0Aa8KBfl6saRD8DMFtILHLMPCUOJ+l16Pcj/
gfipOZEpiBDHY4EIzYJRq/yYMHmw5SrvZeoPKdIPgDjJ7ovB9fteTwmLVSaiccM0
nd7TTMYiqm6PPnh7FEZWopfMEvx35LSxxI5Q6nuNqgQ33EgtYtM5yJC1cISJH1gf
HwQWmr2yqwZ/xgEQc0jI4LO6ISTx4xsY0NKM5aUr6ayzbmE5kRRfnNkUKpDd8by9
CpuFg898XIO+9Cd9do0MvLjVxg62WU8YxPiC+bcwc5tGKAiVjQaTeTyfYrLDmGaj
sGCmWIXC8dGrrqsaGthw
=ioaG
-END PGP SIGNATURE-



smime.p7s
Description: S/MIME Cryptographic Signature

Re: OK to take hourly snapshots, then cull older ones?

[Avi Miller]

On 15/10/2013, at 4:05 PM, David Madden d...@mersenne.com wrote:

 I haven't looked at the wiki carefully enough to understand this, but
 could one reasonably back up the snapshots of one BTRFS filesystem to
 an independent BTRFS filesystem, in a more efficient way than just
 dump/restore or cpio or something?


Yes, it's called btrfs send/receive. Take a look at the incremental backup page 
for usage:

https://btrfs.wiki.kernel.org/index.php/Incremental_Backup

--
Oracle http://www.oracle.com
Avi Miller | Principal Program Manager | +61 (412) 229 687
Oracle Linux and Virtualization
417 St Kilda Road, Melbourne, Victoria 3004 Australia

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html