Re: [Bug] btrfs-progs v4.3.1, mkfs.btrfs manpage, profiles table missing raid1

[David Sterba]

On Fri, Nov 20, 2015 at 11:05:44AM +, Duncan wrote:
> As in the title...
> 
> The btrfs-progs v4.3.1 mkfs.btrfs manpage has a quite nice profiles table 
> listing the various profiles, single/dup/raid0/raid10/raid5/raid6.
> 
> It's missing raid1. =:^(

Oh, that's not intentional, I'll fix it.

> Also, it calls raid5/6 "copies" rather than "parity".  Perhaps add 
> another column for parity, change the redundancy column to copies, and 
> adjust accordingly?  Alternatively, keep the single redundancy column and 
> just change raid5 to 1 parity and raid6 to 2 parity.

Yeah, parity would be better. I'll split it to copy and parity, where a
copy really means 1:1 byte copy.

> Other than that, I really like the nice profiles table layout.
> Kudos to whoever set that up. =:^)

Thanks ;)
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad extent [5993525264384, 5993525280768), type mismatch with chunk

[Christoph Anton Mitterer]

On Mon, 2015-11-23 at 09:10 +0800, Qu Wenruo wrote:
> Also, you won't want compiler to do extra optimization
I did the following:
$ export CFLAGS="-g -O0 -Wall -D_FORTIFY_SOURCE=2"
$ ./configure --disable-convert --disable-documentation

So if you want me to get rid of _FORTIFY_SOURCE, please tell.


> 
> After make, you won't need to install the btrfs-progs, you can just
> use 
> gdb to debug local ./btrfsck and add new breakpoints to do the trick.
# gdb ./btrfs
GNU gdb (Debian 7.10-1) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./btrfs...done.
(gdb) break cmds-check.c:4421
Breakpoint 1 at 0x42d000: file cmds-check.c, line 4421.
(gdb) run check /dev/mapper/data-b
Starting program: /home/calestyo/bfsck/btrfs-tools-4.3/btrfs check 
/dev/mapper/data-b
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
...
bad extent [6619620278272, 6619620294656), type mismatch with chunk
bad extent [6619620294656, 6619620311040), type mismatch with chunk
bad extent [6619620311040, 6619620327424), type mismatch with chunk
checking free space cache
checking fs roots

with not breakpoint reached. And I've actually did that with both btrfs
where the problem occurred (the master, and the one that send/received
snapshots incrementally from it).


Hope that helps... anything further to do?

Chris.

smime.p7s
Description: S/MIME cryptographic signature

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Austin S Hemmelgarn]

On 2015-11-23 12:56, David Sterba wrote:

On Mon, Nov 23, 2015 at 08:56:13PM +0800, Anand Jain wrote:

Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
be compatible w any set of older/newer kernels.

So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
skinny-metadata even if the running kernel does not supports it, and
so the mount fails on the running.


So the default behaviour of mkfs will try to best guess the feature set
of currently running kernel. I think this is is the most common scenario
and justifies the change in default behaviours.
I feel that Christoph's suggestion in the other sub-thread to have it 
spit out a notice that it disabled something because of the kernel it's 
running on is worth adding also.  We should probably also spit out a 
warning if the user asks for a feature that isn't supported on the 
current kernel (but still let them create the filesystem regardless).


For the other cases I'd like to introduce some human-readable shortcuts
to the --features option. Eg. 'mkfs.btrfs -O compat-3.2' will pick all
options supported by the unpatched mainline kernel of version 3.2. This
would be present for all version, regardless if there was a change in the
options or not.

Similarly for convenience, add 'running' that would pick the options
from running kernel but will be explicit.
Is the intent to enable stuff that the devs consider stable that's 
supported by the running kernel, or all the features supported by the 
running kernel?  It's probably best to use the first as the defaults, 
and then have an option to pull in everything the running kernel 
supports (possibly name that option something like 'running-all').


A remaining option should override the 'running' behaviour and pick the
latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
name is yet to be determined.

Maybe something like 'recommended' or 'suggested'?

It might also be nice to have an option to tell it to turn on everything 
the tools support (possibly call that one something like 
'max-features'), though this is probably less useful due to the fact 
that most mkfs features in BTRFS are incompat features.




smime.p7s
Description: S/MIME Cryptographic Signature

Re: btrfs send reproducibly fails for a specific subvolume after sending 15 GiB, scrub reports no errors

[Austin S Hemmelgarn]

On 2015-11-22 16:59, Nils Steinger wrote:

Hi,

I recently ran into a problem while trying to back up some of my btrfs
subvolumes over the network:
`btrfs send` works flawlessly on snapshots of most subvolumes, but keeps
failing on snapshots of a certain subvolume — always after sending 15 GiB:

btrfs send /btrfs/snapshots/home/2015-11-17_03:28:14_BOOT-AUTOSNAPSHOT |
pv | ssh kappa "btrfs receive /mnt/300gb/backups/snapshots/zeta/home/"
At subvol /btrfs/snapshots/home/2015-11-17_03:28:14_BOOT-AUTOSNAPSHOT
At subvol 2015-11-17_03:28:14_BOOT-AUTOSNAPSHOT
ERROR: send ioctl failed with -2: No such file or directory
   15GB 0:34:34 [7,41MB/s]

I've tried piping the output to /dev/null instead of ssh and got the
same error (again after sending 15 GiB), so this seems to be on the
sending side.
This is an issue that comes up sometimes with send, it's not well 
understood or documented, but sometimes something in source FS can get 
into a state that send chokes on, and then crashes.  I've actually been 
trying to reproduce this myself on a small filesystem so that it's 
easier to debug, but so far been unsuccessful.  I have yet to find any 
reliable way to reproduce this, and thus have no reliable way to prevent 
it from happening either.


However, btrfs scrub reports no errors and I don't get any messages in
dmesg when the btrfs send fails.
Scrub is intended to fix corruption due to hardware failures.  In almost 
all cases that I've seen of what you are getting, it wasn't a provable 
hardware issue, and scrub returned no errors.


What could cause this kind of error?
And is there a way to fix it, preferably without recreating the FS?
In general (assuming you are seeing the same issue I run into from time 
to time), there are two options other than recreating the filesystem:
1. Recreate the file that scrub is choking on.  You can see what file by 
adding -vv to the receive command-li9ne, although be ready for lots of 
output.  It's important to note that mv won't work for this unless 
you're moving the data to a different filesystem (if it's a directory, 
copy everything out and then recreate the directory, then copy 
everything back in).  The downside to this option is that you will 
usually run into multiple files that send chokes on, and the only way to 
find them all is to keep repeating the process until send completes 
successfully.
2. Run a full balance on the FS (this doesn't work anywhere near as 
reliably as the first option, but is the only way to fix some issues 
caused by doing batch deduplication on some older kernels).




smime.p7s
Description: S/MIME Cryptographic Signature

Re: btrfs: poor performance on deleting many large files

[Austin S Hemmelgarn]

On 2015-11-22 20:43, Mitch Fossen wrote:

Hi all,

I have a btrfs setup of 4x2TB HDDs for /home in btrfs RAID0 on Ubuntu
15.10 (kernel 4.2) and btrfs-progs 4.3.1. Root is on a separate SSD
also running btrfs.

About 6 people use it via ssh and run simulations. One of these
simulations generates a lot of intermediate data that can be discarded
after it is run, it usually ends up being around 100GB to 300GB spread
across dozens of files 500M to 5GB apiece.

The problem is that, when it comes time to do a "rm -rf
~/working_directory" the entire machine locks up and sporadically
allows other IO requests to go through, with a 5 to 10 minute delay
before other requests seem to be served. It can end up taking half an
hour or more to fully remove the offending directory, with the hangs
happening frequently enough to be frustrating. This didn't seem to
happen when the system was using ext4 on LVM.
Based on this description, this sounds to me like an issue with 
fragmentation.


Is there a way to fix this performance issue or at least mitigate it?
Would using ionice and the CFQ scheduler help? As far as I know Ubuntu
uses deadline by default which ignores ionice values.
This depends on a number of factors.  If you are on a new enough kernel, 
you may actually be using the blk-mq code instead of one of the 
traditional I/O schedulers, which does honor ionice values, and is 
generally a lot better than CFQ or deadline at actual fairness and 
performance.  If you aren't running on that code path, then whether 
deadline or CFQ is better is pretty hard to determine.  In general, CFQ 
needs some serious effort and benchmarking to get reasonable performance 
out of it.  CFQ can beat deadline in performance when properly tuned to 
the workload (except if you have really small rotational media (smaller 
than 32G or so), or if you absolutely need deterministic scheduling), 
but when you don't take the time to tune CFQ, deadline is usually better 
(except on SSD's, where CFQ is generally better than deadline even 
without performance tuning).


Alternatively, would balancing and defragging data more often help?
The current mount options are compress=lzo and space_cache, and I will
try it with autodefrag enabled as well to see if that helps.
Balance is not likely to help much, but defragmentation might.  I would 
suggest running the defrag when nobody has any other data on the 
filesystem, as it will likely cause a severe drop in performance the 
first time it's run.  Autodefrag might help, but it may also make 
performance worse while writing the files in the first place.  You might 
also try with compress=none, depending on your storage hardware, using 
in-line compression can actually make things go significantly slower (I 
see this a lot with SSD's, and also with some high-end storage 
controllers, and especially when dealing with large data-sets that 
aren't very compressible).


For now I think I'll recommend that everyone use subvolumes for these
runs and then enable user_subvol_rm_allowed.
As Duncan said, this is probably the best option short term.  It is 
worth noting however that removing a subvolume still has some overhead 
(which appears to scale linearly with the amount of data in the 
subvolume).  This overhead isn't likely to be an issue however unless a 
bunch of subvolumes get removed in bulk however.





smime.p7s
Description: S/MIME Cryptographic Signature

[PATCH v2 2/5] btrfs-progs: add framework to check features supported by sysfs

[Anand Jain]

This adds a framework to check the /sys/fs/btrfs/features for the list
of supported features by the btrfs kernel. Which I hope by using it the
mkfs and btrfs-convert could tune to set features as supported by the
running kernel.


Signed-off-by: Anand Jain 
---
 utils.c | 66 +
 utils.h |  1 +
 2 files changed, 59 insertions(+), 8 deletions(-)

diff --git a/utils.c b/utils.c
index 24042e5..48a1989 100644
--- a/utils.c
+++ b/utils.c
@@ -577,22 +577,23 @@ out:
  */
 static const struct btrfs_fs_feature {
const char *name;
+   const char *name_ker;
u64 flag;
const char *desc;
const char *min_ker_ver;
 } mkfs_features[] = {
-   { "mixed-bg", BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS,
+   { "mixed-bg", "mixed_groups", BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS,
"mixed data and metadata block groups", "2.7.31"},
-   { "extref", BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF,
+   { "extref", "extended_iref", BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF,
"increased hardlink limit per file to 65536", "3.7"},
-   { "raid56", BTRFS_FEATURE_INCOMPAT_RAID56,
+   { "raid56", "raid56", BTRFS_FEATURE_INCOMPAT_RAID56,
"raid56 extended format", "3.9"},
-   { "skinny-metadata", BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA,
+   { "skinny-metadata", "skinny_metadata", 
BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA,
"reduced-size metadata extent refs", "3.10"},
-   { "no-holes", BTRFS_FEATURE_INCOMPAT_NO_HOLES,
+   { "no-holes", "no_holes", BTRFS_FEATURE_INCOMPAT_NO_HOLES,
"no explicit hole extents for files", "3.14"},
/* Keep this one last */
-   { "list-all", BTRFS_FEATURE_LIST_ALL, NULL }
+   { "list-all", "", BTRFS_FEATURE_LIST_ALL, NULL }
 };
 
 static int parse_one_fs_feature(const char *name, u64 *flags)
@@ -602,10 +603,12 @@ static int parse_one_fs_feature(const char *name, u64 
*flags)
 
for (i = 0; i < ARRAY_SIZE(mkfs_features); i++) {
if (name[0] == '^' &&
-   !strcmp(mkfs_features[i].name, name + 1)) {
+   (!strcmp(mkfs_features[i].name, name + 1) ||
+   !strcmp(mkfs_features[i].name_ker, name + 1))) {
*flags &= ~ mkfs_features[i].flag;
found = 1;
-   } else if (!strcmp(mkfs_features[i].name, name)) {
+   } else if (!strcmp(mkfs_features[i].name, name) ||
+   !strcmp(mkfs_features[i].name_ker, name)) {
*flags |= mkfs_features[i].flag;
found = 1;
}
@@ -3147,3 +3150,50 @@ u64 btrfs_features_allowed_by_kernel(void)
}
return (features);
 }
+
+int check_or_load_btrfs_ko()
+{
+   int fd;
+
+   /*
+* open will load btrfs kernel module if its not loaded,
+* and if the kernel has CONFIG auto load set?
+*/
+   fd = open("/dev/btrfs-control", O_RDONLY);
+   if (fd < 0)
+   return -errno;
+
+   close(fd);
+   return 0;
+}
+
+int btrfs_features_allowed_by_sysfs(u64 *features)
+{
+   int ret;
+   DIR *dir;
+   struct dirent *ent;
+
+   ret = check_or_load_btrfs_ko();
+   if (ret) {
+   /* returns, -errno */
+   return ret;
+   }
+
+   dir = opendir("/sys/fs/btrfs/features");
+   if (!dir) {
+   /*
+* An old kernel which does not support sysfs/features
+*/
+   return -errno;
+   }
+
+   *features = 0;
+   while((ent = readdir(dir)) != NULL) {
+   if (!strcmp(".", ent->d_name) ||
+   !strcmp("..", ent->d_name))
+   continue;
+   parse_one_fs_feature(ent->d_name, features);
+   }
+   closedir(dir);
+   return 0;
+}
diff --git a/utils.h b/utils.h
index 9044643..af0aa31 100644
--- a/utils.h
+++ b/utils.h
@@ -105,6 +105,7 @@ char* btrfs_parse_fs_features(char *namelist, u64 *flags);
 void btrfs_process_fs_features(u64 flags);
 void btrfs_parse_features_to_string(char *buf, u64 flags);
 u64 btrfs_features_allowed_by_kernel(void);
+int btrfs_features_allowed_by_sysfs(u64 *features);
 
 struct btrfs_mkfs_config {
char *label;
-- 
2.6.2

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Anand Jain]

Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
be compatible w any set of older/newer kernels.

So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
skinny-metadata even if the running kernel does not supports it, and
so the mount fails on the running.

Here in this set of patches will make sure the progs understands the
kernel supported features.

So in this patch, checks if sysfs tells whether the feature is
supported if not, then it will relay on static kernel version which
provided that feature (skinny-metadata here in this example), next
if for some reason the running kernel does not provide the kernel
version, then it will fall back to the original method to enable
the feature with a hope that kernel will support it.

Also the last patch adds a warning when we fail to read either
sysfs features or the running kernel version.

With this I hope all the concerns from the review comments are
addressed.


Anand Jain (5):
  btrfs-progs: introduce framework to check kernel supported features
  btrfs-progs: add framework to check features supported by sysfs
  btrfs-progs: kernel based default features for mkfs
  btrfs-progs: kernel based default features for btrfs-convert
  btrfs-progs: add warning when we fail to read sysfs or kernel version

 btrfs-convert.c |  18 ++-
 mkfs.c  |  22 -
 utils.c | 146 +++-
 utils.h |   2 +
 4 files changed, 173 insertions(+), 15 deletions(-)

-- 
2.6.2

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 3/5] btrfs-progs: kernel based default features for mkfs

[Anand Jain]

Mkfs from latest btrfs-progs will enable latest default features,
and if the kernel is down-rev and does not support a latest default
feature then mount fails, as expected.

This patch disables default features based on the running kernel.

Signed-off-by: Anand Jain 
---
v2: Check if sysfs tells what features are supported

 mkfs.c | 16 +++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/mkfs.c b/mkfs.c
index a5802f7..4f46ad9 100644
--- a/mkfs.c
+++ b/mkfs.c
@@ -1357,10 +1357,24 @@ int main(int ac, char **av)
int dev_cnt = 0;
int saved_optind;
char fs_uuid[BTRFS_UUID_UNPARSED_SIZE] = { 0 };
-   u64 features = BTRFS_MKFS_DEFAULT_FEATURES;
+   u64 features;
struct mkfs_allocation allocation = { 0 };
struct btrfs_mkfs_config mkfs_cfg;
 
+   /*
+* If kernel could tell supported features by sysfs then use it,
+* if not, then set it by static kernel version, if the this also
+* fails then default to the progs default, which is set as per
+* BTRFS_MKFS_DEFAULT_FEATURES
+*/
+   if (btrfs_features_allowed_by_sysfs())
+   features = btrfs_features_allowed_by_kernel();
+
+   if (features)
+   features &= BTRFS_MKFS_DEFAULT_FEATURES;
+   else
+   features = BTRFS_MKFS_DEFAULT_FEATURES;
+
while(1) {
int c;
static const struct option long_options[] = {
-- 
2.6.2

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 5/5] btrfs-progs: add warning when we fail to read sysfs or version

[Anand Jain]

Signed-off-by: Anand Jain 
---
 btrfs-convert.c | 10 +-
 mkfs.c  |  8 +++-
 2 files changed, 16 insertions(+), 2 deletions(-)

diff --git a/btrfs-convert.c b/btrfs-convert.c
index 52ea12a..b0a998b 100644
--- a/btrfs-convert.c
+++ b/btrfs-convert.c
@@ -2898,14 +2898,22 @@ int main(int argc, char *argv[])
char fslabel[BTRFS_LABEL_SIZE];
u64 features;
 
-   if (btrfs_features_allowed_by_sysfs())
+   if (btrfs_features_allowed_by_sysfs()) {
+   fprintf(stderr,
+   "Warning: Failed/Unsupported to obtain feature list from 
sysfs\n");
+
features = btrfs_features_allowed_by_kernel();
+   if (!features)
+   fprintf(stderr,
+   "Warning: Failed/Unsupported to get running kernel version\n");
+   }
 
if (features)
features &= BTRFS_MKFS_DEFAULT_FEATURES;
else
features = BTRFS_MKFS_DEFAULT_FEATURES;
 
+
while(1) {
enum { GETOPT_VAL_NO_PROGRESS = 256 };
static const struct option long_options[] = {
diff --git a/mkfs.c b/mkfs.c
index 4f46ad9..6cb998b 100644
--- a/mkfs.c
+++ b/mkfs.c
@@ -1367,8 +1367,14 @@ int main(int ac, char **av)
 * fails then default to the progs default, which is set as per
 * BTRFS_MKFS_DEFAULT_FEATURES
 */
-   if (btrfs_features_allowed_by_sysfs())
+   if (btrfs_features_allowed_by_sysfs()) {
+   fprintf(stderr,
+   "Warning: Failed/Unsupported to obtain feature list from 
sysfs\n");
features = btrfs_features_allowed_by_kernel();
+   if (!features)
+   fprintf(stderr,
+   "Warning: Failed/Unsupported to get running kernel version\n");
+   }
 
if (features)
features &= BTRFS_MKFS_DEFAULT_FEATURES;
-- 
2.6.2

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 1/5] btrfs-progs: introduce framework to check kernel supported features

[Anand Jain]

In the newer kernel, supported kernel features can be known from
  /sys/fs/btrfs/features
however this interface was introduced only after 3.14, and most the
incompatible FS features were introduce before 3.14.

This patch proposes to maintain kernel version against the feature list,
and so that will be the minimum kernel version needed to use the feature.

Further, for features supported later than 3.14 this list can still be
updated, so it serves as a repository which can be displayed for easy
reference.

Signed-off-by: Anand Jain 
---
v2: Check for condition that what happens when we fail to read kernel
version. Now the code will fail back to use the default as set by
the progs.

 utils.c | 80 -
 utils.h |  1 +
 2 files changed, 76 insertions(+), 5 deletions(-)

diff --git a/utils.c b/utils.c
index b754686..24042e5 100644
--- a/utils.c
+++ b/utils.c
@@ -32,10 +32,12 @@
 #include 
 #include 
 #include 
+#include 
 #include 
 #include 
 #include 
 #include 
+#include 
 #include 
 
 #include "kerncompat.h"
@@ -567,21 +569,28 @@ out:
return ret;
 }
 
+/*
+ * min_ker_ver: update with minimum kernel version at which the feature
+ * was integrated into the mainline. For the transit period, that is
+ * feature not yet in mainline but in mailing list and for testing,
+ * please use "0.0" to indicate the same.
+ */
 static const struct btrfs_fs_feature {
const char *name;
u64 flag;
const char *desc;
+   const char *min_ker_ver;
 } mkfs_features[] = {
{ "mixed-bg", BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS,
-   "mixed data and metadata block groups" },
+   "mixed data and metadata block groups", "2.7.31"},
{ "extref", BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF,
-   "increased hardlink limit per file to 65536" },
+   "increased hardlink limit per file to 65536", "3.7"},
{ "raid56", BTRFS_FEATURE_INCOMPAT_RAID56,
-   "raid56 extended format" },
+   "raid56 extended format", "3.9"},
{ "skinny-metadata", BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA,
-   "reduced-size metadata extent refs" },
+   "reduced-size metadata extent refs", "3.10"},
{ "no-holes", BTRFS_FEATURE_INCOMPAT_NO_HOLES,
-   "no explicit hole extents for files" },
+   "no explicit hole extents for files", "3.14"},
/* Keep this one last */
{ "list-all", BTRFS_FEATURE_LIST_ALL, NULL }
 };
@@ -3077,3 +3086,64 @@ unsigned int get_unit_mode_from_arg(int *argc, char 
*argv[], int df_mode)
 
return unit_mode;
 }
+
+static int version_to_code(char *v)
+{
+   int i = 0;
+   char *b[3] = {NULL};
+   char *save_b = NULL;
+
+   for (b[i] = strtok_r(v, ".", _b);
+   b[i] != NULL;
+   b[i] = strtok_r(NULL, ".", _b))
+   i++;
+
+   if (b[2] == NULL)
+   return KERNEL_VERSION(atoi(b[0]), atoi(b[1]), 0);
+   else
+   return KERNEL_VERSION(atoi(b[0]), atoi(b[1]), atoi(b[2]));
+
+}
+
+static int get_kernel_code()
+{
+   int ret;
+   struct utsname utsbuf;
+   char *version;
+
+   ret = uname();
+   if (ret)
+   return -ret;
+
+   if (!strlen(utsbuf.release))
+   return -EINVAL;
+
+   version = strtok(utsbuf.release, "-");
+
+   return version_to_code(version);
+}
+
+u64 btrfs_features_allowed_by_kernel(void)
+{
+   int i;
+   int local_kernel_code = get_kernel_code();
+   u64 features = 0;
+
+   /*
+* When system did not provide the kernel version then just
+* return 0, the caller has to depend on the intelligence as
+* per btrfs-progs version
+*/
+   if (local_kernel_code <= 0)
+   return 0;
+
+   for (i = 0; i < ARRAY_SIZE(mkfs_features) - 1; i++) {
+   char *ver = strdup(mkfs_features[i].min_ker_ver);
+
+   if (local_kernel_code >= version_to_code(ver))
+   features |= mkfs_features[i].flag;
+
+   free(ver);
+   }
+   return (features);
+}
diff --git a/utils.h b/utils.h
index 192f3d1..9044643 100644
--- a/utils.h
+++ b/utils.h
@@ -104,6 +104,7 @@ void btrfs_list_all_fs_features(u64 mask_disallowed);
 char* btrfs_parse_fs_features(char *namelist, u64 *flags);
 void btrfs_process_fs_features(u64 flags);
 void btrfs_parse_features_to_string(char *buf, u64 flags);
+u64 btrfs_features_allowed_by_kernel(void);
 
 struct btrfs_mkfs_config {
char *label;
-- 
2.6.2

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 4/5] btrfs-progs: kernel based default features for btrfs-convert

[Anand Jain]

btrfs-convert convert FS with latest default features enabled, and
if the kernel is down-rev and does not support a latest feature then
mount fails, as expected.

This patch disables default features based on the running kernel.

Signed-off-by: Anand Jain 
---
v2: Check if sysfs tells what features are supported

 btrfs-convert.c | 10 +-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/btrfs-convert.c b/btrfs-convert.c
index 73ac584..52ea12a 100644
--- a/btrfs-convert.c
+++ b/btrfs-convert.c
@@ -2896,7 +2896,15 @@ int main(int argc, char *argv[])
int progress = 1;
char *file;
char fslabel[BTRFS_LABEL_SIZE];
-   u64 features = BTRFS_MKFS_DEFAULT_FEATURES;
+   u64 features;
+
+   if (btrfs_features_allowed_by_sysfs())
+   features = btrfs_features_allowed_by_kernel();
+
+   if (features)
+   features &= BTRFS_MKFS_DEFAULT_FEATURES;
+   else
+   features = BTRFS_MKFS_DEFAULT_FEATURES;
 
while(1) {
enum { GETOPT_VAL_NO_PROGRESS = 256 };
-- 
2.6.2

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: shall distros run btrfsck on boot?

[Wang Shilong]

On Tue, Nov 24, 2015 at 12:02 PM, Christoph Anton Mitterer
 wrote:
> Hey.
>
> Short question since that came up on debian-devel.
>
> Now that btrfs check get's more and more useful, are the developers
> going to recommend running it periodically on boot (of course that
> wouldn't work right now, as it would *always* check)?
>
> Plus... is btrfs check (without any arguments) non-desctructive, or are
> there corner-cases where it may lead to any changes on the devices?

As far as i know, 'btrfs check' did not repair anything in default.
only output errors if repair options not specified.

Regards,
Shilong
>
> Thanks,
> Chris.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: shall distros run btrfsck on boot?

[Eric Sandeen]

On 11/23/15 10:35 PM, Duncan wrote:
> Christoph Anton Mitterer posted on Tue, 24 Nov 2015 05:02:34 +0100 as
> excerpted:
> 
>> Hey.
>>
>> Short question since that came up on debian-devel.
>>
>> Now that btrfs check get's more and more useful, are the developers
>> going to recommend running it periodically on boot (of course that
>> wouldn't work right now, as it would *always* check)?
> 
> I'm a list regular and btrfs user, not a dev, but all the indications 
> continue to point to _not_ running it automatically at boot, nobody even 
> _suggesting_ otherwise.  The btrfs kernel code itself detects and often 
> corrects many problems, and btrfs check is simply not recommended for 
> automatic at-boot scheduling -- if the kernel code can't fix it without 
> intervention, then the problem is too serious to be fixed without 
> intervention by some scheduled btrfs check run, as well.
> 
> In fact, take a look at the shipped fsck.btrfs shell-script, based upon 
> the xfs one.

FWIW, fsck.xfs is a no-op because xfs is a *journaling filesystem* and an
unclean shutdown is not a reason to force a filesystem repair on the next
boot.

...that's what the journal was for...!  The filesystem should be 100%
consistent after a mount & a log replay.

The only reason fsck.xfs came into existence was to satisfy initscripts,
IIRC.

-Eric

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: shall distros run btrfsck on boot?

[Christoph Anton Mitterer]

On Tue, 2015-11-24 at 04:35 +, Duncan wrote:
> I'm a list regular and btrfs user, not a dev, but all the indications
 
> continue to point to _not_ running it automatically at boot, nobody
> even 
> _suggesting_ otherwise.
Sure, I just asked because maybe that would have just been an
anachronism from the days btrfsck was much more alpha.


>   The btrfs kernel code itself detects and often 
> corrects many problems, and btrfs check is simply not recommended for
> automatic at-boot scheduling -- if the kernel code can't fix it
> without 
> intervention, then the problem is too serious to be fixed without 
> intervention by some scheduled btrfs check run, as well.
I once had an issue with a btrfs, where the kernel didn't show anything
but btrfsck did...(not the one Qu's currently looking into).
And I though the same is basically the case for other filesystems like
ext.


> In fact, take a look at the shipped fsck.btrfs shell-script, based
> upon 
> the xfs one.  As both the code and the comments suggest, it's 
> specifically designed to simply return success
Sure, but that could have simply been forgotten to update...


Thanks for the update on the status in that matter :)
Cheers,
Chris.

smime.p7s
Description: S/MIME cryptographic signature

[PATCH] btrfs-progs: fsck: Fix a false alert where extent record has wrong metadata flag

[Qu Wenruo]

In process_extent_item(), it gives 'metadata' initial value 0, but for
non-skinny-metadata case, metadata extent can't be judged just from key
type and it forgot that case.

This causes a lot of false alert in non-skinny-metadata filesystem.

Fix it by set correct metadata value before calling add_extent_rec().

Reported-by: Christoph Anton Mitterer 
Signed-off-by: Qu Wenruo 
---
 cmds-check.c | 4 
 1 file changed, 4 insertions(+)

diff --git a/cmds-check.c b/cmds-check.c
index fd661d9..c235702 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -5134,6 +5134,10 @@ static int process_extent_item(struct btrfs_root *root,
 
ei = btrfs_item_ptr(eb, slot, struct btrfs_extent_item);
refs = btrfs_extent_refs(eb, ei);
+   if (btrfs_extent_flags(eb, ei) == BTRFS_EXTENT_FLAG_TREE_BLOCK)
+   metadata = 1;
+   else
+   metadata = 0;
 
add_extent_rec(extent_cache, NULL, 0, key.objectid, num_bytes,
   refs, 0, 0, 0, metadata, 1, num_bytes);
-- 
2.6.2

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: shall distros run btrfsck on boot?

[Duncan]

Christoph Anton Mitterer posted on Tue, 24 Nov 2015 05:02:34 +0100 as
excerpted:

> Hey.
> 
> Short question since that came up on debian-devel.
> 
> Now that btrfs check get's more and more useful, are the developers
> going to recommend running it periodically on boot (of course that
> wouldn't work right now, as it would *always* check)?

I'm a list regular and btrfs user, not a dev, but all the indications 
continue to point to _not_ running it automatically at boot, nobody even 
_suggesting_ otherwise.  The btrfs kernel code itself detects and often 
corrects many problems, and btrfs check is simply not recommended for 
automatic at-boot scheduling -- if the kernel code can't fix it without 
intervention, then the problem is too serious to be fixed without 
intervention by some scheduled btrfs check run, as well.

In fact, take a look at the shipped fsck.btrfs shell-script, based upon 
the xfs one.  As both the code and the comments suggest, it's 
specifically designed to simply return success (well, if the given device 
exists, if not it does error out on that), without actually checking 
anything (beyond simple device existence).  If it's run in auto mode, as 
it would be with an on-boot run, it does nothing else.  If it's not run 
in auto mode, suggesting that a misinformed user attempted to run it, it 
prints a message redirecting them to btrfs check.

That's pretty clear as to intent -- btrfs check is simply not designed or 
intended to ever be run at boot.

> Plus... is btrfs check (without any arguments) non-desctructive, or are
> there corner-cases where it may lead to any changes on the devices?

Btrfs check, without arguments, is designed to be read-only.  AFAIK, the 
only way it wouldn't be would be if there was a serious bug.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

subvols and parents - how?

[Christoph Anton Mitterer]

Hey.

I'd have a, mainly administrative, question.

When I use subvolumes than these have always a parent subvolume (except
ID5), so I can basically decide between two ways:

a) make child subvolumes, e.g.
5
|
+-root   (=subvol, mountpoint /)
  +-boot/
  +-root/
  +-lib/
  +-home/ (=subvolume)
and soon on... perhaps the whole thing without the dedicated root-
subovlume (although that's probably not so smart, I guess).

b) place at least some of the subvolumes directly below the top-level
and mount them e.g. via /etc/fstab, e.g.
5
|
+-root   (=subvol, mountpoint /)
| +-boot/
| +-root/
| +-lib/
+-home/ (=subvolume, mountpoint /home)


Now I wondered whether this has any technical implications, but neither
the wiki, nor the manpages seem to explain a lot here.


The "differences", AFAIU, are the follows:
- When I mount a given subvolume,.. it's childs are automatically 
  "there".
  Whereas when I don't have them as childs (as in (b)) I must of
  course mount them somehow manually.
- Analogously for umounting.
- I can move existing subvols to higher/lower levels, and the parent
  IDs will change accordingly.

So basically it makes no difference, right? Or is there anything more
technical going on? E.g. with the ref-links or so?
Right now, there are, AFAIK, neither recursive snapshots (and
especially not atomic ones) nor recursive send/receive, right?
If that should ever be implemented, would I perhaps have problems with 
(a) or (b)?


Thanks,
Chris.

btw, for a developer:
$ btrfs subvolume list /mnt/ -pacguqt
ID  gen cgenparent  top level   parent_uuid uuidpath
--  --- --  -   --- 
257 16  8   5   5   -   
64f8a75b-5cb5-6e4d-9f30-d45fe3d9e060/root

There seem to be some colum mis-aglinment issues in the output =)
btrfsprogs 4.3

smime.p7s
Description: S/MIME cryptographic signature

Re: Weird space issues (with more than plenty left; over 100GiB free on 465.63GiB RAID partition)

[Duncan]

Steven Hoff posted on Mon, 23 Nov 2015 19:31:43 -0700 as excerpted:

> BTRFS Community,
> 
> I seem to be having a bit of an issue. A comment on
> https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_get_.22
No_space_left_on_device.22_errors.2C_but_df_says_I.27ve_got_lots_of_space
> suggested reporting this to the mailing list.
> 
> Issue: BTRFS claims to be running out of space
> 
> Setup:
> /home: 2 500GB hard drives in MDADM RAID1 with BTRFS being the primary
> filesystem
> 
> Output of various commands:
> 
> $ sudo btrfs fi df
> /home Data, single: total=334.00GiB, used=333.84GiB
> System, DUP: total=32.00MiB, used=80.00KiB
> Metadata, DUP: total=1.50GiB, used=987.17MiB
> unknown, single: total=336.00MiB, used=0.00
> 
> $ sudo btrfs fi show /home
> Label: none  uuid: (private information removed)
>  Total devices 1 FS bytes used 334.80GiB
>  devid1 size 465.63GiB used 337.06GiB path /dev/md0p1
> Btrfs v3.12
> 
> $ df /home
> Filesystem 1K-blocks  Used Available Use% Mounted on
> /dev/md0p1 488252416 352418008 134988424  73% /home
> 
> 
> I would like to note that "total" in the following line:
> Data, single: total=334.00GiB, used=333.84GiB gets smaller whenever data
> is deleted or moved from /home. I'm not sure if that is by design, or if
> it is a bug. Either way, every once in a while applications will start
> closing unexpectedly complaining that there is no space left.


Critical information not provided: kernel version.  However, if it's 
anything as old as the 3.12 btrfs-progs userspace that's being reported, 
you're running seriously ancient versions, and an upgrade to something 
semi-recent is *strongly* recommended.

Normally, the kernel version is most critical, and running either a 
current release series (4.3), or an LTS (long term support) version no 
further back than the penultimate (next to latest, 3.18 as 4.1 is the 
latest LTS, but the currently in development 4.4 has already been 
announced as another LTS so people on 3.18 should be strongly considering 
an upgrade to 4.1 by now).

As for btrfs userspace, while it's generally not so critical, 3.12 is 
getting a bit ridiculously old now, and a good rule of thumb is to run at 
least the version matching that of your kernel series, since a userspace 
release of a particular version comes shortly after the kernel release, 
and is being coded at the same time, so they're designed with each other 
in mind.  Tho running a newer userspace shouldn't be a problem, either.  
Following that rule of thumb, if you're following the kernel rule as 
well, you'll be running at least a 3.18 series kernel, so should be 
running at least a 3.18 series userspace.  That will ensure at least that 
it doesn't get a ridiculously old as your current 3.12 userspace.

But you do seem to be running at least a newer kernel than your 
userspace, as otherwise you'd not be getting that unknown line in btrfs 
fi df.  FWIW, newer userspace correctly reports that as global reserve.

And while it doesn't yet report it as such, the global reserve (again, 
reported as unknown on your ancient userspace) actually comes from the 
metadata number, so can be added to it.  There's actually some very 
recent on-list discussion on this topic, suggesting either making the 
global reserve a sub-entry under metadata, or eliminating it entirely, 
simply adding the global reserve size to metadata's used figure.

If you do that, the metadata line becomes 1.5 gig size, 1.3 gig used.  As 
metadata chunks are nominally a quarter gig but your metadata is dup 
mode, so writing will be to two of them at once, you're into your last 
already allocated set of metadata chunks, and similarly, data usage is 
into its last data chunk (nominally 1 GiB in size).

So that means before you do much of anything, either data or metadata 
will run out on its existing allocation, and new chunks should be 
allocated.

Now according to btrfs fi show, you have plenty of unallocated space 
(device line, 337 gig allocated, of 465 gig size, so over a hundred gig 
of unallocated free space), so allocating fresh chunks shouldn't be a 
problem at all.

But evidently in your case, at least some of the time, those allocations 
are failing.  Certainly, in the past, that was known to happen with some 
versions.

But that would be kernel code, and you didn't post what kernel version 
you're running.

Assuming based on your positively ancient btrfs userspace and the above 
hints that your kernel version, while newer, isn't exactly current... the 
first thing to do would be to get a current 4.3 series kernel, at least 
for testing, and see if the problem goes away.  If so and you you want to 
return to an older version and you weren't already on one of them, try 
the latest release of either the 3.18 or 4.1 kernel series, and see if 
the bug is still gone or is there again.

Based on the results of the above, we'll know whether it's a current 
kernel bug, or corrected in current but still in the 

Re: shall distros run btrfsck on boot?

[Qu Wenruo]

Christoph Anton Mitterer wrote on 2015/11/24 05:43 +0100:

On Tue, 2015-11-24 at 04:35 +, Duncan wrote:

I'm a list regular and btrfs user, not a dev, but all the indications



continue to point to _not_ running it automatically at boot, nobody
even
_suggesting_ otherwise.

Sure, I just asked because maybe that would have just been an
anachronism from the days btrfsck was much more alpha.



   The btrfs kernel code itself detects and often
corrects many problems, and btrfs check is simply not recommended for
automatic at-boot scheduling -- if the kernel code can't fix it
without
intervention, then the problem is too serious to be fixed without
intervention by some scheduled btrfs check run, as well.

I once had an issue with a btrfs, where the kernel didn't show anything
but btrfsck did...(not the one Qu's currently looking into).
And I though the same is basically the case for other filesystems like
ext.



In fact, take a look at the shipped fsck.btrfs shell-script, based
upon
the xfs one.  As both the code and the comments suggest, it's
specifically designed to simply return success

Sure, but that could have simply been forgotten to update...


Thanks for the update on the status in that matter :)
Cheers,
Chris.



Another point of never running btrfsck at boot time is, it's super 
SLOOOW!

And you should have experienced it during the false alert investigation. :)

It's SLOOW because it will need to iterate all the 
metadata of filesystems to make sure every thing devs can think about is OK.


Unlike traditional filesystem (maybe only ext*), it has some dirty flag 
in superblock or things like that to indicate a filesystem needs extra 
check, and e2fsck will only do comprehensive check if the filesystem is 
dirty.
That's why e2fsck can be run at boot time, as most of time, it just 
finds the fs is clean and no extra check.


But btrfs doesn't use the old-fashion method, it uses COW to protect 
metadata. So btrfsck is only designed to do comprehensive check and it's 
very SLW (not to mention sometimes it's buggy and too 
sensitive to give false alert).


Thanks,
Qu

--
This message has been scanned for viruses and
dangerous content by FCNIC, and is
believed to be clean.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad extent [5993525264384, 5993525280768), type mismatch with chunk

[Qu Wenruo]

Christoph Anton Mitterer wrote on 2015/11/24 04:02 +0100:

On Tue, 2015-11-24 at 10:54 +0800, Qu Wenruo wrote:

And it would be even better if you want to be a lab mouse for
incoming fixing patches.

Sure,.. if I get some cheese... and it would be great if you could give
me patches that apply to 4.3.


Hopes you didn't wait too long.

The fixing patch is CCed to you, or you can get it from patchwork:
https://patchwork.kernel.org/patch/7687611/





  (It won't hurt nor destroy your data)

wouldn't  matter... it's already backuped again and that fs is for
playing now ;-)


Just tell me in case you need to give up so that I can start re-using
the device then.


If that patch fixed the false alert, you are OK to re-use it.

Thanks,
Qu



Cheers,
Chris.



--
This message has been scanned for viruses and
dangerous content by FCNIC, and is
believed to be clean.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs send reproducibly fails for a specific subvolume after sending 15 GiB, scrub reports no errors

[Duncan]

Nils Steinger posted on Mon, 23 Nov 2015 22:10:12 +0100 as excerpted:

> Do we anything about what might cause a filesystem to enter a state
> which `send` chokes on?
> I've only seen a small sample of the corrupted files before growing
> tired of the process and just recreating the whole thing, but all of
> them were database files (presumably SQLite). Could it be that the files
> were being written to during an unclean shutdown, leading to some kind
> of corruption of the FS? Unfortunately, I was a little triggerhappy when
> cleaning up old snapshots, so there aren't any left to aid in
> troubleshooting this problem further…

Austin's the one attempting to trace down the problem, so he'd have the 
most direct answer there.  (My use-case doesn't involve snapshotting or 
send/receive at all.)

But if any type of files would be likely to create issues, it'd be 
something like database or VM image files, since the random-file-rewrite-
pattern they typically have is in general the most problematic for copy-
on-write (COW) filesystems such as btrfs.  Without some sort of 
additional fragmentation management (like the autodefrag mount option), 
these files will end up _highly_ fragmented on btrfs, often thousands of 
fragments, tens of thousands when the files in question are multi-gig.

For the typically smallish sqlite database files, autodefrag can help 
with the fragmentation such a rewrite pattern generally triggers with 
COW, and it'd be recommended in general if all such files on the 
filesystem are smallish (quarter to a half gig or smaller), but if you're 
running large VM images, etc, autodefrag doesn't scale so well to them, 
and much more complex fragmentation management will be needed.  But 
that'd be for a different post as I don't yet know if it applies here, 
and I'm trying to keep this one short.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Duncan]

Austin S Hemmelgarn posted on Mon, 23 Nov 2015 15:14:44 -0500 as
excerpted:

>> A remaining option should override the 'running' behaviour and pick the
>> latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
>> name is yet to be determined.

> Maybe something like 'recommended' or 'suggested'?

Just keep it simple.  If it enables the latest defaults, just call it 
"latest".  =:^)

Better yet, match compat-3.2, etc, with compat-running and compat-latest, 
making it crystal clear they're parallel options, as well as labeling 
them all compat-*, making it crystal clear what the intended 
functionality is.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: shall distros run btrfsck on boot?

[Duncan]

Christoph Anton Mitterer posted on Tue, 24 Nov 2015 05:43:31 +0100 as
excerpted:

[Duncan wrote...]
>> The btrfs kernel code itself detects and often
>> corrects many problems, and btrfs check is simply not recommended for
>> automatic at-boot scheduling -- if the kernel code can't fix it without
>> intervention, then the problem is too serious to be fixed without
>> intervention by some scheduled btrfs check run, as well.

> I once had an issue with a btrfs, where the kernel didn't show anything
> but btrfsck did...(not the one Qu's currently looking into).


That wouldn't be entirely uncommon, because as Eric mentions, btrfs check 
is intended to be thorough, where the kernel mount-time check is intended 
to be fast.

But of course, as Eric also mentions, that's yet another reason you don't 
want btrfs check running at boot... it's *SSLLLOOW*, because it's 
being thorough.

There are times you'd want to run btrfs check, the reason it's even there 
as a subcommand, but that would be when the filesystem either fails to 
mount, or when it exhibits other serious symptoms of problems... send and 
balance fails, remount-readonlys, nasty messages in the kernel log, etc.  

Careful admins may want to run btrfs check periodically in any case, but 
because it /is/ slow, and because it /isn't/ supposed to be needed for 
routine issues, they probably don't want to be doing it at boot when 
everything else is waiting on it.

Because you don't want your boot taking hours...  And in many cases, if a 
check /is/ going to take hours and even then not guarantee that the 
filesystem is workable, admins may want to simply blow away the 
filesystem, recreate it, and copy everything back to it from backups, 
rather than spending that much time on something that still has a good 
chance of being broken afterward.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: shall distros run btrfsck on boot?

[Duncan]

Duncan posted on Tue, 24 Nov 2015 06:46:18 + as excerpted:

> That wouldn't be entirely uncommon, because as Eric mentions, btrfs
> check is intended to be thorough, where the kernel mount-time check is
> intended to be fast.
> 
> But of course, as Eric also mentions, that's yet another reason you
> don't want btrfs check running at boot... it's *SSLLLOOW*, because
> it's being thorough.

Oops!  Mis-attribution.  Qu not Eric.

(I had read both replies in my email but only saw Eric's on the list, 
which I read in my news client via gmane's list2news service, when I 
composed the above.  So I presumed the points I remembered being made 
were from Eric's post, when it was Qu's.) 

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad extent [5993525264384, 5993525280768), type mismatch with chunk

[Qu Wenruo]

Christoph Anton Mitterer wrote on 2015/11/23 19:12 +0100:

On Mon, 2015-11-23 at 09:10 +0800, Qu Wenruo wrote:

Also, you won't want compiler to do extra optimization

I did the following:
$ export CFLAGS="-g -O0 -Wall -D_FORTIFY_SOURCE=2"


Wow, I didn't ever know it's possible to override FORTIFY_SOURCE to 
suppress the warning.


Great tip!


$ ./configure --disable-convert --disable-documentation

So if you want me to get rid of _FORTIFY_SOURCE, please tell.




After make, you won't need to install the btrfs-progs, you can just
use
gdb to debug local ./btrfsck and add new breakpoints to do the trick.

# gdb ./btrfs
GNU gdb (Debian 7.10-1) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./btrfs...done.
(gdb) break cmds-check.c:4421
Breakpoint 1 at 0x42d000: file cmds-check.c, line 4421.


Yes, that's one possible code where set the bad_extent flag.

But there are also some other places like line 4411, 4394 and 4387.

So there are still 3 breakpoint needs to add.

At least, we ruled out one possible case.

Thanks for all the result you provide, we are firmly getting to the 
result now.


Thanks,
Qu


(gdb) run check /dev/mapper/data-b
Starting program: /home/calestyo/bfsck/btrfs-tools-4.3/btrfs check 
/dev/mapper/data-b
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
...
bad extent [6619620278272, 6619620294656), type mismatch with chunk
bad extent [6619620294656, 6619620311040), type mismatch with chunk
bad extent [6619620311040, 6619620327424), type mismatch with chunk
checking free space cache
checking fs roots

with not breakpoint reached. And I've actually did that with both btrfs
where the problem occurred (the master, and the one that send/received
snapshots incrementally from it).


Hope that helps... anything further to do?

Chris.



--
This message has been scanned for viruses and
dangerous content by FCNIC, and is
believed to be clean.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 00/25] Btrfs-convert rework to support native separate

[Qu Wenruo]

David Sterba wrote on 2015/11/23 18:33 +0100:

On Fri, Nov 20, 2015 at 11:24:04AM +0800, Qu Wenruo wrote:

Here comes the 1st version of btrfs-convert rework.
Any test is welcomed, and it can already pass the convert test from
btrfs-progs. (Since the test doesn't test rollback function)


I went through the patches, looks mostly ok akin to the proposed
changes. I'll take the independent patches rightaway. Unfortunatelly
there are code changes that clash significantly with the pending
reiserfs addition to convert, I'm afraid you'll have to rework your
patchset on top of that.  The code is now pushed to branch
'dev/convert-reiser'.


Thanks for the check, David.

I'll rebase it soon.

Thanks,
Qu

--
This message has been scanned for viruses and
dangerous content by FCNIC, and is
believed to be clean.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad extent [5993525264384, 5993525280768), type mismatch with chunk

[Qu Wenruo]

Christoph Anton Mitterer wrote on 2015/11/24 02:53 +0100:

On Tue, 2015-11-24 at 08:46 +0800, Qu Wenruo wrote:

But there are also some other places like line 4411, 4394 and 4387.

Ah of course, I didn't have a look for further places

$ grep -n "rec->wrong_chunk_type = 1" cmds-check.c
4387:   rec->wrong_chunk_type = 1;
4394:   rec->wrong_chunk_type = 1;
4411:   rec->wrong_chunk_type = 1;
4421:   rec->wrong_chunk_type = 1;




So there are still 3 breakpoint needs to add.

GNU gdb (Debian 7.10-1) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./btrfs...done.
(gdb) break cmds-check.c:4387
Breakpoint 1 at 0x42cf2b: file cmds-check.c, line 4387.
(gdb) break cmds-check.c:4394
Breakpoint 2 at 0x42cf57: file cmds-check.c, line 4394.
(gdb) break cmds-check.c:4411
Breakpoint 3 at 0x42cfa6: file cmds-check.c, line 4411.
(gdb) break cmds-check.c:4421
Breakpoint 4 at 0x42d000: file cmds-check.c, line 4421.
(gdb) run check /dev/mapper/data-b
Starting program: /home/calestyo/bfsck/btrfs-tools-4.3/btrfs check 
/dev/mapper/data-b
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Checking filesystem on /dev/mapper/data-b
UUID: 250ddae1-7b37-4b22-89e9-4dc5886c810f
checking extents

Breakpoint 1, check_extent_type (rec=0x20a6740) at cmds-check.c:4387
4387rec->wrong_chunk_type = 1;
(gdb) continue
Continuing.

Breakpoint 1, check_extent_type (rec=0x20a6740) at cmds-check.c:4387
4387rec->wrong_chunk_type = 1;
(gdb) cont 100
Will ignore next 99 crossings of breakpoint 1.  Continuing.

Breakpoint 1, check_extent_type (rec=0x20a9880) at cmds-check.c:4387
4387rec->wrong_chunk_type = 1;
(gdb) cont 1000
Will ignore next 999 crossings of breakpoint 1.  Continuing.


Great, that's the direct cause.

After some quick glance, it seems that extent_record->metadata does not 
always reflect the right status of the extent.


I'll dig further to see what's causing the problem.

Thanks for all the debug info, it really helps a lot!
Qu


That goes on for a few millions... until we get the:
bad extent [6619620016128, 6619620032512), type mismatch with chunk
bad extent [6619620032512, 6619620048896), type mismatch with chunk
again.. and the check exits normally with:
checking free space cache
checking fs roots
checking csums
checking root refs
found 5862373889375 bytes used err is 0
total csum bytes: 5715302800
total tree bytes: 9903816704
total fs tree bytes: 2475769856
total extent tree bytes: 938393600
btree space waste bytes: 1072581913
file data blocks allocated: 9170230497280
  referenced 9281014861824
btrfs-progs v4.3
[Inferior 1 (process 18130) exited normally]


So it's the one in 4378.

Anything further to do? :)

Chris.



--
This message has been scanned for viruses and
dangerous content by FCNIC, and is
believed to be clean.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad extent [5993525264384, 5993525280768), type mismatch with chunk

[Christoph Anton Mitterer]

On Tue, 2015-11-24 at 08:46 +0800, Qu Wenruo wrote:
> But there are also some other places like line 4411, 4394 and 4387.
Ah of course, I didn't have a look for further places

$ grep -n "rec->wrong_chunk_type = 1" cmds-check.c
4387:   rec->wrong_chunk_type = 1;
4394:   rec->wrong_chunk_type = 1;
4411:   rec->wrong_chunk_type = 1;
4421:   rec->wrong_chunk_type = 1;



> So there are still 3 breakpoint needs to add.
GNU gdb (Debian 7.10-1) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./btrfs...done.
(gdb) break cmds-check.c:4387
Breakpoint 1 at 0x42cf2b: file cmds-check.c, line 4387.
(gdb) break cmds-check.c:4394
Breakpoint 2 at 0x42cf57: file cmds-check.c, line 4394.
(gdb) break cmds-check.c:4411
Breakpoint 3 at 0x42cfa6: file cmds-check.c, line 4411.
(gdb) break cmds-check.c:4421
Breakpoint 4 at 0x42d000: file cmds-check.c, line 4421.
(gdb) run check /dev/mapper/data-b
Starting program: /home/calestyo/bfsck/btrfs-tools-4.3/btrfs check 
/dev/mapper/data-b
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Checking filesystem on /dev/mapper/data-b
UUID: 250ddae1-7b37-4b22-89e9-4dc5886c810f
checking extents

Breakpoint 1, check_extent_type (rec=0x20a6740) at cmds-check.c:4387
4387rec->wrong_chunk_type = 1;
(gdb) continue 
Continuing.

Breakpoint 1, check_extent_type (rec=0x20a6740) at cmds-check.c:4387
4387rec->wrong_chunk_type = 1;
(gdb) cont 100
Will ignore next 99 crossings of breakpoint 1.  Continuing.

Breakpoint 1, check_extent_type (rec=0x20a9880) at cmds-check.c:4387
4387rec->wrong_chunk_type = 1;
(gdb) cont 1000
Will ignore next 999 crossings of breakpoint 1.  Continuing.

That goes on for a few millions... until we get the:
bad extent [6619620016128, 6619620032512), type mismatch with chunk
bad extent [6619620032512, 6619620048896), type mismatch with chunk
again.. and the check exits normally with:
checking free space cache
checking fs roots
checking csums
checking root refs
found 5862373889375 bytes used err is 0
total csum bytes: 5715302800
total tree bytes: 9903816704
total fs tree bytes: 2475769856
total extent tree bytes: 938393600
btree space waste bytes: 1072581913
file data blocks allocated: 9170230497280
 referenced 9281014861824
btrfs-progs v4.3
[Inferior 1 (process 18130) exited normally]


So it's the one in 4378.

Anything further to do? :)

Chris.

smime.p7s
Description: S/MIME cryptographic signature

Re: bad extent [5993525264384, 5993525280768), type mismatch with chunk

[Qu Wenruo]

Christoph Anton Mitterer wrote on 2015/11/24 03:48 +0100:

On Tue, 2015-11-24 at 10:09 +0800, Qu Wenruo wrote:

I'll dig further to see what's causing the problem.

I guess you'd prefer if I keep the fs for later verification?


That would be the best.

And it would be even better if you want to be a lab mouse for incoming 
fixing patches. (It won't hurt nor destroy your data)


Thanks,
Qu




Thanks for all the debug info, it really helps a lot!

Well thanks for your efforts as well :)

Chris.



--
This message has been scanned for viruses and
dangerous content by FCNIC, and is
believed to be clean.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad extent [5993525264384, 5993525280768), type mismatch with chunk

[Christoph Anton Mitterer]

On Tue, 2015-11-24 at 10:54 +0800, Qu Wenruo wrote:
> And it would be even better if you want to be a lab mouse for
> incoming fixing patches.
Sure,.. if I get some cheese... and it would be great if you could give
me patches that apply to 4.3.


>  (It won't hurt nor destroy your data)
wouldn't  matter... it's already backuped again and that fs is for
playing now ;-)


Just tell me in case you need to give up so that I can start re-using
the device then.

Cheers,
Chris.

smime.p7s
Description: S/MIME cryptographic signature

Weird space issues (with more than plenty left; over 100GiB free on 465.63GiB RAID partition)

[Steven Hoff]

BTRFS Community,

I seem to be having a bit of an issue. A comment on
https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_get_.22No_space_left_on_device.22_errors.2C_but_df_says_I.27ve_got_lots_of_space
suggested reporting this to the mailing list.

Issue: BTRFS claims to be running out of space

Setup:
/home: 2 500GB hard drives in MDADM RAID1 with BTRFS being the primary 
filesystem


Output of various commands:

$ sudo btrfs fi df /home
Data, single: total=334.00GiB, used=333.84GiB
System, DUP: total=32.00MiB, used=80.00KiB
Metadata, DUP: total=1.50GiB, used=987.17MiB
unknown, single: total=336.00MiB, used=0.00

$ sudo btrfs fi show /home
Label: none  uuid: (private information removed)
Total devices 1 FS bytes used 334.80GiB
devid1 size 465.63GiB used 337.06GiB path /dev/md0p1
Btrfs v3.12

$ df /home
Filesystem 1K-blocks  Used Available Use% Mounted on
/dev/md0p1 488252416 352418008 134988424  73% /home


I would like to note that "total" in the following line:
Data, single: total=334.00GiB, used=333.84GiB
gets smaller whenever data is deleted or moved from /home. I'm not sure if
that is by design, or if it is a bug. Either way, every once in a while 
applications will

start closing unexpectedly complaining that there is no space left. I've run
sudo btrfs fi balance /home
with no discernible result (other than relocating ~300 blocks). I have a 
system
monitoring tool that updates usage stats on my hard drives once every 
couple
seconds. It shows no abnormal disk usage during the times when 
applications start

complaining about no room left.

Any thoughts/ideas/help would be appreciated.

Regards,

Lectrode

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad extent [5993525264384, 5993525280768), type mismatch with chunk

[Christoph Anton Mitterer]

On Tue, 2015-11-24 at 10:09 +0800, Qu Wenruo wrote:
> I'll dig further to see what's causing the problem.
I guess you'd prefer if I keep the fs for later verification?


> Thanks for all the debug info, it really helps a lot!
Well thanks for your efforts as well :)

Chris.

smime.p7s
Description: S/MIME cryptographic signature

shall distros run btrfsck on boot?

[Christoph Anton Mitterer]

Hey.

Short question since that came up on debian-devel.

Now that btrfs check get's more and more useful, are the developers
going to recommend running it periodically on boot (of course that
wouldn't work right now, as it would *always* check)?

Plus... is btrfs check (without any arguments) non-desctructive, or are
there corner-cases where it may lead to any changes on the devices?

Thanks,
Chris.

smime.p7s
Description: S/MIME cryptographic signature

Re: [PATCH 2/2] generic/15[78]: fix error messages in the golden output

[Christoph Hellwig]

On Mon, Nov 23, 2015 at 01:25:33PM -0800, Darrick J. Wong wrote:
> > Shouldn't these be Invalid argument just like the
> > to a device case above or the clone case?
> 
> I was trying to mirror the behavior of reflink, which spits out
> EOPNOTSUPP when the destination isn't a regular file and EINVAL
> when the source isn't a regular file.

clone is called on the destination and takes the source from the
ioctl argument.  dedupe is called on the source and then opens the
destinations, so they're not really comparable.  Btrfs currently
returns EACCES for non-dir, non-regular destinations which look
wrong and I think EINVAL for a mismatch between source and destination
types would make most sense.  I've also prepared a btrfs patch for
this and clone, but I'd like to have consensus on the exact error
first.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] Btrfs: fix race between cleaner kthread and space cache writeout

[fdmanana]

From: Filipe Manana 

When a block group becomes unused and the cleaner kthread is currently
running, we can end up getting the current transaction aborted with error
-ENOENT when we try to commit the transaction, leading to the following
trace:

  [59779.258768] WARNING: CPU: 3 PID: 5990 at fs/btrfs/extent-tree.c:3740 
btrfs_write_dirty_block_groups+0x17c/0x214 [btrfs]()
  [59779.272594] BTRFS: Transaction aborted (error -2)
  (...)
  [59779.291137] Call Trace:
  [59779.291621]  [] dump_stack+0x4e/0x79
  [59779.292543]  [] warn_slowpath_common+0x9f/0xb8
  [59779.293435]  [] ? 
btrfs_write_dirty_block_groups+0x17c/0x214 [btrfs]
  [59779.295000]  [] warn_slowpath_fmt+0x48/0x50
  [59779.296138]  [] ? 
write_one_cache_group.isra.32+0x77/0x82 [btrfs]
  [59779.297663]  [] 
btrfs_write_dirty_block_groups+0x17c/0x214 [btrfs]
  [59779.299141]  [] commit_cowonly_roots+0x1de/0x261 [btrfs]
  [59779.300359]  [] btrfs_commit_transaction+0x4c4/0x99c 
[btrfs]
  [59779.301805]  [] btrfs_sync_fs+0x145/0x1ad [btrfs]
  [59779.302893]  [] sync_filesystem+0x7f/0x93
  (...)
  [59779.318186] ---[ end trace 577e2daff90da33a ]---

The following diagram illustrates a sequence of steps leading to this
problem:

   CPU 1 CPU 2

   

adds bg A to list
fs_info->unused_bgs

adds bg B to list
fs_info->unused_bgs

   

  cleaner kthread
delete_unused_bgs()

  sees bg A in list
  fs_info->unused_bgs

  btrfs_start_transaction()

   

  deletes bg A

update_block_group(bg C)

  --> adds bg C to list
  
fs_info->unused_bgs

  deletes bg B

  sees bg C in the list
  fs_info->unused_bgs

  btrfs_remove_chunk(bg C)
btrfs_remove_block_group(bg C)

  --> checks if the block group
  is in a dirty list, and
  because it isn't now, it
  does nothing

  --> the block group item
  is deleted from the
  extent tree

  --> adds bg C to list
  
transaction->dirty_bgs

 some task calls
 
btrfs_commit_transaction(t N + 1)
   
commit_cowonly_roots()
 
btrfs_write_dirty_block_groups()
   --> sees bg C in 
cur_trans->dirty_bgs
   --> calls 
write_one_cache_group()
   which 
returns -ENOENT because
   it did not 
find the block group
   item in the 
extent tree
   --> transaction 
aborte with -ENOENT
   because 
write_one_cache_group()
   returned 
that error

So fix this by adding a block group to the list of dirty block groups
before adding it to the list of unused block groups.

This happened on a stress test using fsstress plus concurrent calls to
fallocate 20G and truncate (releasing part of the space allocated with
fallocate).

Signed-off-by: Filipe Manana 
---
 fs/btrfs/extent-tree.c | 29 -
 1 file changed, 16 insertions(+), 13 deletions(-)

diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 8fd14b6..e563e5a 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -5915,19 +5915,6 @@ static int update_block_group(struct btrfs_trans_handle 
*trans,
set_extent_dirty(info->pinned_extents,
 bytenr, bytenr + num_bytes - 1,
 GFP_NOFS | __GFP_NOFAIL);
-   /*
-* No longer have used bytes in this block group, queue
-* it for deletion.
-*/
-   if (old_val == 0) {
-   spin_lock(>unused_bgs_lock);
-   if (list_empty(>bg_list)) {
-  

Re: [PATCH v2 3/5] btrfs-progs: kernel based default features for mkfs

[Austin S Hemmelgarn]

On 2015-11-23 10:57, Christoph Anton Mitterer wrote:

Hey.

On Mon, 2015-11-23 at 20:56 +0800, Anand Jain wrote:

This patch disables default features based on the running kernel.

Not sure if that's very realistic in practise (most people will have
some distro, whose btrfsprogs version probably matches the kernel), but
purely from the end-user PoV:

I would find it useful if btrfs gives a warning if it creates a
filesystem which (because unsupported in the current kernel) lacks
features which are considered default by then.
It should give a warning if the user requests a feature that is 
unsupported by the kernel it's being run on, but it should not by 
default try to enable something that isn't supported by the kernel it's 
running on.



AFAIU, really "clonding" (I mean including all snapshots, subvols,
etc.) a btrfs is not possible right now (or is it?), so a btrfs is
something that rather should stay longer (as one cannot easily
copy it to/with a new one)... so for me as an end-user, it may be
easier to switch to a newer kernel, in order to get a feature which is
default, than to migrate the fs later.
It is actually possible to clone a btrfs filesystem, just not in a way 
that people used to stuff like ext4 would recognize.  In essence, you 
need to take the FS mostly off-line, force all subvolumes to read-only, 
then use send-receive to transfer things, and finally make the 
subvolumes writable again.  I've been considering doing a script to do 
this automatically, but have never gotten around to it as it's not 
something that is quick to code, and it's not something I do very often.





smime.p7s
Description: S/MIME Cryptographic Signature

Re: [PATCH v2 3/5] btrfs-progs: kernel based default features for mkfs

[Christoph Anton Mitterer]

On Mon, 2015-11-23 at 11:05 -0500, Austin S Hemmelgarn wrote:
> I would find it useful if btrfs gives a warning if it creates a
> > filesystem which (because unsupported in the current kernel) lacks
> > features which are considered default by then.
> It should give a warning if the user requests a feature that is 
> unsupported by the kernel it's being run on, but it should not by 
> default try to enable something that isn't supported by the kernel
> it's 
> running on.
Well that as well, and of course it shouldn't try to enable a feature
that wouldn't work, but what I meant was, e.g. if I create a fs with
btrfs-progs 4.3 (where skinny-extents are default) but on such an old
kernel where this isn't supported yet,... it should tell me "Normally
I'd create the fs with skinny-extents, but I don't as your kernel is
too old".


> It is actually possible to clone a btrfs filesystem, just not in a
> way 
> that people used to stuff like ext4 would recognize.  In essence, you
> need to take the FS mostly off-line, force all subvolumes to read-
> only, 
> then use send-receive to transfer things, and finally make the 
> subvolumes writable again.  I've been considering doing a script to
> do 
> this automatically, but have never gotten around to it as it's not 
> something that is quick to code, and it's not something I do very
> often.
And that would also keep all ref-links, etc.? I.e. the copied fs
wouldn't eat up much more space than the original?
Well than such script should be part of btrfs-progs :-)

Cheers,
Chris.

smime.p7s
Description: S/MIME cryptographic signature

Re: [PATCH v2 3/5] btrfs-progs: kernel based default features for mkfs

[Austin S Hemmelgarn]

On 2015-11-23 11:14, Christoph Anton Mitterer wrote:

On Mon, 2015-11-23 at 11:05 -0500, Austin S Hemmelgarn wrote:

I would find it useful if btrfs gives a warning if it creates a

filesystem which (because unsupported in the current kernel) lacks
features which are considered default by then.

It should give a warning if the user requests a feature that is
unsupported by the kernel it's being run on, but it should not by
default try to enable something that isn't supported by the kernel
it's
running on.

Well that as well, and of course it shouldn't try to enable a feature
that wouldn't work, but what I meant was, e.g. if I create a fs with
btrfs-progs 4.3 (where skinny-extents are default) but on such an old
kernel where this isn't supported yet,... it should tell me "Normally
I'd create the fs with skinny-extents, but I don't as your kernel is
too old".

Ah, I misunderstood your meaning, that would be nice to have.

It is actually possible to clone a btrfs filesystem, just not in a
way
that people used to stuff like ext4 would recognize.  In essence, you
need to take the FS mostly off-line, force all subvolumes to read-
only,
then use send-receive to transfer things, and finally make the
subvolumes writable again.  I've been considering doing a script to
do
this automatically, but have never gotten around to it as it's not
something that is quick to code, and it's not something I do very
often.

And that would also keep all ref-links, etc.? I.e. the copied fs
wouldn't eat up much more space than the original?
Well than such script should be part of btrfs-progs :-)
There's no way to reliably preserve _all_ ref-links, but between 
snapshots and the subvolumes they are snapshots of, it should, assuming 
that you either send them all at once (which would take a long time 
probably), or that you do proper incremental transfers.  The other 
disadvantage is that it may not (depending on the features of course) 
cause any new features to be used except on new files.





smime.p7s
Description: S/MIME Cryptographic Signature

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[David Sterba]

On Mon, Nov 23, 2015 at 08:56:13PM +0800, Anand Jain wrote:
> Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
> be compatible w any set of older/newer kernels.
> 
> So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
> skinny-metadata even if the running kernel does not supports it, and
> so the mount fails on the running.

So the default behaviour of mkfs will try to best guess the feature set
of currently running kernel. I think this is is the most common scenario
and justifies the change in default behaviours.

For the other cases I'd like to introduce some human-readable shortcuts
to the --features option. Eg. 'mkfs.btrfs -O compat-3.2' will pick all
options supported by the unpatched mainline kernel of version 3.2. This
would be present for all version, regardless if there was a change in the
options or not.

Similarly for convenience, add 'running' that would pick the options
from running kernel but will be explicit.

A remaining option should override the 'running' behaviour and pick the
latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
name is yet to be determined.

> Here in this set of patches will make sure the progs understands the
> kernel supported features.
> 
> So in this patch, checks if sysfs tells whether the feature is
> supported if not, then it will relay on static kernel version which
> provided that feature (skinny-metadata here in this example), next
> if for some reason the running kernel does not provide the kernel
> version, then it will fall back to the original method to enable
> the feature with a hope that kernel will support it.
> 
> Also the last patch adds a warning when we fail to read either
> sysfs features or the running kernel version.

Your patchset is a good start, the additional options I've described can
be added on top of that. We might need to switch the version
representation from string to KERNEL_VERSION but that's an
implementation detail.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 00/25] Btrfs-convert rework to support native separate

[David Sterba]

On Fri, Nov 20, 2015 at 11:24:04AM +0800, Qu Wenruo wrote:
> Here comes the 1st version of btrfs-convert rework.
> Any test is welcomed, and it can already pass the convert test from
> btrfs-progs. (Since the test doesn't test rollback function)

I went through the patches, looks mostly ok akin to the proposed
changes. I'll take the independent patches rightaway. Unfortunatelly
there are code changes that clash significantly with the pending
reiserfs addition to convert, I'm afraid you'll have to rework your
patchset on top of that.  The code is now pushed to branch
'dev/convert-reiser'.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 3/5] btrfs-progs: kernel based default features for mkfs

[Christoph Anton Mitterer]

Hey.

On Mon, 2015-11-23 at 20:56 +0800, Anand Jain wrote:
> This patch disables default features based on the running kernel.
Not sure if that's very realistic in practise (most people will have
some distro, whose btrfsprogs version probably matches the kernel), but
purely from the end-user PoV:

I would find it useful if btrfs gives a warning if it creates a
filesystem which (because unsupported in the current kernel) lacks
features which are considered default by then.


AFAIU, really "clonding" (I mean including all snapshots, subvols,
etc.) a btrfs is not possible right now (or is it?), so a btrfs is
something that rather should stay longer (as one cannot easily
copy it to/with a new one)... so for me as an end-user, it may be
easier to switch to a newer kernel, in order to get a feature which is
default, than to migrate the fs later.


Cheers,
Chris.

smime.p7s
Description: S/MIME cryptographic signature

Re: btrfs send reproducibly fails for a specific subvolume after sending 15 GiB, scrub reports no errors

[Nils Steinger]

On Mon, Nov 23, 2015 at 05:49:05AM +, Duncan wrote:
> Btrfs scrub?  Why do you believe it will detect/fix the problem?

I was under the impression that btrfs scrub would detect all kinds of
inconsistencies (not just data-checksum mismatches), including whatever caused
btrfs send to fail.
Thanks for clearing up that misconception!


In this case, I ended up following Austin's first advice: I ran `btrfs receive
-vv` and moved the directory where it failed (something inside my Chromium
profile) off the filesystem and back onto it. When I reran `send` it failed
inside a neighbouring directory, so I recreated that one too. Cue some more
repetitions of that (with files from my Firefox and Skype profile directories).
In the end, I just rsync'd my entire home directory to a new subvolume, which
finally seems to have done the trick — at least I could `btrfs send` a snapshot
of the new subvolume to /dev/null.

Do we anything about what might cause a filesystem to enter a state which
`send` chokes on?
I've only seen a small sample of the corrupted files before growing tired of
the process and just recreating the whole thing, but all of them were database
files (presumably SQLite). Could it be that the files were being written to
during an unclean shutdown, leading to some kind of corruption of the FS?
Unfortunately, I was a little triggerhappy when cleaning up old snapshots, so
there aren't any left to aid in troubleshooting this problem further…


Regards,
Nils
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/2] generic/15[78]: fix error messages in the golden output

[Darrick J. Wong]

On Sat, Nov 21, 2015 at 10:06:44AM -0800, Christoph Hellwig wrote:
> > --- a/tests/generic/158.out
> > +++ b/tests/generic/158.out
> >  Try to dedupe a device
> > -XFS_IOC_FILE_EXTENT_SAME: Permission denied
> > +XFS_IOC_FILE_EXTENT_SAME: Invalid argument
> >  Try to dedupe to a dir
> > -/mnt/test-158/dir1: Is a directory
> > +TEST_DIR/test-158/dir1: Is a directory
> >  Try to dedupe to a device
> > -dedupe: Permission denied
> > +dedupe: Operation not supported
> >  Try to dedupe to a fifo
> > -dedupe: Permission denied
> > +dedupe: Operation not supported
> 
> Shouldn't these be Invalid argument just like the
> to a device case above or the clone case?

I was trying to mirror the behavior of reflink, which spits out EOPNOTSUPP when
the destination isn't a regular file and EINVAL when the source isn't a regular
file.

--D

> 
> Otherwise looks good,
> 
> Reviewed-by: Christoph Hellwig 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html