Re: dduper - Offline btrfs deduplication tool

[Lakshmipathi.G]

> 
> One question:
> Why not ioctl_fideduperange?
> i.e. you kill most of benefits from that ioctl - atomicity.
> 
I plan to add fideduperange as an option too. User can
choose between fideduperange and ficlonerange call.

If I'm not wrong, with fideduperange, kernel performs
comparsion check before dedupe. And it will increase
time to dedupe files.

I believe the risk involved with ficlonerange is  minimized 
by having a backup file(reflinked). We can revert to older 
original file, if we encounter some problems.

> 
> -- 
> Have a nice day,
> Timofey.

Cheers.
Lakshmipathi.G

dduper - Offline btrfs deduplication tool

[Lakshmipathi.G]

Hi -

dduper is an offline dedupe tool. Instead of reading whole file blocks and
computing checksum, It works by fetching checksum from BTRFS csum tree. This 
hugely improves the performance. 

dduper works like:
- Read csum for given two files.
- Find matching location.
- Pass the location to ioctl_ficlonerange directly
  instead of ioctl_fideduperange

By default, dduper adds safty check to above steps by creating a 
backup reflink file and compares the md5sum after dedupe. 
If the backup file matches new deduped file, then backup file is 
removed. You can skip this check by passing --skip option. Here is 
sample cli usage [1] and quick demo [2]  

Some performance numbers: (with -skip option)

Dedupe two 1GB files with same  content - 1.2 seconds
Dedupe two 5GB files with same  content - 8.2 seconds
Dedupe two 10GB files with same  content - 13.8 seconds

dduper requires `btrfs inspect-internal dump-csum` command, you can use 
this branch [3] or apply patch by yourself [4] 

[1] 
https://gitlab.collabora.com/laks/btrfs-progs/blob/dump_csum/Documentation/dduper_usage.md
[2] http://giis.co.in/btrfs_dedupe.gif
[3] git clone https://gitlab.collabora.com/laks/btrfs-progs.git -b  dump_csum
[4] https://patchwork.kernel.org/patch/10540229/ 

Please remember its version-0.1, so test it out, if you plan to use dduper real 
data.
Let me know, if you have suggestions or feedback or bugs :)

Cheers.
Lakshmipathi.G

[PATCH] v3: btrfs-progs: Introduce 'btrfs inspect-internal dump-csum' option

[Lakshmipathi.G]

Print csum for a given file on stdout.

Sample usage:
btrfs inspect-internal dump-csum /btrfs/50gbfile /dev/sda4
Signed-off-by: Lakshmipathi.G 
---
 Makefile |   2 +-
 cmds-inspect-dump-csum.c | 239 +++
 cmds-inspect-dump-csum.h |  22 +
 cmds-inspect.c   |   3 +
 4 files changed, 265 insertions(+), 1 deletion(-)
 create mode 100644 cmds-inspect-dump-csum.c
 create mode 100644 cmds-inspect-dump-csum.h

diff --git a/Makefile b/Makefile
index 544410e6..8ad28012 100644
--- a/Makefile
+++ b/Makefile
@@ -122,7 +122,7 @@ cmds_objects = cmds-subvolume.o cmds-filesystem.o 
cmds-device.o cmds-scrub.o \
   cmds-quota.o cmds-qgroup.o cmds-replace.o check/main.o \
   cmds-restore.o cmds-rescue.o chunk-recover.o super-recover.o \
   cmds-property.o cmds-fi-usage.o cmds-inspect-dump-tree.o \
-  cmds-inspect-dump-super.o cmds-inspect-tree-stats.o cmds-fi-du.o 
\
+  cmds-inspect-dump-super.o cmds-inspect-tree-stats.o cmds-fi-du.o 
cmds-inspect-dump-csum.o \
   mkfs/common.o check/mode-common.o check/mode-lowmem.o
 libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o 
\
   kernel-lib/crc32c.o messages.o \
diff --git a/cmds-inspect-dump-csum.c b/cmds-inspect-dump-csum.c
new file mode 100644
index ..06dba368
--- /dev/null
+++ b/cmds-inspect-dump-csum.c
@@ -0,0 +1,239 @@
+/*
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License v2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; if not, write to the
+ * Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+ * Boston, MA 021110-1307, USA.
+ */
+
+#include "kerncompat.h"
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include "ctree.h"
+#include "disk-io.h"
+#include "print-tree.h"
+#include "transaction.h"
+#include "list.h"
+#include "utils.h"
+#include "commands.h"
+#include "crc32c.h"
+#include "cmds-inspect-dump-csum.h"
+#include "help.h"
+#include "volumes.h"
+
+
+const char * const cmd_inspect_dump_csum_usage[] = {
+   "btrfs inspect-internal dump-csum  ",
+   "Get csums for the given file.",
+   NULL
+};
+
+int btrfs_lookup_csums(struct btrfs_trans_handle *trans, struct btrfs_root 
*root,
+   struct btrfs_path *path, u64 bytenr, int cow, int total_csums)
+{
+   int ret;
+   int i;
+   int start_pos = 0;
+   struct btrfs_key file_key;
+   struct btrfs_key found_key;
+   struct btrfs_csum_item *item;
+   struct extent_buffer *leaf;
+   u64 csum_offset = 0;
+   u16 csum_size =
+   btrfs_super_csum_size(root->fs_info->super_copy);
+   int csums_in_item = 0;
+   unsigned int tree_csum = 0;
+   int pending_csums = total_csums;
+   static int cnt=1;
+
+   file_key.objectid = BTRFS_EXTENT_CSUM_OBJECTID;
+   file_key.offset = bytenr;
+   file_key.type = BTRFS_EXTENT_CSUM_KEY;
+   ret = btrfs_search_slot(trans, root, _key, path, 0, cow);
+   if (ret < 0)
+   goto fail;
+   while(1){
+   leaf = path->nodes[0];
+   if (ret > 0) {
+   ret = 1;
+   if (path->slots[0] == 0)
+   goto fail;
+   path->slots[0]--;
+   btrfs_item_key_to_cpu(leaf, _key, path->slots[0]);
+   if (found_key.type != BTRFS_EXTENT_CSUM_KEY){
+   fprintf(stderr, "\nInvalid key found.");
+   goto fail;
+   }
+
+   csum_offset = ((bytenr - found_key.offset) / 
root->fs_info->sectorsize) * csum_size;
+   csums_in_item = btrfs_item_size_nr(leaf, 
path->slots[0]);
+   csums_in_item /= csum_size;
+   csums_in_item -= ( bytenr - found_key.offset ) / 
root->fs_info->sectorsize;
+   start_pos=csum_offset;
+   }
+   if (path->slots[0] >= btrfs_header_nritems(leaf)) {
+   if (pending_csums > 0){
+   ret = btrfs_next_leaf(root, path);
+   if (ret == 0)
+ conti

[PATCH] v2: btrfs-progs: Introduce 'btrfs inspect-internal dump-csum' option

[Lakshmipathi.G]

Dump csum for a given file.

Sample usage:
btrfs inspect-internal dump-csum /btrfs/50gbfile /dev/sda4
csum for /btrfs/50gbfile dumped to /btrfs/50gbfile.csumdump

Signed-off-by: Lakshmipathi.G 
---
 Makefile |   2 +-
 cmds-inspect-dump-csum.c | 256 +++
 cmds-inspect-dump-csum.h |  22 
 cmds-inspect.c   |   3 +
 4 files changed, 282 insertions(+), 1 deletion(-)
 create mode 100644 cmds-inspect-dump-csum.c
 create mode 100644 cmds-inspect-dump-csum.h

diff --git a/Makefile b/Makefile
index 544410e..8ad2801 100644
--- a/Makefile
+++ b/Makefile
@@ -122,7 +122,7 @@ cmds_objects = cmds-subvolume.o cmds-filesystem.o 
cmds-device.o cmds-scrub.o \
   cmds-quota.o cmds-qgroup.o cmds-replace.o check/main.o \
   cmds-restore.o cmds-rescue.o chunk-recover.o super-recover.o \
   cmds-property.o cmds-fi-usage.o cmds-inspect-dump-tree.o \
-  cmds-inspect-dump-super.o cmds-inspect-tree-stats.o cmds-fi-du.o 
\
+  cmds-inspect-dump-super.o cmds-inspect-tree-stats.o cmds-fi-du.o 
cmds-inspect-dump-csum.o \
   mkfs/common.o check/mode-common.o check/mode-lowmem.o
 libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o 
\
   kernel-lib/crc32c.o messages.o \
diff --git a/cmds-inspect-dump-csum.c b/cmds-inspect-dump-csum.c
new file mode 100644
index 000..38afcc3
--- /dev/null
+++ b/cmds-inspect-dump-csum.c
@@ -0,0 +1,256 @@
+/*
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License v2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; if not, write to the
+ * Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+ * Boston, MA 021110-1307, USA.
+ */
+
+#include "kerncompat.h"
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include "ctree.h"
+#include "disk-io.h"
+#include "print-tree.h"
+#include "transaction.h"
+#include "list.h"
+#include "utils.h"
+#include "commands.h"
+#include "crc32c.h"
+#include "cmds-inspect-dump-csum.h"
+#include "help.h"
+#include "volumes.h"
+
+
+const char * const cmd_inspect_dump_csum_usage[] = {
+   "btrfs inspect-internal dump-csum  ",
+   "Get csums for the given file.",
+   NULL
+};
+
+int btrfs_lookup_csums(struct btrfs_trans_handle *trans, struct btrfs_root 
*root,
+   struct btrfs_path *path, u64 bytenr, int cow, int total_csums, FILE *fp)
+{
+   int ret;
+   int i;
+   int start_pos = 0;
+   struct btrfs_key file_key;
+   struct btrfs_key found_key;
+   struct btrfs_csum_item *item;
+   struct extent_buffer *leaf;
+   u64 csum_offset = 0;
+   u16 csum_size =
+   btrfs_super_csum_size(root->fs_info->super_copy);
+   int csums_in_item = 0;
+   unsigned int tree_csum = 0;
+   int pending_csums = total_csums;
+   static int cnt=1;
+
+   file_key.objectid = BTRFS_EXTENT_CSUM_OBJECTID;
+   file_key.offset = bytenr;
+   file_key.type = BTRFS_EXTENT_CSUM_KEY;
+   ret = btrfs_search_slot(trans, root, _key, path, 0, cow);
+   if (ret < 0)
+   goto fail;
+   while(1){
+   leaf = path->nodes[0];
+   if (ret > 0) {
+   ret = 1;
+   if (path->slots[0] == 0)
+   goto fail; 
+   path->slots[0]--;
+   btrfs_item_key_to_cpu(leaf, _key, path->slots[0]);
+   if (found_key.type != BTRFS_EXTENT_CSUM_KEY){
+   fprintf(stderr, "\nInvalid key found.");
+   goto fail;
+   }
+
+   csum_offset = ((bytenr - found_key.offset) / 
root->fs_info->sectorsize) * csum_size;
+   csums_in_item = btrfs_item_size_nr(leaf, 
path->slots[0]);
+   csums_in_item /= csum_size;
+   csums_in_item -= ( bytenr - found_key.offset ) / 
root->fs_info->sectorsize;
+   start_pos=csum_offset;
+   }
+   if (path->slots[0] >= btrfs_header_nritems(leaf)) {
+   if (pending_csums > 0){
+   ret = btrfs_next_leaf(root, path);
+  

Re: [PATCH v2.2 04/10] btrfs: qgroup: Use separate meta reservation type for delalloc

[Lakshmipathi.G]

> And for anyone who wants to test the latest version of the patchset, it
> can be fetched from the following branch:

I started a build with qgroup_meta_rework from [0] and its currently
running [1],
will update the results  once it completes in few hours time.

[0]: https://github.com/adam900710/linux :
[1]: http://btrfs.in:3000/r/Lakshmipathi


Cheers,
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: xfstests/btrfs/100 and 151 failure

[Lakshmipathi.G]

> Thanks for the config file, will check with this and update the status.

script/100 passed on today's run [1] with my kernel config too. May be
a false alarm :-)

[1]: 
https://github.com/Lakshmipathi/btrfsqa/blob/master/results/results_2017-12-22_07:12/

thanks!

Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org http://www.btrfs.in
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: xfstests/btrfs/100 and 151 failure

[Lakshmipathi.G]

>
> I've attached my master config before compiling a kernel with it I just
> do make olddefconfig to sync it with latest kbuild options.
>
>>

Thanks for the config file, will check with this and update the status.


Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: xfstests/btrfs/100 and 151 failure

[Lakshmipathi.G]

> Actually 151 has been failing for me as well but not 100
>
Okay, can you share the kernel .config file? I'll give it a try
with those config and check 100, sometime tomorrow.


Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

xfstests/btrfs/100 and 151 failure

[Lakshmipathi.G]

Hi.

While checking with latest misc-next branch from btrfs-devel.
btrfs/100 and 151 failed.
Details available on below bugzilla links. thanks

https://bugzilla.kernel.org/show_bug.cgi?id=198085
https://bugzilla.kernel.org/show_bug.cgi?id=198087


Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 4/4] btrfs: factor __btrfs_open_devices() to create btrfs_open_one_device()

[Lakshmipathi.G]

Hi Anand,

Applied your fix, now the issue resolved. thanks!


Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org


On Mon, Nov 27, 2017 at 5:17 PM, Anand Jain <anand.j...@oracle.com> wrote:
>
> Hi Lakshmipathi,
>
>  Oops I can see the same. I am trying to narrow down.
>
> Thanks, Anand
>
>
> On 11/27/2017 02:47 PM, Lakshmipathi.G wrote:
>>
>> Hi Anand,
>>
>> With this patch applied, btrfs-progs/misc-test/021 error out. Is this
>> same for you?
>>
>> Without this patch: https://asciinema.org/a/RJmE5469mHlL3S1BIOCifWVn6
>> With this patch:https://asciinema.org/a/1h5UX6DIFNsvvMXgLo4GiEgdE
>>
>> thanks!
>> 
>> Cheers,
>> Lakshmipathi.G
>> http://www.giis.co.in http://www.webminal.org
>>
>>
>> On Thu, Nov 9, 2017 at 9:15 PM, Anand Jain <anand.j...@oracle.com> wrote:
>>>
>>> No functional changes, create btrfs_open_one_device() from
>>> __btrfs_open_devices(). This is a preparatory work to add dynamic
>>> device scan.
>>>
>>> Signed-off-by: Anand Jain <anand.j...@oracle.com>
>>> ---
>>>   fs/btrfs/volumes.c | 126
>>> +
>>>   1 file changed, 69 insertions(+), 57 deletions(-)
>>>
>>> diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
>>> index 0857b580014d..d24e966ee29f 100644
>>> --- a/fs/btrfs/volumes.c
>>> +++ b/fs/btrfs/volumes.c
>>> @@ -601,6 +601,73 @@ void btrfs_free_stale_device(struct btrfs_device
>>> *cur_dev)
>>>  }
>>>   }
>>>
>>> +static int btrfs_open_one_device(struct btrfs_fs_devices *fs_devices,
>>> +   struct btrfs_device *device, fmode_t flags,
>>> +   void *holder)
>>> +{
>>> +   struct request_queue *q;
>>> +   struct block_device *bdev;
>>> +   struct buffer_head *bh;
>>> +   struct btrfs_super_block *disk_super;
>>> +   u64 devid;
>>> +   int ret;
>>> +
>>> +   if (device->bdev)
>>> +   return -EINVAL;
>>> +   if (!device->name)
>>> +   return -EINVAL;
>>> +
>>> +   ret = btrfs_get_bdev_and_sb(device->name->str, flags, holder, 1,
>>> +   , );
>>> +   if (ret)
>>> +   return ret;
>>> +
>>> +   disk_super = (struct btrfs_super_block *)bh->b_data;
>>> +   devid = btrfs_stack_device_id(_super->dev_item);
>>> +   if (devid != device->devid)
>>> +   goto error_brelse;
>>> +
>>> +   if (memcmp(device->uuid, disk_super->dev_item.uuid,
>>> +  BTRFS_UUID_SIZE))
>>> +   goto error_brelse;
>>> +
>>> +   device->generation = btrfs_super_generation(disk_super);
>>> +
>>> +   if (btrfs_super_flags(disk_super) & BTRFS_SUPER_FLAG_SEEDING) {
>>> +   device->writeable = 0;
>>> +   fs_devices->seeding = 1;
>>> +   } else {
>>> +   device->writeable = !bdev_read_only(bdev);
>>> +   }
>>> +
>>> +   q = bdev_get_queue(bdev);
>>> +   if (blk_queue_discard(q))
>>> +   device->can_discard = 1;
>>> +   if (!blk_queue_nonrot(q))
>>> +   fs_devices->rotating = 1;
>>> +
>>> +   device->bdev = bdev;
>>> +   device->in_fs_metadata = 0;
>>> +   device->mode = flags;
>>> +
>>> +   fs_devices->open_devices++;
>>> +   if (device->writeable &&
>>> +   device->devid != BTRFS_DEV_REPLACE_DEVID) {
>>> +   fs_devices->rw_devices++;
>>> +   list_add(>dev_alloc_list,
>>> +_devices->alloc_list);
>>> +   }
>>> +   brelse(bh);
>>> +
>>> +   return 0;
>>> +
>>> +error_brelse:
>>> +   brelse(bh);
>>> +   blkdev_put(bdev, flags);
>>> +
>>> +   return -EINVAL;
>>> +}
>>> +
>>>   /*
>>>* Add new device to list of registered devices
>>>*
>>> @@ -978,69 +1045,14 @@ static int __btrfs_open_devices(struct
>>> btrfs_fs_devices *fs_devices,
>>>  flags |= FMODE_EXCL;
>>>
>>>  list_fo

Re: [PATCH 4/4] btrfs: factor __btrfs_open_devices() to create btrfs_open_one_device()

[Lakshmipathi.G]

Hi Anand,

With this patch applied, btrfs-progs/misc-test/021 error out. Is this
same for you?

Without this patch: https://asciinema.org/a/RJmE5469mHlL3S1BIOCifWVn6
With this patch:https://asciinema.org/a/1h5UX6DIFNsvvMXgLo4GiEgdE

thanks!

Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org


On Thu, Nov 9, 2017 at 9:15 PM, Anand Jain <anand.j...@oracle.com> wrote:
> No functional changes, create btrfs_open_one_device() from
> __btrfs_open_devices(). This is a preparatory work to add dynamic
> device scan.
>
> Signed-off-by: Anand Jain <anand.j...@oracle.com>
> ---
>  fs/btrfs/volumes.c | 126 
> +
>  1 file changed, 69 insertions(+), 57 deletions(-)
>
> diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
> index 0857b580014d..d24e966ee29f 100644
> --- a/fs/btrfs/volumes.c
> +++ b/fs/btrfs/volumes.c
> @@ -601,6 +601,73 @@ void btrfs_free_stale_device(struct btrfs_device 
> *cur_dev)
> }
>  }
>
> +static int btrfs_open_one_device(struct btrfs_fs_devices *fs_devices,
> +   struct btrfs_device *device, fmode_t flags,
> +   void *holder)
> +{
> +   struct request_queue *q;
> +   struct block_device *bdev;
> +   struct buffer_head *bh;
> +   struct btrfs_super_block *disk_super;
> +   u64 devid;
> +   int ret;
> +
> +   if (device->bdev)
> +   return -EINVAL;
> +   if (!device->name)
> +   return -EINVAL;
> +
> +   ret = btrfs_get_bdev_and_sb(device->name->str, flags, holder, 1,
> +   , );
> +   if (ret)
> +   return ret;
> +
> +   disk_super = (struct btrfs_super_block *)bh->b_data;
> +   devid = btrfs_stack_device_id(_super->dev_item);
> +   if (devid != device->devid)
> +   goto error_brelse;
> +
> +   if (memcmp(device->uuid, disk_super->dev_item.uuid,
> +  BTRFS_UUID_SIZE))
> +   goto error_brelse;
> +
> +   device->generation = btrfs_super_generation(disk_super);
> +
> +   if (btrfs_super_flags(disk_super) & BTRFS_SUPER_FLAG_SEEDING) {
> +   device->writeable = 0;
> +   fs_devices->seeding = 1;
> +   } else {
> +   device->writeable = !bdev_read_only(bdev);
> +   }
> +
> +   q = bdev_get_queue(bdev);
> +   if (blk_queue_discard(q))
> +   device->can_discard = 1;
> +   if (!blk_queue_nonrot(q))
> +   fs_devices->rotating = 1;
> +
> +   device->bdev = bdev;
> +   device->in_fs_metadata = 0;
> +   device->mode = flags;
> +
> +   fs_devices->open_devices++;
> +   if (device->writeable &&
> +   device->devid != BTRFS_DEV_REPLACE_DEVID) {
> +   fs_devices->rw_devices++;
> +   list_add(>dev_alloc_list,
> +_devices->alloc_list);
> +   }
> +   brelse(bh);
> +
> +   return 0;
> +
> +error_brelse:
> +   brelse(bh);
> +   blkdev_put(bdev, flags);
> +
> +   return -EINVAL;
> +}
> +
>  /*
>   * Add new device to list of registered devices
>   *
> @@ -978,69 +1045,14 @@ static int __btrfs_open_devices(struct 
> btrfs_fs_devices *fs_devices,
> flags |= FMODE_EXCL;
>
> list_for_each_entry(device, head, dev_list) {
> -   struct request_queue *q;
> -   struct block_device *bdev;
> -   struct buffer_head *bh;
> -   struct btrfs_super_block *disk_super;
> -   u64 devid;
> -
> -   if (device->bdev)
> -   continue;
> -   if (!device->name)
> -   continue;
> -
> /* Just open everything we can; ignore failures here */
> -   if (btrfs_get_bdev_and_sb(device->name->str, flags, holder, 1,
> -   , ))
> +   ret = btrfs_open_one_device(fs_devices, device, flags, 
> holder);
> +   if (ret)
> continue;
>
> -   disk_super = (struct btrfs_super_block *)bh->b_data;
> -   devid = btrfs_stack_device_id(_super->dev_item);
> -   if (devid != device->devid)
> -   goto error_brelse;
> -
> -   if (memcmp(device->uuid, disk_super->dev_item.uuid,
> -  BTRFS_UUID_SIZE))
> -   goto error_brelse;
> -
> -   device->

Re: misc/021-image-multi-devices fails on latest btrfs-devel/misc-next

[Lakshmipathi.G]

Hi Qu,

I checked with David, Seems like his github/repo.or.cz repo has latest
dev patches
before pushing them into kernel.org repo.

--
With the help of 'git bisect', managed to debug this issue further.
With this patch[1]
applied we are running into this issue.

https://asciinema.org/a/OMLXpd69iHaej0Xfsrr2ihKiP

[1] btrfs: factor __btrfs_open_devices() to create btrfs_open_one_device()
https://github.com/kdave/btrfs-devel/commit/25908a59bb5c54f896c646ed3a1f73b2727ad9b5


Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org


On Fri, Nov 24, 2017 at 5:35 PM, Lakshmipathi.G
<lakshmipath...@gmail.com> wrote:
>>>>
>>>> To make it clear, what kernel config and commit are you using when the
>>>> test fails?
>>>>
>>> When I use attached kernel config along with tar from misc-next branch
>>> (wget https://github.com/kdave/btrfs-devel/archive/misc-next.zip) this
>>> issue happens.
>>
>> BTW, David seems to update his branch more often in his git.kernel.org
>> branch than github.
>>
>> So it's possible that the branch from github is out dated.
>
> Okay, I didn't know that one, let me checkout his git.kernel.org branch and
> check the results there.
>
>>
>> For v4.14-rc7, even I enabled all the btrfs related configs, I still
>> can't reproduce it.
>> --
>> $ zcat /proc/config.gz  | grep BTRFS
>> CONFIG_BTRFS_FS=m
>> CONFIG_BTRFS_FS_POSIX_ACL=y
>> CONFIG_BTRFS_FS_CHECK_INTEGRITY=y
>> CONFIG_BTRFS_FS_RUN_SANITY_TESTS=y
>> CONFIG_BTRFS_DEBUG=y
>> CONFIG_BTRFS_ASSERT=y
>> --
>>
>> So there is something strange happened.
>>
>
> Yes. something strange going on. I use the same kernel config
> for misc-next, 4.14 and 4.14.2 - issue happens only with
> misc-next, as you mentioned it may not be up-to-date.
>
>
> 
> Cheers,
> Lakshmipathi.G
> http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: misc/021-image-multi-devices fails on latest btrfs-devel/misc-next

[Lakshmipathi.G]

>>>
>>> To make it clear, what kernel config and commit are you using when the
>>> test fails?
>>>
>> When I use attached kernel config along with tar from misc-next branch
>> (wget https://github.com/kdave/btrfs-devel/archive/misc-next.zip) this
>> issue happens.
>
> BTW, David seems to update his branch more often in his git.kernel.org
> branch than github.
>
> So it's possible that the branch from github is out dated.

Okay, I didn't know that one, let me checkout his git.kernel.org branch and
check the results there.

>
> For v4.14-rc7, even I enabled all the btrfs related configs, I still
> can't reproduce it.
> --
> $ zcat /proc/config.gz  | grep BTRFS
> CONFIG_BTRFS_FS=m
> CONFIG_BTRFS_FS_POSIX_ACL=y
> CONFIG_BTRFS_FS_CHECK_INTEGRITY=y
> CONFIG_BTRFS_FS_RUN_SANITY_TESTS=y
> CONFIG_BTRFS_DEBUG=y
> CONFIG_BTRFS_ASSERT=y
> --
>
> So there is something strange happened.
>

Yes. something strange going on. I use the same kernel config
for misc-next, 4.14 and 4.14.2 - issue happens only with
misc-next, as you mentioned it may not be up-to-date.



Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: misc/021-image-multi-devices fails on latest btrfs-devel/misc-next

[Lakshmipathi.G]

>
> No, it's not about btrfs-image itself, the flag is introduced long long
> long way ago.
>
Okay.

> It's about kernel change which doesn't allow image with such flag to be
> mounted AFAIK.

No luck yet! It works on 4.14.0
-
-bash-4.4$ sudo make TEST=021\* test-misc
[TEST]   misc-tests.sh
[TEST/misc]   021-image-multi-devices
-bash-4.4$ sudo dmesg
[  162.635824] BTRFS: device fsid f0c92f1b-c1ba-4576-8a07-8ce4f7b5bacd
devid 1 transid 5 /dev/loop0
[  162.672952] BTRFS: device fsid f0c92f1b-c1ba-4576-8a07-8ce4f7b5bacd
devid 2 transid 5 /dev/loop1
[  162.692848] BTRFS info (device loop1): disk space caching is enabled
[  162.695725] BTRFS info (device loop1): has skinny extents
[  162.698003] BTRFS info (device loop1): flagging fs with big metadata feature
[  162.701643] BTRFS info (device loop1): creating UUID tree
[  163.792851] BTRFS warning (device loop0): unrecognized super flag:
17179869184
[  163.796350] BTRFS info (device loop0): disk space caching is enabled
[  163.807644] BTRFS info (device loop0): has skinny extents
[  163.811613] BTRFS info (device loop0): checking UUID tree
[  163.841625] BTRFS error (device loop0): 1 errors while writing supers
[  163.845457] BTRFS: error (device loop0) in write_all_supers:3709:
errno=-5 IO failure (1 errors while writing supers)
[  163.851520] BTRFS info (device loop0): forced readonly
[  163.855184] BTRFS warning (device loop0): Skipping commit of
aborted transaction.
[  163.859121] BTRFS: error (device loop0) in
cleanup_transaction:1873: errno=-5 IO failure
[  163.862577] BTRFS info (device loop0): delayed_refs has NO entry
[  163.865623] BTRFS error (device loop0): commit super ret -5
[  163.868303] BTRFS error (device loop0): cleaner transaction attach
returned -30
-bash-4.4$ uname -a
Linux ip-172-31-33-196.us-west-2.compute.internal 4.14.0 #1 SMP Fri
Nov 24 09:48:11 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
-

Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: misc/021-image-multi-devices fails on latest btrfs-devel/misc-next

[Lakshmipathi.G]

Okay thanks for the details on btrfs-image. I was about to try with
previous version of
btrfs-image.

I'll use v4.14 (https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.14.tar.xz)
and check
again.


Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org


On Fri, Nov 24, 2017 at 1:36 PM, Qu Wenruo <quwenruo.bt...@gmx.com> wrote:
>
>
> On 2017年11月24日 15:47, Lakshmipathi.G wrote:
>> Hi Qu,
>>
>> I'm using btrfs-progs devel branch. mount command is failing with misc/021.
>> When I tried to manually mount it. dmesg shows  "unrecognized super
>> flag: 17179869184"
>> and interestingly "df -h" shows 16E as available!
>
> That flag is meta dump flag, any image recovered from btrfs-image should
> have that flag.
>
> 16E may also related to that, but not a big problem as btrfs-image
> result should only be used by developers, and all its data is wiped out.
> (Maybe one day we will not allow RW mount for such image)
>
>
> The problem happens when you try to mount the image, there may be some
> new commits doing more restrict check for rw mount in devel branch.
>
> Would you please try v4.14 kernel?
>
> Thanks,
> Qu
>
>>
>> https://asciinema.org/a/4JGYg7YGQz1PHtdBRJ6AVw0fh
>>
>> Let me check further.
>> 
>> Cheers,
>> Lakshmipathi.G
>> http://www.giis.co.in http://www.webminal.org
>>
>>
>> On Fri, Nov 24, 2017 at 9:55 AM, Lakshmipathi.G
>> <lakshmipath...@gmail.com> wrote:
>>> Hi Qu,
>>>
>>> I'm using the same instance. Let me check again with and without the
>>> commits list on Nov-23
>>> (https://github.com/kdave/btrfs-devel/commits/misc-next) and get back
>>> with results.
>>>
>>> 
>>> Cheers,
>>> Lakshmipathi.G
>>> http://www.giis.co.in http://www.webminal.org
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: misc/021-image-multi-devices fails on latest btrfs-devel/misc-next

[Lakshmipathi.G]

Hi Qu,

I'm using btrfs-progs devel branch. mount command is failing with misc/021.
When I tried to manually mount it. dmesg shows  "unrecognized super
flag: 17179869184"
and interestingly "df -h" shows 16E as available!

https://asciinema.org/a/4JGYg7YGQz1PHtdBRJ6AVw0fh

Let me check further.

Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org


On Fri, Nov 24, 2017 at 9:55 AM, Lakshmipathi.G
<lakshmipath...@gmail.com> wrote:
> Hi Qu,
>
> I'm using the same instance. Let me check again with and without the
> commits list on Nov-23
> (https://github.com/kdave/btrfs-devel/commits/misc-next) and get back
> with results.
>
> 
> Cheers,
> Lakshmipathi.G
> http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: misc/021-image-multi-devices fails on latest btrfs-devel/misc-next

[Lakshmipathi.G]

Hi Qu,

I'm using the same instance. Let me check again with and without the
commits list on Nov-23
(https://github.com/kdave/btrfs-devel/commits/misc-next) and get back
with results.


Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: misc/021-image-multi-devices fails on latest btrfs-devel/misc-next

[Lakshmipathi.G]

Quick update: It continues to fail:
https://asciinema.org/a/8HAL3uZ4MG7eb5gXnlgq3jm7q

Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org


On Sat, Nov 18, 2017 at 12:59 PM, Lakshmipathi.G
<lakshmipath...@gmail.com> wrote:
> Hi.
>
> Few days back(Nov14th) this(misc/021-image-multi-devices) test script
> used to pass and seems like its failing
> now. may be due to recent commits? Logs,screen-casts available here:
> https://lakshmipathi.github.io/btrfsqa/
>
> ----
> Cheers,
> Lakshmipathi.G
> http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

misc/021-image-multi-devices fails on latest btrfs-devel/misc-next

[Lakshmipathi.G]

Hi.

Few days back(Nov14th) this(misc/021-image-multi-devices) test script
used to pass and seems like its failing
now. may be due to recent commits? Logs,screen-casts available here:
https://lakshmipathi.github.io/btrfsqa/


Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

btrfs test bed

[Lakshmipathi.G]

Hi,

I'm trying to setup a public test bed for btrfs. Idea is to quickly
test user scripts and check
its valid or not for specific kernel/btrfs-progs versions. Also keep
running them daily and
record/save the logs. Its ran for few days and results can be view on
below link[2].

If you have suggestion/idea/scripts to improve this setup, let me know. thanks!

[1] Setup: https://github.com/Lakshmipathi/btrfsqa
[2] Results: https://lakshmipathi.github.io/btrfsqa/



Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/2] btrfs-progs: Replace some BUG_ON by return

[Lakshmipathi.G]

Hi Gu,

I applied these two patches:

https://patchwork.kernel.org/patch/9939879/
https://patchwork.kernel.org/patch/9939881/

Still the test "sudo make TEST=003\* test-fuzz" is failing like:

 btrfs-progs/btrfs check --check-data-csum
btrfs-progs/tests/fuzz-tests/images/bko-156731.raw.restored
checking extents
cmds-check.c:5660: check_owner_ref: BUG_ON `rec->is_root` triggered, value 1
 btrfs-progs/btrfs[0x44dce4]
 btrfs-progs/btrfs[0x45b396]
 btrfs-progs/btrfs[0x45c2f0]
 btrfs-progs/btrfs[0x45ce92]
 btrfs-progs/btrfs(cmd_check+0x17ee)[0x460ee7]
 btrfs-progs/btrfs(main+0x15d)[0x40b523]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7faf98a4d830]
 btrfs-progs/btrfs(_start+0x29)[0x40aff9]
Checking filesystem on
btrfs-progs/tests/fuzz-tests/images/bko-156731.raw.restored
UUID: 52975385-d05a-482b-8fb8-613a86a66042
failed (ignored, ret=134):  btrfs-progs/btrfs check --check-data-csum
btrfs-progs/tests/fuzz-tests/images/bko-156731.raw.restored
mayfail: returned code 134 (SIGABRT), not ignored
test failed for case 003-multi-check-unmounted

Did I miss some other patch? thanks.


Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org


On Wed, Sep 6, 2017 at 7:12 AM, Gu Jinxiang <g...@cn.fujitsu.com> wrote:
> The following test failed becasuse there is no data/metadata/system 
> block_group.
> Use return ret to replace BUG_ON(ret) to avoid system crash, because there is
> enough message for user to understand what happened.
>
> $sudo TEST=003\* make test-fuzz
> Unable to find block group for 0
> Unable to find block group for 0
> Unable to find block group for 0
> extent-tree.c:2693: btrfs_reserve_extent: BUG_ON `ret` triggered, value -28
> /home/fnst/btrfs/btrfs-progs/btrfs[0x419966]
> /home/fnst/btrfs/btrfs-progs/btrfs(btrfs_reserve_extent+0xb16)[0x41f500]
> /home/fnst/btrfs/btrfs-progs/btrfs(btrfs_alloc_free_block+0x55)[0x41f59b]
> /home/fnst/btrfs/btrfs-progs/btrfs[0x46a6ce]
> /home/fnst/btrfs/btrfs-progs/btrfs(cmd_check+0x1012)[0x47c885]
> /home/fnst/btrfs/btrfs-progs/btrfs(main+0x127)[0x40b055]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x2aae83e89f45]
> /home/fnst/btrfs/btrfs-progs/btrfs[0x40a939]
> Creating a new CRC tree
> Checking filesystem on 
> /home/fnst/btrfs/btrfs-progs/tests/fuzz-tests/images/bko-155621-bad-block-group-offset.raw.restored
> UUID: 5cb33553-6f6d-4ce8-83fd-20af5a2f8181
> Reinitialize checksum tree
> failed (ignored, ret=134): /home/fnst/btrfs/btrfs-progs/btrfs check 
> --init-csum-tree 
> /home/fnst/btrfs/btrfs-progs/tests/fuzz-tests/images/bko-155621-bad-block-group-offset.raw.restored
> mayfail: returned code 134 (SIGABRT), not ignored
> test failed for case 003-multi-check-unmounted
>
> Signed-off-by: Gu Jinxiang <g...@cn.fujitsu.com>
> ---
>  extent-tree.c | 16 ++--
>  transaction.c |  8 ++--
>  2 files changed, 16 insertions(+), 8 deletions(-)
>
> diff --git a/extent-tree.c b/extent-tree.c
> index eed56886..14838a5d 100644
> --- a/extent-tree.c
> +++ b/extent-tree.c
> @@ -2678,11 +2678,13 @@ int btrfs_reserve_extent(struct btrfs_trans_handle 
> *trans,
> ret = do_chunk_alloc(trans, info,
>  num_bytes,
>  BTRFS_BLOCK_GROUP_METADATA);
> -   BUG_ON(ret);
> +   if (ret)
> +   goto out;
> }
> ret = do_chunk_alloc(trans, info,
>  num_bytes + SZ_2M, data);
> -   BUG_ON(ret);
> +   if (ret)
> +   goto out;
> }
>
> WARN_ON(num_bytes < info->sectorsize);
> @@ -2690,9 +2692,11 @@ int btrfs_reserve_extent(struct btrfs_trans_handle 
> *trans,
>search_start, search_end, hint_byte, ins,
>trans->alloc_exclude_start,
>trans->alloc_exclude_nr, data);
> -   BUG_ON(ret);
> +   if (ret)
> +   goto out;
> clear_extent_dirty(>free_space_cache,
>ins->objectid, ins->objectid + ins->offset - 1);
> +out:
> return ret;
>  }
>
> @@ -2761,7 +2765,8 @@ static int alloc_tree_block(struct btrfs_trans_handle 
> *trans,
> int ret;
> ret = btrfs_reserve_extent(trans, root, num_bytes, empty_size,
>hint_byte, search_end, ins, 0);
> -   BUG_ON(ret);
> +   if (ret)
> +   goto out;
>
> if (root_objectid == BTRFS_EXTENT_TREE_OBJECTID) {
> struct pending_extent_op *extent_op;
> @@ -2792,6 +2797,7 @@ static int al

Re: kernel BUG at fs/btrfs/ctree.h:3457

[Lakshmipathi.G]

> Please skip that patch, as the check timing has its problem.
>

Okay, I'll skip the patch and run the tests hereafter. thanks.


Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kernel BUG at fs/btrfs/ctree.h:3457

[Lakshmipathi.G]

Seems like output is messed by my email client: Here is formatted one:
https://bugzilla.kernel.org/show_bug.cgi?id=197587#c6


Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org


On Mon, Nov 6, 2017 at 10:47 AM, Lakshmipathi.G
<lakshmipath...@gmail.com> wrote:
> Hi.
>
> While running  xfstests/065 with 4.14.0-rc7 and latest btrfs-progs
> from devel branch, it crashes the system after dumping below log:
> (Note: For btrfs-devel, I'm still using SANITY_TESTS=y and
> FS_INTEGRITY=y and applied patch
> https://patchwork.kernel.org/patch/10038011/)
>
>
> Nov 06 05:08:09 ip-172-31-32-145.us-west-2.compute.internal kernel:
> BTRFS info (device xvdf): disk space caching is enabled
> Nov 06 05:08:09 ip-172-31-32-145.us-west-2.compute.internal kernel:
> BTRFS info (device xvdf): has skinny extents
> Nov 06 05:08:09 ip-172-31-32-145.us-west-2.compute.internal kernel:
> BTRFS info (device xvdf): flagging fs with big metadata feature
> Nov 06 05:08:09 ip-172-31-32-145.us-west-2.compute.internal kernel:
> BTRFS info (device xvdf): enabling ssd optimizations
> Nov 06 05:08:09 ip-172-31-32-145.us-west-2.compute.internal kernel:
> BTRFS info (device xvdf): creating UUID tree
> Nov 06 05:08:10 ip-172-31-32-145.us-west-2.compute.internal kernel:
> BTRFS info (device xvdf): dev_replace from /dev/xvdd (devid 2) to
> /dev/xvdg started
> Nov 06 05:08:12 ip-172-31-32-145.us-west-2.compute.internal kernel:
> BTRFS info (device xvdf): dev_replace from /dev/xvdd (devid 2) to
> /dev/xvdg finished
> Nov 06 05:08:16 ip-172-31-32-145.us-west-2.compute.internal kernel:
> BTRFS: device fsid 3ebc32b5-ba0a-405b-8acf-4785b0a4b81a devid 1
> transid 5 /dev/xvdc
> Nov 06 05:08:16 ip-172-31-32-145.us-west-2.compute.internal kernel:
> BTRFS: device fsid 3ebc32b5-ba0a-405b-8acf-4785b0a4b81a devid 2
> transid 5 /dev/xvdd
> Nov 06 05:08:17 ip-172-31-32-145.us-west-2.compute.internal kernel:
> BTRFS: device fsid 3ebc32b5-ba0a-405b-8acf-4785b0a4b81a devid 3
> transid 5 /dev/xvde
> Nov 06 05:08:17 ip-172-31-32-145.us-west-2.compute.internal kernel:
> BTRFS: device fsid 3ebc32b5-ba0a-405b-8acf-4785b0a4b81a devid 4
> transid 5 /dev/xvdf
> Nov 06 05:08:17 ip-172-31-32-145.us-west-2.compute.internal kernel:
> BTRFS info (device xvdf): disk space caching is enabled
> Nov 06 05:08:17 ip-172-31-32-145.us-west-2.compute.internal kernel:
> BTRFS info (device xvdf): has skinny extents
> Nov 06 05:08:17 ip-172-31-32-145.us-west-2.compute.internal kernel:
> BTRFS info (device xvdf): flagging fs with big metadata feature
> Nov 06 05:08:17 ip-172-31-32-145.us-west-2.compute.internal kernel:
> BTRFS info (device xvdf): enabling ssd optimizations
> Nov 06 05:08:17 ip-172-31-32-145.us-west-2.compute.internal kernel:
> BTRFS info (device xvdf): creating UUID tree
> Nov 06 05:08:18 ip-172-31-32-145.us-west-2.compute.internal kernel:
> BTRFS info (device xvdf): dev_replace from /dev/xvdd (devid 2) to
> /dev/xvdg started
> Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel:
> BTRFS critical (device xvdf): corrupt leaf: root=9 block=4400742400
> slot=0, invalid nritems, have 0 should not be 0 for non-root leaf
> Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel:
> BTRFS info (device xvdf): leaf 4400742400 total ptrs 0 free space
> 16283
> Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel:
> assertion failed: 0, file: fs/btrfs/disk-io.c, line: 540
> Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel:
> [ cut here ]
> Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel:
> kernel BUG at fs/btrfs/ctree.h:3457!
> Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel:
> invalid opcode:  [#1] SMP
> Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel:
> Modules linked in: cirrus drm_kms_helper ttm drm intel_rapl sb_edac
> ppdev crct10dif_pclmul crc32_pclmul crc32c_intel xen_netfront
> ghash_clmulni_intel intel_rapl_perf parport_pc parport i2c_piix4
> xen_blkfront serio_raw ata_generic pata_acpi
> Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel:
> CPU: 1 PID: 13413 Comm: fsstress Not tainted 4.14.0-rc7 #1
> Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel:
> Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/12/2016
> Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel:
> task: 9d2331a48000 task.stack: bc0f43fd4000
> Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel:
> RIP: 0010:assfail.constprop.49+0x1c/0x22
> Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel:
> RSP: 0018:bc0f43fd79c0 EFLAGS: 00010282
> Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel:
>

kernel BUG at fs/btrfs/ctree.h:3457

[Lakshmipathi.G]

pi
Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel:
CPU: 1 PID: 13413 Comm: fsstress Tainted: G  D 4.14.0-rc7
#1
Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel:
Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/12/2016
Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel:
task: 9d2331a48000 task.stack: bc0f43fd4000
Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel:
RIP: 0010:do_exit+0x51/0xb40
Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel:
RSP: 0018:bc0f43fd7ed8 EFLAGS: 00010206
Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel:
RAX: bc0f43fd7d40 RBX: 9d2331a48000 RCX: bc0f43fd7d50



Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

btrfs-progs:fuzz-tests/003-multi-check-unmounted broken?

[Lakshmipathi.G]

Hi.

Seems like fuzz-test/003 is broken for quite sometime now (it works on
v4.7.3 but not after that).
Can anyone else able to run this test successfully? If this test is
not required anymore, shall
we remove it from fuzz-tests?

TEST=003\* tests/fuzz-tests.sh
[TEST/fuzz]   003-multi-check-unmounted
btrfs-progs/tests/common: line 170:  1439 Aborted
(core dumped) $INSTRUMENT "$@" >> "$RESULTS" 2>&1
mayfail: returned code 134 (SIGABRT), not ignored
test failed for case 003-multi-check-unmounted

Tried applying these two patches
https://patchwork.kernel.org/patch/9939881/ still script doesn't
succeed.

thanks.


Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: xfstests:btrfs/025, 026 fails

[Lakshmipathi.G]

Okay I narrowed it down to

/bin/mount -t btrfs -o context=system_u:object_r:root_t:s0 -o compress
/dev/xvdc /mnt/scratch

command from xfstests/btrfs/026.

Cheers.

On 11/4/17, Lakshmipathi.G <lakshmipath...@gmail.com> wrote:
> Hi .
>
> While running xfstests with 4.14.0-rc7 the test hangs on btrfs/026
> (after running 025) and produces below trace:
>
> [ 2134.147940] run fstests btrfs/026 at 2017-11-04 09:46:19
> [ 2134.388024] BTRFS: device fsid d3de0b02-8b7e-422a-8165-c074f834e909
> devid 1 transid 5 /dev/xvdc
> [ 2134.405001] general protection fault:  [#1] SMP
> [ 2134.408660] Modules linked in: cirrus drm_kms_helper ttm drm ppdev
> intel_rapl sb_edac parport_pc crct10dif_pclmul parport crc32_pclmul
> crc32c_intel ghash_clmulni_intel xen_netfront intel_rapl_perf
> i2c_piix4 xen_blkfront serio_raw ata_generic pata_acpi
> [ 2134.420112] CPU: 1 PID: 8478 Comm: mount Not tainted 4.14.0-rc7 #1
> [ 2134.423711] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
> [ 2134.427557] task: 8ec43c672600 task.stack: a90fc886c000
> [ 2134.431351] RIP: 0010:btrfs_compress_str2level+0x18/0x4e
> [ 2134.434764] RSP: 0018:a90fc886fa10 EFLAGS: 00010202
> [ 2134.438128] RAX: 0001 RBX: 8ec434c8 RCX:
> 0004
> [ 2134.442160] RDX: 5485389ff3298000 RSI: 5485389ff3298000 RDI:
> 94cdb929
> [ 2134.446266] RBP: a90fc886fac0 R08:  R09:
> 5485389ff3298000
> [ 2134.450252] R10: 002c R11: 94a917c0 R12:
> 94d1ec8a
> [ 2134.454875] R13: 8ec434d1c720 R14:  R15:
> 
> [ 2134.459141] FS:  7f9e870c8480() GS:8ec43fe4()
> knlGS:
> [ 2134.464051] CS:  0010 DS:  ES:  CR0: 80050033
> [ 2134.467519] CR2: 7f5ae9499710 CR3: 0004343e4002 CR4:
> 001606e0
> [ 2134.471549] DR0:  DR1:  DR2:
> 
> [ 2134.475601] DR3:  DR6: fffe0ff0 DR7:
> 0400
> [ 2134.480049] Call Trace:
> [ 2134.482285]  ? btrfs_parse_options+0x9fd/0x1020
> [ 2134.485502]  open_ctree+0x1244/0x24df
> [ 2134.488245]  ? open_ctree+0x1244/0x24df
> [ 2134.490954]  btrfs_mount+0xe16/0xefc
> [ 2134.493481]  ? find_next_bit+0xb/0x10
> [ 2134.496095]  ? cpumask_next+0x1b/0x20
> [ 2134.498643]  mount_fs+0x32/0x150
> [ 2134.501001]  ? __alloc_percpu+0x15/0x20
> [ 2134.504016]  vfs_kern_mount.part.22+0x5d/0x120
> [ 2134.507094]  vfs_kern_mount+0x13/0x20
> [ 2134.509737]  btrfs_mount+0x1a8/0xefc
> [ 2134.512338]  ? find_next_bit+0xb/0x10
> [ 2134.514963]  mount_fs+0x32/0x150
> [ 2134.517289]  ? __alloc_percpu+0x15/0x20
> [ 2134.519917]  vfs_kern_mount.part.22+0x5d/0x120
> [ 2134.522754]  do_mount+0x5c7/0xca0
> [ 2134.525101]  ? kmem_cache_alloc_trace+0x165/0x1d0
> [ 2134.528554]  ? copy_mount_options+0x2c/0x220
> [ 2134.531459]  SyS_mount+0x98/0xe0
> [ 2134.533880]  entry_SYSCALL_64_fastpath+0x1a/0xa5
> [ 2134.536942] RIP: 0033:0x7f9e8611c28a
> [ 2134.539496] RSP: 002b:7fff6cddbf68 EFLAGS: 0246 ORIG_RAX:
> 00a5
> [ 2134.543632] RAX: ffda RBX: 7f9e86c9c7ef RCX:
> 7f9e8611c28a
> [ 2134.547646] RDX: 5562a4af82f0 RSI: 5562a4af83b0 RDI:
> 5562a4af8390
> [ 2134.552254] RBP: 7f9e86eae184 R08: 5562a4af8310 R09:
> 7fff6cddae68
> [ 2134.556487] R10: c0ed R11: 0246 R12:
> 5562a4af8140
> [ 2134.560647] R13: 7fff6cddc288 R14: 5562a359f4a0 R15:
> 
> [ 2134.564622] Code: ad 31 db eb a9 0f 1f 40 00 66 2e 0f 1f 84 00 00
> 00 00 00 0f 1f 44 00 00 55 48 89 fa b9 04 00 00 00 48 89 d6 48 c7 c7
> 29 b9 cd 94  a6 48 89 e5 40 0f 97 c6 0f 92 c1 31 c0 40 38 ce 75 06
> 80 7a
> [ 2134.574756] RIP: btrfs_compress_str2level+0x18/0x4e RSP:
> a90fc886fa10
> [ 2134.578890] ---[ end trace 47a5851d0ed3c8fd ]---
>
>
> https://bugzilla.kernel.org/show_bug.cgi?id=197771
>
> 
> Cheers,
> Lakshmipathi.G
> http://www.giis.co.in http://www.webminal.org
>


-- 

Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

xfstests:btrfs/025, 026 fails

[Lakshmipathi.G]

Hi .

While running xfstests with 4.14.0-rc7 the test hangs on btrfs/026
(after running 025) and produces below trace:

[ 2134.147940] run fstests btrfs/026 at 2017-11-04 09:46:19
[ 2134.388024] BTRFS: device fsid d3de0b02-8b7e-422a-8165-c074f834e909
devid 1 transid 5 /dev/xvdc
[ 2134.405001] general protection fault:  [#1] SMP
[ 2134.408660] Modules linked in: cirrus drm_kms_helper ttm drm ppdev
intel_rapl sb_edac parport_pc crct10dif_pclmul parport crc32_pclmul
crc32c_intel ghash_clmulni_intel xen_netfront intel_rapl_perf
i2c_piix4 xen_blkfront serio_raw ata_generic pata_acpi
[ 2134.420112] CPU: 1 PID: 8478 Comm: mount Not tainted 4.14.0-rc7 #1
[ 2134.423711] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
[ 2134.427557] task: 8ec43c672600 task.stack: a90fc886c000
[ 2134.431351] RIP: 0010:btrfs_compress_str2level+0x18/0x4e
[ 2134.434764] RSP: 0018:a90fc886fa10 EFLAGS: 00010202
[ 2134.438128] RAX: 0001 RBX: 8ec434c8 RCX: 0004
[ 2134.442160] RDX: 5485389ff3298000 RSI: 5485389ff3298000 RDI: 94cdb929
[ 2134.446266] RBP: a90fc886fac0 R08:  R09: 5485389ff3298000
[ 2134.450252] R10: 002c R11: 94a917c0 R12: 94d1ec8a
[ 2134.454875] R13: 8ec434d1c720 R14:  R15: 
[ 2134.459141] FS:  7f9e870c8480() GS:8ec43fe4()
knlGS:
[ 2134.464051] CS:  0010 DS:  ES:  CR0: 80050033
[ 2134.467519] CR2: 7f5ae9499710 CR3: 0004343e4002 CR4: 001606e0
[ 2134.471549] DR0:  DR1:  DR2: 
[ 2134.475601] DR3:  DR6: fffe0ff0 DR7: 0400
[ 2134.480049] Call Trace:
[ 2134.482285]  ? btrfs_parse_options+0x9fd/0x1020
[ 2134.485502]  open_ctree+0x1244/0x24df
[ 2134.488245]  ? open_ctree+0x1244/0x24df
[ 2134.490954]  btrfs_mount+0xe16/0xefc
[ 2134.493481]  ? find_next_bit+0xb/0x10
[ 2134.496095]  ? cpumask_next+0x1b/0x20
[ 2134.498643]  mount_fs+0x32/0x150
[ 2134.501001]  ? __alloc_percpu+0x15/0x20
[ 2134.504016]  vfs_kern_mount.part.22+0x5d/0x120
[ 2134.507094]  vfs_kern_mount+0x13/0x20
[ 2134.509737]  btrfs_mount+0x1a8/0xefc
[ 2134.512338]  ? find_next_bit+0xb/0x10
[ 2134.514963]  mount_fs+0x32/0x150
[ 2134.517289]  ? __alloc_percpu+0x15/0x20
[ 2134.519917]  vfs_kern_mount.part.22+0x5d/0x120
[ 2134.522754]  do_mount+0x5c7/0xca0
[ 2134.525101]  ? kmem_cache_alloc_trace+0x165/0x1d0
[ 2134.528554]  ? copy_mount_options+0x2c/0x220
[ 2134.531459]  SyS_mount+0x98/0xe0
[ 2134.533880]  entry_SYSCALL_64_fastpath+0x1a/0xa5
[ 2134.536942] RIP: 0033:0x7f9e8611c28a
[ 2134.539496] RSP: 002b:7fff6cddbf68 EFLAGS: 0246 ORIG_RAX:
00a5
[ 2134.543632] RAX: ffda RBX: 7f9e86c9c7ef RCX: 7f9e8611c28a
[ 2134.547646] RDX: 5562a4af82f0 RSI: 5562a4af83b0 RDI: 5562a4af8390
[ 2134.552254] RBP: 7f9e86eae184 R08: 5562a4af8310 R09: 7fff6cddae68
[ 2134.556487] R10: c0ed R11: 0246 R12: 5562a4af8140
[ 2134.560647] R13: 7fff6cddc288 R14: 5562a359f4a0 R15: 
[ 2134.564622] Code: ad 31 db eb a9 0f 1f 40 00 66 2e 0f 1f 84 00 00
00 00 00 0f 1f 44 00 00 55 48 89 fa b9 04 00 00 00 48 89 d6 48 c7 c7
29 b9 cd 94  a6 48 89 e5 40 0f 97 c6 0f 92 c1 31 c0 40 38 ce 75 06
80 7a
[ 2134.574756] RIP: btrfs_compress_str2level+0x18/0x4e RSP: a90fc886fa10
[ 2134.578890] ---[ end trace 47a5851d0ed3c8fd ]---


https://bugzilla.kernel.org/show_bug.cgi?id=197771


Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kernel BUG at fs/btrfs/ctree.h:3457!

[Lakshmipathi.G]

Yes, the patch works.  Enabled both CONFIG_BTRFS_FS_CHECK_INTEGRITY
and CONFIG_BTRFS_FS_RUN_SANITY_TESTS.
And applied above patch. This method also resolved the issue. thanks.

Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kernel BUG at fs/btrfs/ctree.h:3457!

[Lakshmipathi.G]

> BTW, it would be better to see if this patch also solves your problem.
>
> https://patchwork.kernel.org/patch/10038011/

Okay sure, let me apply this patch and check the results.



Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kernel BUG at fs/btrfs/ctree.h:3457!

[Lakshmipathi.G]

After disabling FS_CHECK_INTEGRITY and FS_RUN_SANITY_TESTS in the
config. Now the issue resolved. thanks.

cast: https://asciinema.org/a/Dy6eHhhWPEIxotVbVUBWrhFeF

Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org


On Thu, Nov 2, 2017 at 12:55 PM, Lakshmipathi.G
<lakshmipath...@gmail.com> wrote:
> Okay thanks, I'll disable above mentioned entry in kernel config and
> run the tests.
> Will get back with  updated results.
>
>
> 
> Cheers,
> Lakshmipathi.G
> http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kernel BUG at fs/btrfs/ctree.h:3457!

[Lakshmipathi.G]

Okay thanks, I'll disable above mentioned entry in kernel config and
run the tests.
Will get back with  updated results.



Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kernel BUG at fs/btrfs/ctree.h:3457!

[Lakshmipathi.G]

>
> I'll try to reproduce it with FS_CHECK_INTEGRITY enabled.
>

Okay thanks for the details.  Here's the kernel config file which hits
this issue:
https://github.com/Lakshmipathi/btrfsqa/blob/master/setup/config/kernel.config#L3906

thanks!

Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

kernel BUG at fs/btrfs/ctree.h:3457!

[Lakshmipathi.G]

Hi.

I'm constantly hitting this bug while running btrfs-progs:fsck-test:012 .
Screencast: https://asciinema.org/a/wQxZjCeVvX2kVqKjVBGyR0klq

Logged more details: https://bugzilla.kernel.org/show_bug.cgi?id=197587
Anyone else got this issue or something wrong with my test environment?


Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Building a BTRFS test machine

[Lakshmipathi.G]

Last few days, made small progress on this. Though Python/bash script
is working, its not yet perfect.
Gladly welcome any feedback and/or scripts :-)

https://github.com/Lakshmipathi/btrfsqa


Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"

[Lakshmipathi.G]

> I will gladly repeat this process, but I am very concerned why this
> corruption happened in the first place.
>
Can you share these steps as simple bash script? Currently I'm running
few tests, I can check your script and share the results.


> The only thing I could think of is that the btrfs version that I used to mkfs
> was not up to date. Is there a way to determine which version was used to
> create the filesystem?
>

If I'm not wrong, i don't think we can get mkfs version from existing file
system layout. what was your kernel version?


Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 1/1] btrfs-progs: tests/common: Display warning only after searching for btrfs kernel module

[Lakshmipathi.G]

Signed-off-by: Lakshmipathi.G <lakshmipath...@giis.co.in>
---
 tests/common | 8 ++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/tests/common b/tests/common
index eb525a4..773b1e5 100644
--- a/tests/common
+++ b/tests/common
@@ -428,8 +428,12 @@ run_check_umount_test_dev()
 check_kernel_support()
 {
if ! grep -iq 'btrfs' /proc/filesystems; then
-   echo "WARNING: btrfs filesystem not listed in 
/proc/filesystems, some tests might fail"
-   return 1
+   run_check $SUDO_HELPER modprobe btrfs
+   if ! grep -iq 'btrfs' /proc/filesystems; then
+   echo "WARNING: btrfs filesystem not listed in \
+/proc/filesystems, some tests might fail"
+   return 1
+   fi
fi
return 0
 }
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"

[Lakshmipathi.G]

1. I guess you should be able to dump tree details via
'btrfs-debug-tree' and then map the extent/data (from scrub
offline output) and track it back to inode-object. Store output of
both btrfs-debug-tree and scrub-offline in different files and then
play around with grep to extract required data.

2. I think normal scrub(online) fails to detect these csum errors for
some reason,I don't have much idea about online scrub.

3. I assume, the issue is not related to hardware. Since the offline
scrub able to get available (corrupted) csum.

Yes, offline scrub will try to fix corruption whenever it is possible.
And also you have quite lot of "all mirror(s) corrupted, can't be repaired",
which will be hard to recovery.

I suggest running offline scrub on all devices. Then online scrub
and finally track those corrupted files with the help of extent info.
----
Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org


On Wed, Oct 25, 2017 at 7:22 AM, Zak Kohler <y...@y2kbugger.com> wrote:
> I apologize for the bad line wrapping on the last post...will be
> setting up mutt soon.
>
> This is the final result for the offline scrub:
> Doing offline scrub [O] [681/683]
> Scrub result:
> Tree bytes scrubbed: 5234491392
> Tree extents scrubbed: 638975
> Data bytes scrubbed: 4353723572224
> Data extents scrubbed: 374300
> Data bytes without csum: 533200896
> Read error: 0
> Verify error: 0
> Csum error: 175
>
> The offline scrub apparently corrected some metadata extents while
> scanning /dev/sdn
>
>
> I also ran the online scrub directly on the /dev/sdn, "0 errors":
>
> $ btrfs scrub status /dev/sdn
> scrub status for 88406942-e3e1-42c6-ad71-e23bb315caa7
> scrub started at Tue Oct 24 06:55:12 2017 and finished after 01:52:44
> total bytes scrubbed: 677.35GiB with 0 errors
>
> The csum mismatches are still missed by the online scrub when choosing
> a single . Now I am doing offline scrub on the other devices
> to see if they are clean.
>
> $ lsblk -o +SERIAL
> NAME  MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT SERIAL
> sdh 8:112  0  1.8T  0 disk WD-WMAZA370
> sdi 8:128  0  1.8T  0 disk WD-WCAZA569
> sdn 8:208  0  1.8T  0 disk WD-WCAZA580
>
> $ btrfs scrub start --offline --progress /dev/sdh
> ERROR: data at bytenr 5365456896 ...
> ERROR: extent 5341712384 ...
> ...
>
> One thing to note is that a /dev/sdh is also having csum errors
> detected despite it having never been mentioned dmesg. I understand
> that you may have the ability to run two offline checks at once but
> the error message I get is slightly misleading.
>
> $ btrfs scrub start --offline --progress /dev/sdi
> ERROR: cannot open device '/dev/sdn': Device or resource busy
> ERROR: cannot open file system
>
> I get an error about sdn when the device I am trying to scan is sdi,
> and the device that is currently being scanned is sdh.
>
> On Tue, Oct 24, 2017 at 2:00 AM, Zak Kohler <y...@y2kbugger.com> wrote:
>> Yes, it is finding much more than just one error.
>>
>> From dmesg
>> [89520.441354] BTRFS warning (device sdn): csum failed ino 4708 off
>> 27529216 csum 2615801759 expected csum 874979996
>>
>> $ sudo btrfs scrub start --offline --progress /dev/sdn
>> ERROR: data at bytenr 68431499264 mirror 1 csum mismatch, have
>> 0x5aa0d40f expect 0xd4a15873
>> ERROR: extent 68431474688 len 14467072 CORRUPTED, all mirror(s)
>> corrupted, can't be repaired
>> ERROR: data at bytenr 83646357504 mirror 1 csum mismatch, have
>> 0xfc0baabe expect 0x7f9cb681
>> ERROR: extent 83519741952 len 134217728 CORRUPTED, all mirror(s)
>> corrupted, can't be repaired
>> ERROR: data at bytenr 121936633856 mirror 1 csum mismatch, have
>> 0x507016a5 expect 0x50609afe
>> ERROR: extent 121858334720 len 134217728 CORRUPTED, all mirror(s)
>> corrupted, can't be repaired
>> ERROR: data at bytenr 144872591360 mirror 1 csum mismatch, have
>> 0x33964d73 expect 0xf9937032
>> ERROR: extent 144822386688 len 61231104 CORRUPTED, all mirror(s)
>> corrupted, can't be repaired
>> ERROR: data at bytenr 167961075712 mirror 1 csum mismatch, have
>> 0xf43bd0e3 expect 0x5be589bb
>> ERROR: extent 167950999552 len 27537408 CORRUPTED, all mirror(s)
>> corrupted, can't be repaired
>> ERROR: data at bytenr 175643619328 mirror 1 csum mismatch, have
>> 0x1e168ca1 expect 0xd413b1e0
>> ERROR: data at bytenr 175643754496 mirror 1 csum mismatch, have
>> 0x6cfdc8ae expect 0xa6f8f5ef
>> ERROR: extent 175640539136 len 6381568 CORRUPTED, all mirror(s)
>> corrupted, can't be repaired
>> ERROR: data at bytenr 183316750336 mirror 1 csum m

Re: btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"

[Lakshmipathi.G]

> Does anyone know why scrub did not catch these errors that show up in dmesg?

Can you try offline scrub from this repo
https://github.com/gujx2017/btrfs-progs/tree/offline_scrub and see
whether it
detects the issue?  "btrfs scrub start --offline "



Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3] btrfs-progs: RAID5: Inject data stripe corruption and verify scrub fixes it and retains correct parity.

[Lakshmipathi.G]

Any review comments on this RAID5 test script? I understand it depends
on 'dump-tree' output, but I assume, right now, we have no other
choice other than this method.

I'll modify this patch to reflect coding practices as mentioned in
tests/README.md, if current method is okay.

Cheers.
Lakshmipathi.G

On Mon, Feb 20, 2017 at 11:57 PM, Lakshmipathi.G
<lakshmipath...@giis.co.in> wrote:
> Test script to create file with specific data-stripe layout.Computes stripe 
> locations.
> Inject corruption into data-stripe and verify scrubbing process fixes 
> corrupted block.
> It also validates the corresponding parity stripe.
>
> Signed-off-by: Lakshmipathi.G <lakshmipath...@giis.co.in>
> ---
>  .../test.sh| 402 
> +
>  1 file changed, 402 insertions(+)
>  create mode 100755 
> tests/misc-tests/020-raid5-datastripe-corruption-parity-check/test.sh
>
> diff --git 
> a/tests/misc-tests/020-raid5-datastripe-corruption-parity-check/test.sh 
> b/tests/misc-tests/020-raid5-datastripe-corruption-parity-check/test.sh
> new file mode 100755
> index 000..0129a75
> --- /dev/null
> +++ b/tests/misc-tests/020-raid5-datastripe-corruption-parity-check/test.sh
> @@ -0,0 +1,402 @@
> +#!/bin/bash
> +#
> +# Raid5: Inject data stripe corruption and fix them using scrub.
> +#
> +# Script will perform the following:
> +# 1) Create Raid5 using 3 loopback devices.
> +# 2) Ensure file layout is created in a predictable manner.
> +#Each data stripe(64KB) should uniquely start with 'DN',
> +#where N represents the data stripe number.(ex:D0,D1 etc)
> +# 3) Once file is created with specific layout, check whether file
> +#has single extent. At the moment, script wont handle multi-extent
> +#files.
> +# 4) If file has single extent with the help of 'dump-tree' compute data and
> +#parity stripe details like devicename, position and actual on-disk data.
> +# 5) $STRIPEINFO_COMPLETE file will have all necessary data at this stage.
> +# 6) Inject corruption into given data-stripe by zero'ing out its first 4 
> bytes.
> +# 7) After injecting corruption, running online-scrub is expected to fix
> +#the corrupted data stripe with the help of parity block and
> +#corresponding data stripe.
> +# 8) If scrub successful, verify the data stripe has original un-corrupted 
> value.
> +# 9) If scrub successful, verify parity stripe is valid, otherwise its a 
> parity bug.
> +# 10) If no issues found, cleanup files and devices. Repeat the process for
> +#different file size and data-stripe.
> +#
> +#  Note: This script corrupts only data-stripe blocks.
> +#  Known Limitations (will be addressed later):
> +# - Script expects even number of data-stripes in file.
> +# - Script expects the file to have single extent.
> +
> +source $TOP/tests/common
> +
> +check_prereq btrfs
> +check_prereq mkfs.btrfs
> +check_prereq btrfs-debugfs
> +
> +setup_root_helper
> +prepare_test_dev 1024M
> +
> +ndevs=3
> +declare -a devs
> +declare -a parity_offsets
> +stripe_entry=""
> +device_name=""
> +stripe_offset=""
> +stripe_content=""
> +
> +#Complete stripe layout
> +STRIPEINFO_COMPLETE=$(mktemp --tmpdir 
> btrfs-progs-raid5-stripe-complete.infoXX)
> +#dump-tree output file
> +DUMPTREE_OUTPUT=$(mktemp --tmpdir btrfs-progs-raid5-tree-dump.infoXX)
> +#tmp files
> +STRIPEINFO_PARTIAL=$(mktemp --tmpdir 
> btrfs-progs-raid5-stripe-partial.infoXX)
> +STRIPE_TMP=$(mktemp --tmpdir btrfs-progs-raid5-stripetmp.infoXX)
> +MULTI_EXTENT_CHECK=$(mktemp --tmpdir 
> btrfs-progs-raid5-extent-check.infoXX)
> +EXTENT_WITH_SIZE=$(mktemp --tmpdir btrfs-progs-raid5-extent-size.infoXX)
> +PARITY_LOC1=$(mktemp --tmpdir btrfs-progs-raid5-parity-loc1.infoXX)
> +PARITY_LOCATION=$(mktemp --tmpdir 
> btrfs-progs-raid5-parity-locations.infoXX)
> +
> +
> +prepare_devices()
> +{
> +   for i in `seq $ndevs`; do
> +   touch img$i
> +   chmod a+rw img$i
> +   truncate -s0 img$i
> +   truncate -s512M img$i
> +   devs[$i]=`run_check_stdout $SUDO_HELPER losetup --find --show 
> img$i`
> +   done
> +   truncate -s0 $STRIPE_TMP
> +   truncate -s0 $STRIPEINFO_PARTIAL
> +   truncate -s0 $STRIPEINFO_COMPLETE
> +}
> +
> +cleanup_devices()
> +{
> +   for dev in ${devs[@]}; do
> +   run_check $SUDO_HELPER losetup -d $dev
> +   done
> +   for i in `seq $ndevs`; do
> +   truncate -s0 img$i
> +   done
> +   run_check $SUDO_HELPER losetup --all
> +}
> +
> +test_

Re: btrfs-progs task tracking on github

[Lakshmipathi.G]

Existing project list is good.

Reg: "Project tests:  Build and test coverage: pre-built images on
docker hub"  If we don't have pre-built images on docker hub, I'll
pick up this task and start working on it. thanks.

----
Cheers,
Lakshmipathi.G



On Sat, Sep 9, 2017 at 5:43 AM, Qu Wenruo <quwenruo.bt...@gmx.com> wrote:
>
>
> On 2017年09月09日 01:36, David Sterba wrote:
>>
>> Hi,
>>
>> in order to make more visible which tasks for btrfs-progs are in
>> progress or desired, I've started to populate the github Projects [1]
>> some time ago. I haven't fleshed out the workflow so this hasn't been
>> announced yet.
>>
>> The aim is to help contributors to join progs development and start from
>> one point with the tasks to choose from. Current practice is to either
>> ask on IRC or find something on the wiki, hoping that the projects are
>> still up to date. Not all of them are, wiki is not the best tool for
>> task tracking.
>>
>> Using github issues and projects is diversion from the current model
>> similar to the kernel part of btrfs, ie. mailinglist for everything.
>> This does suite all the needs but I don't want to completely switch to
>> github-only workflow.  Leaving both options means that bug/patchset
>> discussions will not be visible to both audiences directly, but web
>> archives or publicly accessible github pages should be enough for
>> cross-references.
>>
>> The exact workflow from a task to a merge is not yet defined, I hope
>> we'll be able to find out and document it once the first guinea pig
>> volunteers. The Projects use the kanban-style, I made a bit familiar
>> with that but have no prior experience with it.
>>
>> There are categories for tasks and some brief descriptions. The rough
>> idea is to:
>>
>> 1. move a task from TODO to WIP
>> 2. create issue from the task, for discussions etc
>> 3. work on patches, review cycle
>> 4. when finished, move to next stage on the project page, and ask for
>> merge
>> 5. task is done, code merged and released
>>
>> Who does what and how exactly will have to be found on the way and
>> documented eventually.
>>
>> [1] https://github.com/kdave/btrfs-progs/projects
>
>
> This looks quite good.
>
> BTW, is that only repo owner be able to modify the status?
> In fact I found that some projects like offline scrub doesn't show up in
> either WIP or TODO list.
>
> Thanks,
> Qu
>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Building a BTRFS test machine

[Lakshmipathi.G]

Definitely it will be useful.  For quite sometime, I was thinking about:
1. Creating test cases for each BTRFS features. (ex:
https://btrfs.wiki.kernel.org/index.php/Scrub_corruption_cases)
2. Automate these  feature specific test scripts using bash/python
while maintaining them in public git repos.
3. Launching BTRFS test machines in cloud (aws spot instance are very cheap) .
4. Run those feature specific tests and publish the results into wiki
or/and bugs into bugzilla.
5. Destroy the test machines.

Such setup will require quite good effort. As Qu mentioned, the
challenging part will be integrating existing different tests and make
it automatic.

Another thought on BTRFS testing:
At this moment, when someone sends a patch - i assume, they
automatically pushed into patchwork.kernel.org (based on some back-end
configuration I guess).

On Btrfs mailing-list,we have lot of bug reports so using something
like [1] we can write a
program which parses mailing-list for specific keyword like
"BUGREPORT:" similar to "PATCH:"
and create a new
bug or update the details on bugzilla.kernel.org? This will ensure we
can keep track of bugs on
bugzilla.

I think updating bugs based on mailing-list responses will be little
tricky than creating new bugs.
something like that can be useful as well.

I can help a bit on such testbed setup  :-) Thanks!


Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org


On Mon, Aug 14, 2017 at 6:31 AM, Cerem Cem ASLAN <cerem...@ceremcem.net> wrote:
> Would that be useful to build a BTRFS test machine, which will perform
> both software tests (btrfs send | btrfs receive, read/write random
> data etc.) and hardware tests, such as abrupt power off test, abruptly
> removing a raid-X disk physically, etc.
>
> If it would be useful, what tests should it cover?
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 4/5] btrfs-progs: tests: fix typo in convert-tests/008-readonly-image

[Lakshmipathi.G]

>  if [ $? -ne 1 ]; then
> echo "after convert ext2_save/image is not read-only"
Though this mayn't affect test-script result, please change the typo
in this location also. thanks.

Cheers.
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs-convert --no-datasum & btrfs check: check_owner_ref: BUG_ON `rec->is_root` triggered, value 1

[Lakshmipathi.G]

Looked like I missed few patch.  After checking with latest branch. Its working.


Cheers,
Lakshmipathi.G



On Sun, Jul 2, 2017 at 7:30 PM, Lakshmipathi.G <lakshmipath...@gmail.com> wrote:
> Hi.
> While using btrfs-convert with '--no-datasum' option & running btrfs
> check crashes:
>
> steps:
> mkfs.ext4 tests/t1
> mount tests/t1 /data
> cp -r *.c /data
> umount /data
> ./btrfs-convert --no-datasum tests/t1
> ./btrfs check tests/t1
> 
> cmds-check.c:5488: check_owner_ref: BUG_ON `rec->is_root` triggered, value 1
>
> Is this a bug or something wrong with my environment?
>
> More info: https://bugzilla.kernel.org/show_bug.cgi?id=196217
> 
> Cheers,
> Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

btrfs-convert --no-datasum & btrfs check: check_owner_ref: BUG_ON `rec->is_root` triggered, value 1

[Lakshmipathi.G]

Hi.
While using btrfs-convert with '--no-datasum' option & running btrfs
check crashes:

steps:
mkfs.ext4 tests/t1
mount tests/t1 /data
cp -r *.c /data
umount /data
./btrfs-convert --no-datasum tests/t1
./btrfs check tests/t1

cmds-check.c:5488: check_owner_ref: BUG_ON `rec->is_root` triggered, value 1

Is this a bug or something wrong with my environment?

More info: https://bugzilla.kernel.org/show_bug.cgi?id=196217

Cheers,
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Btrfs-progs: convert: do not clear nodatasum flag in inode item

[Lakshmipathi.G]

Could you please add also a test-script for this interesting issue
under tests/convert-tests
(If its not already there)?. thanks.


Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org


On Thu, Jun 29, 2017 at 2:59 AM, Liu Bo <bo.li@oracle.com> wrote:
> With the current btrfs-convert, if we convert a ext4 without data checksum,
> it'd not set nodatasum flag in inode item, nor create csum item, reading
> file ends up with checksum errors.
>
> Signed-off-by: Liu Bo <bo.li@oracle.com>
> ---
>  convert/source-ext2.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/convert/source-ext2.c b/convert/source-ext2.c
> index 1b0576b..64b95c5 100644
> --- a/convert/source-ext2.c
> +++ b/convert/source-ext2.c
> @@ -742,7 +742,7 @@ static int ext2_check_state(struct btrfs_convert_context 
> *cctx)
>  static void ext2_convert_inode_flags(struct btrfs_inode_item *dst,
>  struct ext2_inode *src)
>  {
> -   u64 flags = 0;
> +   u64 flags = btrfs_stack_inode_flags(dst);
>
> COPY_ONE_EXT2_FLAG(flags, src, APPEND);
> COPY_ONE_EXT2_FLAG(flags, src, SYNC);
> --
> 2.5.0
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v4] btrfs-progs: btrfs-convert: Add larger device support

[Lakshmipathi.G]

> > -   u32 free_inodes_count;
> > +   u64 first_data_block;
> > +   u64 block_count;
> > +   u64 inodes_count;
> > +   u64 free_inodes_count;
> 
> I've split this change from the patch as it does not logically belong to
> the same patch, altough the change is simple.

Okay sure, thanks. 

Cheers.
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Stability status of btrfs-convert...

[Lakshmipathi.G]

>Is this tool supported, at least? Can I use this tool as a part of
>software upgrade to change the data filesystem to btrfs?

Yes its supported. If your existing FS is too large (>22TB) we have a bug for it
https://bugzilla.kernel.org/show_bug.cgi?id=194795

AFAIK, there are quite number of people converted their EXT4 to BTRFS .
IMO, btrfs-convert is fairly stable, especially if you have only GB's of data

Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2] btrfs-progs: Fix 'btrfs device stats --check' cli option

[Lakshmipathi.G]

Bug 194961 - btrfs device stats --check  does not work
https://bugzilla.kernel.org/show_bug.cgi?id=194961

Reported-by: Tomas Thiemel<thie...@centrum.cz>
Signed-off-by: Lakshmipathi.G <lakshmipath...@giis.co.in>
---
 cmds-device.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/cmds-device.c b/cmds-device.c
index 5e016a7..4337eb2 100644
--- a/cmds-device.c
+++ b/cmds-device.c
@@ -400,6 +400,7 @@ static int cmd_device_stats(int argc, char **argv)
while (1) {
int c;
static const struct option long_options[] = {
+   {"check", no_argument, NULL, 'c'},
{"reset", no_argument, NULL, 'z'},
{NULL, 0, NULL, 0}
};
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] Fix 'btrfs device stats --check' cli option

[Lakshmipathi.G]

Reported by Tomas Thiemel
Bug 194961 - btrfs device stats --check  does not work

---
 cmds-device.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/cmds-device.c b/cmds-device.c
index 5e016a7..4337eb2 100644
--- a/cmds-device.c
+++ b/cmds-device.c
@@ -400,6 +400,7 @@ static int cmd_device_stats(int argc, char **argv)
while (1) {
int c;
static const struct option long_options[] = {
+   {"check", no_argument, NULL, 'c'},
{"reset", no_argument, NULL, 'z'},
{NULL, 0, NULL, 0}
};
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v4 00/20] Btrfs-progs offline scrub

[Lakshmipathi.G]

> For any one who wants to try it, it can be get from my repo:
> https://github.com/adam900710/btrfs-progs/tree/offline_scrub

While running single corruption script receive  message like

"REPARIED: corrupted data with good P/Q, repaired" along with
message from script which says "Parity stripe check passed."

complete output here:
https://github.com/Lakshmipathi/btrfs_offline_scrub/blob/master/logs-june18/single_corruption_misc-tests-results.txt


multiple-corruption:  While corrupting continuous blocks received:

>Filename=file256k.txt Total Stripes=4 Data Stripe to be corrupted=2,3
..
..//debugfs-tree output ends
>ERROR: full stripe 145358848 CORRUPTED: too many read error or corrupted 
>devices
>ERROR: full stripe 145358848: tolerance: 1, missing: 0, read error: 0, csum 
>error: 2


While corrupting non-continuous blocks:
>Filename=file512k.txt Total Stripes=8 Data Stripe to be corrupted=1,3,5
..
..//debugfs-tree output ends
>full stripe 145227776 REPARIED: corrupted data with good P/Q, repaired
>full stripe 145358848 REPARIED: corrupted data with good P/Q, repaired
>full stripe 145489920 REPARIED: corrupted data with good P/Q, repaired

output for other combinations like :
>Filename=file768k.txt Total Stripes=12   Data Stripe to be 
>corrupted=10,8,6,4
>Filename=file1m.txt Total Stripes=16   Data Stripe to be 
>corrupted=14,12,10,8,7,6
>Filename=file2m.txt Total Stripes=32   Data Stripe to be 
>corrupted=23,22,21,20,19,18,16
>Filename=file4m.txt Total Stripes=64   Data Stripe to be 
>corrupted=34,33,32,31,30,20,18,15,10,8,5
>Filename=file8m.txt Total Stripes=128   Data Stripe to be 
>corrupted=100,90,80,70,60,50,40,30,20,10

can be found here:
https://github.com/Lakshmipathi/btrfs_offline_scrub/blob/master/logs-june18/multiple_corruptions_misc-tests-results.txt

Outputs are looking fine, but more testing required with different
file-types :-) thanks.


Cheers,
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs-convert: whats minimum free space requirement?

[Lakshmipathi.G]

On Fri, Jun 09, 2017 at 03:38:31PM +0800, Qu Wenruo wrote:
> >I was thinking about adding a free-space check (based on total blocks/free 
> >block
> >stat from source superblock) before btrfs-convert begins, looks like its more
> >complex than i thought.
> 
> BTW, it's already done, since it's complex, btrfs-convert will try to
> calculate it early on, so even it fails to find the needed space, it will
> not screw up the old fs.
> 
> Thanks,
> Qu
> >

oh, I didn't realize that, thanks. Will test/try with little free space and 
check the
returning error messages. 

Cheers.
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs-convert: whats minimum free space requirement?

[Lakshmipathi.G]

On Fri, Jun 09, 2017 at 01:35:12PM +0800, Qu Wenruo wrote:
> >ex: Like 5% of ext3/4 free space is needed for
> >btrfs-convert to succeed?
> 
> That's hard to say.
> 
> It's not only dependent on how much free space is here, but also how
> continuous the free space is.
> 
> In convert, it needs the following continuous space at least:
> 
> 1) 64K for temporary btrfs super block
> 2) 4M for system chunk
> 3) 32M for metadata chunk.
> 
> Further more, we want to keep chunks at reasonable size, that's to say we
> want each used data space on old fs is covered by a chunk >= 32M.
> 
> For worst case, if the free space in old fs is so scattered, either we can't
> find continuous for sys/meta chunk, or used data space is so scattered that
> we can't find free space after rounding up data chunks, then we can't do
> convert.
> 
> Thanks,
> Qu

Thanks for the details, Qu.

I was thinking about adding a free-space check (based on total blocks/free block
stat from source superblock) before btrfs-convert begins, looks like its more 
complex than i thought. 

Cheers.
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

btrfs-convert: whats minimum free space requirement?

[Lakshmipathi.G]

Hi.
Just wanted to check whether do we have any numbers
on whats the minimum free space requirement on 
source file system for btrfs-convert to work?

ex: Like 5% of ext3/4 free space is needed for
btrfs-convert to succeed? 

Cheers.
Lakshmipathi.G


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v4] btrfs-progs: btrfs-convert: Add larger device support

[Lakshmipathi.G]

With larger file system (in this case its 22TB), ext2fs_open() returns
EXT2_ET_CANT_USE_LEGACY_BITMAPS error message with ext2fs_read_block_bitmap().

To overcome this issue, (a) we need pass EXT2_FLAG_64BITS flag with ext2fs_open.
(b) use 64-bit functions like ext2fs_get_block_bitmap_range2,
ext2fs_inode_data_blocks2,ext2fs_read_ext_attr2. (c) use 64bit types with
btrfs_convert_context fields.

bug: https://bugzilla.kernel.org/show_bug.cgi?id=194795
Signed-off-by: Lakshmipathi.G <lakshmipath...@giis.co.in>
---
 convert/common.h  |  8 
 convert/source-ext2.c | 11 ++-
 2 files changed, 10 insertions(+), 9 deletions(-)

diff --git a/convert/common.h b/convert/common.h
index 0d3adea..2f4ea48 100644
--- a/convert/common.h
+++ b/convert/common.h
@@ -30,10 +30,10 @@ struct btrfs_mkfs_config;
 
 struct btrfs_convert_context {
u32 blocksize;
-   u32 first_data_block;
-   u32 block_count;
-   u32 inodes_count;
-   u32 free_inodes_count;
+   u64 first_data_block;
+   u64 block_count;
+   u64 inodes_count;
+   u64 free_inodes_count;
u64 total_bytes;
char *volume_name;
const struct btrfs_convert_operations *convert_ops;
diff --git a/convert/source-ext2.c b/convert/source-ext2.c
index 1b0576b..275cb89 100644
--- a/convert/source-ext2.c
+++ b/convert/source-ext2.c
@@ -34,8 +34,9 @@ static int ext2_open_fs(struct btrfs_convert_context *cctx, 
const char *name)
ext2_filsys ext2_fs;
ext2_ino_t ino;
u32 ro_feature;
+   int open_flag = EXT2_FLAG_SOFTSUPP_FEATURES | EXT2_FLAG_64BITS;
 
-   ret = ext2fs_open(name, 0, 0, 0, unix_io_manager, _fs);
+   ret = ext2fs_open(name, open_flag, 0, 0, unix_io_manager, _fs);
if (ret) {
fprintf(stderr, "ext2fs_open: %s\n", error_message(ret));
return -1;
@@ -148,7 +149,7 @@ static int ext2_read_used_space(struct 
btrfs_convert_context *cctx)
return -ENOMEM;
 
for (i = 0; i < fs->group_desc_count; i++) {
-   ret = ext2fs_get_block_bitmap_range(fs->block_map, blk_itr,
+   ret = ext2fs_get_block_bitmap_range2(fs->block_map, blk_itr,
block_nbytes * 8, block_bitmap);
if (ret) {
error("fail to get bitmap from ext2, %s",
@@ -353,7 +354,7 @@ static int ext2_create_symlink(struct btrfs_trans_handle 
*trans,
int ret;
char *pathname;
u64 inode_size = btrfs_stack_inode_size(btrfs_inode);
-   if (ext2fs_inode_data_blocks(ext2_fs, ext2_inode)) {
+   if (ext2fs_inode_data_blocks2(ext2_fs, ext2_inode)) {
btrfs_set_stack_inode_size(btrfs_inode, inode_size + 1);
ret = ext2_create_file_extents(trans, root, objectid,
btrfs_inode, ext2_fs, ext2_ino,
@@ -627,9 +628,9 @@ static int ext2_copy_extended_attrs(struct 
btrfs_trans_handle *trans,
ret = -ENOMEM;
goto out;
}
-   err = ext2fs_read_ext_attr(ext2_fs, ext2_inode->i_file_acl, buffer);
+   err = ext2fs_read_ext_attr2(ext2_fs, ext2_inode->i_file_acl, buffer);
if (err) {
-   fprintf(stderr, "ext2fs_read_ext_attr: %s\n",
+   fprintf(stderr, "ext2fs_read_ext_attr2: %s\n",
error_message(err));
ret = -1;
goto out;
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] btrfs-progs: btrfs-convert: Add larger device support

[Lakshmipathi.G]

> 
> I think we should use the 64bit types just to be safe.
> 
Dave, Okay made those changes and new patch sent.

>If we need to modify tons of things, then please split such modification to
>different patches.

Qu, Seems like only minor changes needed, so used a single patch.

Cheers.
Lakshmipathi.G

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 00/19] Btrfs-progs offline scrub

[Lakshmipathi.G]

>
> Only recovery needs to be implemented now.
>
> Thanks,
> Qu
>
Once recovery is implemented, I'll try again.

Just one suggestion: Optionally, It is possible to print filename for
these detected blocks. For ex, if corruption happened on a
old/unwanted archived log file  (/var/log/nginx_access_20160101.log)
I don't need to panic, as I can live without that file. Or possibly
delete that file, so that scrub doesn't report errors.


-- 

Cheers,
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 00/19] Btrfs-progs offline scrub

[Lakshmipathi.G]

Okay, I did multiple (upto 11) corruption on the same file. Seems like
it says 'CORRUPTED' when we corrupt two continuous data stripes and
reports 'RECOVERABLE' whenever possible.It looks fine.  You can find
the logs and test-scripts on below link. thanks.

https://github.com/Lakshmipathi/btrfs_offline_scrub

-- 

Cheers,
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 00/19] Btrfs-progs offline scrub

[Lakshmipathi.G]

Okay sure, When I find free-time,  will run few more combinations and
let you know the details along with any other suggestions/thoughts on
the interface. thanks.

On 5/22/17, Qu Wenruo <quwen...@cn.fujitsu.com> wrote:
> Good to hear that.
>
> And you could try more combination (although I have done a lot during
> the test) to see if it's good enough to be used as a better scrub
> equivalent.
>
> And of course, any idea on the interface design (--offline under scrub
> or other place) is welcomed.
>
> Thanks,
> Qu
>
> At 05/22/2017 04:30 PM, Lakshmipathi.G wrote:
>> Yes. It detects the injected corruption.
>>
>> #btrfs scrub start --offline /dev/loop6
>> ERROR: data at bytenr 145293312 mirror 0 csum mismatch, have
>> 0x790f1fe1 expect 0xa30cb5c5
>> ERROR: full stripe 145227776 RECOVERABLE: Data stripes corrupted, but
>> P/Q is good
>>
>> Scrub result:
>> Tree bytes scrubbed: 131072
>> Tree extents scrubbed: 8
>> Data bytes scrubbed: 192512
>> Data extents scrubbed: 3
>> Data bytes without csum: 131072
>> Read error: 0
>> Verify error: 0
>> Csum error: 1
>>
>> 
>> Cheers,
>> Lakshmipathi.G
>>
>>
>
>
>


-- 

Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 00/19] Btrfs-progs offline scrub

[Lakshmipathi.G]

Yes. It detects the injected corruption.

#btrfs scrub start --offline /dev/loop6
ERROR: data at bytenr 145293312 mirror 0 csum mismatch, have
0x790f1fe1 expect 0xa30cb5c5
ERROR: full stripe 145227776 RECOVERABLE: Data stripes corrupted, but
P/Q is good

Scrub result:
Tree bytes scrubbed: 131072
Tree extents scrubbed: 8
Data bytes scrubbed: 192512
Data extents scrubbed: 3
Data bytes without csum: 131072
Read error: 0
Verify error: 0
Csum error: 1


Cheers,
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 00/19] Btrfs-progs offline scrub

[Lakshmipathi.G]

>
> As the title said, it's *offline* scrub, while you're still using the
> *online* scrub.
>
> To use offline scrub, you should exec "btrfs scrub start --offline
> "
>
> And then it should detect the same error.
> Output is nothing like kernel.
>
> Thanks,
> Qu
>

Ah,my bad.  I thought the entire binary is changed, didn't realize its
an option.  Will check again with 'offline' option.

-- 

Cheers,
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 00/19] Btrfs-progs offline scrub

[Lakshmipathi.G]

When I ran the script, it produces below output. From the output it
looks like, it detected the error and corrected. But parity-block
corruption fix patch not yet merged or I missed something?

Run: (with scrub_offline repo)
btrfs scrub start /home/laks/centos/laks/BTRFS/scrub_offline/tests/mnt
scrub started on /home/laks/centos/laks/BTRFS/scrub_offline/tests/mnt,
fsid a5cf2e54-e8da-4570-8328-2cee46678de3 (pid=5943)
WARNING: errors detected during scrubbing, corrected.
Scrub corrected value: D1xx ##output from testscript
Scrub corrupted parity stripe.   ##output from testscript

(mkfs.btrfs -f -d raid5 -m raid5 /dev/loop18 /dev/loop19 /dev/loop20)

>>
>> Can  I inject corruption with existing script [1] and expect offline
>> scrub to fix it? If so, I'll give it try and let you know the results.
>>
>> [1] https://patchwork.kernel.org/patch/9583455/
>
> No fixing yet.
>
> This part is focusing on error detection and report.
>
> Feel free to inject corruption and to see if it behave as expected.
>
> Thanks,
> Qu
>

Cheers,
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/2] btrfs-progs: test for restoring multiple devices fs into a single device

[Lakshmipathi.G]

Looks fine, just couple of minor feedback.

> +run_check $SUDO_HELPER $TOP/mkfs.btrfs -f $loop1 $loop2

Please add quotation around variable in the script, as suggested in
tests/README.md 'Coding style, best practices' section. This section
is added recently, we are updating older scripts.

> +
> +# Cleanup loop devices.
> +run_check $SUDO_HELPER losetup -d $loop1
> +run_check $SUDO_HELPER losetup -d $loop2
> +rm -f dev1 dev2
> +
> +# Compare the file digests.
> +[ $orig_md5 == $new_md5 ] || _fail "File digests do not match"

"Cleanup loop devices" can be done after comparing file integrity?
This way, if integrity fails, the setup will be there to debug
further. thanks.

-- 

Cheers,
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs-progs: tests: remove variable quotation from convert-tests

[Lakshmipathi.G]

Oops, sorry, I introduced those two issues in recent patches and
missed (skipped?) them while testing.  With above patch, 008/009
test-cases are working fine now.  thanks.

On 5/16/17, Tsutomu Itoh <t-i...@jp.fujitsu.com> wrote:
> In btrfs-progs-v4.11-rc1, the following convert-tests failed.
>
> [TEST/conv]   008-readonly-image
> [TEST/conv] readonly image test, btrfs defaults
> failed: mke2fs -t ext4 -b 4096 -F
> /Build/btrfs-progs-v4.11-rc1/tests/test.img
> test failed for case 008-readonly-image
> Makefile:271: recipe for target 'test-convert' failed
> make: *** [test-convert] Error 1
> [TEST/conv]   009-common-inode-flags
> [TEST/conv] common inode flags test, btrfs defaults
> failed: mke2fs -t ext4 -b 4096 -F
> /Build/btrfs-progs-v4.11-rc1/tests/test.img
> test failed for case 009-common-inode-flags
> Makefile:271: recipe for target 'test-convert' failed
> make: *** [test-convert] Error 1
>
> So, remove quotes from $default_mke2fs.
>
> Signed-off-by: Tsutomu Itoh <t-i...@jp.fujitsu.com>
> ---
>  tests/convert-tests/008-readonly-image/test.sh | 2 +-
>  tests/convert-tests/009-common-inode-flags/test.sh | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/tests/convert-tests/008-readonly-image/test.sh
> b/tests/convert-tests/008-readonly-image/test.sh
> index b2f1ae37..27c9373e 100755
> --- a/tests/convert-tests/008-readonly-image/test.sh
> +++ b/tests/convert-tests/008-readonly-image/test.sh
> @@ -10,7 +10,7 @@ check_prereq btrfs-convert
>
>  default_mke2fs="mke2fs -t ext4 -b 4096"
>  convert_test_preamble '' 'readonly image test' 16k "$default_mke2fs"
> -convert_test_prep_fs "$default_mke2fs"
> +convert_test_prep_fs $default_mke2fs
>  run_check_umount_test_dev
>  convert_test_do_convert
>  run_check_mount_test_dev
> diff --git a/tests/convert-tests/009-common-inode-flags/test.sh
> b/tests/convert-tests/009-common-inode-flags/test.sh
> index a5828790..02823e14 100755
> --- a/tests/convert-tests/009-common-inode-flags/test.sh
> +++ b/tests/convert-tests/009-common-inode-flags/test.sh
> @@ -11,7 +11,7 @@ check_prereq btrfs-convert
>  fail=0
>  default_mke2fs="mke2fs -t ext4 -b 4096"
>  convert_test_preamble '' 'common inode flags test' 16k "$default_mke2fs"
> -convert_test_prep_fs "$default_mke2fs"
> +convert_test_prep_fs $default_mke2fs
>
>  # create file with specific flags
>  run_check $SUDO_HELPER touch "$TEST_MNT/flag_test"
> --
> 2.12.2
>
> 
> Tsutomu Itoh  t-i...@jp.fujitsu.com
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


-- 

Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] btrfs-progs: btrfs-convert: Add larger device support

[Lakshmipathi.G]

On Mon, May 15, 2017 at 09:40:29AM +0800, Qu Wenruo wrote:
> >bug: https://bugzilla.kernel.org/show_bug.cgi?id=194795
> 
> Errr, it seems that you forgot to update ext2_open_fs() to update how we get
> cctx->block_counts.
> 
> Without that update, we still get wrong total size of original fs, so
> converted image will be corrupted.
> 
> In this 22T case, it can't pass convert test since after conversion,
> converted image can't pass e2fsck.
> 
> Thanks,
> Qu
> 

Thanks for pointing out the issue. So we need to update 
common.h/cctx->block_count from u32 to u64?

do we also need to change other fields like inodes_count and
free_inode_count?

Cheers.
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 00/19] Btrfs-progs offline scrub

[Lakshmipathi.G]

>
> Ping?
>
> Any comments?
>
> Thanks,
> Qu

Can  I inject corruption with existing script [1] and expect offline
scrub to fix it? If so, I'll give it try and let you know the results.

[1] https://patchwork.kernel.org/patch/9583455/


Cheers,
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2] btrfs-progs: btrfs-convert: Add larger device support

[Lakshmipathi.G]

With larger file system (in this case its 22TB), ext2fs_open() returns
EXT2_ET_CANT_USE_LEGACY_BITMAPS error message with ext2fs_read_block_bitmap().

To overcome this issue, we need pass EXT2_FLAG_64BITS flag with ext2fs_open
and also use 64-bit functions like ext2fs_get_block_bitmap_range2,
ext2fs_inode_data_blocks2,ext2fs_read_ext_attr2

bug: https://bugzilla.kernel.org/show_bug.cgi?id=194795

Signed-off-by: Lakshmipathi.G <lakshmipath...@giis.co.in>
---
 convert/source-ext2.c | 11 ++-
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/convert/source-ext2.c b/convert/source-ext2.c
index 1b0576b..275cb89 100644
--- a/convert/source-ext2.c
+++ b/convert/source-ext2.c
@@ -34,8 +34,9 @@ static int ext2_open_fs(struct btrfs_convert_context *cctx, 
const char *name)
ext2_filsys ext2_fs;
ext2_ino_t ino;
u32 ro_feature;
+   int open_flag = EXT2_FLAG_SOFTSUPP_FEATURES | EXT2_FLAG_64BITS;
 
-   ret = ext2fs_open(name, 0, 0, 0, unix_io_manager, _fs);
+   ret = ext2fs_open(name, open_flag, 0, 0, unix_io_manager, _fs);
if (ret) {
fprintf(stderr, "ext2fs_open: %s\n", error_message(ret));
return -1;
@@ -148,7 +149,7 @@ static int ext2_read_used_space(struct 
btrfs_convert_context *cctx)
return -ENOMEM;
 
for (i = 0; i < fs->group_desc_count; i++) {
-   ret = ext2fs_get_block_bitmap_range(fs->block_map, blk_itr,
+   ret = ext2fs_get_block_bitmap_range2(fs->block_map, blk_itr,
block_nbytes * 8, block_bitmap);
if (ret) {
error("fail to get bitmap from ext2, %s",
@@ -353,7 +354,7 @@ static int ext2_create_symlink(struct btrfs_trans_handle 
*trans,
int ret;
char *pathname;
u64 inode_size = btrfs_stack_inode_size(btrfs_inode);
-   if (ext2fs_inode_data_blocks(ext2_fs, ext2_inode)) {
+   if (ext2fs_inode_data_blocks2(ext2_fs, ext2_inode)) {
btrfs_set_stack_inode_size(btrfs_inode, inode_size + 1);
ret = ext2_create_file_extents(trans, root, objectid,
btrfs_inode, ext2_fs, ext2_ino,
@@ -627,9 +628,9 @@ static int ext2_copy_extended_attrs(struct 
btrfs_trans_handle *trans,
ret = -ENOMEM;
goto out;
}
-   err = ext2fs_read_ext_attr(ext2_fs, ext2_inode->i_file_acl, buffer);
+   err = ext2fs_read_ext_attr2(ext2_fs, ext2_inode->i_file_acl, buffer);
if (err) {
-   fprintf(stderr, "ext2fs_read_ext_attr: %s\n",
+   fprintf(stderr, "ext2fs_read_ext_attr2: %s\n",
error_message(err));
ret = -1;
goto out;
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs-progs: btrfs-convert: Add larger device support

[Lakshmipathi.G]

>
> Does the mke2fs command exist on your system? We maybe want to use
> mkfs.ext4 directly.

Yes, mke2fs is there. (I'll check using mkfs.ext4 directly)

# ./convert-tests/009-common-inode-flags/test.sh
[TEST/conv] common inode flags test, btrfs defaults
failed: mke2fs -t ext4 -b 4096 -F
/home/laks/centos/laks/BTRFS/btrfs-progs/tests/test.img

After failure, I can run failed command manually.
# mke2fs -t ext4 -b 4096 -F
/home/laks/centos/laks/BTRFS/btrfs-progs/tests/test.imgmke2fs 1.42.13
(17-May-2015)
Discarding device blocks: done
Creating filesystem with 131072 4k blocks and 32768 inodes
Filesystem UUID: 659dbb16-ea5e-48e0-8b1a-30474f54c282
Superblock backups stored on blocks:
32768, 98304

Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

-- 

Cheers,
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs-progs: btrfs-convert: Add larger device support

[Lakshmipathi.G]

created a test script but it fail to detect mk2fs. Test script 009
also produces:

# ./convert-tests/009-common-inode-flags/test.sh
[TEST/conv] common inode flags test, btrfs defaults
failed: mke2fs -t ext4 -b 4096 -F
/home/laks/centos/laks/BTRFS/btrfs-progs/tests/test.img

Seems like run_check fails to detect mke2fs command. Something wrong
with my setup or issue with run_check?

-- 

Cheers,
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 9/9] btrfs-progs: modify: Introduce option to specify the pattern to fill mirror

[Lakshmipathi.G]

okay, but I'm not sure whether it can be useful here other than the
sample cli usage. The reason, this tool relies heavily on Python/C
layer[1]. The framework works like this:

C-api(fs-progs) <--> Python/C api <--> Python program.

C-api is typical fs library (like libext2fs/libbtrfs) and C/Python
used  to export every on-disk fs structure as an python-object along
with few fs-library functions. Finally in python script we can perform
stuffs like:

--
import fslib as fs

disk_object=fs.get_inode("/file/path")
#set few values finally make a call
retval = fs.corrupt_object(disk_object, offset, size, optype, fixcrc)
---

at C-python layer above python-object converted into C appropriate
data-type and passed on to C-layer.

hard-part will be troubleshooting issues, tracking them is difficult
because both gdb and pdb can help only upto certain point.

[1]: https://docs.python.org/2/extending/extending.html

Cheers.
Lakshmipathi.G


On 5/5/17, Qu Wenruo <quwen...@cn.fujitsu.com> wrote:
>
>
> At 05/05/2017 12:53 AM, David Sterba wrote:
>> On Sun, Apr 23, 2017 at 01:12:42PM +0530, Lakshmipathi.G wrote:
>>> Thanks for the example and details. I understood some and need to
>>> re-read couple of more times to understand the remaining.
>>>
>>> btw, I created a corruption framework(with previous org), the sample
>>> usage and example is below. It looks similar to Btrfs corruption tool.
>>> thanks.
>>>
>>> --
>>> corrupt.py --help
>> [...]
>>
>> Interesting, can you please share the script? This is another
>> alternative that seems more plausible for rapid prototyping of various
>> corruption scenarios. The C utility (either existing btrfs-corrupt-block
>> or the proposed btrfs-modify) can become tedious to change, but can be
>> compiled and distributed without the python dependency.
>>
>> I wanted to use something python-based for tests when Hans announced the
>> python-btrfs project, but it has broader goals than just the testsuite
>> needs.  So we could have our own corrupt.py, just for our internal use.
>>
>> I'm not sure if a compiled tool like btrfs-modify is really needed, but
>> why we can't have both.
>>
>>
> Python based tool is always a good idea.
>
> If python based lib can provide the same capability of current C
> facilities (btrfs_search_slot() at least) and have better encapsulation,
> I'm totally committed to python based one.
>
> Although I'm a little afraid that the python script is not using the low
> level btrfs_search_slot(), but just getting chunk layout then do
> mathematics to get stripe location.
>
> Thanks,
> Qu
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs-progs: btrfs-convert: Add larger device support

[Lakshmipathi.G]

> file is ~160MB. This sounds much better. So please write a test, using
> something like
> 
> truncate -s3T image
> mkfs.ext4 image
> mount && write some data
> convert && rollback
> 
> Thanks.  Later we might need to add some mkfs.ext4 option coverage.
Sure, will send the above test script in upcoming days, then look
into mkfs.ext4 options. thanks.

Cheers.
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs-progs: btrfs-convert: Add larger device support

[Lakshmipathi.G]

On Tue, May 02, 2017 at 05:06:01PM +0200, David Sterba wrote:
> On Thu, Apr 27, 2017 at 03:35:34PM +0530, Lakshmipathi.G wrote:
> > Bug: Fail to convert 22TB EXT4 to BTRFS
> > https://bugzilla.kernel.org/show_bug.cgi?id=194795
> 
> Thanks for working on the bug, the proposed fix looks good to me judging
> by the use of extended API. This would need a test though. I'm trying
> this on a sparse file, 3TB occupies 50G and is slow to create.
> Conversion works and hasn't finished yet, so this is not suitable for
> tests run by default.
> 
> Please write a more detailed changelog, a reference to bugzilla is not
> really sufficient.

Sure, will send a patch with better commit description about the bug and
proposed fix. 

Cheers.
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/1] btrfs-progs: send: fail on first -ENODATA only

[Lakshmipathi.G]

Hi Christian, thanks for fixing it quickly :) I don't have permission
to change the bug state to fixed, if you or Nazar have those rights, i
think this bug can be closed.

Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org


On Mon, May 1, 2017 at 6:39 PM, Christian Brauner
<christian.brau...@canonical.com> wrote:
> Hi,
>
> The original bug-reporter verified that my patch fixes the bug. See
>
> https://bugzilla.kernel.org/show_bug.cgi?id=195597
>
> Christian
>
> On Sat, Apr 29, 2017 at 11:54:05PM +0200, Christian Brauner wrote:
>> Returning -ENODATA is only considered invalid on the first run of the loop.
>>
>> Signed-off-by: Christian Brauner <christian.brau...@ubuntu.com>
>> ---
>>  cmds-receive.c | 20 ++--
>>  1 file changed, 14 insertions(+), 6 deletions(-)
>>
>> diff --git a/cmds-receive.c b/cmds-receive.c
>> index b59f00e4..72e9c8f3 100644
>> --- a/cmds-receive.c
>> +++ b/cmds-receive.c
>> @@ -1091,6 +1091,7 @@ static int do_receive(struct btrfs_receive *rctx, 
>> const char *tomnt,
>>   char *dest_dir_full_path;
>>   char root_subvol_path[PATH_MAX];
>>   int end = 0;
>> + int iterations = 0;
>>
>>   dest_dir_full_path = realpath(tomnt, NULL);
>>   if (!dest_dir_full_path) {
>> @@ -1198,13 +1199,18 @@ static int do_receive(struct btrfs_receive *rctx, 
>> const char *tomnt,
>>rctx,
>>rctx->honor_end_cmd,
>>max_errors);
>> - if (ret < 0 && ret == -ENODATA) {
>> + if (ret < 0) {
>> + if (ret != -ENODATA)
>> + goto out;
>> +
>>   /* Empty stream is invalid */
>> - error("empty stream is not considered valid");
>> - ret = -EINVAL;
>> - goto out;
>> - } else if (ret < 0) {
>> - goto out;
>> + if (iterations == 0) {
>> + error("empty stream is not considered valid");
>> + ret = -EINVAL;
>> + goto out;
>> + }
>> +
>> + ret = 1;
>>   }
>>   if (ret > 0)
>>   end = 1;
>> @@ -1213,6 +1219,8 @@ static int do_receive(struct btrfs_receive *rctx, 
>> const char *tomnt,
>>   ret = finish_subvol(rctx);
>>   if (ret < 0)
>>   goto out;
>> +
>> + iterations++;
>>   }
>>   ret = 0;
>>
>> --
>> 2.11.0
>>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/2 v2] btrfs-progs: fix btrfs send & receive with -e flag

[Lakshmipathi.G]

Hi.

Adding the bug reporter, Nazar for the discussion (as I'm not familiar
with send/receive feature/code).


Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org

On Fri, Apr 28, 2017 at 3:25 PM, Christian Brauner
<christian.brau...@canonical.com> wrote:
>
> Hi,
>
> On Fri, Apr 28, 2017 at 02:55:31PM +0530, Lakshmipathi.G wrote:
> > Seems like user reported an issue with this patch. please check
> > https://bugzilla.kernel.org/show_bug.cgi?id=195597
>
> I can take a look. What I'm wondering about is why it fails only in the HDD
> to SSD case. If -ENODATA is returned with this patch it should mean that there
> was no header data. So is the user sure that this doesn't indicate a valid
> error?
>
> Christian
>
> >
> > 
> > Cheers,
> > Lakshmipathi.G
> >
> >
> > On Tue, Apr 4, 2017 at 1:51 AM, Christian Brauner <
> > christian.brau...@ubuntu.com> wrote:
> > > The old check here tried to ensure that empty streams are not considered
> > valid.
> > > The old check however, will always fail when only one run through the
> > while(1)
> > > loop is needed and honor_end_cmd is set. So this:
> > >
> > > btrfs send /some/subvol | btrfs receive -e /some/
> > >
> > > will consistently fail because -e causes honor_cmd_to be set and
> > > btrfs_read_and_process_send_stream() to correctly return 1. So the
> > command will
> > > be successful but btrfs receive will error out because the send - receive
> > > concluded in one run through the while(1) loop.
> > >
> > > If we want to exclude empty streams we need a way to tell the difference
> > between
> > > btrfs_read_and_process_send_stream() returning 1 because read_buf() did
> > not
> > > detect any data and read_and_process_cmd() returning 1 because
> > honor_end_cmd was
> > > set. Without introducing too many changes the best way to me seems to have
> > > btrfs_read_and_process_send_stream() return -ENODATA in the first case.
> > The rest
> > > stays the same. We can then check for -ENODATA in do_receive() and report
> > a
> > > proper error in this case. This should also be backwards compatible to
> > previous
> > > versions of btrfs receive. They will fail on empty streams because a
> > negative
> > > value is returned. The only thing that they will lack is a nice error
> > message.
> > >
> > > Signed-off-by: Christian Brauner <christian.brau...@ubuntu.com>
> > > ---
> > > Changelog: 2017-04-03
> > > - no changes
> > > ---
> > >  cmds-receive.c | 13 +
> > >  send-stream.c  |  2 +-
> > >  2 files changed, 6 insertions(+), 9 deletions(-)
> > >
> > > diff --git a/cmds-receive.c b/cmds-receive.c
> > > index 6cf22637..b59f00e4 100644
> > > --- a/cmds-receive.c
> > > +++ b/cmds-receive.c
> > > @@ -1091,7 +1091,6 @@ static int do_receive(struct btrfs_receive *rctx,
> > const char *tomnt,
> > > char *dest_dir_full_path;
> > > char root_subvol_path[PATH_MAX];
> > > int end = 0;
> > > -   int count;
> > >
> > > dest_dir_full_path = realpath(tomnt, NULL);
> > > if (!dest_dir_full_path) {
> > > @@ -1186,7 +1185,6 @@ static int do_receive(struct btrfs_receive *rctx,
> > const char *tomnt,
> > > if (ret < 0)
> > > goto out;
> > >
> > > -   count = 0;
> > > while (!end) {
> > > if (rctx->cached_capabilities_len) {
> > > if (g_verbose >= 3)
> > > @@ -1200,16 +1198,15 @@ static int do_receive(struct btrfs_receive *rctx,
> > const char *tomnt,
> > >  rctx,
> > >
> >  rctx->honor_end_cmd,
> > >  max_errors);
> > > -   if (ret < 0)
> > > -   goto out;
> > > -   /* Empty stream is invalid */
> > > -   if (ret && count == 0) {
> > > +   if (ret < 0 && ret == -ENODATA) {
> > > +   /* Empty stream is invalid */
> > > error("empty stream is not considered valid");
> > > ret = -EINVAL;
> > > goto out;
> > > +   } else if (ret < 0) {
>

Re: [PATCH] btrfs-progs: btrfs-convert: Add larger device support

[Lakshmipathi.G]

Hi Qu,

Seems like its a known issue with older bitmaps.
http://linux-ext4.vger.kernel.narkive.com/wkNbJe0b/patch-debugfs-open-with-ext2-flag-64bits

Cheers.
Lakshmipathi.G

On Fri, Apr 28, 2017 at 09:48:10AM +0800, Qu Wenruo wrote:
> 
> 
> At 04/28/2017 09:27 AM, Qu Wenruo wrote:
> >At 04/27/2017 06:05 PM, Lakshmipathi.G wrote:
> >>Bug: Fail to convert 22TB EXT4 to BTRFS
> >>https://bugzilla.kernel.org/show_bug.cgi?id=194795
> >
> >Thanks for fixing it.
> >
> >While testing manually using LVM thin provision, although the convert
> >itself works, the converted image has something wrong on its size.
> >
> >The converted image is only 4T sized, not the correct size (20T) of my
> >original setup.
> >
> >This may not be the problem of your patch.
> >I'll double check the original code to ensure it works.
> 
> It's the method we get blocks count wrong.
> 
> We should do it just as dumpe2fs does:
> --
> static __u64 e2p_blocks_count(struct ext2_super_block *super)
> {
>   return super->s_blocks_count |
>   (ext2fs_has_feature_64bit(super) ?
>   (__u64) super->s_blocks_count_hi << 32 : 0);
> }
> --
> 
> Current super->s_blocks_count can only handle U32_MAX, higher bits are in
> super->s_blocks_count_hi.
> 
> Thanks,
> Qu
> 
> >
> >Thanks,
> >Qu
> >
> >>
> >>Signed-off-by: Lakshmipathi.G <lakshmipath...@giis.co.in>
> >>---
> >>  convert/source-ext2.c | 11 ++-
> >>  1 file changed, 6 insertions(+), 5 deletions(-)
> >>
> >>diff --git a/convert/source-ext2.c b/convert/source-ext2.c
> >>index 1b0576b..275cb89 100644
> >>--- a/convert/source-ext2.c
> >>+++ b/convert/source-ext2.c
> >>@@ -34,8 +34,9 @@ static int ext2_open_fs(struct btrfs_convert_context
> >>*cctx, const char *name)
> >>  ext2_filsys ext2_fs;
> >>  ext2_ino_t ino;
> >>  u32 ro_feature;
> >>+int open_flag = EXT2_FLAG_SOFTSUPP_FEATURES | EXT2_FLAG_64BITS;
> >>-ret = ext2fs_open(name, 0, 0, 0, unix_io_manager, _fs);
> >>+ret = ext2fs_open(name, open_flag, 0, 0, unix_io_manager, _fs);
> >>  if (ret) {
> >>  fprintf(stderr, "ext2fs_open: %s\n", error_message(ret));
> >>  return -1;
> >>@@ -148,7 +149,7 @@ static int ext2_read_used_space(struct
> >>btrfs_convert_context *cctx)
> >>  return -ENOMEM;
> >>  for (i = 0; i < fs->group_desc_count; i++) {
> >>-ret = ext2fs_get_block_bitmap_range(fs->block_map, blk_itr,
> >>+ret = ext2fs_get_block_bitmap_range2(fs->block_map, blk_itr,
> >>  block_nbytes * 8, block_bitmap);
> >>  if (ret) {
> >>  error("fail to get bitmap from ext2, %s",
> >>@@ -353,7 +354,7 @@ static int ext2_create_symlink(struct
> >>btrfs_trans_handle *trans,
> >>  int ret;
> >>  char *pathname;
> >>  u64 inode_size = btrfs_stack_inode_size(btrfs_inode);
> >>-if (ext2fs_inode_data_blocks(ext2_fs, ext2_inode)) {
> >>+if (ext2fs_inode_data_blocks2(ext2_fs, ext2_inode)) {
> >>  btrfs_set_stack_inode_size(btrfs_inode, inode_size + 1);
> >>  ret = ext2_create_file_extents(trans, root, objectid,
> >>  btrfs_inode, ext2_fs, ext2_ino,
> >>@@ -627,9 +628,9 @@ static int ext2_copy_extended_attrs(struct
> >>btrfs_trans_handle *trans,
> >>  ret = -ENOMEM;
> >>  goto out;
> >>  }
> >>-err = ext2fs_read_ext_attr(ext2_fs, ext2_inode->i_file_acl, buffer);
> >>+err = ext2fs_read_ext_attr2(ext2_fs, ext2_inode->i_file_acl,
> >>buffer);
> >>  if (err) {
> >>-fprintf(stderr, "ext2fs_read_ext_attr: %s\n",
> >>+fprintf(stderr, "ext2fs_read_ext_attr2: %s\n",
> >>  error_message(err));
> >>  ret = -1;
> >>  goto out;
> >>
> 
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] btrfs-progs: btrfs-convert: Add larger device support

[Lakshmipathi.G]

Bug: Fail to convert 22TB EXT4 to BTRFS
https://bugzilla.kernel.org/show_bug.cgi?id=194795

Signed-off-by: Lakshmipathi.G <lakshmipath...@giis.co.in>
---
 convert/source-ext2.c | 11 ++-
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/convert/source-ext2.c b/convert/source-ext2.c
index 1b0576b..275cb89 100644
--- a/convert/source-ext2.c
+++ b/convert/source-ext2.c
@@ -34,8 +34,9 @@ static int ext2_open_fs(struct btrfs_convert_context *cctx, 
const char *name)
ext2_filsys ext2_fs;
ext2_ino_t ino;
u32 ro_feature;
+   int open_flag = EXT2_FLAG_SOFTSUPP_FEATURES | EXT2_FLAG_64BITS;
 
-   ret = ext2fs_open(name, 0, 0, 0, unix_io_manager, _fs);
+   ret = ext2fs_open(name, open_flag, 0, 0, unix_io_manager, _fs);
if (ret) {
fprintf(stderr, "ext2fs_open: %s\n", error_message(ret));
return -1;
@@ -148,7 +149,7 @@ static int ext2_read_used_space(struct 
btrfs_convert_context *cctx)
return -ENOMEM;
 
for (i = 0; i < fs->group_desc_count; i++) {
-   ret = ext2fs_get_block_bitmap_range(fs->block_map, blk_itr,
+   ret = ext2fs_get_block_bitmap_range2(fs->block_map, blk_itr,
block_nbytes * 8, block_bitmap);
if (ret) {
error("fail to get bitmap from ext2, %s",
@@ -353,7 +354,7 @@ static int ext2_create_symlink(struct btrfs_trans_handle 
*trans,
int ret;
char *pathname;
u64 inode_size = btrfs_stack_inode_size(btrfs_inode);
-   if (ext2fs_inode_data_blocks(ext2_fs, ext2_inode)) {
+   if (ext2fs_inode_data_blocks2(ext2_fs, ext2_inode)) {
btrfs_set_stack_inode_size(btrfs_inode, inode_size + 1);
ret = ext2_create_file_extents(trans, root, objectid,
btrfs_inode, ext2_fs, ext2_ino,
@@ -627,9 +628,9 @@ static int ext2_copy_extended_attrs(struct 
btrfs_trans_handle *trans,
ret = -ENOMEM;
goto out;
}
-   err = ext2fs_read_ext_attr(ext2_fs, ext2_inode->i_file_acl, buffer);
+   err = ext2fs_read_ext_attr2(ext2_fs, ext2_inode->i_file_acl, buffer);
if (err) {
-   fprintf(stderr, "ext2fs_read_ext_attr: %s\n",
+   fprintf(stderr, "ext2fs_read_ext_attr2: %s\n",
error_message(err));
ret = -1;
goto out;
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] btrfs-progs: Introduce 'btrfs inspect-internal dump-csum' option

[Lakshmipathi.G]

Example usage:
btrfs inspect-internal dump-csum /btrfs/50gbfile /dev/sda4
csum for /btrfs/50gbfile dumped to /btrfs/50gbfile.csumdump

Signed-off-by: Lakshmipathi.G <lakshmipath...@giis.co.in>
---
 Makefile |   3 +-
 cmds-inspect-dump-csum.c | 256 +++
 cmds-inspect-dump-csum.h |  24 +
 cmds-inspect.c   |   3 +
 4 files changed, 285 insertions(+), 1 deletion(-)
 create mode 100644 cmds-inspect-dump-csum.c
 create mode 100644 cmds-inspect-dump-csum.h

diff --git a/Makefile b/Makefile
index 81598df..6e63e18 100644
--- a/Makefile
+++ b/Makefile
@@ -102,7 +102,7 @@ cmds_objects = cmds-subvolume.o cmds-filesystem.o 
cmds-device.o cmds-scrub.o \
   cmds-restore.o cmds-rescue.o chunk-recover.o super-recover.o \
   cmds-property.o cmds-fi-usage.o cmds-inspect-dump-tree.o \
   cmds-inspect-dump-super.o cmds-inspect-tree-stats.o cmds-fi-du.o 
\
-  mkfs/common.o
+  cmds-inspect-dump-csum.o mkfs/common.o
 libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o 
\
   kernel-lib/crc32c.o messages.o \
   uuid-tree.o utils-lib.o rbtree-utils.o
@@ -191,6 +191,7 @@ btrfs_convert_cflags = 
-DBTRFSCONVERT_EXT2=$(BTRFSCONVERT_EXT2)
 btrfs_fragments_libs = -lgd -lpng -ljpeg -lfreetype
 btrfs_debug_tree_objects = cmds-inspect-dump-tree.o
 btrfs_show_super_objects = cmds-inspect-dump-super.o
+btrfs_dump_csum_objects = cmds-inspect-dump-csum.o
 btrfs_calc_size_objects = cmds-inspect-tree-stats.o
 
 # collect values of the variables above
diff --git a/cmds-inspect-dump-csum.c b/cmds-inspect-dump-csum.c
new file mode 100644
index 000..6065eb4
--- /dev/null
+++ b/cmds-inspect-dump-csum.c
@@ -0,0 +1,256 @@
+/*
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License v2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; if not, write to the
+ * Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+ * Boston, MA 021110-1307, USA.
+ */
+
+#include "kerncompat.h"
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include "ctree.h"
+#include "disk-io.h"
+#include "print-tree.h"
+#include "transaction.h"
+#include "list.h"
+#include "utils.h"
+#include "commands.h"
+#include "crc32c.h"
+#include "cmds-inspect-dump-csum.h"
+#include "help.h"
+#include "volumes.h"
+
+
+const char * const cmd_inspect_dump_csum_usage[] = {
+   "btrfs inspect-internal dump-csum  ",
+   "Get csums for the given file.",
+   NULL
+};
+
+int btrfs_lookup_csums(struct btrfs_trans_handle *trans, struct btrfs_root 
*root,
+   struct btrfs_path *path, u64 bytenr, int cow, int total_csums, FILE *fp)
+{
+   int ret;
+   int i;
+   int start_pos = 0;
+   struct btrfs_key file_key;
+   struct btrfs_key found_key;
+   struct btrfs_csum_item *item;
+   struct extent_buffer *leaf;
+   u64 csum_offset = 0;
+   u16 csum_size =
+   btrfs_super_csum_size(root->fs_info->super_copy);
+   int csums_in_item = 0;
+   unsigned int tree_csum = 0;
+   int pending_csums = total_csums;
+   static int cnt=1;
+
+   file_key.objectid = BTRFS_EXTENT_CSUM_OBJECTID;
+   file_key.offset = bytenr;
+   file_key.type = BTRFS_EXTENT_CSUM_KEY;
+   ret = btrfs_search_slot(trans, root, _key, path, 0, cow);
+   if (ret < 0)
+   goto fail;
+   while(1){
+   leaf = path->nodes[0];
+   if (ret > 0) {
+   ret = 1;
+   if (path->slots[0] == 0)
+   goto fail; 
+   path->slots[0]--;
+   btrfs_item_key_to_cpu(leaf, _key, path->slots[0]);
+   if (found_key.type != BTRFS_EXTENT_CSUM_KEY){
+   fprintf(stderr, "\nInvalid key found.");
+   goto fail;
+   }
+
+   csum_offset = ((bytenr - found_key.offset) / 
root->sectorsize) * csum_size;
+   csums_in_item = btrfs_item_size_nr(leaf, 
path->slots[0]);
+   csums_in_item /= csum_size;
+   csums_in_item -= ( bytenr - found_key.offset ) / 
root->sectorsize;
+   start_pos=csum_offset

[PATCH] btrfs-progs: tests: add variable quotation to fsck-tests

[Lakshmipathi.G]

Signed-off-by: Lakshmipathi.G <lakshmipath...@giis.co.in>
---
 tests/fsck-tests/006-bad-root-items/test.sh|  6 +++---
 tests/fsck-tests/012-leaf-corruption/test.sh   | 24 +++---
 tests/fsck-tests/013-extent-tree-rebuild/test.sh   | 22 ++--
 tests/fsck-tests/018-leaf-crossing-stripes/test.sh |  4 ++--
 .../fsck-tests/019-non-skinny-false-alert/test.sh  |  4 ++--
 tests/fsck-tests/020-extent-ref-cases/test.sh  |  4 ++--
 .../021-partially-dropped-snapshot-case/test.sh|  4 ++--
 tests/fsck-tests/022-qgroup-rescan-halfway/test.sh |  4 ++--
 tests/fsck-tests/023-qgroup-stack-overflow/test.sh |  4 ++--
 tests/fsck-tests/024-clear-space-cache/test.sh | 16 +++
 tests/fsck-tests/025-file-extents/test.sh  |  2 +-
 tests/fsck-tests/026-check-inode-link/test.sh  |  2 +-
 12 files changed, 48 insertions(+), 48 deletions(-)

diff --git a/tests/fsck-tests/006-bad-root-items/test.sh 
b/tests/fsck-tests/006-bad-root-items/test.sh
index 8433234..bf3ef78 100755
--- a/tests/fsck-tests/006-bad-root-items/test.sh
+++ b/tests/fsck-tests/006-bad-root-items/test.sh
@@ -1,15 +1,15 @@
 #!/bin/bash
 
-source $TOP/tests/common
+source "$TOP/tests/common"
 
 check_prereq btrfs
 
-echo "extracting image default_case.tar.xz" >> $RESULTS
+echo "extracting image default_case.tar.xz" >> "$RESULTS"
 tar --no-same-owner -xJf default_case.tar.xz || \
_fail "failed to extract default_case.tar.xz"
 check_image test.img
 
-echo "extracting image skinny_case.tar.xz" >> $RESULTS
+echo "extracting image skinny_case.tar.xz" >> "$RESULTS"
 tar --no-same-owner -xJf skinny_case.tar.xz || \
_fail "failed to extract skinny_case.tar.xz"
 check_image test.img
diff --git a/tests/fsck-tests/012-leaf-corruption/test.sh 
b/tests/fsck-tests/012-leaf-corruption/test.sh
index a308727..43b0e6d 100755
--- a/tests/fsck-tests/012-leaf-corruption/test.sh
+++ b/tests/fsck-tests/012-leaf-corruption/test.sh
@@ -1,6 +1,6 @@
 #!/bin/bash
 
-source $TOP/tests/common
+source "$TOP/tests/common"
 
 check_prereq btrfs-image
 
@@ -37,16 +37,16 @@ leaf_no_data_ext_list=(
 generate_leaf_corrupt_no_data_ext()
 {
dest=$1
-   echo "generating leaf_corrupt_no_data_ext.btrfs-image" >> $RESULTS
+   echo "generating leaf_corrupt_no_data_ext.btrfs-image" >> "$RESULTS"
tar --no-same-owner -xJf ./no_data_extent.tar.xz || \
_fail "failed to extract leaf_corrupt_no_data_ext.btrfs-image"
-   $TOP/btrfs-image -r test.img.btrfs-image $dest || \
+   "$TOP/btrfs-image" -r test.img.btrfs-image "$dest" || \
_fail "failed to extract leaf_corrupt_no_data_ext.btrfs-image"
 
# leaf at 4206592 and 20905984 contains no regular data
# extent, clear its csum to corrupt the leaf.
for x in 4206592 20905984; do
-   dd if=/dev/zero of=$dest bs=1 count=32 conv=notrunc seek=$x \
+   dd if=/dev/zero of="$dest" bs=1 count=32 conv=notrunc seek="$x" 
\
1>/dev/null 2>&1
done
 }
@@ -60,21 +60,21 @@ check_inode()
name=$5
 
# Check whether the inode exists
-   exists=$($SUDO_HELPER find $path -inum $ino)
+   exists=$($SUDO_HELPER find "$path" -inum "$ino")
if [ -z "$exists" ]; then
_fail "inode $ino not recovered correctly"
fi
 
# Check inode type
-   found_mode=$(printf "%o" 0x$($SUDO_HELPER stat $exists -c %f))
-   if [ $found_mode -ne $mode ]; then
+   found_mode=$(printf "%o" 0x$($SUDO_HELPER stat "$exists" -c %f))
+   if [ "$found_mode" -ne "$mode" ]; then
echo "$found_mode"
_fail "inode $ino modes not recovered"
fi
 
# Check inode size
-   found_size=$($SUDO_HELPER stat $exists -c %s)
-   if [ $mode -ne 41700 -a $found_size -ne $size ]; then
+   found_size=$($SUDO_HELPER stat "$exists" -c %s)
+   if [ $mode -ne 41700 -a "$found_size" -ne "$size" ]; then
_fail "inode $ino size not recovered correctly"
fi
 
@@ -90,11 +90,11 @@ check_inode()
 check_leaf_corrupt_no_data_ext()
 {
image=$1
-   $SUDO_HELPER mount -o loop $image -o ro $TEST_MNT
+   $SUDO_HELPER mount -o loop "$image" -o ro "$TEST_MNT"
 
i=0
while [ $i -lt ${#leaf_no_data_ext_list[@]} ]; do
-   check_inode $TEST_MNT/lost+found \
+   check_inode "$TEST_MNT/lost+found" \
${leaf_no_data_ext_list[i]} \
${leaf_no_data_ext_list[i + 1]}

[PATCH] btrfs-progs: tests: add variable quotation to convert-tests

[Lakshmipathi.G]

Signed-off-by: Lakshmipathi.G <lakshmipath...@giis.co.in>
---
 tests/convert-tests.sh |  2 +-
 tests/convert-tests/001-ext2-basic/test.sh |  4 ++--
 tests/convert-tests/002-ext3-basic/test.sh |  4 ++--
 tests/convert-tests/003-ext4-basic/test.sh |  4 ++--
 .../004-ext2-backup-superblock-ranges/test.sh  | 22 +++---
 .../convert-tests/005-delete-all-rollback/test.sh  |  6 +++---
 tests/convert-tests/006-large-hole-extent/test.sh  |  6 +++---
 .../007-unsupported-block-sizes/test.sh|  8 
 tests/convert-tests/008-readonly-image/test.sh |  8 
 tests/convert-tests/009-common-inode-flags/test.sh | 14 +++---
 10 files changed, 39 insertions(+), 39 deletions(-)

diff --git a/tests/convert-tests.sh b/tests/convert-tests.sh
index c566336..819e7e2 100755
--- a/tests/convert-tests.sh
+++ b/tests/convert-tests.sh
@@ -37,7 +37,7 @@ run_one_test() {
./test.sh
if [ $? -ne 0 ]; then
_fail "test failed for case $testname"
-   if [[ $TEST_LOG =~ dump ]]; then
+   if [[ "$TEST_LOG" =~ dump ]]; then
cat "$RESULTS"
fi
fi
diff --git a/tests/convert-tests/001-ext2-basic/test.sh 
b/tests/convert-tests/001-ext2-basic/test.sh
index 8f4f935..07f1d6e 100755
--- a/tests/convert-tests/001-ext2-basic/test.sh
+++ b/tests/convert-tests/001-ext2-basic/test.sh
@@ -1,7 +1,7 @@
 #!/bin/bash
 
-source $TOP/tests/common
-source $TOP/tests/common.convert
+source "$TOP/tests/common"
+source "$TOP/tests/common.convert"
 
 setup_root_helper
 prepare_test_dev 512M
diff --git a/tests/convert-tests/002-ext3-basic/test.sh 
b/tests/convert-tests/002-ext3-basic/test.sh
index aeb111e..be8a929 100755
--- a/tests/convert-tests/002-ext3-basic/test.sh
+++ b/tests/convert-tests/002-ext3-basic/test.sh
@@ -1,7 +1,7 @@
 #!/bin/bash
 
-source $TOP/tests/common
-source $TOP/tests/common.convert
+source "$TOP/tests/common"
+source "$TOP/tests/common.convert"
 
 setup_root_helper
 prepare_test_dev 512M
diff --git a/tests/convert-tests/003-ext4-basic/test.sh 
b/tests/convert-tests/003-ext4-basic/test.sh
index 531c81b..44ee592 100755
--- a/tests/convert-tests/003-ext4-basic/test.sh
+++ b/tests/convert-tests/003-ext4-basic/test.sh
@@ -1,7 +1,7 @@
 #!/bin/bash
 
-source $TOP/tests/common
-source $TOP/tests/common.convert
+source "$TOP/tests/common"
+source "$TOP/tests/common.convert"
 
 setup_root_helper
 prepare_test_dev 512M
diff --git a/tests/convert-tests/004-ext2-backup-superblock-ranges/test.sh 
b/tests/convert-tests/004-ext2-backup-superblock-ranges/test.sh
index c56650b..0ce62f7 100755
--- a/tests/convert-tests/004-ext2-backup-superblock-ranges/test.sh
+++ b/tests/convert-tests/004-ext2-backup-superblock-ranges/test.sh
@@ -10,7 +10,7 @@
 # 4) Overlap file extents
 # 5) Unable to rollback
 
-source $TOP/tests/common
+source "$TOP/tests/common"
 
 check_prereq btrfs-convert
 check_prereq btrfs
@@ -23,20 +23,20 @@ prepare_test_dev 512M
 # override common function
 function check_image() {
TEST_DEV="$1"
-   run_check e2fsck -n -f $TEST_DEV
-   run_check $TOP/btrfs-convert $TEST_DEV
-   run_check $TOP/btrfs check $TEST_DEV
-   run_check $TOP/btrfs inspect-internal dump-super $TEST_DEV
+   run_check e2fsck -n -f "$TEST_DEV"
+   run_check "$TOP/btrfs-convert" "$TEST_DEV"
+   run_check "$TOP/btrfs" check "$TEST_DEV"
+   run_check "$TOP/btrfs" inspect-internal dump-super "$TEST_DEV"
 
run_check_mount_test_dev
-   run_check $SUDO_HELPER e2fsck -n -f $TEST_MNT/ext2_saved/image
-   run_check $SUDO_HELPER umount $TEST_MNT
+   run_check $SUDO_HELPER e2fsck -n -f "$TEST_MNT/ext2_saved/image"
+   run_check $SUDO_HELPER umount "$TEST_MNT"
 
-   run_check $TOP/btrfs check $TEST_DEV
-   run_check $TOP/btrfs-convert -r $TEST_DEV
-   run_check e2fsck -n -f $TEST_DEV
+   run_check "$TOP/btrfs" check "$TEST_DEV"
+   run_check "$TOP/btrfs-convert" -r "$TEST_DEV"
+   run_check e2fsck -n -f "$TEST_DEV"
 
-   rm -f $TEST_DEV
+   rm -f "$TEST_DEV"
 }
 
 check_all_images
diff --git a/tests/convert-tests/005-delete-all-rollback/test.sh 
b/tests/convert-tests/005-delete-all-rollback/test.sh
index cf576e7..c6f7466 100755
--- a/tests/convert-tests/005-delete-all-rollback/test.sh
+++ b/tests/convert-tests/005-delete-all-rollback/test.sh
@@ -2,8 +2,8 @@
 # create a base image, convert to btrfs, remove all files, rollback the ext4 
image
 # note: ext4 only
 
-source $TOP/tests/common
-source $TOP/tests/common.convert
+source "$TOP/tests/common"
+source "

Re: [PATCH 9/9] btrfs-progs: modify: Introduce option to specify the pattern to fill mirror

[Lakshmipathi.G]

Thanks for the example and details. I understood some and need to
re-read couple of more times to understand the remaining.

btw, I created a corruption framework(with previous org), the sample
usage and example is below. It looks similar to Btrfs corruption tool.
thanks.

--
corrupt.py --help
Usage: corrupt.py [options]...]

Options:
  --version show program's version number and exit
  -h, --helpshow this help message and exit
  -t object, --object=object
The object type to be corrupted [ choose from (inode,

l_inner,l_master,l_leaf,fileinode_inner,fileinode_leaf,dirinode_in
ner,dirinode_leaf,data_block,fec_block,baddr)
default: inode]
  -p /fs/path, --path=/fs/path
The file or directory path in /fs
  -q optype, --optype=optype
The corruption operation on the object [choose from
(set,xor,add) default: set]
  -o offset, --offset=offset
The valid offset value of the object [default:0]
  -m mirror, --mirror=mirror
The mirror number [default: 1]
  -s size, --size=size  The offset field/byte to be corrupted [default: 1]
  -c fixcrc, --fixcrc=fixcrc
This fixes the CRC's of the corrupted objects
  -n lbn, --lbn=lbn The lbn of the data block to be corrupted
  -b baddr, --baddr=baddr
The baddr to be corrupted
  --log=LOG_FILEThis logs the result into the log file

Example usage:
corrupt.py --object=inode --offset=0 --size=4 --path=/fs/path/to/file.txt
Will corrupt first 4 bytes of inode structure.

corrupt.py --object=dirinode_inner --offset=0 --size=4
--path=/fs/path/to/file.txt
Will corrupt first 4 bytes of directory-inode structure.

corrupt.py --object=dirinode_leaf --offset=0 --size=4
--path=/fs/path/to/file.txt
Will corrupt first 4 bytes of directory-inode leaf structure.
(file.txt should be
large enough to contain leafs).

corrupt.py --object=inode --offset=0 --size=4
--path=/fs/path/to/file.txt  --mirror=2
Will corrupt first 4 bytes of inode structure that reside on Node/drive-2.

--optype=optype ==> default is set. It sets '0' to specified offset.
add => Will try given value to existing value. For example.
offset=(nlink's offset) --optype=add will increase the nlink count by
1.
(running fsck, expected to find this and fix it)

where as optype xor => should flip the bits at given offset.
--


Cheers,
Lakshmipathi.G
FOSS Programmer.
http://www.giis.co.in http://www.webminal.org

On Thu, Apr 20, 2017 at 6:30 AM, Qu Wenruo <quwen...@cn.fujitsu.com> wrote:
>
>
>
> At 04/19/2017 03:46 PM, Lakshmipathi.G wrote:
>>
>> Thanks. This should make easier to create corruption test scripts.
>>
>> when using --root-ino-offset 5,257,0  Does final entry 0 has to be default 
>> value ? Does it represent any value at
>> this moment?
>
>
> For case 5,257,0, it means the start point of the corruption is root 5, inode 
> 257 and file offset 0.
>
> So let's take the following file layout as an example (a little complex one, 
> to show off all the ability)
>
> Root 5 inode 257:
>
>\/ This is file offset
> Extent 0:   disk bytenr X, disk len 4K, nr_bytes 4K offset 0  plain
> Extent 4K:  disk bytenr Y, disk len 8K, nr_bytes 4K offset 0  plain
> Extent 8K:  disk bytenr Z, disk len 8K, nr_bytes 4K offset 4K plain
>
> And if you specified --root-ino-offset 5,257,0 and --length 12288
>
> Then the following on-disk range will be corrupted:
> [X, X+4K)
> [Y, Y+4K)
> [Z+4K, Z+8K)
>
> Making all the 0~12K of root5, inode257 get corrupted, while unrelated ranges 
> like [Z, Z+4K) and [Y+4K, Y+8K) not modified.
>
> While for RAID1/RAID10/DUP usage, just create a file on a new fs, then pass 
> --root-ino-offset 5,257,0 with --length  will corrupt the whole 
> file for you, allowing us to test recovery.
>
>>
>> One more thing, does btrfs-modify command has option to print out existing 
>> parity value (all parity of a file or specific a parity)?
>> something like get_parity option?
>
>
> 1) For print parity value
> I didn't see the need to print parity value.
> As if you want to verify if the parity is good or not, just use offline scrub.
> It will detect parity/data corruption.
>
> 2) For printing/corrupting value of a file in RAID56 profile
> Not as easy as one may think.
>
> Think about the following case:
>
> Root 5 ino 257:
> Extent 0: disk bytenr X(rounded to stripe boundary), disk len 32K, nr_bytes 
> 32K, offset 0.
> Extent 1: disk bytenr X+32K, disk len 64K, nr_bytes 64K, offset 0
>
> On disk it will be:
>
> |Dev  1   |   Dev   2|   Dev 3  |
> X X+64K  X+128K X+192K
>

Re: [PATCH 9/9] btrfs-progs: modify: Introduce option to specify the pattern to fill mirror

[Lakshmipathi.G]

Nice. With this new option, it should be possible to achieve something
like RAID56 corruption test-script say,
https://patchwork.kernel.org/patch/9583455/  right?

Cheers,
Lakshmipathi.G
FOSS Programmer.
http://www.giis.co.in http://www.webminal.org


On Mon, Apr 17, 2017 at 8:56 AM, Qu Wenruo <quwen...@cn.fujitsu.com> wrote:
> Introduce a new option, --pattern, to specify the pattern to fill
> mirror.
>
> Signed-off-by: Qu Wenruo <quwen...@cn.fujitsu.com>
> ---
>  Documentation/btrfs-modify.asciidoc |  5 +
>  modify/mirror.c | 11 ++-
>  2 files changed, 15 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/btrfs-modify.asciidoc 
> b/Documentation/btrfs-modify.asciidoc
> index ae2ada65..d82fb0df 100644
> --- a/Documentation/btrfs-modify.asciidoc
> +++ b/Documentation/btrfs-modify.asciidoc
> @@ -41,6 +41,11 @@ stripe number 1.
>  P for RAID56 1st parity stripe. Only RAID5 and RAID6 support stripe number P.
>  Q for RAID6 2nd parity stripe.
>
> +--pattern 
> +Specify to pattern to modify the mirror data5f536652.
> ++
> +Default value is "0x66524842" (ascii "BHRf", part of the btrfs magic 
> "_BHRfS_M").
> +
>  EXIT STATUS
>  ---
>  *btrfs-modify* returns a zero exit status if all its operations succeed.
> diff --git a/modify/mirror.c b/modify/mirror.c
> index c89927f6..dd717fbe 100644
> --- a/modify/mirror.c
> +++ b/modify/mirror.c
> @@ -373,13 +373,15 @@ int modify_mirror(int argc, char **argv)
> char *device;
> u64 length = (u64)-1;
> u64 logical = (u64)-1;
> +   u32 pattern = 0x66524842;
> int stripe = STRIPE_UNINITILIZED;
> int ret;
>
> while (1) {
> int c;
> enum { GETOPT_VAL_LOGICAL = 257, GETOPT_VAL_LENGTH,
> -   GETOPT_VAL_STRIPE, GETOPT_VAL_ROOT_INO_OFFSET };
> +   GETOPT_VAL_STRIPE, GETOPT_VAL_ROOT_INO_OFFSET,
> +   GETOPT_VAL_PATTERN };
> static const struct option long_options[] = {
> { "logical", required_argument, NULL,
> GETOPT_VAL_LOGICAL },
> @@ -388,6 +390,8 @@ int modify_mirror(int argc, char **argv)
> { "stripe", required_argument, NULL, 
> GETOPT_VAL_STRIPE },
> { "root-ino-offset", required_argument, NULL,
> GETOPT_VAL_ROOT_INO_OFFSET},
> +   { "pattern", required_argument, NULL,
> +   GETOPT_VAL_PATTERN},
> { NULL, 0, NULL, 0 }
> };
>
> @@ -407,6 +411,9 @@ int modify_mirror(int argc, char **argv)
> case GETOPT_VAL_ROOT_INO_OFFSET:
> parse_root_ino_offset(, optarg);
> break;
> +   case GETOPT_VAL_PATTERN:
> +   pattern = arg_strtou32(optarg);
> +   break;
> case '?':
> case 'h':
> usage(modify_mirror_usage);
> @@ -439,6 +446,8 @@ int modify_mirror(int argc, char **argv)
> stripe = 0;
> }
>
> +   memset(write_buf, pattern, sizeof(write_buf) / sizeof(pattern));
> +
> fs_info = open_ctree_fs_info(device, 0, 0, 0, OPEN_CTREE_WRITES);
> if (!fs_info) {
> error("failed to open btrfs on device %s\n", device);
> --
> 2.12.2
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs-progs: cli-tests: Convert non-raid filesystem to raid

[Lakshmipathi.G]

On Mon, Mar 13, 2017 at 06:47:50PM +0100, David Sterba wrote:
> On Sun, Jan 29, 2017 at 08:14:55PM +0530, Lakshmipathi.G wrote:
> > Simple script to verify non-raid filesystem conversion.
> 
> The cli (command line interface) tests are supposed to cover the common
> usecases from the point of option combinations etc, not really verifying
> the result. It would be good if you add more than just one simple test,
> especially for the balance filters. Adding the helpers to prepare the
> filesystem (number of devices, raid profiles) would be a good start.
> Some code for that is in the mkfs tests, but this belongs to the common
> scripts so we could easily use that.

Ok. I'll try to add helpers to common. So that option like this can
be re-used easily.

Cheers.
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3] btrfs-progs: misc-tests: Superblock corruption and recovery using backup.

[Lakshmipathi.G]

On Mon, Mar 13, 2017 at 06:15:04PM +0100, David Sterba wrote:
> On Fri, Feb 17, 2017 at 04:01:59AM +0530, Lakshmipathi.G wrote:
> > Test script to recover damaged primary superblock using backup superblock.
> > 
> > Signed-off-by: Lakshmipathi.G <lakshmipath...@giis.co.in>
> 
> Thanks for the test, a few comments below.
> 
> > ---
> >  .../019-fix-superblock-corruption/test.sh  | 36 
> > ++
> >  1 file changed, 36 insertions(+)
> >  create mode 100755 tests/misc-tests/019-fix-superblock-corruption/test.sh
> > 
> > diff --git a/tests/misc-tests/019-fix-superblock-corruption/test.sh 
> > b/tests/misc-tests/019-fix-superblock-corruption/test.sh
> > new file mode 100755
> > index 000..95815e4
> > --- /dev/null
> > +++ b/tests/misc-tests/019-fix-superblock-corruption/test.sh
> > @@ -0,0 +1,36 @@
> > +#!/bin/bash
> > +#
> > +# Corrupt primary superblock and restore it using backup superblock.
> > +#
> > +
> > +source $TOP/tests/common
> > +
> > +check_prereq btrfs-select-super
> > +check_prereq btrfs
> > +
> > +setup_root_helper
> > +prepare_test_dev 512M
> > +
> > +FIRST_SUPERBLOCK_OFFSET=65536
> > +
> > +test_superblock_restore()
> > +{
> > +   run_check $SUDO_HELPER $TOP/mkfs.btrfs -f $TEST_DEV
> > +
> > +   # Corrupt superblock checksum
> > +dd if=/dev/zero of=$TEST_DEV seek=$FIRST_SUPERBLOCK_OFFSET bs=1 \
> > +count=4  conv=notrunc &> /dev/null
> > +   # Run btrfs check to detect corruption
> > +   $TOP/btrfs check $TEST_DEV >& /dev/null && \
> > +   _fail "btrfs check should detect corruption"
> 
> No run_check? In general all test output is desired and should be in the
> test log so you can use 'run_mayfail'. Also, please keep the entier
> output and don't redirect it to /dev/null .
> 
> > +   # Copy backup superblock to primary
> > +   run_check $TOP/btrfs-select-super -s 1 $TEST_DEV
> > +
> > +   echo "Performing btrfs check" &>> $RESULTS
> 
> Use _log for that
> 
> > +   $TOP/btrfs check $TEST_DEV &>> $RESULTS
> 
> Use run_check
> 
> > +if [ $? -ne 0 ]; then
> > +   _fail "Failed to fix superblock."
> 
> And this would be unnecessary.
> 
> I've added a section to tests/README.md to describe the common coding
> practices. Feel free to read/update/fix. Older tests could violate the
> recommendations so at least new tests are in line and we'll update the
> rest incrementally.

Just sent a new patch(v4), addressing above review comments.thanks.

Cheers.
Lakshmipathi.G


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v4] btrfs-progs: misc-tests: Superblock corruption and recovery using backup

[Lakshmipathi.G]

Signed-off-by: Lakshmipathi.G <lakshmipath...@giis.co.in>
---
 .../020-fix-superblock-corruption/test.sh  | 35 ++
 1 file changed, 35 insertions(+)
 create mode 100755 tests/misc-tests/020-fix-superblock-corruption/test.sh

diff --git a/tests/misc-tests/020-fix-superblock-corruption/test.sh 
b/tests/misc-tests/020-fix-superblock-corruption/test.sh
new file mode 100755
index 000..0faebad
--- /dev/null
+++ b/tests/misc-tests/020-fix-superblock-corruption/test.sh
@@ -0,0 +1,35 @@
+#!/bin/bash
+#
+# Corrupt primary superblock and restore it using backup superblock.
+#
+
+source $TOP/tests/common
+
+check_prereq btrfs-select-super
+check_prereq btrfs
+
+setup_root_helper
+prepare_test_dev
+
+FIRST_SUPERBLOCK_OFFSET=65536
+
+test_superblock_restore()
+{
+   run_check $SUDO_HELPER "$TOP/mkfs.btrfs" -f "$TEST_DEV"
+
+   # Corrupt superblock checksum
+run_check $SUDO_HELPER dd if=/dev/zero of="$TEST_DEV" \
+   seek="$FIRST_SUPERBLOCK_OFFSET" bs=1 count=4 conv=notrunc
+
+   # Run btrfs check to detect corruption
+   run_mayfail "$TOP/btrfs" check "$TEST_DEV" && \
+   _fail "btrfs check should detect corruption"
+
+   # Copy backup superblock to primary
+   run_check "$TOP/btrfs-select-super" -s 1 "$TEST_DEV"
+
+   #Performing btrfs check
+   run_check "$TOP/btrfs" check "$TEST_DEV"
+}
+
+test_superblock_restore
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v4] btrfs-progs: fsck-tests: verify 'btrfs check --repair' fixes corrupted nlink field

[Lakshmipathi.G]

commit 5cdcc57b8d599f03fe692e0285ee9e17f32267fa
Author: Lakshmipathi.G <lakshmipath...@giis.co.in>
Date:   Sat Apr 15 14:26:20 2017 +0530

btrfs-progs: fsck-tests: verify 'btrfs check --repair' fixes corrupted 
nlink field

Signed-off-by: Lakshmipathi.G <lakshmipath...@giis.co.in>

diff --git a/tests/fsck-tests/026-check-inode-link/test.sh 
b/tests/fsck-tests/026-check-inode-link/test.sh
new file mode 100755
index 000..3a7a3c1
--- /dev/null
+++ b/tests/fsck-tests/026-check-inode-link/test.sh
@@ -0,0 +1,38 @@
+#!/bin/bash
+# verify that 'btrfs check --repair' fixes corrupted inode nlink field
+
+source $TOP/tests/common
+
+check_prereq btrfs-corrupt-block
+check_prereq mkfs.btrfs
+
+setup_root_helper
+prepare_test_dev
+
+test_inode_nlink_field()
+{
+   run_check $SUDO_HELPER "$TOP/mkfs.btrfs" -f "$TEST_DEV"
+
+   run_check_mount_test_dev
+   run_check $SUDO_HELPER touch "$TEST_MNT/test_nlink.txt"
+
+   # find inode_number
+   inode_number=`stat -c%i "$TEST_MNT/test_nlink.txt"`
+   run_check_umount_test_dev
+
+   # corrupt nlink field of inode object
+run_check $SUDO_HELPER "$TOP/btrfs-corrupt-block" -i "$inode_number" \
+   -f nlink "$TEST_DEV"
+
+   check_image "$TEST_DEV"
+
+   # verify link count
+   run_check_mount_test_dev
+   link_count=`stat -c%h "$TEST_MNT/test_nlink.txt"`
+   run_check_umount_test_dev
+   if [ "$link_count" != 1 ];then
+   _fail "btrfs check --repair: Unable to fix nlink corruption."
+   fi
+}
+
+test_inode_nlink_field
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3] btrfs-progs: fsck-tests: verify 'btrfs check --repair' fixes corrupted nlink field

[Lakshmipathi.G]

On Thu, Mar 30, 2017 at 05:52:11PM +0200, David Sterba wrote:
> On Thu, Feb 02, 2017 at 01:35:02PM +0530, Lakshmipathi.G wrote:
> > Signed-off-by: Lakshmipathi.G <lakshmipath...@giis.co.in>
> > ---
> >  tests/fsck-tests/026-check-inode-link/test.sh | 30 
> > +++
> >  1 file changed, 30 insertions(+)
> >  create mode 100755 tests/fsck-tests/026-check-inode-link/test.sh
> > 
> > diff --git a/tests/fsck-tests/026-check-inode-link/test.sh 
> > b/tests/fsck-tests/026-check-inode-link/test.sh
> > new file mode 100755
> > index 000..6822ee2
> > --- /dev/null
> > +++ b/tests/fsck-tests/026-check-inode-link/test.sh
> > @@ -0,0 +1,30 @@
> > +#!/bin/bash
> > +# verify that 'btrfs check --repair' fixes corrupted inode nlink field
> > +
> > +source $TOP/tests/common
> > +
> > +check_prereq btrfs-corrupt-block
> > +check_prereq mkfs.btrfs
> > +
> > +setup_root_helper
> > +prepare_test_dev 512M
> 
> Please use default size unless you really need 512M for the test
> purposes.
> 
> > +
> > +test_inode_nlink_field()
> > +{
> > +   run_check $SUDO_HELPER $TOP/mkfs.btrfs -f $TEST_DEV
> > +
> > +   run_check_mount_test_dev
> > +   run_check $SUDO_HELPER touch $TEST_MNT/test_nlink.txt
> > +
> > +   # find inode_number
> > +   inode_number=`stat -c%i $TEST_MNT/test_nlink.txt`
> > +   run_check_umount_test_dev
> > +
> > +   # corrupt nlink field of inode object
> > +run_check $SUDO_HELPER $TOP/btrfs-corrupt-block -i $inode_number \
> > +   -f nlink $TEST_DEV
> > +
> > +   check_image $TEST_DEV
> 
> The wrapper runs a pre-check that must detect errors, then runs --repair
> and then again a plain check. Is this intended here?
> 
Yes its intentional. The test should fail, if its unable to detect the 
corruption.
This is to ensure btrfs-corrupt-block -f nlink works as expected.
> I think we should try to mount the fixed image again and read the file
> or check link count.
> 
Right, I missed the verification part, will added this and change 
prepare_test_dev 
to default size.
> Please add shell quotation around all variables.
Now quotation is added.

Cheers.
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: List of corruption cases for scrub

[Lakshmipathi.G]

On Mon, Feb 27, 2017 at 11:40:31AM +0800, Qu Wenruo wrote:
> 
> 
> At 02/24/2017 10:32 AM, Lakshmipathi.G wrote:
> >Hi.
> >
> >I tried to a create list of corruption test scenarios for scrubbing process 
> >with RAID5.
> >Here's the list: 
> >https://btrfs.wiki.kernel.org/index.php/Scrub_corruption_cases
> >
> >If similar list already exists in btrfs wiki, let me know, I'll move this 
> >data
> >to there. Feel free to add/update/delete entries or suggest new corrupt 
> >ideas!
> 
> What about adding testing result for latest kernel release?
> 
> Thanks,
> Qu

True, Ideally, the wiki should have list-corruption cases along with result for
each latest kernel version.

Planning to add more cases, gradually. For example, I think inode object
with RAID5 can be replicated across drives. If thats correct, can scrub 
fix those corruption too? cases like this may be more important than other
to-do items like raid5 with features like snapshot/quota etc

I think it should be possible to add test-results for few cases but list has
repeated-cases with different sizes,drives etc for these cases we need some
kind of automated scripts.

Hope to extend the cases first and then start updating results with manual or 
automated tests.

Cheers.
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: List of corruption cases for scrub

[Lakshmipathi.G]

On Fri, Feb 24, 2017 at 06:39:20PM +0100, Goffredo Baroncelli wrote:
> Hi,
> On 2017-02-24 03:32, Lakshmipathi.G wrote:
> > Hi.
> > 
> > I tried to a create list of corruption test scenarios for scrubbing process 
> > with RAID5.
> > Here's the list: 
> > https://btrfs.wiki.kernel.org/index.php/Scrub_corruption_cases
> > 
> > If similar list already exists in btrfs wiki, let me know, I'll move this 
> > data
> > to there. Feel free to add/update/delete entries or suggest new corrupt 
> > ideas!
> 
> Thanks for doing that. About the test "Device-corruption", I suppose that you 
> want to test a scenario where one disk fails. If so you do not have to always 
> corrupt "D0" block, but the first block of the file which is stored in the 
> failing disk (which could be different from the D0).
> 
Yes, that's right. Updated (Device-corruption) the list now.

> Another scenario which should be tested, is the reshaping of the raid: add a 
> drive (or remove a drive) then re-balance the system (with or without some 
> corruption). IIRC this use the same scrub code.
> 
Added this as 'Rebalance-corruption' on the wiki. thanks!

Cheers.
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

List of corruption cases for scrub

[Lakshmipathi.G]

Hi.

I tried to a create list of corruption test scenarios for scrubbing process 
with RAID5.
Here's the list: https://btrfs.wiki.kernel.org/index.php/Scrub_corruption_cases

If similar list already exists in btrfs wiki, let me know, I'll move this data
to there. Feel free to add/update/delete entries or suggest new corrupt ideas!

Cheers.
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Large no.of extents for a single small file with a fresh RAID5 setup

[Lakshmipathi.G]

> 
> >I'm using Ubuntu(16.04) desktop version (not server) running Xorg and 
> >others. May be its possible its
> >flushing data to disk at constant time. If you like to give it a try again 
> >with servers, below is the
> >exact script along with its timing & multiextent output.
> 
> Archlinux here, with my btrfs vm and browser and music player, still one
> extent.
> 
looks like I have better setup to produce single-extent files than yours :-)
> Are you under heavy memory usage?
> 
no, not as per top and free. 'top' shows mem.usage as:
KiB Mem :  7092516 total,  5961796 free,   794836 used,   335884 buff/cache

where 'compiz' process uses around 2.5% of memory and 'free' command reports
almost 5gb as free. May be some 'sysctl' or others needs to be adjusted on my 
setup?
> >
> >one more curious thing, is it fine to have extents with size 4096 on RAID5 
> >setup?
> >
> 
> Sure, why not?
> 
> Any size aligned to sector size is valid, no matter the profile.
> 
> For 4K extent, RAID5/6 will do read-modify-write, which is very common.
> 
> Thanks,
> Qu
> 
While looking into mapping from logical->physical chunks on RAID5. I was 
wondering 
how these 4KB extents will fit into 64KB data-stripe?

I didn't have idea about read-modify-write, just found few wiki links on it, 
will 
check. thanks

Cheers.
Lakshmipathi.G

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs-progs: RAID5:Inject data stripe corruption and verify scrub fixes it.

[Lakshmipathi.G]

> >
> >Looked into patch description:
> >
> >After scrubbing dev3 only:
> >0xcdcd (Good)  |  0xcdcd  | 0xcdcd (Bad)
> >(D1)  (D2)(P)
> >
> >So the Parity stripe (P) always get replaced by exact content of D1/D2 
> >(data-stripe)
> >or by random  data?
> 
> Neither. it's just XOR result of D2(never changed, 0xcdcd) and old D1(wrong,
> 0x).
> 0xcdcd XOR 0x = 0xcdcd
> 
> So you got 0xcdcd, bad result.
> 
> If you corrupt D1 with random data, then parity will be random then.
> 
> >If it always  get replaced by exact value from either
> >D1 or D2.  I think current script can be modified to detect that bug. If 
> >parity gets
> >replaced by random value, then it will the make task more difficult.
> 
> Not hard to detect.
> As the content is completely under your control, you know the correct parity
> value, and you can verify it very easy.
> 

version-3 of this script calculates exact data/parity location, instead of 
dumping data 
and searching locations. Tested with upto 8MB file, from the output all 128 
data-stripes 
and 64 parity stripe location seems fine. It constantly hit the parity bug with 
the script.


If the script gets accepted, will add slightly other corruption variants likes:
- corrupt all even stripe (D2,D4..)
- corrupt all odd stripe  (D1,D3..)
- corrupt all parity stripes
- corrupt all both data stripe (D0 & D1) and expect error message
(Cover above cases for RAID6)

thanks.

Cheers.
Lakshmipathi.G

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Large no.of extents for a single small file with a fresh RAID5 setup

[Lakshmipathi.G]

> >>How did you create the 2m file?
> >
> >Yes,this is the problem. script invokes 'dd' with 'notrunc' for each
> >byte. If the file is 1kb, 'dd' invoked 1024 times with notrunc.
> >This seems to be creating the issue.
> 
> IIRC that's not the direct cause though.
> 
> Since kernel is using delayed allocation, each time we trigger buffered
> write, kernel just info fs to do accounting and copy the data into page
> cache, no real write is triggered.
> 
> Only sync/fsync and memory pressure will make us to write pages into disc,
> and until then we allocate space for them.
> 
> So unless you're trying such operation on a busy server which triggered
> several sync/fsync/memory pressure during the several seconds of dd, it
> won't cause so many fragments.
> 
> I also tried your dd bs=1 method, no problem and still one single extent.
> 
> Thanks,
> Qu
I'm using Ubuntu(16.04) desktop version (not server) running Xorg and others. 
May be its possible its
flushing data to disk at constant time. If you like to give it a try again with 
servers, below is the
exact script along with its timing & multiextent output.

one more curious thing, is it fine to have extents with size 4096 on RAID5 
setup?

Cheers.
Lakshmipathi.G

#time /root/check_extents.sh f2MB 32
real1m30.284s
user1m22.796s
sys 0m39.324s

laks/btrfs-progs# ./btrfs-debugfs -f tests/mnt/f2MB
(263 0): ram 110592 disk 151015424 disk_size 114688
(263 110592): ram 4096 disk 147103744 disk_size 4096
(263 114688): ram 110592 disk 151130112 disk_size 114688
(263 225280): ram 696320 disk 151257088 disk_size 700416
(263 921600): ram 4096 disk 147107840 disk_size 4096
(263 925696): ram 700416 disk 152043520 disk_size 704512
(263 1626112): ram 4096 disk 147111936 disk_size 4096
(263 1630208): ram 344064 disk 153223168 disk_size 348160
(263 1974272): ram 4096 disk 147116032 disk_size 4096
(263 1978368): ram 118784 disk 152748032 disk_size 118784
file: tests/mnt/f2MB extents 10 disk size 2117632 logical size 2097152 ratio 
0.99

--
$cat /root/check_extents.sh 
#!/bin/bash
#$1 Filename
#$2 Expected no.of data stripes for the file.

create_layout(){
fname=$1
size=$(( $2 * 65536 ))
n=0
bs_value=1
stripe=0
while (( $n < $size ))
do
if [ $(( $n % 65536 )) -eq 0 ]; then
val='D'$stripe
echo -n $val
stripe=$(( $stripe+1 ))
# ensure proper value   
bs_value=`echo "${#val}"` 
else
echo -n 'x'
bs_value=1
fi
n=$(( $n+$bs_value ))
done | dd of="/home/laks/btrfs-progs/tests/mnt/$fname" bs=$bs_value 
conv=notrunc &> /dev/null
##EDIT above hard-coded path
}
create_layout $1 $2
--
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v3] btrfs-progs: RAID5: Inject data stripe corruption and verify scrub fixes it and retains correct parity.

[Lakshmipathi.G]

Test script to create file with specific data-stripe layout.Computes stripe 
locations.
Inject corruption into data-stripe and verify scrubbing process fixes corrupted 
block.
It also validates the corresponding parity stripe.

Signed-off-by: Lakshmipathi.G <lakshmipath...@giis.co.in>
---
 .../test.sh| 402 +
 1 file changed, 402 insertions(+)
 create mode 100755 
tests/misc-tests/020-raid5-datastripe-corruption-parity-check/test.sh

diff --git 
a/tests/misc-tests/020-raid5-datastripe-corruption-parity-check/test.sh 
b/tests/misc-tests/020-raid5-datastripe-corruption-parity-check/test.sh
new file mode 100755
index 000..0129a75
--- /dev/null
+++ b/tests/misc-tests/020-raid5-datastripe-corruption-parity-check/test.sh
@@ -0,0 +1,402 @@
+#!/bin/bash
+#
+# Raid5: Inject data stripe corruption and fix them using scrub.
+# 
+# Script will perform the following:
+# 1) Create Raid5 using 3 loopback devices.
+# 2) Ensure file layout is created in a predictable manner. 
+#Each data stripe(64KB) should uniquely start with 'DN',   
+#where N represents the data stripe number.(ex:D0,D1 etc)
+# 3) Once file is created with specific layout, check whether file 
+#has single extent. At the moment, script wont handle multi-extent
+#files.
+# 4) If file has single extent with the help of 'dump-tree' compute data and 
+#parity stripe details like devicename, position and actual on-disk data.
+# 5) $STRIPEINFO_COMPLETE file will have all necessary data at this stage.
+# 6) Inject corruption into given data-stripe by zero'ing out its first 4 
bytes.
+# 7) After injecting corruption, running online-scrub is expected to fix 
+#the corrupted data stripe with the help of parity block and 
+#corresponding data stripe. 
+# 8) If scrub successful, verify the data stripe has original un-corrupted 
value.
+# 9) If scrub successful, verify parity stripe is valid, otherwise its a 
parity bug.
+# 10) If no issues found, cleanup files and devices. Repeat the process for 
+#different file size and data-stripe.
+#
+#  Note: This script corrupts only data-stripe blocks. 
+#  Known Limitations (will be addressed later):
+# - Script expects even number of data-stripes in file.
+# - Script expects the file to have single extent.
+
+source $TOP/tests/common
+
+check_prereq btrfs
+check_prereq mkfs.btrfs
+check_prereq btrfs-debugfs
+
+setup_root_helper
+prepare_test_dev 1024M
+
+ndevs=3
+declare -a devs
+declare -a parity_offsets
+stripe_entry=""
+device_name=""
+stripe_offset=""
+stripe_content=""
+
+#Complete stripe layout
+STRIPEINFO_COMPLETE=$(mktemp --tmpdir 
btrfs-progs-raid5-stripe-complete.infoXX)
+#dump-tree output file
+DUMPTREE_OUTPUT=$(mktemp --tmpdir btrfs-progs-raid5-tree-dump.infoXX)
+#tmp files
+STRIPEINFO_PARTIAL=$(mktemp --tmpdir 
btrfs-progs-raid5-stripe-partial.infoXX)
+STRIPE_TMP=$(mktemp --tmpdir btrfs-progs-raid5-stripetmp.infoXX)
+MULTI_EXTENT_CHECK=$(mktemp --tmpdir btrfs-progs-raid5-extent-check.infoXX)
+EXTENT_WITH_SIZE=$(mktemp --tmpdir btrfs-progs-raid5-extent-size.infoXX)
+PARITY_LOC1=$(mktemp --tmpdir btrfs-progs-raid5-parity-loc1.infoXX)
+PARITY_LOCATION=$(mktemp --tmpdir 
btrfs-progs-raid5-parity-locations.infoXX)
+
+
+prepare_devices()
+{
+   for i in `seq $ndevs`; do
+   touch img$i
+   chmod a+rw img$i
+   truncate -s0 img$i
+   truncate -s512M img$i
+   devs[$i]=`run_check_stdout $SUDO_HELPER losetup --find --show 
img$i`
+   done
+   truncate -s0 $STRIPE_TMP
+   truncate -s0 $STRIPEINFO_PARTIAL
+   truncate -s0 $STRIPEINFO_COMPLETE
+}
+
+cleanup_devices()
+{
+   for dev in ${devs[@]}; do
+   run_check $SUDO_HELPER losetup -d $dev
+   done
+   for i in `seq $ndevs`; do
+   truncate -s0 img$i
+   done
+   run_check $SUDO_HELPER losetup --all
+}
+
+test_do_mkfs()
+{
+   run_check $SUDO_HELPER $TOP/mkfs.btrfs -f   \
+   $@
+}
+
+test_mkfs_multi()
+{
+   test_do_mkfs $@ ${devs[@]}
+}
+
+#$1 Filename
+#$2 Expected no.of data stripes for the file.
+create_layout(){
+   fname=$1
+   size=$(( $2 * 65536 ))
+   n=0
+   bs_value=1
+   stripe=0
+   while (( $n < $size ))
+   do
+   if [ $(( $n % 65536 )) -eq 0 ]; then
+   val='D'$stripe
+   echo -n $val
+   stripe=$(( $stripe+1 ))
+   # ensure proper value   
+   bs_value=`echo "${#val}"` 
+   else
+   echo -n 'x'
+   bs_value=1
+   fi
+n=$(( $n+$bs_value ))
+   done | dd of=/tmp/$fname bs=$bs_value conv=notrunc &> /dev/null
+   cp /tmp/$fname $TEST_MNT
+   rm -f /tmp/$fna

Re: Large no.of extents for a single small file with a fresh RAID5 setup

[Lakshmipathi.G]

On Mon, Feb 20, 2017 at 03:53:15PM +0800, Qu Wenruo wrote:
> 
> 
> At 02/18/2017 10:07 AM, Lakshmipathi.G wrote:
> >Hi,
> >
> >With RAID5, using 3 loop-back devices each has 4GB. On a fresh setup, 
> >created a 2mb file.
> >While checking no.of extents,it shows like:
> >
> ># ./btrfs-debugfs -f /mnt/file2m.txt
> >(257 0): ram 110592 disk 145227776 disk_size 114688
> >(257 110592): ram 4096 disk 145461248 disk_size 4096
> >(257 114688): ram 114688 disk 145342464 disk_size 118784
> >(257 229376): ram 712704 disk 145620992 disk_size 716800
> >(257 942080): ram 4096 disk 145096704 disk_size 4096
> >(257 946176): ram 737280 disk 146407424 disk_size 741376
> >(257 1683456): ram 413696 disk 147193856 disk_size 413696
> >file: /mnt/file2m.txt extents 7 disk size 2113536 logical size 2097152 ratio 
> >0.99
> >
> >Is this expected behaviour? Is there any mount option to minimize no.of 
> >extents for
> >a smaller file?
> 
> What mount option did you use?
> Are you using compression?

Used the default mount options (mount /dev/loop0 /mnt) and compression not 
enabled.

> How did you create the 2m file?

Yes,this is the problem. script invokes 'dd' with 'notrunc' for each
byte. If the file is 1kb, 'dd' invoked 1024 times with notrunc.
This seems to be creating the issue.

Tried using open-fd (exec FD<>/path/f, write finally close FD) method
even this runs into possible sync and creates multiple extents.

so finally found a simple trick, created the file outside btrfs then copy it
into btrfs-mnt-pt :) this workaround helps to solve the script file-creation
issue.

> 
> This seems quite strange, if you go through normal buffered write without
> compression, 2m can easily be embedded into one file extent.
> 
> And according to extent at offset 0, 112K, 224K, its data extent is larger
> than its file extent, which means it goes through CoW.
> 
> Either you did fsync/sync between writes or you goes through Direct IO.
> 
> Anyway, the way you create the 2M file is quite important here now.
> 
> Thanks,
> Qu
> >

thanks for the help.

Cheers.
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Large no.of extents for a single small file with a fresh RAID5 setup

[Lakshmipathi.G]

Hi,

With RAID5, using 3 loop-back devices each has 4GB. On a fresh setup, created a 
2mb file.
While checking no.of extents,it shows like:

# ./btrfs-debugfs -f /mnt/file2m.txt 
(257 0): ram 110592 disk 145227776 disk_size 114688
(257 110592): ram 4096 disk 145461248 disk_size 4096
(257 114688): ram 114688 disk 145342464 disk_size 118784
(257 229376): ram 712704 disk 145620992 disk_size 716800
(257 942080): ram 4096 disk 145096704 disk_size 4096
(257 946176): ram 737280 disk 146407424 disk_size 741376
(257 1683456): ram 413696 disk 147193856 disk_size 413696
file: /mnt/file2m.txt extents 7 disk size 2113536 logical size 2097152 ratio 
0.99

Is this expected behaviour? Is there any mount option to minimize no.of extents 
for
a smaller file?

Cheers.
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2] btrfs-progs: RAID5:Inject data stripe corruption and verify scrub fixes it.

[Lakshmipathi.G]

Test script to create file with specific data-stripe layout and pre-computed 
parity values.
Inject corruption into data-stripe and verify scrubbing process fixes corrupted 
block.
It also attempts to validate the corresponding parity block value.

Signed-off-by: Lakshmipathi.G <lakshmipath...@giis.co.in>
---
 .../020-raid5-datastripe-corruption/test.sh| 275 +
 1 file changed, 275 insertions(+)
 create mode 100755 tests/misc-tests/020-raid5-datastripe-corruption/test.sh

diff --git a/tests/misc-tests/020-raid5-datastripe-corruption/test.sh 
b/tests/misc-tests/020-raid5-datastripe-corruption/test.sh
new file mode 100755
index 000..3c743eb
--- /dev/null
+++ b/tests/misc-tests/020-raid5-datastripe-corruption/test.sh
@@ -0,0 +1,275 @@
+#!/bin/bash
+#
+# Raid5: Inject data stripe corruption and fix them using scrub.
+# 
+# Script will perform the following:
+# 1) Create Raid5 using 3 loopback devices.
+# 2) Ensure file layout is created in a predictable manner. 
+#Each data stripe(64KB) should uniquely start with 'DN',   
+#where N represents the data stripe number.(ex:D0,D1 etc)
+# 3) Once file is created with specific layout, gather data stripe details 
+#like devicename, position and actual on-disk data.
+# 4) Now use 'dd' to verify the data-stripe against its expected value.
+# 5) At this stage, gather possible parity location for this specific data
+#stripe. The parity values is pre-computed for known data-stripes.
+# 6) Inject corruption into given data-stripe by zero'ing out its first 4 
bytes.
+# 7) After injecting corruption, running online-scrub is expected to fix 
+#the corrupted data stripe with the help of parity block and 
+#corresponding data stripe. 
+# 8) If scrub successful, verify the data stripe has original un-corrupted 
value.
+# 9) If scrub successful, re-generate list of possible parity blocks, this
+#list must match one prepared in step 5. Otherwise report it as possible 
parity 
+#bug issue.
+# 10) If no issues found, cleanup files and devices. Repeat the process for 
+#different file size and data-stripe.
+#
+#  Note: This script doesn't handle parity block corruption.
+
+source $TOP/tests/common
+
+check_prereq btrfs
+check_prereq mkfs.btrfs
+
+setup_root_helper
+prepare_test_dev 512M
+
+ndevs=3
+declare -a devs
+device_name=""
+stripe_offset=""
+stripe_content=""
+known_parity_value="0001"
+
+LAYOUT_TMP=$(mktemp --tmpdir btrfs-progs-raid5-file.layoutXX)
+STRIPEINFO_TMP=$(mktemp --tmpdir btrfs-progs-raid5-file.infoXX)
+PARITYINFO1_TMP=$(mktemp --tmpdir btrfs-progs-raid5-initial.parityXX)
+PARITYINFO2_TMP=$(mktemp --tmpdir btrfs-progs-raid5-end.parityXX)
+
+prepare_devices()
+{
+   for i in `seq $ndevs`; do
+   touch img$i
+   chmod a+rw img$i
+   truncate -s0 img$i
+   truncate -s512M img$i
+   devs[$i]=`run_check_stdout $SUDO_HELPER losetup --find --show 
img$i`
+   done
+}
+
+cleanup_devices()
+{
+   for dev in ${devs[@]}; do
+   run_check $SUDO_HELPER losetup -d $dev
+   done
+   for i in `seq $ndevs`; do
+   truncate -s0 img$i
+   done
+   run_check $SUDO_HELPER losetup --all
+}
+
+test_do_mkfs()
+{
+   run_check $SUDO_HELPER $TOP/mkfs.btrfs -f   \
+   $@
+}
+
+test_mkfs_multi()
+{
+   test_do_mkfs $@ ${devs[@]}
+}
+
+#$1 Filename
+#$2 Expected no.of data stripes for the file.
+create_layout(){
+   fname=$1
+   size=$(( $2 * 65536 ))
+   n=0
+   bs_value=1
+   stripe=0
+   while (( $n < $size ))
+   do
+   if [ $(( $n % 65536 )) -eq 0 ]; then
+   val='D'$stripe
+   echo -n $val
+   stripe=$(( $stripe+1 ))
+   # ensure proper value   
+   bs_value=`echo "${#val}"` 
+   else
+   echo -n 'x'
+   bs_value=1
+   fi
+n=$(( $n+$bs_value ))
+   done | dd of="$TEST_MNT"/$fname bs=$bs_value conv=notrunc &> /dev/null
+}
+
+find_data_stripe_details(){
+   for dev in ${devs[@]}; do
+   echo $dev >> $LAYOUT_TMP
+   $SUDO_HELPER cat $dev | hexdump -e '"%010_ad|" 16/1 "%_p" 
"|\n"' |
+   grep -P 'D[0-9]+xx'  >> $LAYOUT_TMP
+   done
+}
+#$1 - file to save possible parity locations
+find_parity_stripe_count(){
+   PARITY_COUNT=$1
+   for dev in ${devs[@]}; do
+   echo "Possible parity stripes locations on device: $dev" >> 
$PARITY_COUNT
+   $SUDO_HELPER cat $dev | hexdump -e '"%010_ad|" 16/1 "%02X" 
"\n"' |
+   grep $known_parity_value >> $PARITY_

Re: [PATCH v2] btrfs-progs: misc-tests: Superblock corruption and recovery using backup.

[Lakshmipathi.G]

On Thu, Feb 16, 2017 at 09:05:02AM +0800, Qu Wenruo wrote:
> 
> 
> At 02/16/2017 04:50 AM, Lakshmipathi.G wrote:
> >Signed-off-by: Lakshmipathi.G <lakshmipath...@giis.co.in>
> 
> Looks good to me.
> 
> Reviewed by: Qu Wenruo <quwen...@cn.fujitsu.com>
> 
> BTW, it would be better to have some commit message.

ok, sent the v3 patch with commit message.

Cheers.
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v3] btrfs-progs: misc-tests: Superblock corruption and recovery using backup.

[Lakshmipathi.G]

Test script to recover damaged primary superblock using backup superblock.

Signed-off-by: Lakshmipathi.G <lakshmipath...@giis.co.in>
---
 .../019-fix-superblock-corruption/test.sh  | 36 ++
 1 file changed, 36 insertions(+)
 create mode 100755 tests/misc-tests/019-fix-superblock-corruption/test.sh

diff --git a/tests/misc-tests/019-fix-superblock-corruption/test.sh 
b/tests/misc-tests/019-fix-superblock-corruption/test.sh
new file mode 100755
index 000..95815e4
--- /dev/null
+++ b/tests/misc-tests/019-fix-superblock-corruption/test.sh
@@ -0,0 +1,36 @@
+#!/bin/bash
+#
+# Corrupt primary superblock and restore it using backup superblock.
+#
+
+source $TOP/tests/common
+
+check_prereq btrfs-select-super
+check_prereq btrfs
+
+setup_root_helper
+prepare_test_dev 512M
+
+FIRST_SUPERBLOCK_OFFSET=65536
+
+test_superblock_restore()
+{
+   run_check $SUDO_HELPER $TOP/mkfs.btrfs -f $TEST_DEV
+
+   # Corrupt superblock checksum
+dd if=/dev/zero of=$TEST_DEV seek=$FIRST_SUPERBLOCK_OFFSET bs=1 \
+count=4  conv=notrunc &> /dev/null
+   # Run btrfs check to detect corruption
+   $TOP/btrfs check $TEST_DEV >& /dev/null && \
+   _fail "btrfs check should detect corruption"
+   # Copy backup superblock to primary
+   run_check $TOP/btrfs-select-super -s 1 $TEST_DEV
+
+   echo "Performing btrfs check" &>> $RESULTS
+   $TOP/btrfs check $TEST_DEV &>> $RESULTS
+if [ $? -ne 0 ]; then
+   _fail "Failed to fix superblock."
+fi
+}
+
+test_superblock_restore
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs-progs: RAID5:Inject data stripe corruption and verify scrub fixes it.

[Lakshmipathi.G]

On Thu, Feb 16, 2017 at 09:12:31AM +0800, Qu Wenruo wrote:
> 
> 
> At 02/16/2017 04:56 AM, Lakshmipathi.G wrote:
> >On Wed, Feb 15, 2017 at 05:29:33PM +0800, Qu Wenruo wrote:
> >>
> >>
> >>At 02/15/2017 05:03 PM, Lakshmipathi.G wrote:
> >>>Signed-off-by: Lakshmipathi.G <lakshmipath...@giis.co.in>
> >>>---
> >>>.../020-raid5-datastripe-corruption/test.sh| 224 
> >>>+
> >>>1 file changed, 224 insertions(+)
> >>>create mode 100755 tests/misc-tests/020-raid5-datastripe-corruption/test.sh
> >>>
> >>>diff --git a/tests/misc-tests/020-raid5-datastripe-corruption/test.sh 
> >>>b/tests/misc-tests/020-raid5-datastripe-corruption/test.sh
> >>>new file mode 100755
> >>>index 000..d04c430
> >>>--- /dev/null
> >>>+++ b/tests/misc-tests/020-raid5-datastripe-corruption/test.sh
> >>>@@ -0,0 +1,224 @@
> >>>+#!/bin/bash
> >>>+#
> >>>+# Raid5: Inject data stripe corruption and fix them using scrub.
> >>>+#
> >>>+# Script will perform the following:
> >>>+# 1) Create Raid5 using 3 loopback devices.
> >>>+# 2) Ensure file layout is created in a predictable manner.
> >>>+#Each data stripe(64KB) should uniquely start with 'DN',
> >>>+#where N represents the data stripe number.(ex:D0,D1 etc)
> >>
> >>If you want really predictable layout, you could just upload compressed
> >>images for this purpose.
> >>
> >>Which makes things super easy, and unlike fstests, btrfs-progs self-test
> >>accepts such images.
> >>
> >>>+# 3) Once file is created with specific layout, gather data stripe details
> >>>+#like devicename, position and actual on-disk data.
> >>>+# 4) Now use 'dd' to verify the data-stripe against its expected value
> >>>+#and inject corruption by zero'ing out contents.
> >>>+# 5) After injecting corruption, running online-scrub is expected to fix
> >>>+#the corrupted data stripe with the help of parity block and
> >>>+#corresponding data stripe.
> >>
> >>You should also verify parity stripe is not corrupted.
> >>It's already known that RAID5/6 will corrupted parity while recovering data
> >>stripe.
> >>
> >>Kernel patch for this, with detailed bug info.
> >>https://patchwork.kernel.org/patch/9553581/
> >>
> >>>+# 6) Finally, validate the data stripe has original un-corrupted value.
> >>>+#
> >>>+#  Note: This script doesn't handle parity block corruption.
> >>
> >>Normally such test case should belong to xfstests (renamed to fstests
> >>recently) as we're verifying kernel behavior, not btrfs-progs behavior.
> >>
> >>But since fstests test case should be as generic as possible, and we don't
> >>have a good enough tool to corrupt given data/parity stripe, my previously
> >>submitted test case is rejected.
> >>
> >>Personally speaking, this seems to be a dilemma for me.
> >>
> >>We really need a test case for this, bugs has been spotted that RAID5/6
> >>scrub will corrupt P/Q while recovering data stripe.
> >>But we need to enhance btrfs-corrupt-block to a better shape to make fstests
> >>to accept it, and it won't take a short time.
> >>
> >>So I really have no idea what should we do for such test.
> >>
> >>Thanks,
> >>Qu
> >
> >Will check compressed images for parity strpe testing. I assume at the 
> >moment,
> >we currently support single static compressed image. Adding more than one 
> >static
> >compressed images like disk1.img disk2.img disk3.img for RAID is supported in
> >existing test framework?
> 
> Not yet, but since you can use test.sh instead of running check_image() from
> test frameset, it's never a big problem.
> 
ok, will check it out.
> >
> >Using compressed images for checking parity seems little easier than 
> >computing
> >via scripting.
> >
> >Looked into patch description:
> >
> >After scrubbing dev3 only:
> >0xcdcd (Good)  |  0xcdcd  | 0xcdcd (Bad)
> >(D1)  (D2)(P)
> >
> >So the Parity stripe (P) always get replaced by exact content of D1/D2 
> >(data-stripe)
> >or by random  data?
> 
> Neither. it's just XOR result of D2(never changed, 0xcdcd) and old D1(wrong,
> 0x).
> 0xcdcd XOR 0x = 0xcdcd
> 
> So y

Re: How to dump/find parity of RAID-5 file?

[Lakshmipathi.G]

On Wed, Feb 15, 2017 at 07:24:55PM +0100, Goffredo Baroncelli wrote:
> The chunk-tree maps the logical address [145096704...145096704+134217728) 
> [size=128MB] to the physical ones 
>   devid3 : [6368..6368+67108864) [size=64MB]
>   devid1 : [6368..6368+67108864) [size=64MB]
>   devid2 : [83034112..83034112+67108864) [size=64MB]
> 
> So because the logical address is divided in pieces of 64k, interleaved by 
> the parity, we know that:
> * first 128kb
> logical address [145096704  ..145096704+64k)   -> devid1, [6368
> ..6368+64k)
> logical address [145096704+64k  ..145096704+2x64k) -> devid2, [83034112
> ..83034112+64k)
> parity:-> devid3, [6368
> ..6368+64k)
> * second 128kb
> logical address [145096704+2x64k..145096704+3x64k) -> devid2, 
> [83034112+64k..83034112+2x64k)
> logical address [145096704+3x64k..145096704+4x64k) -> devid3, 
> [6368+64k..6368+2x64k)
> parity:-> devid1, 
> [6368+64k..6368+2x64k)
> And so on...
> 
> (NB: 145096704+2x64k == 145227776)
> 
> The fs-tree, maps the file content [0..131072) [size=128k] to the logical 
> address [145227776..145227776+131072) [size=128k]
> 
> So the file content is stored starting from the disk devid2, at 
> 83034112+64k=83099648 (first 64k). The second 64k is placed in disk devid3 at 
> 6368+64k=63176704; the parity is stored at disk1, 6368+64k = 63176704
> 
> 
> BR
> G.Baroncelli
> 

Thanks for the detailed example with exact numbers. Now understood the address 
mapping better. With this as a reference, I think
it should be possible to access parity/data-stripes more sensible manner 
instead using expensive "cat /device/ | hexdump | grep"
combination. thanks.

Cheers.
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs-progs: RAID5:Inject data stripe corruption and verify scrub fixes it.

[Lakshmipathi.G]

On Wed, Feb 15, 2017 at 05:29:33PM +0800, Qu Wenruo wrote:
> 
> 
> At 02/15/2017 05:03 PM, Lakshmipathi.G wrote:
> >Signed-off-by: Lakshmipathi.G <lakshmipath...@giis.co.in>
> >---
> > .../020-raid5-datastripe-corruption/test.sh| 224 
> > +
> > 1 file changed, 224 insertions(+)
> > create mode 100755 tests/misc-tests/020-raid5-datastripe-corruption/test.sh
> >
> >diff --git a/tests/misc-tests/020-raid5-datastripe-corruption/test.sh 
> >b/tests/misc-tests/020-raid5-datastripe-corruption/test.sh
> >new file mode 100755
> >index 000..d04c430
> >--- /dev/null
> >+++ b/tests/misc-tests/020-raid5-datastripe-corruption/test.sh
> >@@ -0,0 +1,224 @@
> >+#!/bin/bash
> >+#
> >+# Raid5: Inject data stripe corruption and fix them using scrub.
> >+#
> >+# Script will perform the following:
> >+# 1) Create Raid5 using 3 loopback devices.
> >+# 2) Ensure file layout is created in a predictable manner.
> >+#Each data stripe(64KB) should uniquely start with 'DN',
> >+#where N represents the data stripe number.(ex:D0,D1 etc)
> 
> If you want really predictable layout, you could just upload compressed
> images for this purpose.
> 
> Which makes things super easy, and unlike fstests, btrfs-progs self-test
> accepts such images.
> 
> >+# 3) Once file is created with specific layout, gather data stripe details
> >+#like devicename, position and actual on-disk data.
> >+# 4) Now use 'dd' to verify the data-stripe against its expected value
> >+#and inject corruption by zero'ing out contents.
> >+# 5) After injecting corruption, running online-scrub is expected to fix
> >+#the corrupted data stripe with the help of parity block and
> >+#corresponding data stripe.
> 
> You should also verify parity stripe is not corrupted.
> It's already known that RAID5/6 will corrupted parity while recovering data
> stripe.
> 
> Kernel patch for this, with detailed bug info.
> https://patchwork.kernel.org/patch/9553581/
> 
> >+# 6) Finally, validate the data stripe has original un-corrupted value.
> >+#
> >+#  Note: This script doesn't handle parity block corruption.
> 
> Normally such test case should belong to xfstests (renamed to fstests
> recently) as we're verifying kernel behavior, not btrfs-progs behavior.
> 
> But since fstests test case should be as generic as possible, and we don't
> have a good enough tool to corrupt given data/parity stripe, my previously
> submitted test case is rejected.
> 
> Personally speaking, this seems to be a dilemma for me.
> 
> We really need a test case for this, bugs has been spotted that RAID5/6
> scrub will corrupt P/Q while recovering data stripe.
> But we need to enhance btrfs-corrupt-block to a better shape to make fstests
> to accept it, and it won't take a short time.
> 
> So I really have no idea what should we do for such test.
> 
> Thanks,
> Qu

Will check compressed images for parity strpe testing. I assume at the moment,
we currently support single static compressed image. Adding more than one static
compressed images like disk1.img disk2.img disk3.img for RAID is supported in
existing test framework?

Using compressed images for checking parity seems little easier than computing
via scripting.

Looked into patch description:

After scrubbing dev3 only:
0xcdcd (Good)  |  0xcdcd  | 0xcdcd (Bad) 
(D1)  (D2)(P) 

So the Parity stripe (P) always get replaced by exact content of D1/D2 
(data-stripe)
or by random  data? If it always  get replaced by exact value from either
D1 or D2.  I think current script can be modified to detect that bug. If parity 
gets
replaced by random value, then it will the make task more difficult.

Yes, without better support for RAID with tools like btrfs-corrupt-block, it 
will be
hard to play-around with RAID to create test scripts.

Cheers.
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html