Re: Unable to open ctree

2016-10-02 Thread Chris Murphy 
On Sun, Oct 2, 2016 at 12:17 PM, Leonidas Spyropoulos
 wrote:
> Hi all,
>
> I'm a everyday user of btrfs. The system is a laptop with UEFI partition
> and the following partition scheme:
> # blkid
> /dev/sda1: UUID="." TYPE="vfat" PARTUUID="."
> /dev/sda2: UUID="." TYPE="swap" PARTUUID="."
> /dev/sda3: LABEL="root" UUID="." UUID_SUB="." TYPE="btrfs" PARTUUID="."
> /dev/sda4: LABEL="home" UUID="." UUID_SUB="." TYPE="btrfs" PARTUUID="."
>
> The partitions are surely not out of space:
> # btrfs fi show
> Label: 'root'  uuid: c7a773e0-e2be-4482-acdc-dbfa4c13c98c
> Total devices 1 FS bytes used 9.98GiB
> devid1 size 74.52GiB used 13.03GiB path /dev/sda3
>
> Label: 'home'  uuid: 63f5219f-399a-4cce-be86-a8c7400853bd
> Total devices 1 FS bytes used 17.65GiB
> devid1 size 188.87GiB used 25.03GiB path /dev/sda4
>
> When the "root" partition is to be mounted during boot up it fails and I
> am dropped on an emergency shell. The dmesg output is:
>
> [  716.434137] BTRFS info (device sda3): disk space caching is enabled
> [  716.434147] BTRFS info (device sda3): has skinny extents
> [  716.436538] BTRFS info (device sda3): bdev /dev/sda3 errs: wr 0, rd
> 0, flush 0, corrupt 32, gen 0
> [  716.439182] BTRFS warning (device sda3): sda3 checksum verify failed
> on 11185160192 wanted 2670F0AB found FDB2F5BD level 0
> [  716.439195] BTRFS error (device sda3): failed to read block groups:
> -5
> [  716.481402] BTRFS: open_ctree failed
>
> The btrfs check /dev/sdc3 output is https://ptpb.pw/nc1d
>
> dmesg output: https://ptpb.pw/G0Wv
>
> The kernel I was running is right before this failure was 4.7.5 (with ck
> patchset custom compiled) and there was not any kernel panics - just a
> restart. Currently runnign through latest Archlinux liveCD
>
> # uname -a
> Linux archiso 4.7.5-1-ARCH #1 SMP PREEMPT Sat Sep 24 13:04:22 CEST 2016
> x86_64 GNU/Linux
>
> # btrfs --version
> btrfs-progs v4.7.3
>
> As this is the root partition I am not able to access it without a
> liveCD. Can I perform any kind of steps to recaclulate checksums?

Well short of a bug, the problem aren't the checksums. The problem is
the metadata is wrong, so if you recalculate checksums you're likely
end up with an even more corrupted file system because it'll start out
trusting bad metadata.

If you're prepared to lose this filesystem, use --repair and see if it
can fix the metadata problems despite csum failures. If it were me,
I'd take a btrfs-image before --repair. In theory, if it makes things
worse you can restore the image, or donate the image to making the
btrfsck better.

If --repair doesn't work, try -b --repair.

If that doesn't work then I'd probably use --init-extent-tree which,
while it's a heavy hammer, at least still isn't going to pretend bad
metadata is good which is what --init-csum-tree will end up doing.

But before all of that I'm curious what you get for:

btrfs-debug-tree -b 11185160192 /dev/sda3
btrfs-find-root /dev/sda3





-- 
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unable to open ctree

2016-10-02 Thread Leonidas Spyropoulos 
Hi Roman,


On 03/10/16, Roman Mamedov wrote:
> On Sun, 2 Oct 2016 13:29:56 -0600
> Chris Murphy  wrote:
> 
> > Well short of a bug, the problem aren't the checksums. The problem is
> > the metadata is wrong, so if you recalculate checksums you're likely
> > end up with an even more corrupted file system because it'll start out
> > trusting bad metadata.
> 
> It appears that metadata got created with profile "single", because the device
> is SSD. If this was DUP metadata, this entire problem wouldn't happen.
> It is a terrible idea to downgrade metadata to single on detecting SSDs at
> mkfs. The original rationale was that "SSDs will deduplicate it anyways", but
> there are many ways things can corrupt way before reaching the SSD (from the
> point of view of which it will look like the computer sent two different
> metadata blocks, if one got corrupted in flight), and secondly, the ability of
> SSDs to perfectly deduplicate small 4K sized pieces of data at hundreds of
> megabytes in read/write speeds is VASTLY overestimated here.

I agree that the wear in SSDs due to metadata dublications is
overestimated and since my partition appears to be screwed I will make
sure next time to force it to have another copy there.

My progress from the repairs which both failed are: https://ptpb.pw/CtbC

Next step is initialize the extent tree
btrfs check --init-extent-tree --repair /dev/sda3

Thanks,

-- 
Leonidas Spyropoulos

A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Unable to open ctree

2016-10-02 Thread Leonidas Spyropoulos 
Hi all,

I'm a everyday user of btrfs. The system is a laptop with UEFI partition
and the following partition scheme:
# blkid 
/dev/sda1: UUID="." TYPE="vfat" PARTUUID="."
/dev/sda2: UUID="." TYPE="swap" PARTUUID="."
/dev/sda3: LABEL="root" UUID="." UUID_SUB="." TYPE="btrfs" PARTUUID="."
/dev/sda4: LABEL="home" UUID="." UUID_SUB="." TYPE="btrfs" PARTUUID="."

The partitions are surely not out of space:
# btrfs fi show
Label: 'root'  uuid: c7a773e0-e2be-4482-acdc-dbfa4c13c98c
Total devices 1 FS bytes used 9.98GiB
devid1 size 74.52GiB used 13.03GiB path /dev/sda3

Label: 'home'  uuid: 63f5219f-399a-4cce-be86-a8c7400853bd
Total devices 1 FS bytes used 17.65GiB
devid1 size 188.87GiB used 25.03GiB path /dev/sda4

When the "root" partition is to be mounted during boot up it fails and I
am dropped on an emergency shell. The dmesg output is:

[  716.434137] BTRFS info (device sda3): disk space caching is enabled
[  716.434147] BTRFS info (device sda3): has skinny extents
[  716.436538] BTRFS info (device sda3): bdev /dev/sda3 errs: wr 0, rd
0, flush 0, corrupt 32, gen 0
[  716.439182] BTRFS warning (device sda3): sda3 checksum verify failed
on 11185160192 wanted 2670F0AB found FDB2F5BD level 0
[  716.439195] BTRFS error (device sda3): failed to read block groups:
-5
[  716.481402] BTRFS: open_ctree failed

The btrfs check /dev/sdc3 output is https://ptpb.pw/nc1d

dmesg output: https://ptpb.pw/G0Wv

The kernel I was running is right before this failure was 4.7.5 (with ck
patchset custom compiled) and there was not any kernel panics - just a
restart. Currently runnign through latest Archlinux liveCD

# uname -a
Linux archiso 4.7.5-1-ARCH #1 SMP PREEMPT Sat Sep 24 13:04:22 CEST 2016
x86_64 GNU/Linux

# btrfs --version
btrfs-progs v4.7.3

As this is the root partition I am not able to access it without a
liveCD. Can I perform any kind of steps to recaclulate checksums?

Thanks

-- 
Leonidas Spyropoulos

A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unable to open ctree

2016-10-02 Thread Chris Murphy 
On Sun, Oct 2, 2016 at 2:22 PM, Roman Mamedov  wrote:
> On Sun, 2 Oct 2016 13:29:56 -0600
> Chris Murphy  wrote:
>
>> Well short of a bug, the problem aren't the checksums. The problem is
>> the metadata is wrong, so if you recalculate checksums you're likely
>> end up with an even more corrupted file system because it'll start out
>> trusting bad metadata.
>
> It appears that metadata got created with profile "single", because the device
> is SSD. If this was DUP metadata, this entire problem wouldn't happen.
> It is a terrible idea to downgrade metadata to single on detecting SSDs at
> mkfs.

That's uncertain, we've seen file systems with dup also not mount with
csum errors in metadata.



-- 
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unable to open ctree

2016-10-02 Thread Roman Mamedov 
On Sun, 2 Oct 2016 13:29:56 -0600
Chris Murphy  wrote:

> Well short of a bug, the problem aren't the checksums. The problem is
> the metadata is wrong, so if you recalculate checksums you're likely
> end up with an even more corrupted file system because it'll start out
> trusting bad metadata.

It appears that metadata got created with profile "single", because the device
is SSD. If this was DUP metadata, this entire problem wouldn't happen.
It is a terrible idea to downgrade metadata to single on detecting SSDs at
mkfs. The original rationale was that "SSDs will deduplicate it anyways", but
there are many ways things can corrupt way before reaching the SSD (from the
point of view of which it will look like the computer sent two different
metadata blocks, if one got corrupted in flight), and secondly, the ability of
SSDs to perfectly deduplicate small 4K sized pieces of data at hundreds of
megabytes in read/write speeds is VASTLY overestimated here.


-- 
With respect,
Roman


pgpPEC1J40RIX.pgp
Description: OpenPGP digital signature

Re: Unable to open ctree

2016-10-02 Thread Leonidas Spyropoulos 
Hi Chris,

Thanks for your suggestions

On 02/10/16, Chris Murphy wrote:
> Well short of a bug, the problem aren't the checksums. The problem is
> the metadata is wrong, so if you recalculate checksums you're likely
> end up with an even more corrupted file system because it'll start out
> trusting bad metadata.
> 
> If you're prepared to lose this filesystem, use --repair and see if it
> can fix the metadata problems despite csum failures. If it were me,
> I'd take a btrfs-image before --repair. In theory, if it makes things
> worse you can restore the image, or donate the image to making the
> btrfsck better.
> 
> If --repair doesn't work, try -b --repair.
> 
> If that doesn't work then I'd probably use --init-extent-tree which,
> while it's a heavy hammer, at least still isn't going to pretend bad
> metadata is good which is what --init-csum-tree will end up doing.
> 
> But before all of that I'm curious what you get for:
> 
> btrfs-debug-tree -b 11185160192 /dev/sda3
> btrfs-find-root /dev/sda3

The output from btrfs-debug-tree command is: https://ptpb.pw/0weU
The btrfs-find-root: https://ptpb.pw/PIZe
The btrfs-show-super -f : https://ptpb.pw/O4C9

I tried the btrfs-image but failed
# btrfs-image /dev/sda3 /root/sda3-btrfs-image.bin
with output https://ptpb.pw/jtDJ

Unless there's some other idea I will continue with the following:
# btrfs check --repair
# btrfs check -b --repair
# btrfs check --init-extent-tree --repair

and see if I can get something

Thanks,

-- 
Leonidas Spyropoulos

A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unable to open ctree

2016-10-02 Thread Leonidas Spyropoulos 
Hello,

After the command to rebuild the extent tree
# btrfs check --init-extent-tree --repair /dev/sda3

I was able to mount and run successfully a scrub (not sure if it would
detect something though).

Next was reboot and try to boot which was successful as well.

I'll keep an eye on the dmesh log next days but seems it's ok.

Thanks for suggestions all,

Regards,

-- 
Leonidas Spyropoulos

A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V21 01/19] Btrfs: subpage-blocksize: extent_clear_unlock_delalloc: Prevent page from being unlocked more than once

2016-10-02 Thread Chandan Rajendra 
extent_clear_unlock_delalloc() can unlock a page more than once as shown
below (assume 4k as the block size and 64k as the page size).

cow_file_range
  create 4k ordered extent corresponding to page offsets 0 - 4095
  extent_clear_unlock_delalloc corresponding to page offsets 0 - 4095
unlock page
  create 4k ordered extent corresponding to page offsets 4096 - 8191
  extent_clear_unlock_delalloc corresponding to page offsets 4096 - 8191
unlock page

To prevent such a scenario this commit passes "delalloc end" to
extent_clear_unlock_delalloc() to help decide whether the page can be unlocked
or not.

NOTE: Since extent_clear_unlock_delalloc() is used by compression code
as well, the commit passes ordered extent "end" as the value for the
argument corresponding to "delalloc end" for invocations made from
compression code path. This will be fixed by a future commit that gets
compression to work in subpage-blocksize scenario.

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/extent_io.c | 16 +++
 fs/btrfs/extent_io.h |  5 ++--
 fs/btrfs/inode.c | 78 +---
 3 files changed, 57 insertions(+), 42 deletions(-)

diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
index f669240..dc60c604 100644
--- a/fs/btrfs/extent_io.c
+++ b/fs/btrfs/extent_io.c
@@ -1708,9 +1708,8 @@ out_failed:
 }
 
 void extent_clear_unlock_delalloc(struct inode *inode, u64 start, u64 end,
-struct page *locked_page,
-unsigned clear_bits,
-unsigned long page_ops)
+   u64 delalloc_end, struct page *locked_page,
+   unsigned clear_bits, unsigned long page_ops)
 {
struct extent_io_tree *tree = _I(inode)->io_tree;
int ret;
@@ -1718,6 +1717,7 @@ void extent_clear_unlock_delalloc(struct inode *inode, 
u64 start, u64 end,
unsigned long index = start >> PAGE_SHIFT;
unsigned long end_index = end >> PAGE_SHIFT;
unsigned long nr_pages = end_index - index + 1;
+   u64 page_end;
int i;
 
clear_extent_bit(tree, start, end, clear_bits, 1, 0, NULL, GFP_NOFS);
@@ -1748,8 +1748,14 @@ void extent_clear_unlock_delalloc(struct inode *inode, 
u64 start, u64 end,
SetPageError(pages[i]);
if (page_ops & PAGE_END_WRITEBACK)
end_page_writeback(pages[i]);
-   if (page_ops & PAGE_UNLOCK)
-   unlock_page(pages[i]);
+
+   if (page_ops & PAGE_UNLOCK) {
+   page_end = page_offset(pages[i]) +
+   PAGE_SIZE - 1;
+   if ((page_end <= end)
+   || (end == delalloc_end))
+   unlock_page(pages[i]);
+   }
put_page(pages[i]);
}
nr_pages -= ret;
diff --git a/fs/btrfs/extent_io.h b/fs/btrfs/extent_io.h
index 06b6f14..0948bca 100644
--- a/fs/btrfs/extent_io.h
+++ b/fs/btrfs/extent_io.h
@@ -430,9 +430,8 @@ int map_private_extent_buffer(struct extent_buffer *eb, 
unsigned long offset,
 void extent_range_clear_dirty_for_io(struct inode *inode, u64 start, u64 end);
 void extent_range_redirty_for_io(struct inode *inode, u64 start, u64 end);
 void extent_clear_unlock_delalloc(struct inode *inode, u64 start, u64 end,
-struct page *locked_page,
-unsigned bits_to_clear,
-unsigned long page_ops);
+   u64 delalloc_end, struct page *locked_page,
+   unsigned bits_to_clear, unsigned long page_ops);
 struct bio *
 btrfs_bio_alloc(struct block_device *bdev, u64 first_sector, int nr_vecs,
gfp_t gfp_flags);
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 3440b52..3e4feac 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -560,12 +560,13 @@ cont:
 * we don't need to create any more async work items.
 * Unlock and free up our temp pages.
 */
-   extent_clear_unlock_delalloc(inode, start, end, NULL,
-clear_flags, PAGE_UNLOCK |
-PAGE_CLEAR_DIRTY |
-PAGE_SET_WRITEBACK |
-page_error_op |
-PAGE_END_WRITEBACK);
+   extent_clear_unlock_delalloc(inode, start, end, end,
+   NULL, clear_flags,
+

[PATCH V21 03/19] Btrfs: subpage-blocksize: Use PG_Uptodate flag to track block uptodate status

2016-10-02 Thread Chandan Rajendra 
This commit causes a block's uptodate status to be tracked using
struct page's PG_Uptodate flag instead of extent_io_tree's
EXTENT_UPTODATE flag.

This is in preparation for subpage-blocksize patchset which will use a
per-page bitmap for tracking individual block's uptodate status in the
case of blocksize < PAGE_SIZE. We will continue to use PG_Uptodate flag
to track uptodate status for blocksize == PAGE_SIZE scenario.

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/extent_io.c | 61 +++-
 fs/btrfs/extent_io.h |  2 +-
 fs/btrfs/inode.c |  6 ++
 3 files changed, 11 insertions(+), 58 deletions(-)

diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
index dd7faa1..522c943 100644
--- a/fs/btrfs/extent_io.c
+++ b/fs/btrfs/extent_io.c
@@ -1950,12 +1950,9 @@ int test_range_bit(struct extent_io_tree *tree, u64 
start, u64 end,
  * helper function to set a given page up to date if all the
  * extents in the tree for that page are up to date
  */
-static void check_page_uptodate(struct extent_io_tree *tree, struct page *page)
+static void check_page_uptodate(struct page *page)
 {
-   u64 start = page_offset(page);
-   u64 end = start + PAGE_SIZE - 1;
-   if (test_range_bit(tree, start, end, EXTENT_UPTODATE, 1, NULL))
-   SetPageUptodate(page);
+   SetPageUptodate(page);
 }
 
 int free_io_failure(struct extent_io_tree *failure_tree,
@@ -2492,18 +2489,6 @@ static void end_bio_extent_writepage(struct bio *bio)
bio_put(bio);
 }
 
-static void
-endio_readpage_release_extent(struct extent_io_tree *tree, u64 start, u64 len,
- int uptodate)
-{
-   struct extent_state *cached = NULL;
-   u64 end = start + len - 1;
-
-   if (uptodate && tree->track_uptodate)
-   set_extent_uptodate(tree, start, end, , GFP_ATOMIC);
-   unlock_extent_cached(tree, start, end, , GFP_ATOMIC);
-}
-
 /*
  * after a readpage IO is done, we need to:
  * clear the uptodate bits on error
@@ -2525,8 +2510,6 @@ static void end_bio_extent_readpage(struct bio *bio)
u64 start;
u64 end;
u64 len;
-   u64 extent_start = 0;
-   u64 extent_len = 0;
int mirror;
int ret;
int i;
@@ -2612,7 +2595,7 @@ readpage_ok:
off = i_size & (PAGE_SIZE-1);
if (page->index == end_index && off)
zero_user_segment(page, off, PAGE_SIZE);
-   SetPageUptodate(page);
+   check_page_uptodate(page);
} else {
ClearPageUptodate(page);
SetPageError(page);
@@ -2620,32 +2603,10 @@ readpage_ok:
unlock_page(page);
offset += len;
 
-   if (unlikely(!uptodate)) {
-   if (extent_len) {
-   endio_readpage_release_extent(tree,
- extent_start,
- extent_len, 1);
-   extent_start = 0;
-   extent_len = 0;
-   }
-   endio_readpage_release_extent(tree, start,
- end - start + 1, 0);
-   } else if (!extent_len) {
-   extent_start = start;
-   extent_len = end + 1 - start;
-   } else if (extent_start + extent_len == start) {
-   extent_len += end + 1 - start;
-   } else {
-   endio_readpage_release_extent(tree, extent_start,
- extent_len, uptodate);
-   extent_start = start;
-   extent_len = end + 1 - start;
-   }
+   unlock_extent_cached(tree, start, end, NULL, GFP_ATOMIC);
+
}
 
-   if (extent_len)
-   endio_readpage_release_extent(tree, extent_start, extent_len,
- uptodate);
if (io_bio->end_io)
io_bio->end_io(io_bio, bio->bi_error);
bio_put(bio);
@@ -2933,18 +2894,15 @@ static int __do_readpage(struct extent_io_tree *tree,
 
if (cur >= last_byte) {
char *userpage;
-   struct extent_state *cached = NULL;
 
iosize = PAGE_SIZE - pg_offset;
userpage = kmap_atomic(page);
memset(userpage + pg_offset, 0, iosize);
flush_dcache_page(page);
kunmap_atomic(userpage);
-   set_extent_uptodate(tree, cur, cur + iosize - 1,
-   , GFP_NOFS);
unlock_extent_cached(tree, cur,

[PATCH V21 02/19] Btrfs: subpage-blocksize: Make sure delalloc range intersects with the locked page's range

2016-10-02 Thread Chandan Rajendra 
find_delalloc_range indirectly depends on EXTENT_UPTODDATE to make sure that
the delalloc range returned intersects with the file range mapped by the
page. Since we now track "uptodate" state in a per-page
bitmap (i.e. in btrfs_page_private->bstate), this commit makes an explicit
check to make sure that the delalloc range starts from within the file range
mapped by the page.

Reviewed-by: Josef Bacik 
Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/extent_io.c | 12 +---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
index dc60c604..dd7faa1 100644
--- a/fs/btrfs/extent_io.c
+++ b/fs/btrfs/extent_io.c
@@ -1477,6 +1477,7 @@ out:
  * 1 is returned if we find something, 0 if nothing was in the tree
  */
 static noinline u64 find_delalloc_range(struct extent_io_tree *tree,
+   struct page *locked_page,
u64 *start, u64 *end, u64 max_bytes,
struct extent_state **cached_state)
 {
@@ -1485,6 +1486,9 @@ static noinline u64 find_delalloc_range(struct 
extent_io_tree *tree,
u64 cur_start = *start;
u64 found = 0;
u64 total_bytes = 0;
+   u64 page_end;
+
+   page_end = page_offset(locked_page) + PAGE_SIZE - 1;
 
spin_lock(>lock);
 
@@ -1505,7 +1509,8 @@ static noinline u64 find_delalloc_range(struct 
extent_io_tree *tree,
  (state->state & EXTENT_BOUNDARY))) {
goto out;
}
-   if (!(state->state & EXTENT_DELALLOC)) {
+   if (!(state->state & EXTENT_DELALLOC)
+   || (page_end < state->start)) {
if (!found)
*end = state->end;
goto out;
@@ -1643,8 +1648,9 @@ again:
/* step one, find a bunch of delalloc bytes starting at start */
delalloc_start = *start;
delalloc_end = 0;
-   found = find_delalloc_range(tree, _start, _end,
-   max_bytes, _state);
+   found = find_delalloc_range(tree, locked_page,
+   _start, _end,
+   max_bytes, _state);
if (!found || delalloc_end <= *start) {
*start = delalloc_start;
*end = delalloc_end;
-- 
2.5.5

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V21 15/19] Btrfs: subpage-blocksize: Enable dedupe ioctl

2016-10-02 Thread Chandan Rajendra 
The function implementing the dedupe ioctl
i.e. btrfs_ioctl_file_extent_same(), returns with an error in
subpage-blocksize scenario. This was done due to the fact that Btrfs did
not have code to deal with block size < page size. This commit removes
this restriction since we now support "block size < page size".

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/ioctl.c | 10 --
 1 file changed, 10 deletions(-)

diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
index 4077fc1..cf13029 100644
--- a/fs/btrfs/ioctl.c
+++ b/fs/btrfs/ioctl.c
@@ -3321,21 +3321,11 @@ ssize_t btrfs_dedupe_file_range(struct file *src_file, 
u64 loff, u64 olen,
 {
struct inode *src = file_inode(src_file);
struct inode *dst = file_inode(dst_file);
-   u64 bs = BTRFS_I(src)->root->fs_info->sb->s_blocksize;
ssize_t res;
 
if (olen > BTRFS_MAX_DEDUPE_LEN)
olen = BTRFS_MAX_DEDUPE_LEN;
 
-   if (WARN_ON_ONCE(bs < PAGE_SIZE)) {
-   /*
-* Btrfs does not support blocksize < page_size. As a
-* result, btrfs_cmp_data() won't correctly handle
-* this situation without an update.
-*/
-   return -EINVAL;
-   }
-
res = btrfs_extent_same(src, loff, olen, dst, dst_loff);
if (res)
return res;
-- 
2.5.5

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V21 13/19] Btrfs: subpage-blocksize: btrfs_punch_hole: Fix uptodate blocks check

2016-10-02 Thread Chandan Rajendra 
In case of subpage-blocksize, the file blocks to be punched may map only
part of a page. For file blocks inside such pages, we need to check for
the presence of BLK_STATE_UPTODATE flag.

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/file.c | 89 -
 1 file changed, 88 insertions(+), 1 deletion(-)

diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c
index 54602e6..6490e56 100644
--- a/fs/btrfs/file.c
+++ b/fs/btrfs/file.c
@@ -2350,6 +2350,8 @@ static int btrfs_punch_hole(struct inode *inode, loff_t 
offset, loff_t len)
struct btrfs_path *path;
struct btrfs_block_rsv *rsv;
struct btrfs_trans_handle *trans;
+   struct address_space *mapping = inode->i_mapping;
+   pgoff_t start_index, end_index;
u64 lockstart;
u64 lockend;
u64 tail_start;
@@ -2362,6 +2364,7 @@ static int btrfs_punch_hole(struct inode *inode, loff_t 
offset, loff_t len)
int err = 0;
unsigned int rsv_count;
bool same_block;
+   bool same_page;
bool no_holes = btrfs_fs_incompat(root->fs_info, NO_HOLES);
u64 ino_size;
bool truncated_block = false;
@@ -2458,11 +2461,45 @@ static int btrfs_punch_hole(struct inode *inode, loff_t 
offset, loff_t len)
goto out_only_mutex;
}
 
+   start_index = lockstart >> PAGE_SHIFT;
+   end_index = lockend >> PAGE_SHIFT;
+
+   same_page = lockstart >> PAGE_SHIFT
+   == lockend >> PAGE_SHIFT;
+
while (1) {
struct btrfs_ordered_extent *ordered;
+   struct page *start_page = NULL;
+   struct page *end_page = NULL;
+   u64 nr_pages;
+   int start_page_blks_uptodate;
+   int end_page_blks_uptodate;
 
truncate_pagecache_range(inode, lockstart, lockend);
 
+   if (lockstart & (PAGE_SIZE - 1)) {
+   start_page = find_or_create_page(mapping, start_index,
+   GFP_NOFS);
+   if (!start_page) {
+   inode_unlock(inode);
+   return -ENOMEM;
+   }
+   }
+
+   if (!same_page && ((lockend + 1) & (PAGE_SIZE - 1))) {
+   end_page = find_or_create_page(mapping, end_index,
+   GFP_NOFS);
+   if (!end_page) {
+   if (start_page) {
+   unlock_page(start_page);
+   put_page(start_page);
+   }
+   inode_unlock(inode);
+   return -ENOMEM;
+   }
+   }
+
+
lock_extent_bits(_I(inode)->io_tree, lockstart, lockend,
 _state);
ordered = btrfs_lookup_first_ordered_extent(inode, lockend);
@@ -2472,18 +2509,68 @@ static int btrfs_punch_hole(struct inode *inode, loff_t 
offset, loff_t len)
 * and nobody raced in and read a page in this range, if we did
 * we need to try again.
 */
+   nr_pages = round_up(lockend, PAGE_SIZE)
+   - round_down(lockstart, PAGE_SIZE);
+   nr_pages >>= PAGE_SHIFT;
+
+   start_page_blks_uptodate = 0;
+   end_page_blks_uptodate = 0;
+   if (root->sectorsize < PAGE_SIZE) {
+   u64 page_end;
+
+   page_end = round_down(lockstart, PAGE_SIZE)
+   + PAGE_SIZE - 1;
+   page_end = min(page_end, lockend);
+   if (start_page
+   && PagePrivate(start_page)
+   && test_page_blks_state(start_page, 1 << 
BLK_STATE_UPTODATE,
+   lockstart, page_end, 0))
+   start_page_blks_uptodate = 1;
+   if (end_page
+   && PagePrivate(end_page)
+   && test_page_blks_state(end_page, 1 << 
BLK_STATE_UPTODATE,
+   page_offset(end_page), 
lockend, 0))
+   end_page_blks_uptodate = 1;
+   } else {
+   if (start_page && PagePrivate(start_page)
+   && PageUptodate(start_page))
+   start_page_blks_uptodate = 1;
+   if (end_page && PagePrivate(end_page)
+   && PageUptodate(end_page))
+   end_page_blks_uptodate = 1;
+   }
+
if ((!ordered ||

[PATCH V21 12/19] Btrfs: subpage-blocksize: Explicitly track I/O status of blocks of an ordered extent.

2016-10-02 Thread Chandan Rajendra 
In subpage-blocksize scenario a page can have more than one block. So in
addition to PagePrivate2 flag, we would have to track the I/O status of
each block of a page to reliably mark the ordered extent as complete.

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/extent_io.c|  19 +--
 fs/btrfs/extent_io.h|   5 +-
 fs/btrfs/inode.c| 363 ++--
 fs/btrfs/ordered-data.c |  19 +++
 fs/btrfs/ordered-data.h |   4 +
 5 files changed, 294 insertions(+), 116 deletions(-)

diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
index 0832797..df6172c 100644
--- a/fs/btrfs/extent_io.c
+++ b/fs/btrfs/extent_io.c
@@ -4483,11 +4483,10 @@ int extent_invalidatepage(struct extent_io_tree *tree,
  * to drop the page.
  */
 static int try_release_extent_state(struct extent_map_tree *map,
-   struct extent_io_tree *tree,
-   struct page *page, gfp_t mask)
+   struct extent_io_tree *tree,
+   struct page *page, u64 start, u64 end,
+   gfp_t mask)
 {
-   u64 start = page_offset(page);
-   u64 end = start + PAGE_SIZE - 1;
int ret = 1;
 
if (test_range_bit(tree, start, end,
@@ -4521,12 +4520,12 @@ static int try_release_extent_state(struct 
extent_map_tree *map,
  * map records are removed
  */
 int try_release_extent_mapping(struct extent_map_tree *map,
-  struct extent_io_tree *tree, struct page *page,
-  gfp_t mask)
+   struct extent_io_tree *tree, struct page *page,
+   u64 start, u64 end, gfp_t mask)
 {
struct extent_map *em;
-   u64 start = page_offset(page);
-   u64 end = start + PAGE_SIZE - 1;
+   u64 orig_start = start;
+   u64 orig_end = end;
 
if (gfpflags_allow_blocking(mask) &&
page->mapping->host->i_size > SZ_16M) {
@@ -4560,7 +4559,9 @@ int try_release_extent_mapping(struct extent_map_tree 
*map,
free_extent_map(em);
}
}
-   return try_release_extent_state(map, tree, page, mask);
+   return try_release_extent_state(map, tree, page,
+   orig_start, orig_end,
+   mask);
 }
 
 /*
diff --git a/fs/btrfs/extent_io.h b/fs/btrfs/extent_io.h
index ad5b000..491f9b4 100644
--- a/fs/btrfs/extent_io.h
+++ b/fs/btrfs/extent_io.h
@@ -276,8 +276,9 @@ typedef struct extent_map *(get_extent_t)(struct inode 
*inode,
 
 void extent_io_tree_init(struct extent_io_tree *tree, void *private_data);
 int try_release_extent_mapping(struct extent_map_tree *map,
-  struct extent_io_tree *tree, struct page *page,
-  gfp_t mask);
+   struct extent_io_tree *tree, struct page *page,
+   u64 start, u64 end,
+   gfp_t mask);
 int try_release_extent_buffer(struct page *page);
 int lock_extent_bits(struct extent_io_tree *tree, u64 start, u64 end,
 struct extent_state **cached);
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index cf55622..03b9425 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -3077,56 +3077,119 @@ static void finish_ordered_fn(struct btrfs_work *work)
btrfs_finish_ordered_io(ordered_extent);
 }
 
-static int btrfs_writepage_end_io_hook(struct page *page, u64 start, u64 end,
-   struct extent_state *state, int uptodate)
+static void mark_blks_io_complete(struct btrfs_ordered_extent *ordered,
+   u64 blk, u64 nr_blks, int uptodate)
 {
-   struct inode *inode = page->mapping->host;
+   struct inode *inode = ordered->inode;
struct btrfs_root *root = BTRFS_I(inode)->root;
-   struct btrfs_ordered_extent *ordered_extent = NULL;
struct btrfs_workqueue *wq;
btrfs_work_func_t func;
-   u64 ordered_start, ordered_end;
int done;
 
-   trace_btrfs_writepage_end_io_hook(page, start, end, uptodate);
+   while (nr_blks--) {
+   if (test_and_set_bit(blk, ordered->blocks_done)) {
+   blk++;
+   continue;
+   }
 
-   ClearPagePrivate2(page);
-loop:
-   ordered_extent = btrfs_lookup_ordered_range(inode, start,
-   end - start + 1);
-   if (!ordered_extent)
-   goto out;
+   done = btrfs_dec_test_ordered_pending(inode, ,
+   ordered->file_offset
+   + (blk << inode->i_blkbits),
+   root->sectorsize,
+   uptodate);
+   if (done) {
+

[PATCH V21 11/19] Btrfs: subpage-blocksize: Deal with partial ordered extent allocations.

2016-10-02 Thread Chandan Rajendra 
In subpage-blocksize scenario, extent allocations for only some of the
dirty blocks of a page can succeed, while allocation for rest of the
blocks can fail. This patch allows I/O against such pages to be
submitted.

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/extent_io.c | 27 ++-
 fs/btrfs/inode.c | 11 ++-
 2 files changed, 24 insertions(+), 14 deletions(-)

diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
index 9af8237..0832797 100644
--- a/fs/btrfs/extent_io.c
+++ b/fs/btrfs/extent_io.c
@@ -1843,17 +1843,23 @@ void extent_clear_unlock_delalloc(struct inode *inode, 
u64 start, u64 end,
if (page_ops & PAGE_SET_PRIVATE2)
SetPagePrivate2(pages[i]);
 
+   if (page_ops & PAGE_SET_ERROR)
+   SetPageError(pages[i]);
+
if (pages[i] == locked_page) {
put_page(pages[i]);
continue;
}
-   if (page_ops & PAGE_CLEAR_DIRTY)
+
+   if ((page_ops & PAGE_CLEAR_DIRTY)
+   && !PagePrivate2(pages[i]))
clear_page_dirty_for_io(pages[i]);
-   if (page_ops & PAGE_SET_WRITEBACK)
+   if ((page_ops & PAGE_SET_WRITEBACK)
+   && !PagePrivate2(pages[i]))
set_page_writeback(pages[i]);
-   if (page_ops & PAGE_SET_ERROR)
-   SetPageError(pages[i]);
-   if (page_ops & PAGE_END_WRITEBACK)
+
+   if ((page_ops & PAGE_END_WRITEBACK)
+   && !PagePrivate2(pages[i]))
end_page_writeback(pages[i]);
 
if (page_ops & PAGE_UNLOCK) {
@@ -2554,7 +2560,7 @@ void end_extent_writepage(struct page *page, int err, u64 
start, u64 end)
uptodate = 0;
}
 
-   if (!uptodate) {
+   if (!uptodate || PageError(page)) {
ClearPageUptodate(page);
SetPageError(page);
ret = ret < 0 ? ret : -EIO;
@@ -3401,7 +3407,6 @@ static noinline_for_stack int writepage_delalloc(struct 
inode *inode,
   nr_written);
/* File system has been set read-only */
if (ret) {
-   SetPageError(page);
/* fill_delalloc should be return < 0 for error
 * but just in case, we use > 0 here meaning the
 * IO is started, so we don't want to return > 0
@@ -3618,7 +3623,6 @@ static int __extent_writepage(struct page *page, struct 
writeback_control *wbc,
struct inode *inode = page->mapping->host;
struct extent_page_data *epd = data;
u64 start = page_offset(page);
-   u64 page_end = start + PAGE_SIZE - 1;
int ret;
int nr = 0;
size_t pg_offset = 0;
@@ -3661,7 +3665,7 @@ static int __extent_writepage(struct page *page, struct 
writeback_control *wbc,
ret = writepage_delalloc(inode, page, wbc, epd, start, _written);
if (ret == 1)
goto done_unlocked;
-   if (ret)
+   if (ret && !PagePrivate2(page))
goto done;
 
ret = __extent_writepage_io(inode, page, wbc, epd,
@@ -3675,10 +3679,7 @@ done:
set_page_writeback(page);
end_page_writeback(page);
}
-   if (PageError(page)) {
-   ret = ret < 0 ? ret : -EIO;
-   end_extent_writepage(page, ret, start, page_end);
-   }
+
unlock_page(page);
return ret;
 
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 42f844b..cf55622 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -951,6 +951,7 @@ static noinline int cow_file_range(struct inode *inode,
struct btrfs_key ins;
struct extent_map *em;
struct extent_map_tree *em_tree = _I(inode)->extent_tree;
+   struct btrfs_ordered_extent *ordered;
unsigned long page_ops, extent_ops;
int ret = 0;
 
@@ -1049,7 +1050,7 @@ static noinline int cow_file_range(struct inode *inode,
ret = btrfs_reloc_clone_csums(inode, start,
  cur_alloc_size);
if (ret)
-   goto out_drop_extent_cache;
+   goto out_remove_ordered_extent;
}
 
btrfs_dec_block_group_reservations(root->fs_info, ins.objectid);
@@ -1078,6 +1079,14 @@ static noinline int cow_file_range(struct inode *inode,
 out:
return ret;
 
+out_remove_ordered_extent:
+   ordered = btrfs_lookup_ordered_extent(inode, start);
+

[PATCH V21 09/19] Btrfs: subpage-blocksize: Compute free space tree BITMAP_RANGE based on sectorsize

2016-10-02 Thread Chandan Rajendra 
The default bitmap length computation in free space tree sanity tests
assumes PAGE_SIZE as the sectorsize. This commit fixes this by using a
variable sectorsize to calculate BITMAP_RANGE.

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/tests/free-space-tree-tests.c | 79 ++
 1 file changed, 43 insertions(+), 36 deletions(-)

diff --git a/fs/btrfs/tests/free-space-tree-tests.c 
b/fs/btrfs/tests/free-space-tree-tests.c
index 3bf5df1..11d9fb0 100644
--- a/fs/btrfs/tests/free-space-tree-tests.c
+++ b/fs/btrfs/tests/free-space-tree-tests.c
@@ -31,7 +31,7 @@ struct free_space_extent {
  * The test cases align their operations to this in order to hit some of the
  * edge cases in the bitmap code.
  */
-#define BITMAP_RANGE (BTRFS_FREE_SPACE_BITMAP_BITS * PAGE_SIZE)
+#define BITMAP_RANGE(sectorsize) (BTRFS_FREE_SPACE_BITMAP_BITS * (sectorsize))
 
 static int __check_free_space_extents(struct btrfs_trans_handle *trans,
  struct btrfs_fs_info *fs_info,
@@ -203,14 +203,15 @@ static int test_remove_beginning(struct 
btrfs_trans_handle *trans,
 struct btrfs_block_group_cache *cache,
 struct btrfs_path *path)
 {
+   u64 bitmap_range = BITMAP_RANGE(fs_info->tree_root->sectorsize);
struct free_space_extent extents[] = {
-   {cache->key.objectid + BITMAP_RANGE,
-   cache->key.offset - BITMAP_RANGE},
+   {cache->key.objectid + bitmap_range,
+   cache->key.offset - bitmap_range},
};
int ret;
 
ret = __remove_from_free_space_tree(trans, fs_info, cache, path,
-   cache->key.objectid, BITMAP_RANGE);
+   cache->key.objectid, bitmap_range);
if (ret) {
test_msg("Could not remove free space\n");
return ret;
@@ -226,15 +227,16 @@ static int test_remove_end(struct btrfs_trans_handle 
*trans,
   struct btrfs_block_group_cache *cache,
   struct btrfs_path *path)
 {
+   u64 bitmap_range = BITMAP_RANGE(fs_info->tree_root->sectorsize);
struct free_space_extent extents[] = {
-   {cache->key.objectid, cache->key.offset - BITMAP_RANGE},
+   {cache->key.objectid, cache->key.offset - bitmap_range},
};
int ret;
 
ret = __remove_from_free_space_tree(trans, fs_info, cache, path,
cache->key.objectid +
-   cache->key.offset - BITMAP_RANGE,
-   BITMAP_RANGE);
+   cache->key.offset - bitmap_range,
+   bitmap_range);
if (ret) {
test_msg("Could not remove free space\n");
return ret;
@@ -249,16 +251,17 @@ static int test_remove_middle(struct btrfs_trans_handle 
*trans,
  struct btrfs_block_group_cache *cache,
  struct btrfs_path *path)
 {
+   u64 bitmap_range = BITMAP_RANGE(fs_info->tree_root->sectorsize);
struct free_space_extent extents[] = {
-   {cache->key.objectid, BITMAP_RANGE},
-   {cache->key.objectid + 2 * BITMAP_RANGE,
-   cache->key.offset - 2 * BITMAP_RANGE},
+   {cache->key.objectid, bitmap_range},
+   {cache->key.objectid + 2 * bitmap_range,
+   cache->key.offset - 2 * bitmap_range},
};
int ret;
 
ret = __remove_from_free_space_tree(trans, fs_info, cache, path,
-   cache->key.objectid + BITMAP_RANGE,
-   BITMAP_RANGE);
+   cache->key.objectid + bitmap_range,
+   bitmap_range);
if (ret) {
test_msg("Could not remove free space\n");
return ret;
@@ -273,8 +276,9 @@ static int test_merge_left(struct btrfs_trans_handle *trans,
   struct btrfs_block_group_cache *cache,
   struct btrfs_path *path)
 {
+   u64 bitmap_range = BITMAP_RANGE(fs_info->tree_root->sectorsize);
struct free_space_extent extents[] = {
-   {cache->key.objectid, 2 * BITMAP_RANGE},
+   {cache->key.objectid, 2 * bitmap_range},
};
int ret;
 
@@ -287,15 +291,15 @@ static int test_merge_left(struct btrfs_trans_handle 
*trans,
}
 
ret = __add_to_free_space_tree(trans, fs_info, cache, path,
-  cache->key.objectid, BITMAP_RANGE);
+  cache->key.objectid, bitmap_range);
if (ret) {

[PATCH V21 10/19] Btrfs: subpage-blocksize: Allow mounting filesystems where sectorsize < PAGE_SIZE

2016-10-02 Thread Chandan Rajendra 
This commit allows mounting filesystem instances with sectorsize smaller
than the PAGE_SIZE.

Since the code assumes that the super block is either equal to or larger
than sectorsize, this commit brings back the nodesize argument for
btrfs_find_create_tree_block() function. This change allows us to be
able to mount and use filesystems with 2048 bytes as the sectorsize.

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/disk-io.c | 21 -
 fs/btrfs/disk-io.h |  2 +-
 fs/btrfs/extent-tree.c |  4 ++--
 fs/btrfs/extent_io.c   |  3 +--
 fs/btrfs/extent_io.h   |  2 +-
 fs/btrfs/tree-log.c|  2 +-
 fs/btrfs/volumes.c | 10 +++---
 7 files changed, 17 insertions(+), 27 deletions(-)

diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index 5663481..2684438 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -936,7 +936,7 @@ void readahead_tree_block(struct btrfs_root *root, u64 
bytenr)
 {
struct extent_buffer *buf = NULL;
 
-   buf = btrfs_find_create_tree_block(root, bytenr);
+   buf = btrfs_find_create_tree_block(root, bytenr, root->nodesize);
if (IS_ERR(buf))
return;
read_extent_buffer_pages(buf, WAIT_NONE, 0);
@@ -949,7 +949,7 @@ int reada_tree_block_flagged(struct btrfs_root *root, u64 
bytenr,
struct extent_buffer *buf = NULL;
int ret;
 
-   buf = btrfs_find_create_tree_block(root, bytenr);
+   buf = btrfs_find_create_tree_block(root, bytenr, root->nodesize);
if (IS_ERR(buf))
return 0;
 
@@ -979,12 +979,12 @@ struct extent_buffer *btrfs_find_tree_block(struct 
btrfs_fs_info *fs_info,
 }
 
 struct extent_buffer *btrfs_find_create_tree_block(struct btrfs_root *root,
-u64 bytenr)
+u64 bytenr, u32 blocksize)
 {
if (btrfs_is_testing(root->fs_info))
return alloc_test_extent_buffer(root->fs_info->eb_info, bytenr,
-   root->nodesize);
-   return alloc_extent_buffer(root->fs_info, bytenr);
+   blocksize);
+   return alloc_extent_buffer(root->fs_info, bytenr, blocksize);
 }
 
 
@@ -1006,7 +1006,7 @@ struct extent_buffer *read_tree_block(struct btrfs_root 
*root, u64 bytenr,
struct extent_buffer *buf = NULL;
int ret;
 
-   buf = btrfs_find_create_tree_block(root, bytenr);
+   buf = btrfs_find_create_tree_block(root, bytenr, root->nodesize);
if (IS_ERR(buf))
return buf;
 
@@ -3891,17 +3891,12 @@ static int btrfs_check_super_valid(struct btrfs_fs_info 
*fs_info,
 * Check sectorsize and nodesize first, other check will need it.
 * Check all possible sectorsize(4K, 8K, 16K, 32K, 64K) here.
 */
-   if (!is_power_of_2(sectorsize) || sectorsize < 4096 ||
+   if (!is_power_of_2(sectorsize) || sectorsize < 2048 ||
sectorsize > BTRFS_MAX_METADATA_BLOCKSIZE) {
printk(KERN_ERR "BTRFS: invalid sectorsize %llu\n", sectorsize);
ret = -EINVAL;
}
-   /* Only PAGE SIZE is supported yet */
-   if (sectorsize != PAGE_SIZE) {
-   printk(KERN_ERR "BTRFS: sectorsize %llu not supported yet, only 
support %lu\n",
-   sectorsize, PAGE_SIZE);
-   ret = -EINVAL;
-   }
+
if (!is_power_of_2(nodesize) || nodesize < sectorsize ||
nodesize > BTRFS_MAX_METADATA_BLOCKSIZE) {
printk(KERN_ERR "BTRFS: invalid nodesize %llu\n", nodesize);
diff --git a/fs/btrfs/disk-io.h b/fs/btrfs/disk-io.h
index 591f078..5f6263e 100644
--- a/fs/btrfs/disk-io.h
+++ b/fs/btrfs/disk-io.h
@@ -50,7 +50,7 @@ void readahead_tree_block(struct btrfs_root *root, u64 
bytenr);
 int reada_tree_block_flagged(struct btrfs_root *root, u64 bytenr,
 int mirror_num, struct extent_buffer **eb);
 struct extent_buffer *btrfs_find_create_tree_block(struct btrfs_root *root,
-  u64 bytenr);
+  u64 bytenr, u32 blocksize);
 void clean_tree_block(struct btrfs_trans_handle *trans,
  struct btrfs_fs_info *fs_info, struct extent_buffer *buf);
 int open_ctree(struct super_block *sb,
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 03da2f6..25fbfa2 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -8238,7 +8238,7 @@ btrfs_init_new_buffer(struct btrfs_trans_handle *trans, 
struct btrfs_root *root,
 {
struct extent_buffer *buf;
 
-   buf = btrfs_find_create_tree_block(root, bytenr);
+   buf = btrfs_find_create_tree_block(root, bytenr, root->nodesize);
if (IS_ERR(buf))
return buf;
 
@@ -8885,7 +8885,7 @@ static noinline int do_walk_down(struct 
btrfs_trans_handle *trans,
 
next =

[PATCH V21 07/19] Btrfs: subpage-blocksize: Use kmalloc()-ed memory to hold metadata blocks

2016-10-02 Thread Chandan Rajendra 
For subpage-blocksizes this commit uses kmalloc()-ed memory to buffer
metadata blocks in memory.

When reading/writing metadata blocks, We now track the first extent
buffer using bio->bi_private. With kmalloc()-ed memory we cannot use
page->private. Hence when writing dirty extent buffers in
subpage-blocksize scenario, this commit forces each bio to contain a
single extent buffer. For the non subpage-blocksize scenario we continue
to track the corresponding extent buffer using page->private and hence a
single write bio will continue to have more than one dirty extent
buffer.

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/ctree.h |   6 +-
 fs/btrfs/disk-io.c   |  27 +++---
 fs/btrfs/extent_io.c | 204 +--
 fs/btrfs/extent_io.h |   8 +-
 fs/btrfs/tests/extent-io-tests.c |   4 +-
 5 files changed, 158 insertions(+), 91 deletions(-)

diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index b9ee7cf..745284c 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -1491,14 +1491,16 @@ static inline void btrfs_set_token_##name(struct 
extent_buffer *eb, \
 #define BTRFS_SETGET_HEADER_FUNCS(name, type, member, bits)\
 static inline u##bits btrfs_##name(struct extent_buffer *eb)   \
 {  \
-   type *p = page_address(eb->pages[0]);   \
+   type *p = (type *)((u8 *)page_address(eb->pages[0]) \
+   + eb->pg_offset);   \
u##bits res = le##bits##_to_cpu(p->member); \
return res; \
 }  \
 static inline void btrfs_set_##name(struct extent_buffer *eb,  \
u##bits val)\
 {  \
-   type *p = page_address(eb->pages[0]);   \
+   type *p = (type *)((u8 *)page_address(eb->pages[0]) \
+   + eb->pg_offset);   \
p->member = cpu_to_le##bits(val);   \
 }
 
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index 9ff48a7..5663481 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -448,13 +448,10 @@ static int btree_read_extent_buffer_pages(struct 
btrfs_root *root,
  * we only fill in the checksum field in the first page of a multi-page block
  */
 
-static int csum_dirty_buffer(struct btrfs_fs_info *fs_info, struct page *page)
+static int csum_dirty_buffer(struct btrfs_fs_info *fs_info,
+   struct extent_buffer *eb)
 {
-   struct extent_buffer *eb;
 
-   eb = (struct extent_buffer *)page->private;
-   if (page != eb->pages[0])
-   return 0;
ASSERT(memcmp_extent_buffer(eb, fs_info->fsid,
btrfs_header_fsid(), BTRFS_FSID_SIZE) == 0);
 
@@ -557,11 +554,10 @@ static int btree_readpage_end_io_hook(struct btrfs_io_bio 
*io_bio,
int ret = 0;
int reads_done;
 
-   if (!page->private)
+   eb = (io_bio->bio).bi_private;
+   if (!eb)
goto out;
 
-   eb = (struct extent_buffer *)page->private;
-
/* the pending IO might have been the only thing that kept this buffer
 * in memory.  Make sure we have a ref for all this other checks
 */
@@ -646,11 +642,11 @@ out:
return ret;
 }
 
-static int btree_io_failed_hook(struct page *page, int failed_mirror)
+static int btree_io_failed_hook(struct page *page, void *private,
+   int failed_mirror)
 {
-   struct extent_buffer *eb;
+   struct extent_buffer *eb = private;
 
-   eb = (struct extent_buffer *)page->private;
set_bit(EXTENT_BUFFER_READ_ERR, >bflags);
eb->read_mirror = failed_mirror;
atomic_dec(>io_pages);
@@ -829,11 +825,18 @@ int btrfs_wq_submit_bio(struct btrfs_fs_info *fs_info, 
struct bio *bio,
 
 static int btree_csum_one_bio(struct btrfs_fs_info *fs_info, struct bio *bio)
 {
+   struct extent_buffer *eb = bio->bi_private;
struct bio_vec *bvec;
int i, ret = 0;
 
bio_for_each_segment_all(bvec, bio, i) {
-   ret = csum_dirty_buffer(fs_info, bvec->bv_page);
+   if (eb->len >= PAGE_SIZE)
+   eb = (struct extent_buffer *)(bvec->bv_page->private);
+
+   if (bvec->bv_page != eb->pages[0])
+   continue;
+
+   ret = csum_dirty_buffer(fs_info, eb);
if (ret)
break;
}
diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
index 6cac61f..8ace367 100644
--- a/fs/btrfs/extent_io.c
+++ b/fs/btrfs/extent_io.c
@@ -2817,18

[PATCH V21 06/19] Btrfs: subpage-blocksize: Fix whole page write

2016-10-02 Thread Chandan Rajendra 
For the subpage-blocksize scenario, a page can contain multiple
blocks. In such cases, this patch handles writing data to files.

Also, When setting EXTENT_DELALLOC, we no longer set EXTENT_UPTODATE bit on
the extent_io_tree since uptodate status is being tracked either by the
bitmap pointed to by page->private or by the PG_uptodate flag.

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/extent_io.c  | 114 +++---
 fs/btrfs/file.c   |  16 +++
 fs/btrfs/inode.c  |  69 --
 fs/btrfs/relocation.c |   3 ++
 4 files changed, 137 insertions(+), 65 deletions(-)

diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
index b3885cc..6cac61f 100644
--- a/fs/btrfs/extent_io.c
+++ b/fs/btrfs/extent_io.c
@@ -2573,36 +2573,41 @@ void end_extent_writepage(struct page *page, int err, 
u64 start, u64 end)
  */
 static void end_bio_extent_writepage(struct bio *bio)
 {
+   struct btrfs_page_private *pg_private;
struct bio_vec *bvec;
+   unsigned long flags;
u64 start;
u64 end;
+   int clear_writeback;
int i;
 
bio_for_each_segment_all(bvec, bio, i) {
struct page *page = bvec->bv_page;
+   struct btrfs_root *root = BTRFS_I(page->mapping->host)->root;
 
-   /* We always issue full-page reads, but if some block
-* in a page fails to read, blk_update_request() will
-* advance bv_offset and adjust bv_len to compensate.
-* Print a warning for nonzero offsets, and an error
-* if they don't add up to a full page.  */
-   if (bvec->bv_offset || bvec->bv_len != PAGE_SIZE) {
-   if (bvec->bv_offset + bvec->bv_len != PAGE_SIZE)
-   
btrfs_err(BTRFS_I(page->mapping->host)->root->fs_info,
-  "partial page write in btrfs with offset %u 
and length %u",
-   bvec->bv_offset, bvec->bv_len);
-   else
-   
btrfs_info(BTRFS_I(page->mapping->host)->root->fs_info,
-  "incomplete page write in btrfs with offset 
%u and "
-  "length %u",
-   bvec->bv_offset, bvec->bv_len);
-   }
+   pg_private = NULL;
+   flags = 0;
+   clear_writeback = 1;
+
+   start = page_offset(page) + bvec->bv_offset;
+   end = start + bvec->bv_len - 1;
 
-   start = page_offset(page);
-   end = start + bvec->bv_offset + bvec->bv_len - 1;
+   if (root->sectorsize < PAGE_SIZE) {
+   pg_private = (struct btrfs_page_private *)page->private;
+   spin_lock_irqsave(_private->io_lock, flags);
+   }
 
end_extent_writepage(page, bio->bi_error, start, end);
-   end_page_writeback(page);
+
+   if (root->sectorsize < PAGE_SIZE) {
+   clear_page_blks_state(page, 1 << BLK_STATE_IO, start,
+   end);
+   clear_writeback = page_io_complete(page);
+   spin_unlock_irqrestore(_private->io_lock, flags);
+   }
+
+   if (clear_writeback)
+   end_page_writeback(page);
}
 
bio_put(bio);
@@ -3465,7 +3470,6 @@ static noinline_for_stack int 
__extent_writepage_io(struct inode *inode,
u64 block_start;
u64 iosize;
sector_t sector;
-   struct extent_state *cached_state = NULL;
struct extent_map *em;
struct block_device *bdev;
size_t pg_offset = 0;
@@ -3517,20 +3521,29 @@ static noinline_for_stack int 
__extent_writepage_io(struct inode *inode,
 page_end, NULL, 1);
break;
}
-   em = epd->get_extent(inode, page, pg_offset, cur,
-end - cur + 1, 1);
+
+   if (blocksize < PAGE_SIZE
+   && !test_page_blks_state(page, BLK_STATE_DIRTY, cur,
+   cur + blocksize - 1, 1)) {
+   cur += blocksize;
+   continue;
+   }
+
+   pg_offset = cur & (PAGE_SIZE - 1);
+
+   em = epd->get_extent(inode, page, pg_offset, cur, blocksize, 1);
if (IS_ERR_OR_NULL(em)) {
SetPageError(page);
ret = PTR_ERR_OR_ZERO(em);
break;
}
 
-   extent_offset = cur - em->start;
em_end = extent_map_end(em);
BUG_ON(em_end <= cur);
BUG_ON(end < cur);
-

[PATCH V21 08/19] Btrfs: subpage-blocksize: Execute sanity tests on all possible block sizes

2016-10-02 Thread Chandan Rajendra 
This commit executes sanity tests for all valid sectorsize/nodesize
combinations.

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/tests/btrfs-tests.c | 8 +++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/fs/btrfs/tests/btrfs-tests.c b/fs/btrfs/tests/btrfs-tests.c
index dca90d6..0f2afb6 100644
--- a/fs/btrfs/tests/btrfs-tests.c
+++ b/fs/btrfs/tests/btrfs-tests.c
@@ -261,13 +261,19 @@ int btrfs_run_sanity_tests(void)
int ret, i;
u32 sectorsize, nodesize;
u32 test_sectorsize[] = {
-   PAGE_SIZE,
+   4096,
+   8192,
+   16384,
+   32768,
+   65536,
};
ret = btrfs_init_test_fs();
if (ret)
return ret;
for (i = 0; i < ARRAY_SIZE(test_sectorsize); i++) {
sectorsize = test_sectorsize[i];
+   if (sectorsize > PAGE_SIZE)
+   break;
for (nodesize = sectorsize;
 nodesize <= BTRFS_MAX_METADATA_BLOCKSIZE;
 nodesize <<= 1) {
-- 
2.5.5

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V21 05/19] Btrfs: subpage-blocksize: Fix whole page read.

2016-10-02 Thread Chandan Rajendra 
For the subpage-blocksize scenario, a page can contain multiple
blocks. In such cases, this patch handles reading data from files.

To track the status of individual blocks of a page, this patch makes use
of a bitmap pointed to by the newly introduced per-page 'struct
btrfs_page_private'.

The per-page btrfs_page_private->io_lock plays the same role as
BH_Uptodate_Lock (see end_buffer_async_read()) i.e. without the io_lock
we may end up in the following situation,

NOTE: Assume 64k page size and 4k block size. Also assume that the first
12 blocks of the page are contiguous while the next 4 blocks are
contiguous. When reading the page we end up submitting two "logical
address space" bios. So end_bio_extent_readpage function is invoked
twice, once for each bio.

|-+-+-|
| Task A  | Task B  | Task C  |
|-+-+-|
| end_bio_extent_readpage | | |
| process block 0 | | |
| - clear BLK_STATE_IO| | |
| - page_read_complete| | |
| process block 1 | | |
| | | |
| | | |
| | end_bio_extent_readpage | |
| | process block 0 | |
| | - clear BLK_STATE_IO| |
| | - page_read_complete| |
| | process block 1 | |
| | | |
| process block 11| process block 3 | |
| - clear BLK_STATE_IO| - clear BLK_STATE_IO| |
| - page_read_complete| - page_read_complete| |
|   - returns true|   - returns true| |
|   - unlock_page()   | | |
| | | lock_page() |
| |   - unlock_page()   | |
|-+-+-|

We end up incorrectly unlocking the page twice and "Task C" ends up
working on an unlocked page. So private->io_lock makes sure that only
one of the tasks gets "true" as the return value when page_io_complete()
is invoked. As an optimization the patch gets the io_lock only when the
last block of the bio_vec is being processed.

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/extent_io.c | 299 +--
 fs/btrfs/extent_io.h |  76 -
 fs/btrfs/inode.c |  13 +--
 3 files changed, 320 insertions(+), 68 deletions(-)

diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
index 522c943..b3885cc 100644
--- a/fs/btrfs/extent_io.c
+++ b/fs/btrfs/extent_io.c
@@ -23,6 +23,7 @@
 
 static struct kmem_cache *extent_state_cache;
 static struct kmem_cache *extent_buffer_cache;
+static struct kmem_cache *page_private_cache;
 static struct bio_set *btrfs_bioset;
 
 static inline bool extent_state_in_tree(const struct extent_state *state)
@@ -163,10 +164,16 @@ int __init extent_io_init(void)
if (!extent_buffer_cache)
goto free_state_cache;
 
+   page_private_cache = kmem_cache_create("btrfs_page_private",
+   sizeof(struct btrfs_page_private), 0,
+   SLAB_MEM_SPREAD, NULL);
+   if (!page_private_cache)
+   goto free_buffer_cache;
+
btrfs_bioset = bioset_create(BIO_POOL_SIZE,
 offsetof(struct btrfs_io_bio, bio));
if (!btrfs_bioset)
-   goto free_buffer_cache;
+   goto free_page_private_cache;
 
if (bioset_integrity_create(btrfs_bioset, BIO_POOL_SIZE))
goto free_bioset;
@@ -177,6 +184,10 @@ free_bioset:
bioset_free(btrfs_bioset);
btrfs_bioset = NULL;
 
+free_page_private_cache:
+   kmem_cache_destroy(page_private_cache);
+   page_private_cache = NULL;
+
 free_buffer_cache:
kmem_cache_destroy(extent_buffer_cache);
extent_buffer_cache = NULL;
@@ -1311,6 +1322,96 @@ int clear_record_extent_bits(struct extent_io_tree 
*tree, u64 start, u64 end,
  changeset);
 }
 
+static int modify_page_blks_state(struct page *page,
+   unsigned long blk_states,
+   u64 start, u64 end, int set)
+{
+   struct inode *inode = page->mapping->host;
+   unsigned long *bitmap;
+   unsigned long first_state;
+   unsigned long state;
+   u64 nr_blks;
+   u64 blk;
+
+   if

[PATCH V21 18/19] Btrfs: subpage-blocksize: __btrfs_lookup_bio_sums: Set offset when moving to a new bio_vec

2016-10-02 Thread Chandan Rajendra 
In __btrfs_lookup_bio_sums() we set the file offset value at the
beginning of every iteration of the while loop. This is incorrect since
the blocks mapped by the current bvec->bv_page might not yet have been
completely processed.

This commit fixes the issue by setting the file offset value when we
move to the next bvec of the bio.

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/file-item.c | 7 +--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/fs/btrfs/file-item.c b/fs/btrfs/file-item.c
index d0d571c..8fc09c1 100644
--- a/fs/btrfs/file-item.c
+++ b/fs/btrfs/file-item.c
@@ -222,11 +222,11 @@ static int __btrfs_lookup_bio_sums(struct btrfs_root 
*root,
disk_bytenr = (u64)bio->bi_iter.bi_sector << 9;
if (dio)
offset = logical_offset;
+   else
+   offset = page_offset(bvec->bv_page) + bvec->bv_offset;
 
page_bytes_left = bvec->bv_len;
while (bio_index < bio->bi_vcnt) {
-   if (!dio)
-   offset = page_offset(bvec->bv_page) + bvec->bv_offset;
count = btrfs_find_ordered_sum(inode, offset, disk_bytenr,
   (u32 *)csum, nblocks);
if (count)
@@ -301,6 +301,9 @@ found:
goto done;
}
bvec++;
+   if (!dio)
+   offset = page_offset(bvec->bv_page)
+   + bvec->bv_offset;
page_bytes_left = bvec->bv_len;
}
 
-- 
2.5.5

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V21 19/19] Btrfs: subpage-blocksize: Disable compression

2016-10-02 Thread Chandan Rajendra 
The subpage-blocksize patchset does not yet support compression. Hence,
the kernel might crash when executing compression code in
subpage-blocksize scenario. This commit disables enabling compression
feature during 'mount' and also when the  user invokes
'chattr +c ' command.

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/ioctl.c |  8 +++-
 fs/btrfs/super.c | 19 +++
 2 files changed, 26 insertions(+), 1 deletion(-)

diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
index 0fdc0a0..862d97b 100644
--- a/fs/btrfs/ioctl.c
+++ b/fs/btrfs/ioctl.c
@@ -322,6 +322,11 @@ static int btrfs_ioctl_setflags(struct file *file, void 
__user *arg)
} else if (flags & FS_COMPR_FL) {
const char *comp;
 
+   if (root->sectorsize < PAGE_SIZE) {
+   ret = -EINVAL;
+   goto out_drop;
+   }
+
ip->flags |= BTRFS_INODE_COMPRESS;
ip->flags &= ~BTRFS_INODE_NOCOMPRESS;
 
@@ -1342,7 +1347,8 @@ int btrfs_defrag_file(struct inode *inode, struct file 
*file,
return -EINVAL;
 
if (range->flags & BTRFS_DEFRAG_RANGE_COMPRESS) {
-   if (range->compress_type > BTRFS_COMPRESS_TYPES)
+   if ((range->compress_type > BTRFS_COMPRESS_TYPES)
+   || (root->sectorsize < PAGE_SIZE))
return -EINVAL;
if (range->compress_type)
compress_type = range->compress_type;
diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c
index 73a1d8d..3a2e9d7 100644
--- a/fs/btrfs/super.c
+++ b/fs/btrfs/super.c
@@ -392,6 +392,17 @@ static const match_table_t tokens = {
{Opt_err, NULL},
 };
 
+static int can_enable_compression(struct btrfs_fs_info *fs_info)
+{
+   if (btrfs_super_sectorsize(fs_info->super_copy) < PAGE_SIZE) {
+   btrfs_err(fs_info,
+   "Compression is not supported for subpage-blocksize");
+   return 0;
+   }
+
+   return 1;
+}
+
 /*
  * Regular mount options parser.  Everything that is needed only when
  * reading in a new superblock is parsed here.
@@ -502,6 +513,10 @@ int btrfs_parse_options(struct btrfs_root *root, char 
*options,
if (token == Opt_compress ||
token == Opt_compress_force ||
strcmp(args[0].from, "zlib") == 0) {
+   if (!can_enable_compression(info)) {
+   ret = -EINVAL;
+   goto out;
+   }
compress_type = "zlib";
info->compress_type = BTRFS_COMPRESS_ZLIB;
btrfs_set_opt(info->mount_opt, COMPRESS);
@@ -509,6 +524,10 @@ int btrfs_parse_options(struct btrfs_root *root, char 
*options,
btrfs_clear_opt(info->mount_opt, NODATASUM);
no_compress = 0;
} else if (strcmp(args[0].from, "lzo") == 0) {
+   if (!can_enable_compression(info)) {
+   ret = -EINVAL;
+   goto out;
+   }
compress_type = "lzo";
info->compress_type = BTRFS_COMPRESS_LZO;
btrfs_set_opt(info->mount_opt, COMPRESS);
-- 
2.5.5

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V21 17/19] Btrfs: subpage-blocksize: Make file extent relocate code subpage blocksize aware

2016-10-02 Thread Chandan Rajendra 
The file extent relocation code currently assumes blocksize to be same
as PAGE_SIZE. This commit adds code to support subpage blocksize
scenario.

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/relocation.c | 90 ---
 1 file changed, 71 insertions(+), 19 deletions(-)

diff --git a/fs/btrfs/relocation.c b/fs/btrfs/relocation.c
index f724fb5..75e51a3 100644
--- a/fs/btrfs/relocation.c
+++ b/fs/btrfs/relocation.c
@@ -3114,14 +3114,19 @@ static int relocate_file_extent_cluster(struct inode 
*inode,
 {
u64 page_start;
u64 page_end;
+   u64 block_start;
u64 offset = BTRFS_I(inode)->index_cnt;
+   u64 blocksize = BTRFS_I(inode)->root->sectorsize;
+   u64 reserved_space;
unsigned long index;
unsigned long last_index;
struct page *page;
struct file_ra_state *ra;
gfp_t mask = btrfs_alloc_write_mask(inode->i_mapping);
+   int nr_blocks;
int nr = 0;
int ret = 0;
+   int i;
 
if (!cluster->nr)
return 0;
@@ -3141,13 +3146,19 @@ static int relocate_file_extent_cluster(struct inode 
*inode,
if (ret)
goto out;
 
+   page_start = cluster->start - offset;
+   page_end = min_t(u64, round_down(page_start, PAGE_SIZE) + PAGE_SIZE - 1,
+   cluster->end - offset);
+
index = (cluster->start - offset) >> PAGE_SHIFT;
last_index = (cluster->end - offset) >> PAGE_SHIFT;
while (index <= last_index) {
-   ret = btrfs_delalloc_reserve_metadata(inode, PAGE_SIZE);
+   reserved_space = page_end - page_start + 1;
+
+   ret = btrfs_delalloc_reserve_metadata(inode, reserved_space);
if (ret)
goto out;
-
+again:
page = find_lock_page(inode->i_mapping, index);
if (!page) {
page_cache_sync_readahead(inode->i_mapping,
@@ -3157,7 +3168,7 @@ static int relocate_file_extent_cluster(struct inode 
*inode,
   mask);
if (!page) {
btrfs_delalloc_release_metadata(inode,
-   PAGE_SIZE);
+   reserved_space);
ret = -ENOMEM;
goto out;
}
@@ -3169,6 +3180,38 @@ static int relocate_file_extent_cluster(struct inode 
*inode,
   last_index + 1 - index);
}
 
+   if (PageDirty(page)) {
+   u64 pg_offset = page_offset(page);
+
+   unlock_page(page);
+   put_page(page);
+   ret = btrfs_fdatawrite_range(inode, pg_offset,
+   page_start - 1);
+   if (ret) {
+   btrfs_delalloc_release_metadata(inode,
+   reserved_space);
+   goto out;
+   }
+
+   ret = filemap_fdatawait_range(inode->i_mapping,
+   pg_offset, page_start - 1);
+   if (ret) {
+   btrfs_delalloc_release_metadata(inode,
+   reserved_space);
+   goto out;
+   }
+
+   goto again;
+   }
+
+   if (BTRFS_I(inode)->root->sectorsize < PAGE_SIZE) {
+   ClearPageUptodate(page);
+   if (page->private)
+   clear_page_blks_state(page,
+   1 << BLK_STATE_UPTODATE,
+   page_start, page_end);
+   }
+
if (!PageUptodate(page)) {
btrfs_readpage(NULL, page);
lock_page(page);
@@ -3176,35 +3219,40 @@ static int relocate_file_extent_cluster(struct inode 
*inode,
unlock_page(page);
put_page(page);
btrfs_delalloc_release_metadata(inode,
-   PAGE_SIZE);
+   reserved_space);
ret = -EIO;
goto out;
}
}
 
-   page_start = page_offset(page);
-   page_end = page_start + PAGE_SIZE - 1;
-
lock_extent(_I(inode)->io_tree, page_start, page_end);

[PATCH V21 14/19] Btrfs: subpage-blocksize: Fix file defragmentation code

2016-10-02 Thread Chandan Rajendra 
This commit gets file defragmentation code to work in subpage-blocksize
scenario. It does this by keeping track of page offsets that mark block
boundaries and passing them as arguments to the functions that implement
the defragmentation logic.

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/ioctl.c | 198 ++-
 1 file changed, 136 insertions(+), 62 deletions(-)

diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
index a222bad..4077fc1 100644
--- a/fs/btrfs/ioctl.c
+++ b/fs/btrfs/ioctl.c
@@ -902,12 +902,13 @@ out_unlock:
 static int check_defrag_in_cache(struct inode *inode, u64 offset, u32 thresh)
 {
struct extent_io_tree *io_tree = _I(inode)->io_tree;
+   struct btrfs_root *root = BTRFS_I(inode)->root;
struct extent_map *em = NULL;
struct extent_map_tree *em_tree = _I(inode)->extent_tree;
u64 end;
 
read_lock(_tree->lock);
-   em = lookup_extent_mapping(em_tree, offset, PAGE_SIZE);
+   em = lookup_extent_mapping(em_tree, offset, root->sectorsize);
read_unlock(_tree->lock);
 
if (em) {
@@ -997,7 +998,7 @@ static struct extent_map *defrag_lookup_extent(struct inode 
*inode, u64 start)
struct extent_map_tree *em_tree = _I(inode)->extent_tree;
struct extent_io_tree *io_tree = _I(inode)->io_tree;
struct extent_map *em;
-   u64 len = PAGE_SIZE;
+   u64 len = BTRFS_I(inode)->root->sectorsize;
 
/*
 * hopefully we have this extent in the tree already, try without
@@ -1116,37 +1117,47 @@ out:
  * before calling this.
  */
 static int cluster_pages_for_defrag(struct inode *inode,
-   struct page **pages,
-   unsigned long start_index,
-   unsigned long num_pages)
+   struct page **pages,
+   unsigned long start_index,
+   size_t pg_offset,
+   unsigned long num_blks)
 {
-   unsigned long file_end;
u64 isize = i_size_read(inode);
+   u64 start_blk;
+   u64 end_blk;
u64 page_start;
u64 page_end;
u64 page_cnt;
+   u64 blk_cnt;
int ret;
int i;
int i_done;
struct btrfs_ordered_extent *ordered;
struct extent_state *cached_state = NULL;
struct extent_io_tree *tree;
+   struct btrfs_root *root;
gfp_t mask = btrfs_alloc_write_mask(inode->i_mapping);
 
-   file_end = (isize - 1) >> PAGE_SHIFT;
-   if (!isize || start_index > file_end)
+   root = BTRFS_I(inode)->root;
+   start_blk = (start_index << PAGE_SHIFT) + pg_offset;
+   start_blk >>= inode->i_blkbits;
+   end_blk = (isize - 1) >> inode->i_blkbits;
+   if (!isize || start_blk > end_blk)
return 0;
 
-   page_cnt = min_t(u64, (u64)num_pages, (u64)file_end - start_index + 1);
+   blk_cnt = min_t(u64, (u64)num_blks, (u64)end_blk - start_blk + 1);
 
ret = btrfs_delalloc_reserve_space(inode,
-   start_index << PAGE_SHIFT,
-   page_cnt << PAGE_SHIFT);
+   start_blk << inode->i_blkbits,
+   blk_cnt << inode->i_blkbits);
if (ret)
return ret;
i_done = 0;
tree = _I(inode)->io_tree;
 
+   page_cnt = DIV_ROUND_UP(pg_offset + (blk_cnt << inode->i_blkbits),
+   PAGE_SIZE);
+
/* step one, lock all the pages */
for (i = 0; i < page_cnt; i++) {
struct page *page;
@@ -1157,12 +1168,22 @@ again:
break;
 
page_start = page_offset(page);
-   page_end = page_start + PAGE_SIZE - 1;
+
+   if (i == 0)
+   page_start += pg_offset;
+
+   if (i == page_cnt - 1) {
+   page_end = (start_index << PAGE_SHIFT) + pg_offset;
+   page_end += (blk_cnt << inode->i_blkbits) - 1;
+   } else {
+   page_end = page_offset(page) + PAGE_SIZE - 1;
+   }
+
while (1) {
lock_extent_bits(tree, page_start, page_end,
 _state);
-   ordered = btrfs_lookup_ordered_extent(inode,
- page_start);
+   ordered = btrfs_lookup_ordered_range(inode, page_start,
+   page_end - page_start + 
1);
unlock_extent_cached(tree, page_start, page_end,
 _state, GFP_NOFS);
if (!ordered)
@@ -1201,7 +1222,7 @@ again:
}
 
pages[i] = page;
-

[PATCH V21 04/19] Btrfs: Remove extent_io_tree's track_uptodate member

2016-10-02 Thread Chandan Rajendra 
We now track block uptodate status using a page's PG_Uptodate
flag. Hence this commit removes the now unused
extent_io_tree->track_uptodate member.

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/disk-io.c   | 1 -
 fs/btrfs/extent_io.h | 1 -
 fs/btrfs/inode.c | 2 --
 3 files changed, 4 deletions(-)

diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index 03ac601..9ff48a7 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -2085,7 +2085,6 @@ int btrfs_init_eb_info(struct btrfs_fs_info *fs_info)
 
eb_info->fs_info = fs_info;
extent_io_tree_init(_info->io_tree, eb_info);
-   eb_info->io_tree.track_uptodate = 0;
eb_info->io_tree.ops = _extent_io_ops;
extent_io_tree_init(_info->io_failure_tree, eb_info);
INIT_RADIX_TREE(_info->buffer_radix, GFP_ATOMIC);
diff --git a/fs/btrfs/extent_io.h b/fs/btrfs/extent_io.h
index 922f4c1..9aa22f9 100644
--- a/fs/btrfs/extent_io.h
+++ b/fs/btrfs/extent_io.h
@@ -102,7 +102,6 @@ struct extent_io_tree {
struct rb_root state;
void *private_data;
u64 dirty_bytes;
-   int track_uptodate;
spinlock_t lock;
const struct extent_io_ops *ops;
 };
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 652d01d..ac4a7c0 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -9306,8 +9306,6 @@ struct inode *btrfs_alloc_inode(struct super_block *sb)
extent_map_tree_init(>extent_tree);
extent_io_tree_init(>io_tree, inode);
extent_io_tree_init(>io_failure_tree, inode);
-   ei->io_tree.track_uptodate = 1;
-   ei->io_failure_tree.track_uptodate = 1;
atomic_set(>sync_writers, 0);
mutex_init(>log_mutex);
mutex_init(>delalloc_mutex);
-- 
2.5.5

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V21 00/19] Allow I/O on blocks whose size is less than page size

2016-10-02 Thread Chandan Rajendra 
Btrfs assumes block size to be the same as the machine's page
size. This would mean that a Btrfs instance created on a 4k page size
machine (e.g. x86) will not be mountable on machines with larger page
sizes (e.g. PPC64/AARCH64). This patchset aims to resolve this
incompatibility.

This patchset continues with the work posted previously at
http://marc.info/?l=linux-btrfs=146760691422240=2

This patchset is based on top of Josef's
1. Metadata throttling in writeback patches
2. Kill the btree inode patches

The major change in this version is the usage of kmalloc()-ed memory for
holding metadata blocks whose size is less than the machine's page size. This
vastly reduces the complexity of extent buffer mangement (Thanks to Josef's
"Kill the btree inode patches").

When writing back dirty extent buffers, we currently track the corresponding
extent buffers using the pointer at page->private. With kmalloc-ed() memory
this isn't possible and hence we track the first extent buffer under writeback
using bio->bi_private. Also, For kmalloc-ed() extent buffers this patchset
currently limits the number of dirty extent buffers in a "write" bio to
1. This limit will be removed in a future patchset.

The commits for the Btrfs kernel module can be found at
https://github.com/chandanr/linux/tree/btrfs/subpagesize-blocksize.

To create a filesystem with block size < page size, a patched version
of the Btrfs-progs package is required. The corresponding fixes for
Btrfs-progs can be found at
https://github.com/chandanr/btrfs-progs/tree/btrfs/subpagesize-blocksize.

Fstests run status:
1. x86_64
   - With 4k sectorsize, all the tests that succeed with the for-linus-4.8
 branch at
 git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git 
 branch also do so with the patches applied.
2. ppc64
   - With 4k sectorsize, 16k nodesize and with "nospace_cache" mount
 option, except for scrub and compression tests, all the tests
 that succeed with the for-next branch at
 git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git
 branch also do so with the patches applied.
   - With 64k sectorsize & nodesize, all the tests that succeed with
 the for-linus-4.8 branch at
 git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git
 branch also do so with the patches applied.

TODO:
1. On ppc64, btrfsck segfaults when checking a filesystem instance
   having 2k sectorsize.
2. I am planning to fix scrub & compression via a separate patchset.

Changes from V20:
1. Applied all the review comments suggested by Josef for version V20.
   However, There are still some instances of
if (compare_sectorsize_with_page_size)
   /* do something */
   One such instance is in check_page_uptodate() where we would need to check
   for BLK_STATE_UPTODATE only if page_size < sectorsize. For page_size ==
   sectorsize case, we unconditionally set PG_uptodate flag.

Changes from V19:
1. The patchset has been rebased on top of kdave/for-next branch.
2. The patch "Btrfs: subpage-blocksize: extent_clear_unlock_delalloc:
   Prevent page from being unlocked more than once" changes the
   signatures of the functions "cow_file_range" &
   "extent_clear_unlock_delalloc". This patch has now been moved to be
   the first patch in the patchset.
3. A new patch "Btrfs: subpage-blocksize: Rate limit scrub error
   message" has been added. btrfs/073 invokes the scrub ioctl in a
   tight loop. In subpage-blocksize scenario this results in a lot of
   "scrub: size assumption sectorsize != PAGE_SIZE" messages being
   printed on the console. Hence this patch rate limits such error
   messages.

Changes from V18:
1. The per-page bitmap used to track the block status is now allocated
   from a slab cache.
2. The per-page bitmap is allocated and used only in cases where
   sectorsize < PAGE_SIZE.
3. The new patch "Btrfs: subpage-blocksize: Disable compression"
   disables compression in subpage-blocksize scenario.

Changes from V17:
1. Due to mistakes made during git rebase operations, fixes ended up
   in incorrect patches. This patchset gets the fixes in the right
   patches.

Changes from V16:
1. The V15 patchset consisted of patches obtained from an incorrect
   git branch. Apologies for the mistake. All the entries listed under
   "Changes from V15" hold good for V16.

Changes from V15:
1. The invocation of cleancache_get_page() in __do_readpage() assumed
   blocksize to be same as PAGE_SIZE. We now invoke cleancache_get_page()
   only if blocksize is same as PAGE_SIZE. Thanks to David Sterba for
   pointing this out.
2. In __extent_writepage_io() we used to accumulate all the contiguous
   dirty blocks within the page before submitting the file offset range
   for I/O. In some cases this caused the bio to span across more than
   a stripe. For example, With 4k block size, 64K stripe size
   and 64K page size, assume
   - All the blocks mapped by the page are contiguous on the logical

[PATCH V21 16/19] Btrfs: subpage-blocksize: btrfs_clone: Flush dirty blocks of a page that do not map the clone range

2016-10-02 Thread Chandan Rajendra 
After cloning the required extents, we truncate all the pages that map
the file range being cloned. In subpage-blocksize scenario, we could
have dirty blocks before and/or after the clone range in the
leading/trailing pages. Truncating these pages would lead to data
loss. Hence this commit forces such dirty blocks to be flushed to disk
before performing the clone operation.

Signed-off-by: Chandan Rajendra 
---
 fs/btrfs/ioctl.c | 16 
 1 file changed, 16 insertions(+)

diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
index cf13029..0fdc0a0 100644
--- a/fs/btrfs/ioctl.c
+++ b/fs/btrfs/ioctl.c
@@ -3914,6 +3914,7 @@ static noinline int btrfs_clone_files(struct file *file, 
struct file *file_src,
int ret;
u64 len = olen;
u64 bs = root->fs_info->sb->s_blocksize;
+   u64 dest_end;
int same_inode = src == inode;
 
/*
@@ -3974,6 +3975,21 @@ static noinline int btrfs_clone_files(struct file *file, 
struct file *file_src,
goto out_unlock;
}
 
+   if ((round_down(destoff, PAGE_SIZE) < inode->i_size) &&
+   !IS_ALIGNED(destoff, PAGE_SIZE)) {
+   ret = filemap_write_and_wait_range(inode->i_mapping,
+   round_down(destoff, PAGE_SIZE),
+   destoff - 1);
+   }
+
+   dest_end = destoff + len - 1;
+   if ((dest_end < inode->i_size) &&
+   !IS_ALIGNED(dest_end + 1, PAGE_SIZE)) {
+   ret = filemap_write_and_wait_range(inode->i_mapping,
+   dest_end + 1,
+   round_up(dest_end, PAGE_SIZE));
+   }
+
if (destoff > inode->i_size) {
ret = btrfs_cont_expand(inode, inode->i_size, destoff);
if (ret)
-- 
2.5.5

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html