Kernel BUG: btrfs send - Incremental backup

2014-03-07 Thread Swâmi Petaramesh
Hi there,

I tried to perform an incremental backup as described in 
https://btrfs.wiki.kernel.org/index.php/Incremental_Backup  between 2 external 
USB drives,

The 1st "btrfs send foo/snap1 | btrfs receive bar" went well, although it took 
5-6 times the time the same workload takes in ZFS.

Then "btrfs send -v -p foo/snap1 foo/snap2 | btrfs receive bar" crashed on me.

After a reboot and retry, I get a KERNEL BUG again :


# uname -a
Linux tethys 3.13.5-1-ARCH #1 SMP PREEMPT Sun Feb 23 00:25:24 CET 2014 x86_64 
GNU/Linux

btrfs send -v -p foo/snap1 foo/snap2 | btrfs receive bar


mars 08 08:22:01 tethys kernel: [ cut here ]
mars 08 08:22:01 tethys kernel: kernel BUG at fs/btrfs/send.c:4398!
mars 08 08:22:02 tethys kernel: invalid opcode:  [#1] PREEMPT SMP 
mars 08 08:22:02 tethys kernel: Modules linked in: usb_storage md5 fuse ecb 
ecryptfs encrypted_keys hmac trusted tpm ctr ccm intel_rapl 
x86_pkg_temp_thermal rts5139(C) 
mars 08 08:22:02 tethys kernel:  usbhid hid sr_mod cdrom sd_mod atkbd libps2 
crct10dif_pclmul crct10dif_common crc32_pclmul ahci crc32c_intel 
ghash_clmulni_intel libahc
mars 08 08:22:02 tethys kernel: CPU: 0 PID: 2432 Comm: btrfs Tainted: P 
C O 3.13.5-1-ARCH #1
mars 08 08:22:02 tethys kernel: Hardware name: TOSHIBA SATELLITE L735/Base 
Board Product Name, BIOS 2.50 06/26/2012
mars 08 08:22:02 tethys kernel: task: 880041724800 ti: 88003b84a000 
task.ti: 88003b84a000
mars 08 08:22:02 tethys kernel: RIP: 0010:[]  
[] 
changed_cb+0x9e5/0x9f0 [btrfs]
mars 08 08:22:02 tethys kernel: RSP: 0018:88003b84bb48  EFLAGS: 00010283
mars 08 08:22:02 tethys kernel: RAX: 000579d5 RBX: 88013279ea00 
RCX: 
0060
mars 08 08:22:02 tethys kernel: RDX: 88003b84bc65 RSI: 000579cd 
RDI: 
88013279ea00
mars 08 08:22:02 tethys kernel: RBP: 88003b84bbe0 R08: 88003b84bc65 
R09: 
0002
mars 08 08:22:02 tethys kernel: R10: 88005981a900 R11:  
R12: 
0002
mars 08 08:22:02 tethys kernel: R13: 88003b84bc65 R14: 88007c628800 
R15: 
2ad1
mars 08 08:22:02 tethys kernel: FS:  7fceade62880() 
GS:88014f40() knlGS:
mars 08 08:22:02 tethys kernel: CS:  0010 DS:  ES:  CR0: 
80050033
mars 08 08:22:02 tethys kernel: CR2: 7fb4cc8c8000 CR3: 41a99000 
CR4: 000407f0
mars 08 08:22:02 tethys kernel: Stack:
mars 08 08:22:02 tethys kernel:  88003b84bb88 a03206ec 
08ca 
88005981a900
mars 08 08:22:02 tethys kernel:   88005981a900 
88005981a870 
mars 08 08:22:02 tethys kernel:  88003b84bbe0 a03161e9 
88003b84bc76 
88003b84bbe0
mars 08 08:22:02 tethys kernel: Call Trace:
mars 08 08:22:02 tethys kernel:  [] ? 
read_extent_buffer+0xbc/0x110 [btrfs]
mars 08 08:22:02 tethys kernel:  [] ? 
btrfs_get_token_32+0x59/0xe0 [btrfs]
mars 08 08:22:02 tethys kernel:  [] ? 
memcmp_extent_buffer+0xc1/0x120 [btrfs]
mars 08 08:22:02 tethys kernel:  [] 
btrfs_compare_trees+0x790/0x970 [btrfs]
mars 08 08:22:02 tethys kernel:  [] ? 
finish_inode_if_needed+0x4e0/0x4e0 [btrfs]
mars 08 08:22:02 tethys kernel:  [] 
btrfs_ioctl_send+0x8a8/0xc80 
[btrfs]
mars 08 08:22:02 tethys kernel:  [] btrfs_ioctl+0xfee/0x27a0 
[btrfs]
mars 08 08:22:02 tethys kernel:  [] ? 
__enqueue_entity+0x78/0x80
mars 08 08:22:02 tethys kernel:  [] ? 
enqueue_entity+0x29b/0xa10
mars 08 08:22:02 tethys kernel:  [] ? 
enqueue_task_fair+0x107/0x540
mars 08 08:22:02 tethys kernel:  [] ? enqueue_task+0x3a/0x60
mars 08 08:22:02 tethys kernel:  [] ? 
check_preempt_curr+0x75/0xa0
mars 08 08:22:02 tethys kernel:  [] do_vfs_ioctl+0x2e0/0x4c0
mars 08 08:22:02 tethys kernel:  [] ? do_fork+0x12f/0x320
mars 08 08:22:02 tethys kernel:  [] SyS_ioctl+0x81/0xa0
mars 08 08:22:02 tethys kernel:  [] 
system_call_fastpath+0x1a/0x1f
mars 08 08:22:02 tethys kernel: Code: 48 8d 75 b6 45 31 c0 b9 01 00 00 00 4c 
89 e2 4c 89 f7 e8 2f eb f7 ff 85 c0 0f 89 77 ff ff ff e9 a0 fd ff ff 31 c0 e9 
99 fd 
ff ff <
mars 08 08:22:02 tethys kernel: RIP  [] 
changed_cb+0x9e5/0x9f0 
[btrfs]
mars 08 08:22:02 tethys kernel:  RSP 
mars 08 08:22:02 tethys kernel: ---[ end trace 718b3fc11b4d9241 ]---


HTH, TIA.

-- 
Swâmi Petaramesh  http://petaramesh.org PGP 9076E32E

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


RE: ENOSPC errors during raid1 rebalance

2014-03-07 Thread Mike Russo
Alright! After doing:

cd /mymedia; find . -type f | while read file; do mv -v "$file" /dev/shm; 
f2=`basename "$file"`; mv -v "/dev/shm/$f2" "$file"; done 

I finally moved whatever files out of the "single" allocation and back onto the 
new RAID1 profile:

oot@ossy:~# /usr/src/btrfs-progs/btrfs fi df /mymedia
Data, RAID1: total=1.23TiB, used=1000.43GiB
Data, single: total=10.00GiB, used=0.00
System, RAID1: total=32.00MiB, used=184.00KiB
Metadata, RAID1: total=3.00GiB, used=1.34GiB

And then the rebalance finally was able to move those two block groups and I 
got no errors:

root@ossy:~# btrfs balance start -dconvert=raid1,soft /mymedia
Done, had to relocate 2 out of 1264 chunks

And now I'm totally RAID1:

root@ossy:~# /usr/src/btrfs-progs/btrfs fi df /mymedia
Data, RAID1: total=1.23TiB, used=1000.43GiB
System, RAID1: total=32.00MiB, used=184.00KiB
Metadata, RAID1: total=3.00GiB, used=1.34GiB

Yay!! If I want to get back the space I can do another defragment but I don't 
really care right now.  Thanks everyone for all your help on this! Hopefully 
this thread will assist anyone else in the future if this occurs again. 

Sincerely,

Michael Russo, Systems Engineer
PaperSolve, Inc.
268 Watchogue Road
Staten Island, NY 10314

Randomly generated quote of the last 5 minutes:
A good plan today is better than a perfect plan tomorrow.
-- Patton
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [PATCH 4/9] Btrfs: use bitfield instead of integer data type for the some variants in btrfs_root

2014-03-07 Thread Josef Bacik
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 02/20/2014 05:08 AM, Miao Xie wrote:
> Signed-off-by: Miao Xie  --- fs/btrfs/ctree.c
> | 25 ++--- fs/btrfs/ctree.h   | 39
> +-- fs/btrfs/disk-io.c | 33
> +++-- fs/btrfs/extent-tree.c |  6
> +++--- fs/btrfs/file.c|  4 +++- fs/btrfs/inode.c   | 29
> ++--- fs/btrfs/ioctl.c   |  4 ++-- 
> fs/btrfs/relocation.c  | 17 + fs/btrfs/root-tree.c
> |  2 +- fs/btrfs/transaction.c | 33
> + fs/btrfs/tree-defrag.c |  2 +- 
> fs/btrfs/tree-log.c|  9 + 12 files changed, 109
> insertions(+), 94 deletions(-)
> 
> diff --git a/fs/btrfs/ctree.c b/fs/btrfs/ctree.c index
> cbd3a7d..9067d79 100644 --- a/fs/btrfs/ctree.c +++
> b/fs/btrfs/ctree.c @@ -224,7 +224,8 @@ static struct extent_buffer
> *btrfs_read_lock_root_node(struct btrfs_root *root) static void
> add_root_to_dirty_list(struct btrfs_root *root) { 
> spin_lock(&root->fs_info->trans_lock); -  if (root->track_dirty &&
> list_empty(&root->dirty_list)) { +if
> (test_bit(BTRFS_ROOT_TRACK_DIRTY, &root->state) && +
> list_empty(&root->dirty_list)) { list_add(&root->dirty_list, 
> &root->fs_info->dirty_cowonly_roots); } @@ -246,9 +247,10 @@ int
> btrfs_copy_root(struct btrfs_trans_handle *trans, int level; struct
> btrfs_disk_key disk_key;
> 
> - WARN_ON(root->ref_cows && trans->transid != -
> root->fs_info->running_transaction->transid); -
> WARN_ON(root->ref_cows && trans->transid != root->last_trans); +
> WARN_ON(test_bit(BTRFS_ROOT_REF_COWS, &root->state) && +
> trans->transid != root->fs_info->running_transaction->transid); +
> WARN_ON(test_bit(BTRFS_ROOT_REF_COWS, &root->state) && +
> trans->transid != root->last_trans);
> 
> level = btrfs_header_level(buf); if (level == 0) @@ -997,14 +999,14
> @@ int btrfs_block_can_be_shared(struct btrfs_root *root, *
> snapshot and the block was not allocated by tree relocation, * we
> know the block is not shared. */ -if (root->ref_cows && + if
> (test_bit(BTRFS_ROOT_REF_COWS, &root->state) && buf != root->node
> && buf != root->commit_root && (btrfs_header_generation(buf) <= 
> btrfs_root_last_snapshot(&root->root_item) || 
> btrfs_header_flag(buf, BTRFS_HEADER_FLAG_RELOC))) return 1; #ifdef
> BTRFS_COMPAT_EXTENT_TREE_V0 - if (root->ref_cows && + if
> (test_bit(BTRFS_ROOT_REF_COWS, &root->state) && 
> btrfs_header_backref_rev(buf) < BTRFS_MIXED_BACKREF_REV) return 1; 
> #endif @@ -1146,9 +1148,10 @@ static noinline int
> __btrfs_cow_block(struct btrfs_trans_handle *trans,
> 
> btrfs_assert_tree_locked(buf);
> 
> - WARN_ON(root->ref_cows && trans->transid != -
> root->fs_info->running_transaction->transid); -
> WARN_ON(root->ref_cows && trans->transid != root->last_trans); +
> WARN_ON(test_bit(BTRFS_ROOT_REF_COWS, &root->state) && +
> trans->transid != root->fs_info->running_transaction->transid); +
> WARN_ON(test_bit(BTRFS_ROOT_REF_COWS, &root->state) && +
> trans->transid != root->last_trans);
> 
> level = btrfs_header_level(buf);
> 
> @@ -1193,7 +1196,7 @@ static noinline int __btrfs_cow_block(struct
> btrfs_trans_handle *trans, return ret; }
> 
> - if (root->ref_cows) { + if (test_bit(BTRFS_ROOT_REF_COWS,
> &root->state)) { ret = btrfs_reloc_cow_block(trans, root, buf,
> cow); if (ret) return ret; @@ -1556,7 +1559,7 @@ static inline int
> should_cow_block(struct btrfs_trans_handle *trans, 
> !btrfs_header_flag(buf, BTRFS_HEADER_FLAG_WRITTEN) && 
> !(root->root_key.objectid != BTRFS_TREE_RELOC_OBJECTID && 
> btrfs_header_flag(buf, BTRFS_HEADER_FLAG_RELOC)) && -
> !root->force_cow) +   !test_bit(BTRFS_ROOT_REF_COWS,
> &root->state))

This may or may not be the problem.  Thanks,

Josef

-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJTGl2IAAoJEANb+wAKly3Br0EQAKWfEahMQ1ZV9jkgSYF/9c6r
ytoH2/Y3u7HYeeqf+Z/j2lkMyvAaWQ9EQCkWSUyCqoU7RPXHEROSCQ11UCk8qYXh
fhV2r3plOWIZ/KDHFvqPTN5FTg97OLvDalyvOR6UP/Ws6Z4Ycm0ePm7kb+25iK9u
N+PkiqHoQqUVw8Z2EEJ2SN/82SyNnuPDG2RkDD9xW8el5fBplAPgUox8W8Z+ubIo
FPiSNv4euly2Zco+Vs3NDFb2tqQBPyAVzE2IC4Nyq1Hci/vyC9k8YfIcCJnsP0Dk
10mVDhSSStWLuqt2L7fUV8nOTyjKT1gBNgoz/eMBeOzWLDbRKD2hpS6wpkw/6/iQ
/ff9Spikw7a87epYo4dxft32aQsDIu6JfgFPggL+VkXyn114MK4U5z7KNOPQUSBq
neFOVELgN1L75TI9v9/p1qKGeZ47vV1lvd6GP717SDF0yv9wgvHR0Ma47KSaLq79
WlwzmqXDYJYdOedKGQky6GZ7EFji5DDlazx7h1pQTN5rEdiQJTEFxrxtMOKzAttF
3EL9wxVAi2ggD2EYWWsk+SJNJxgU59bxTR9ZiOH+tj29+gFcGUpRsXMsj4KampK1
j0a0BSLj+1yUlLcT8qem05aJk720zxy4UBiZGcRP7qySlrOTwU5Xtk0BXXyfUXLg
guOgjR0pj494irbM53RT
=H5nB
-END PGP SIGNATURE-
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [PATCH 4/9] Btrfs: use bitfield instead of integer data type for the some variants in btrfs_root

2014-03-07 Thread Josef Bacik
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 02/20/2014 05:08 AM, Miao Xie wrote:
> Signed-off-by: Miao Xie  --- fs/btrfs/ctree.c
> | 25 ++--- fs/btrfs/ctree.h   | 39
> +-- fs/btrfs/disk-io.c | 33
> +++-- fs/btrfs/extent-tree.c |  6
> +++--- fs/btrfs/file.c|  4 +++- fs/btrfs/inode.c   | 29
> ++--- fs/btrfs/ioctl.c   |  4 ++-- 
> fs/btrfs/relocation.c  | 17 + fs/btrfs/root-tree.c
> |  2 +- fs/btrfs/transaction.c | 33
> + fs/btrfs/tree-defrag.c |  2 +- 
> fs/btrfs/tree-log.c|  9 + 12 files changed, 109
> insertions(+), 94 deletions(-)
> 

This patch causes xfstests to lock up the box.  I could reproduce by
running

./check generic/070 generic/074 generic/075

in a loop.  It blows up at 075 and usually happens right away, but
sometimes doesn't blow up until the second go around.  I'm dropping it
for now.  Thanks,

Josef

-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJTGlwuAAoJEANb+wAKly3BrE4P+wY0BhYvT1PjMBu6nFSZFxSd
5an4GGW5aoRrQ3G0wZkEpWPJK34exLvuzzaNxHH9FSesz27l1bnibNTUy7BzI7Mv
tj92Dxmckpcvl+z3QI+CcPkI6NpVzBNETRxBpRVtkqxWKioEhk4oOwvctBqbqaI1
lEOxylu6/09k2BjjeehJFLjzhVzq9taTUt7ETAaBaJLsPFdD9/E8S1EO/uJyabSo
yGSWSHEnXUMjRCaXZ1bZJKDowTZ10wviU4rDPyFi99h4h1m6FZe5jNStHojNcNkc
Qt1xv3s2bkuz7yAC9/yJuz4HFiWtiyRHD609vBiIBO93hLCFi5Xc4jXtSRVVC+Xx
891gfD0c8/SVJpviTOFnxewozmIgudcC4y3i2AglRI+IXEkNRr66G6v/H0uZEN+n
lhiNTLQ+/Kd+mxUA7Y3To86Hnb55TqX7G8AL2zdFKBb1LJLBCazcEQZkDDraG2MW
x77H38VvKyQ+pBgv/lZFnKHadPkveFrGJUYCx49SOHAvWDoDVcIx2liayR8WYBMA
reRT6Vd+sH4xkgVXXNhrFcDFsnbJp1R9LkHuk4HW56QASaMcLqwZ/RmwL/3ldX0c
17IyUZkXZx79rbzBwL8EmU9Gb+oDkNeu8nXIzVnBERDBQbDQqSThaLeiPfuJNPjU
mGmvLKc/fXNN3RofWrSx
=N9yr
-END PGP SIGNATURE-
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[PATCH] Btrfs: return EPERM when deleting a default subvolume

2014-03-07 Thread Guangyu Sun
The error message is confusing:

 # btrfs sub delete /mnt/mysub/
 Delete subvolume '/mnt/mysub'
 ERROR: cannot delete '/mnt/mysub' - Directory not empty

The error message does not make sense to me: It's not about deleting a
directory but it's a subvolume, and it doesn't matter if the subvolume is
empty or not.

Maybe EPERM or is more appropriate in this case, combined with an explanatory
kernel log message. (e.g. "subvolume with ID 123 cannot be deleted because
it is configured as default subvolume.")

Reported-by: Koen De Wit 
Signed-off-by: Guangyu Sun 
---
 fs/btrfs/ioctl.c |5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
index a6d8efa..c380854 100644
--- a/fs/btrfs/ioctl.c
+++ b/fs/btrfs/ioctl.c
@@ -1797,7 +1797,10 @@ static noinline int may_destroy_subvol(struct btrfs_root 
*root)
if (di && !IS_ERR(di)) {
btrfs_dir_item_key_to_cpu(path->nodes[0], di, &key);
if (key.objectid == root->root_key.objectid) {
-   ret = -ENOTEMPTY;
+   ret = -EPERM;
+   printk(KERN_ERR "subvolume %llu cannot be deleted "
+  "because it is configured as default "
+  "subvolume.\n", key.objectid);
goto out;
}
btrfs_release_path(path);
-- 
1.7.9.5

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [PATCH v2] xfstests: add test for btrfs send issuing premature rmdir operations

2014-03-07 Thread Josef Bacik
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 02/24/2014 06:54 AM, Filipe David Borba Manana wrote:
> Regression test for btrfs incremental send issue where a rmdir
> instruction is sent against an orphan directory inode which is not
> empty yet, causing btrfs receive to fail when it attempts to remove
> the directory.
> 
> This issue is fixed by the following linux kernel btrfs patch:
> 
> Btrfs: fix send attempting to rmdir non-empty directories
> 
> Signed-off-by: Filipe David Borba Manana  ---

Looks good to me,

Reviewed-by: Josef Bacik 

Thanks,

Josef
-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJTGg5bAAoJEANb+wAKly3BFtMQAMDITxFeFKRw5o5g88NUJT+n
/AQp584QUb+sBhJnyYVET98S47R4C9d7jAu8fUxUJd2FaZ0UPR+rl/LnpdYXy8oJ
29A2tvbSiOZGQp+kWlGpLqKmmEj/3R/0QZwTrB1jsxKwVsH+OGK1fazOwhkahloL
8R6xBtcvMzkBFEQ08T0dmodDaWKPp8dgztcScrVt+RGkePPVoKE9X4hwTwAl8muk
nPv9TvRULOs2emvxuTnEVqZfou0HcOTuKWXAOXN2peXBe324SjfQ0JmAMjtG1Ukw
nAs5wu8eII1k9/rjGLarNhpuSuMaM5tth0tIb2ebYwxwpYpY6o4eNN2RR6vWlCEz
vULou68VKdOG6/c6uO6cXuI8zOwScWuYyYFXFLTFpAhjdKwY1HaCwSanvnamdYy8
kNITDGkYSqs0Abj9wzQDcJcTrjpHDDe/3olb5LMJjrdTneMyDVXc+Ym+FbBhmQUX
d7vC+a7sRVWwN4xGoqxDgZZQLpGBEO60FhA5kIO9lgis2LdG8SXE0ch4sQuWZ8iC
S9Aw46dssmiY+d3BkugCOPW/7IlID/obLGUF+CWcWMkxSMbwaPn6RfNJmW+9k+rv
fkUVj+U75pBtnuwW3LxV7gUq8HwTz8G1fh+96o41yKXpayBQUoIA47AHCUZtP9Fi
RemqwFmZxhi7ztZYfni2
=8Kl8
-END PGP SIGNATURE-
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [PATCH] btrfs: Allow forced conversion of metadata to dup profile on, multiple devices

2014-03-07 Thread Josef Bacik
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 02/26/2014 09:23 AM, Austin S Hemmelgarn wrote:
> Currently, btrfs balance start fails when trying to convert
> metadata or system chunks to dup profile on filesystems with
> multiple devices.  This requires that a conversion from a
> multi-device filesystem to a single device filesystem use the
> following methodology: 1. btrfs balance start -dconvert=single
> -mconvert=single \ -sconvert=single -f / 2. btrfs device delete /
> /dev/sdx 3. btrfs balance start -mconvert=dup -sconvert=dup / This
> results in a period of time (possibly very long if the devices are 
> big) where you don't have the protection guarantees of multiple
> copies of metadata chunks.
> 
> After applying this patch, one can instead use the following
> methodology for conversion from a multi-device filesystem to a
> single device filesystem: 1. btrfs balance start -dconvert=single
> -mconvert=dup \ -sconvert=dup -f / 2. btrfs device delete /
> /dev/sdx This greatly reduces the chances of the operation causing
> data loss due to a read error during the device delete.
> 
> Signed-off-by: Austin S. Hemmelgarn  ---

Fails to build, wait for me to push an updated btrfs-next and rebase
and resubmit

fs/btrfs/volumes.c: In function ‘btrfs_balance’:
fs/btrfs/volumes.c:3223:55: warning: suggest parentheses around ‘&&’
within ‘||’ [-Wparentheses]
   if (((bctl->sys.flags & BTRFS_BALANCE_ARGS_CONVERT) &&
   ^
fs/btrfs/volumes.c:3227:8: error: ‘num_devs’ undeclared (first use in
this function)
   (num_devs > 1)) {
^
fs/btrfs/volumes.c:3227:8: note: each undeclared identifier is
reported only once for each function it appears in
make[1]: *** [fs/btrfs/volumes.o] Error 1
make: *** [_module_fs/btrfs] Error 2

Thanks,

Josef

-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJTGgp0AAoJEANb+wAKly3BrkkQANOaAAk6LJ4VO86f9/N7Hkom
Hf4FK5lQiMkzlLc8pz+RxHeIHSXScXdrVKPv8bIPoGwd171OIrMA1HZa/9iuez8z
OInjO6Zwj6l/+N3/QckAJKkNmbInNH2wgdhUrFyUw+gDFvVnl1YGUcTwx6/udqTB
kfsls2ivD7S9kfSXyaM4oxVMN+tZWpZtOs1TMpf7BDMIz92gr87VADKIocrlu5qh
EZ2YyWGyIP87jz+7zzNPUQ00/BAgC1lPnbZf+ei0L1KQbDtjII3Rl1/rlg78Zdtb
VQfYMBz0hpaOE+UNfVsClgDgMjAkphob7BXTrkzJaChOagJ5tIlSGZRUaHPt3qaz
so1R/3BaPzlTdv7gPCpMg+nSYdNl0x3w5CauCB4EX3L8PtbxK9tQdFtKmpl5GNrZ
gBADD+AAbyPoz4GO2lfQyIN6Hd5FA+drK6cErF/rY+dPlieyihdYfBWRU+nao9RY
ju07FN2UHJaAlSh/K8ZLv4lZUULDObr/tOdg5M1h5piSL9H9jOHUdaLmRzSx+7hZ
j96qRY9/W4L6avEQQLphGbgn3v+aOHcgxLS0MyO/fhXANah2z69PjpS5J7GOXGZ5
UDGTneQIeRqueYms1yqpmZA0Ctwavq0TGoDvYsOlf/x7vucJE6a02qMRitUJbuvV
6m+8a6PFRudGng7zQYef
=KCPn
-END PGP SIGNATURE-
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [PATCH 1/2 v2] Btrfs: rename waiting_dir_move

2014-03-07 Thread Josef Bacik
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 02/27/2014 02:47 AM, Liu Bo wrote:
> This is a preparation work, rename waiting_dir_move to
> send_dir_node. We'd like to share waiting_dir_move structure in new
> did_create_dir() code.
> 
> Signed-off-by: Liu Bo  --- v2: fix wrong
> patch name.
> 
This isn't applying cleanly, rebase once I push out btrfs-next today
and resend.  Thanks,

Josef

-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJTGglEAAoJEANb+wAKly3Bm9YP+wSnaq7jN0pQT8UVORFE2BVI
rTKGWfSnJQtEAUx3sF0xCp6dCQlkG+BBPHmK3W1yGovtHQrkR0rkGufx0aLzqD2m
xpV7qQKa9rAQjK5Xsx0K0vDETqYFzEIGv+bv1Q6jFVKoEBVSSdWDPZoSLkVZ9T0Q
aoYUvC/xd4fkp6ikb11vL+3c7WysWaDpSTflZdEFgrd5PgE+r1CoxmTs5NMCxhIK
V6GEu/l3yKK2/95CNAfhkkeRXyV2NVxf/AzM0Jay7+zyCUBM4u2B7BAuC7ZlCxG2
kUhqTw4TkFUIW6mW/sszKWJM9bjDLyq7bYzmJ08yFeiiS0Pvr9NaYxkNc55HUve0
ZLnrICzAJbjQV0f01+Tn5G2/jKX6PEeGRcyWJ1RgEFBRDPh5AMsbHeme3JKWkikZ
xxtT3gvZMlASm1CFtepoFQiBwtd8KR7VJIS9VAhO7VGyvWu3+DSvGe8di+OKvjIh
kBnx7mgfmQChIe4Ccem3iJsNPkR/e9vpoZalBSGTnBGNiztDc64Ht3Xh+YN034n1
4zw6XjYoKo+NJ/8pz2auKjWtZdC3AUSEkaU3koB4U1+uyQJAqDfki8j0Trye5ehn
TemIj6kSfOvuuiCaGCqYwJJYG5r0UXALRwpGtiMk+SGzUklbjmg8q1LKKJiagV8K
yVTNyMKP8Qldlsw1bAPm
=Nkos
-END PGP SIGNATURE-
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Problems on previously perfect setup

2014-03-07 Thread KC
Recently (a week or so back) I started having those problems on my BTRFS 
system


-
I. My setup:
-
SSD
/boot   ext2defaults0   2
/  btrfs rw,noatime,compress=lzo,ssd,space_cache,autodefrag 0 0 (there 
is a /home subvolume mounted automatically)


HDD
/mnt/Disk_D  btrfs rw,noatime,compress=zlib,autodefrag,nofail 0 0
(I disconnected it, but it changed nothing)


--
II. My problems:
--
- 1 - . During random boots (seems to be a 50/50 chance) the system 
won't boot, because it get's stuck on


[ OK ] Mounted Configuration File system
[ *** ] (1 of 13) A start job is running for dev-disk-by\.

It will slowly cycle from 1 to 13, then come back to 1 to cycle again, 
and again, and after three times, I get this:


Timed out waiting for device dev-disk-by\...
Dependency failed for /boot
Dependency failed for Local File System

The computer freezes and I have to shut it down by holding the power 
button for several seconds.




- 2 - . My successful boot times extended considerably. I am using an 
automatic login and X start without any DM. Before, I could hardly see 
the login command line because it was so quickly covered by started 
OpenBox session. Now it lingers several seconds while the disk LED is 
working hard.




- 3 - . For the first 10-15 seeocnds after getting into OpenBox the 
heaviest processes displayed by Conky are several instances of kworker. 
Also, it might be an impression, but I feel like the starting almost 
every program became sluggish.



--
III. Recent changes to the system:
--
Perhaps they could indicate potential suspects:

- upgraded kernel (now 3.13.5-2)
- upgraded btrfs tools (always using latest David Sterba's unstable 
'integration' branch)
- did "btrfs balance /" because it had write erros due to no space even 
though there was like 30GB free. After doing it, no space problem went away




Does anyone have any idea what could be happening?
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [PATCH 1/3] Btrfs: don't skip the page flush since the enospc is not brought by it

2014-03-07 Thread Josef Bacik
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 02/27/2014 12:58 AM, Miao Xie wrote:
> As we know, btrfs flushes the continuous pages as many as
> possible, but if all the free spaces are small, we will allocate
> the spaces by several times, and if there is something wrong with
> the space reservation, it is very likely that some allocations
> succeed and the others fail. But the current code doesn't take this
> case into account, and set the error flag for the pages though
> their spaces are allocated successfully. It introduces a problem
> that we can not umount the fs after the above problem happens
> because we need wait for the flush of those pages whose spaces are
> allocated. This patch fixes the above problem, and makes the btrfs
> developers happy when they investigate the problem of the space
> reservation.
> 
> Signed-off-by: Miao Xie  --- 
> fs/btrfs/extent_io.c | 15 +-- 1 file changed, 13
> insertions(+), 2 deletions(-)
> 
> diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c index
> fbe501d..97595ea 100644 --- a/fs/btrfs/extent_io.c +++
> b/fs/btrfs/extent_io.c @@ -3198,8 +3198,19 @@ static int
> __extent_writepage(struct page *page, struct writeback_control
> *wbc, &nr_written); /* File system has been set read-only */ if
> (ret) { - SetPageError(page); -   
> goto done; +/* +  
>*
> Private2 means we allocate the space + * for 
> this page
> successfully, and enospc + * error doesn't set 
> the fs to be
> R/O, so +  * we can write out the page and 
> needn't +   * set
> error flag for this page. If so, we +  * can prevent 
> the umount
> task from being +  * blocked. +   
>  */ +   if (!(ret == -ENOSPC
> && PagePrivate2(page))) { +   
> SetPageError(page); +   goto
> done; +   }

So now we just ignore ret == -ENOSPC, we still need to return this
back to the caller.  Thanks,

Josef

-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJTGez/AAoJEANb+wAKly3B9z4QAJ7VWrt7yfW+IpjzVGViPrmb
S/Q+nuUEZjFk9huKePz33MaT76gWZlrFM6ZgfSLIEP3gPDaQIz+mrCmDBDSnD0uT
ZJypte70l74Rfmp0gBHs+kv9VQPQkeDE17PMGCJBHnqvm/P45PgYU81nWAMhh3uQ
Khsu5TDW7Tddi8FHyuD9a8TBdMlZfRLDOvWCTY/9OUPpRhaOhL8RBffw6iKfnq/t
M32nGEbZGH1y/kWWc39JNKlUwzW0m0oclznPKp0wXo+lMFmHwl76TGgBl0tvLyAs
VGhv101dH0b7vuQOerXSajVjBGFUqovIW3lDPwf4aU4XWSSebbXxeaYiEs5YMFU4
ivzXEMpYBHEteSW9TzJ0Th+D60LhG2vApsG10Ewj7xEblCjcaO8mej6L0j7citw0
WtNiSACyvIlw4zCWT3699UJlmIJ2fsrxEb5SKfE5QFpOXshhSciA8YZJyuX2EBVE
je/rUMhTLCqkuY4zKZLzJiqRhNGQQbWZn4+SmRxpg5RsuxzIZzZcWiEEfucdhB6w
7fKhtMBcY4rUzszG2otEbQaPLNvUa/u04448/gsYhvwy9II0WSkLbe3LF0IS1Z6o
DM0vUE3FAI80EL/fwyg2VvyxOOSYSE9feoX5ClJOZ5qJamdSqLI9PND04lMz6Dp0
+UUKtPx2wn6CnqO3vBeH
=94MT
-END PGP SIGNATURE-
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[PATCH 1/3 v2 RFC] btrfs: total_devices vs num_devices

2014-03-07 Thread Anand Jain
The intended usage of total_devices and num_devices
should be recorded in the comments so that these
two counters can be used correctly as originally
intended.

As of now there appears to be slight deviations/bugs
from the original intention, the bugs are apparent
that num_devices does not count seed devices where
as total_devices does, but total_devices does not
count the replacing devices where as num_devices does.

So in this situation the ioctl(BTRFS_IOC_DEVICES_READY)
will fails when there is seed FS. Next, applications
using ioctl(BTRFS_IOC_FS_INFO) and ioctl(BTRFS_IOC_DEV_INFO)
in conjunction to allocate the slots will also fail
as former depend on the num_devices where as latter
would match total_devices (except when replace is running).

This patch will help have the clarity on usage
of these two counter so that bugs related to this can
be fixed.

Signed-off-by: Anand Jain 
---
v2: edit commit

 fs/btrfs/volumes.h | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/fs/btrfs/volumes.h b/fs/btrfs/volumes.h
index 91b7596..99c71aa 100644
--- a/fs/btrfs/volumes.h
+++ b/fs/btrfs/volumes.h
@@ -121,12 +121,18 @@ struct btrfs_fs_devices {
/* the device with this id has the most recent copy of the super */
u64 latest_devid;
u64 latest_trans;
+   /* num_devices contains run time count of devices which are part
+* of this FSID. (the FS devices + replacing devices + ?
+*/
u64 num_devices;
u64 open_devices;
u64 rw_devices;
u64 missing_devices;
u64 total_rw_bytes;
u64 num_can_discard;
+   /* total_devices contains static recorded count of device which
+* are part of this FSID. (the FS devices + seed devices + ?
+*/
u64 total_devices;
struct block_device *latest_bdev;
 
-- 
1.8.5.3

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[PATCH 2/3 RFC] btrfs: total_devices should count replacing devices

2014-03-07 Thread Anand Jain
ioctl(BTRFS_IOC_FS_INFO) returns num_devices which does not
count seed device. num_devices is used to calculate
the number of slots during the ioctl(BTRFS_IOC_DEV_INFO)
but ioctl(BTRFS_IOC_DEV_INFO) would count seed devices as well.
Due to this miss match btrfs_progs get_fs_info() hits the bug..
get_fs_info()
::
BUG_ON(ndevs >= fi_args->num_devices);

what I notice is total_devices should be returned by the
ioctl(BTRFS_IOC_FS_INFO)), however total_devices does not
count (transient) replacing device but num_devices does.

first, num_devices which appears to be a subset of
total_devices helps to count the number of devices
under a FSID which excludes the seed devices but
includes the transient replacing device.

total_devices which includes seed devices is as of
now does not count the tansient replacing device,
which IMO is a bug or the implementation details
are not clear enough to state the role of num_devices
and total_devices.

The user land on the otherhand would want to know
all the devices that would come out of probe
against a FSID

As of now in this patch I am making the ioctl.c local
changes (instead of asking replace thread to update
total_devices) to  include replacing device.

Signed-off-by: Anand Jain 
---
 fs/btrfs/ioctl.c | 13 -
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
index 5036f9d..ec7d5c6 100644
--- a/fs/btrfs/ioctl.c
+++ b/fs/btrfs/ioctl.c
@@ -2510,6 +2510,7 @@ static long btrfs_ioctl_fs_info(struct btrfs_root *root, 
void __user *arg)
struct btrfs_device *next;
struct btrfs_fs_devices *fs_devices = root->fs_info->fs_devices;
int ret = 0;
+   int dev_replace_running = 0;
 
if (!capable(CAP_SYS_ADMIN))
return -EPERM;
@@ -2518,8 +2519,18 @@ static long btrfs_ioctl_fs_info(struct btrfs_root *root, 
void __user *arg)
if (!fi_args)
return -ENOMEM;
 
+   btrfs_dev_replace_lock(&root->fs_info->dev_replace);
+   if (btrfs_dev_replace_is_ongoing(&root->fs_info->dev_replace))
+   dev_replace_running = 1;
+
+   btrfs_dev_replace_unlock(&root->fs_info->dev_replace);
+
mutex_lock(&fs_devices->device_list_mutex);
-   fi_args->num_devices = fs_devices->num_devices;
+   if (dev_replace_running)
+   fi_args->num_devices = fs_devices->total_devices + 1;
+   else
+   fi_args->num_devices = fs_devices->total_devices;
+
memcpy(&fi_args->fsid, root->fs_info->fsid, sizeof(fi_args->fsid));
 
list_for_each_entry_safe(device, next, &fs_devices->devices, dev_list) {
-- 
1.8.5.3

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[PATCH 3/3 RFC] btrfs: show_devname should not consider seed disk

2014-03-07 Thread Anand Jain
most of the user level scripts uses /proc/self/mounts for the
disk-path to mount-point to fsid mapping. But when seed disk is
present which generally has lowest devid, the /proc/self/mounts
would show the seed disk, but seed disk has different fsid from
the actual fsid that's mounted. Due to this miss match these
scripts fails to work. One such example is btrfs-porgs
check_mounted_where().

The solution here is not to loop into the seed disks, but still
return the lowest devid under the given mount point.

Signed-off-by: Anand Jain 
---
 fs/btrfs/super.c | 15 ++-
 1 file changed, 6 insertions(+), 9 deletions(-)

diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c
index f3c0247..c023ffa 100644
--- a/fs/btrfs/super.c
+++ b/fs/btrfs/super.c
@@ -1892,15 +1892,12 @@ static int btrfs_show_devname(struct seq_file *m, 
struct dentry *root)
 
mutex_lock(&fs_info->fs_devices->device_list_mutex);
cur_devices = fs_info->fs_devices;
-   while (cur_devices) {
-   head = &cur_devices->devices;
-   list_for_each_entry(dev, head, dev_list) {
-   if (dev->missing)
-   continue;
-   if (!first_dev || dev->devid < first_dev->devid)
-   first_dev = dev;
-   }
-   cur_devices = cur_devices->seed;
+   head = &cur_devices->devices;
+   list_for_each_entry(dev, head, dev_list) {
+   if (dev->missing)
+   continue;
+   if (!first_dev || dev->devid < first_dev->devid)
+   first_dev = dev;
}
 
if (first_dev) {
-- 
1.8.5.3

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: Understanding btrfs and backups

2014-03-07 Thread Sander
Eric Mesa wrote (ao):
> Duncan - thanks for this comprehensive explanation. For a huge portion of
> your reply...I was all wondering why you and others were saying snapshots
> aren't backups. They certainly SEEMED like backups. But now I see that the
> problem is one of precise terminology vs colloquialisms. In other words,
> snapsshots are not backups in and of themselves. They are like Mac's Time
> Machine. BUT if you take these snapshots and then put them on another media
> - whether that's local or not - THEN you have backups. Am I right, or am I
> still missing something subtle? 

Snapshots are backups, but only protect you against a limited amount of
disasters. Snapshots are very convenient to quickly go back in time for
some or all files and directories. But if the filesystem or underlaying
disk goes up in flames, the snapshots are toast as well. So you need
additional backups, preferably not on the same hardware, for real
protection against data loss.

The convenience of snapshots is that you can (almost) make them as often
as you want, fully automated, with (almost) no impact on performance,
without the need for extra hardware, and a restore is no more than a
simple copy.

Sander
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


RE: ENOSPC errors during raid1 rebalance

2014-03-07 Thread Mike Russo
Thanks Hugo, that makes sense, and maybe leads to a possible way to fix the 
issue in future versions of btrfs-convert or a way to handle it in the balance 
code. 
What I did to find files with extents:

cd /mymedia
find . -type f -print0 | xargs -0 filefrag | grep -v 1\ extent | grep -v 0\ 
extent | awk -F: '{print $1}' > /tmp/extent
cat /tmp/extent | while read file; do mv -v "$file" /dev/shm; f2=`basename 
"$file"`; mv -v "/dev/shm/$f2" "$file"; done

I no longer care that even after doing this I still have a file with multiple 
extents, I just don't want them inside those block groups with >1GB extents. 
(BTW my terminology may be all off here, I just know that in my syslog btrfs 
tries to relocate two particular "block groups" and gets 2 enospc errors for 
them.)

But since I'm still getting the errors the files I care about probably aren't 
in this list so I'll do it for all the files on the system (since I can't find 
out what files are in those block groups).  


-Original Message-
From: Hugo Mills [mailto:h...@carfax.org.uk] 
Sent: Friday, March 07, 2014 3:02 AM
To: Mike Russo
Cc: linux-btrfs@vger.kernel.org
Subject: Re: ENOSPC errors during raid1 rebalance

   The defrag operation, by its nature, _doesn't_ preserve extents, and thus 
can act to break up the large extents, making it possible to balance the chunks 
that the offending extents live on.

   Hugo.

--
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
 --- If the first-ever performance is the première,  is the --- 
  last-ever performance the derrière?   
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: Understanding btrfs and backups

2014-03-07 Thread Eric Mesa
Duncan <1i5t5.duncan  cox.net> writes:
> *But*, btrfs snapshots by themselves remain on the existing btrfs 
> filesystem, and thus are subject to many of the same risks as the 
> filesystem itself.  As you mentioned raid is redundancy not backup, 
> snapshots aren't backup either; snapshots are multiple logical copies 
> thus protecting you from accidental deletion or bad editing, but pointed 
> at the same data blocks without redundancy, and if those data blocks or 
> the entire physical media go bad...
> 
> Which is where real backups, separate copies on separate physical media, 
> come in, which is where btrfs send/receive, as the ars-technica article 
> was describing, comes in.
> 
> The idea is to make a read-only snapshot on the local filesystem, read-
> only so it can't change while it's being sent, and then use btrfs send to 
> send that snapshot to be stored on some other media, which can optionally 
> be over the network to a machine and media at a different site, altho it 
> can be to a different device on the same machine, as well.
> 
> The first time you do this, there's no existing copy at the other end, so 
> btrfs send sends a full copy and btrfs receive writes it out.  After 
> that, the receive side has a snapshot identical to the one created on the 
> send side and further btrfs send/receives to the same set simply 
> duplicate the differences between the reference and the new snapshot from 
> the send end to the receive end.  As with local snapshots, old ones can 
> be deleted on both the send and receive ends, as long as at least one 
> common reference snapshot is maintained on both ends, so diffs taken 
> against the send side reference can be applied to an appropriately 
> identical receive side reference, thereby updating the receive side to 
> match the new read-only snapshot on the send side.
> 
> Hopefully that's clearer now. =:^)
> 


Duncan - thanks for this comprehensive explanation. For a huge portion of
your reply...I was all wondering why you and others were saying snapshots
aren't backups. They certainly SEEMED like backups. But now I see that the
problem is one of precise terminology vs colloquialisms. In other words,
snapsshots are not backups in and of themselves. They are like Mac's Time
Machine. BUT if you take these snapshots and then put them on another media
- whether that's local or not - THEN you have backups. Am I right, or am I
still missing something subtle? 

I think the most important thing you said was at the end and I'd like a
little clarification on that if it's OK with you. 

"As with local snapshots, old ones can 
> be deleted on both the send and receive ends, as long as at least one 
> common reference snapshot is maintained on both ends, so diffs taken 
> against the send side reference can be applied to an appropriately 
> identical receive side reference, thereby updating the receive side to 
> match the new read-only snapshot on the send side."

So, let's say I have everything set up. This means I created the read-only
shot on my home btrfs volume and sent it to the backup drive. I'm making
hourly snapshots and after each snapshot is made, it's sent to the backup
drive. So, obviously the backup drive needs to be at least as big as the
home drive so it can store what's on home plus the snapshot-diffs. Now let's
be extreme and say that in the course of a year I touch and somehow change
every single file on the home drive. That means if I only had one snapshot
I'd need home drive x 2 space. (for used space, not unused space, naturally)
So I might want my backups to have last's year's data, but wouldn't want to
need to upgrade the size of my actual home drive. So I would want to
maintain less snapshots on my home drive than my backup drive. (It's
possible I'm missing something here...something subtle that makes this not
necessary) So do I only need to make sure I have the latest snapshot or
maybe latest plus n-1 on the home drive while the backup drive can have all
snapshots since the beginning? I THINK that can be the case based on reading
your sentence, but I just want to make sure. 

In case you were wondering, this is based on what's happened to me with Back
in Time. I had to reduce the number of backups I was keeping because my home
drive wasn't at 100%, but the backupdrive was at 100% because I'd added and
deleted some VMs and other large files (video files I think). And Back in
Time intelligently does not remove the oldest backup off the top until it
knows it has made a new backup - which it couldn't do because it was at
100%. So I had to delete the top 1 or 2 backups and then tell it to keep
less backups. Your description of snapshots makes it seems much less likely
that this would be an issue. Although Back in Time is an incremental backup,
its takes up more space. If I may venture to see if I've learned something
from your response, is it because when I change a file Back in Time stores
the entire changed file while btrfs only stores the bi

Re: 3.13.5 btrfs read() oops

2014-03-07 Thread Chris Mason
On 03/07/2014 05:55 AM, Daniel J Blueman wrote:
> With kernel 3.13.5 (Ubuntu mainline), when plugging in a (evidently
> twitchy) USB3 stick with a BTRFS filesystem, I hit an oops in read()
> [1].
> 
> Full dmesg output is at:
> https://urldefense.proofpoint.com/v1/url?u=http://quora.org/2014/btrfs-oops.txt&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=6%2FL0lzzDhu0Y1hL9xm%2BQyA%3D%3D%0A&m=l%2B%2Fvx4VlYB25NI5yUJbBz2UM5ao0KdYuSUnSSAQ4pmY%3D%0A&s=0eee1e5d23ceb6100beb27d5a4f2e050d241e6b53e0db678e81ba9c8b88ad351

I've just uploaded a for-stable-3.13 branch that should fix this:

git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git 
for-stable-3.13

-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


3.13.5 btrfs read() oops

2014-03-07 Thread Daniel J Blueman
With kernel 3.13.5 (Ubuntu mainline), when plugging in a (evidently
twitchy) USB3 stick with a BTRFS filesystem, I hit an oops in read()
[1].

Full dmesg output is at:
http://quora.org/2014/btrfs-oops.txt

Thanks,
  Daniel

-- [1]

IP: 0010:[] [] memcpy+0x6/0x110
RSP: 0018:88025fa1b910 EFLAGS: 00010207
RAX: 88005c3d906e RBX: 027e RCX: 027e
RDX: 027e RSI: 00050800 RDI: 88005c3d906e
RBP: 88025fa1b948 R08: 1000 R09: 88025fa1b918
R10:  R11:  R12: 8800560e6350
R13: 1600 R14: 88005c3d92ec R15: 027e
FS: 7f9272f79700() GS:88026f3c() knlGS:
CS: 0010 DS:  ES:  CR0: 80050033
CR2: 7f9264010018 CR3: 00025f79a000 CR4: 001407e0
Stack:
 a036401c 1000 8800837f3800 8801e041a000
  8800763df218 880064c8c4c0 88025fa1ba08
 a0348f9c 0f18  1000
Call Trace:
 [] ? read_extent_buffer+0xbc/0x110 [btrfs]
 [] btrfs_get_extent+0x91c/0x970 [btrfs]
 [] __do_readpage+0x357/0x730 [btrfs]
 [] ? btrfs_real_readdir+0x5b0/0x5b0 [btrfs]
 [] __extent_readpages.constprop.41+0x2a2/0x2c0 [btrfs]
 [] ? btrfs_real_readdir+0x5b0/0x5b0 [btrfs]
 [] extent_readpages+0x1b6/0x1c0 [btrfs]
 [] ? btrfs_real_readdir+0x5b0/0x5b0 [btrfs]
 [] ? alloc_pages_current+0xa3/0x160
 [] btrfs_readpages+0x1f/0x30 [btrfs]
 [] __do_page_cache_readahead+0x1b9/0x270
 [] ondemand_readahead+0x152/0x2a0
 [] page_cache_sync_readahead+0x31/0x50
 [] generic_file_aio_read+0x4c5/0x700
 [] do_sync_read+0x5a/0x90
 [] vfs_read+0x95/0x160
 [] SyS_read+0x49/0xa0
 [] tracesys+0xe1/0xe6
-- 
Daniel J Blueman
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: ENOSPC errors during raid1 rebalance

2014-03-07 Thread Duncan
Hugo Mills posted on Fri, 07 Mar 2014 08:02:13 + as excerpted:

> On Fri, Mar 07, 2014 at 01:13:53AM +, Michael Russo wrote:
>> Duncan <1i5t5.duncan  cox.net> writes:
>> 
>> > But if you're not using compression, /that/ can't explain it...
>> > 
>> > 
>> Ha! Well while that was an interesting discussion of fragmentation, I
>> am only using the default mount options here and so no compression.
> 
>I _think_ the problem here is that there may have been some extents
> created during the conversion which were over 1 GiB in size (or at least
> which run across two or more chunks). This causes problems, because
> there's nowhere that they can be written to by the balance --
> which preserves extents -- because none of the allocation units (chunks)
> are big enough.
> 
>The defrag operation, by its nature, _doesn't_ preserve extents,
> and thus can act to break up the large extents, making it possible to
> balance the chunks that the offending extents live on.

Now /that/ would explain the issue, or at least all I've read of it from 
here.  Nice job! =:^)

Obviously >1 GB files break/fragment on 1-gig data-chunk lines when 
(re)written on btrfs, since that's the biggest allocation unit btrfs has, 
and at least once the filesystem has been (near) full once, btrfs is 
extremely unlikely to be able to place those gig-chunks contiguously, 
thus triggering the issue.

Meanwhile, the move-to-tmpfs-and-back should indeed break up those >1-gig 
blocks too, since the that would force a rewrite and thus a reallocation, 
which would then follow btrfs 1-gig-chunk rules.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: Understanding btrfs and backups

2014-03-07 Thread Wolfgang Mader
Duncan, thank you for this comprehensive post. Really helpful as always!

[...]

> As for restoring, since a snapshot is a copy of the filesystem as it
> existed at that point, and the method btrfs exposes for accessing them is
> to mount that specific snapshot, to restore an individual file from a
> snapshot, you simply mount the snapshot you want somewhere and copy the
> file as it existed in that snapshot over top of your current version
> (which will have presumably already been mounted elsewhere, before you
> mounted the snapshot to retrieve the file from), then unmount the
> snapshot and go about your day. =:^)

Please, how do I list mounted snapshots only?

[...]

> 
> Since a snapshot is an image of the filesystem as it was at that
> particular point in time, and btrfs by nature copies blocks elsewhere
> when they are modified, all (well, not "all" as there's metadata like
> file owner, permissions and group, too, but that's handled the same way)
> the snapshot does is map what blocks composed each file at the time the
> snapshot was taken.

Is it correct, that e.g. ownership is recorded separately from the data 
itself, so if I would change the owner of all my files, the respective 
snapshot would only store the old owner information?

[...]

> 
> The first time you do this, there's no existing copy at the other end, so
> btrfs send sends a full copy and btrfs receive writes it out.  After
> that, the receive side has a snapshot identical to the one created on the
> send side and further btrfs send/receives to the same set simply
> duplicate the differences between the reference and the new snapshot from
> the send end to the receive end.  As with local snapshots, old ones can
> be deleted on both the send and receive ends, as long as at least one
> common reference snapshot is maintained on both ends, so diffs taken
> against the send side reference can be applied to an appropriately
> identical receive side reference, thereby updating the receive side to
> match the new read-only snapshot on the send side.

Is the receiving side a complete file system in its own right? If so, I only 
need to maintain one common reference in order to apply the received snapshot, 
right. If I would in any way get the send and receive side out of sync, such 
that they do not share a common reference any more, only the send/receive 
would fail, but I still would have the complete filesystem on the receiving 
side, and could copy it all over (cp, rscync) to the send side in case of a 
disaster on the send side. Is this correct?

Thank you!
Best,
Wolfgang

-- 
Wolfgang Mader
wolfgang.ma...@fdm.uni-freiburg.de
Telefon: +49 (761) 203-7710
Institute of Physics
Hermann-Herder Str. 3, 79104 Freiburg, Germany
Office: 207
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: ENOSPC errors during raid1 rebalance

2014-03-07 Thread Hugo Mills
On Fri, Mar 07, 2014 at 01:13:53AM +, Michael Russo wrote:
> Duncan <1i5t5.duncan  cox.net> writes:
> 
> > But if you're not using compression, /that/ can't explain it...
> > 
> 
> Ha! Well while that was an interesting discussion of fragmentation,
> I am only using the default mount options here and so no compression. 
> The only reason I'm really even looking at the fragmentation 
> issue is because running the defragment (with varying sizes of
> "-t" which I'm not sure why that's even necessary) seemed
> to force btrfs to move segments around and let me rebalance
> more block groups.  I don't even care if the files are defragged,
> I just want them all in the RAID1 profile.  Hopefully if I move
> each file out to some other FS like /dev/shm and then back
> it will work, I just gotta hack a script together to do so.

   I _think_ the problem here is that there may have been some extents
created during the conversion which were over 1 GiB in size (or at
least which run across two or more chunks). This causes problems,
because there's nowhere that they can be written to by the balance --
which preserves extents -- because none of the allocation units
(chunks) are big enough.

   The defrag operation, by its nature, _doesn't_ preserve extents,
and thus can act to break up the large extents, making it possible to
balance the chunks that the offending extents live on.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
 --- If the first-ever performance is the première,  is the --- 
  last-ever performance the derrière?   


signature.asc
Description: Digital signature