Re: Data single *and* raid?
Hendrik Friedel posted on Fri, 07 Aug 2015 07:16:04 +0200 as excerpted: But then: # btrfs fi df /mnt/__Complete_Disk/ Data, RAID5: total=3.83TiB, used=3.78TiB System, RAID5: total=32.00MiB, used=576.00KiB Metadata, RAID5: total=6.46GiB, used=4.84GiB GlobalReserve, single: total=512.00MiB, used=0.00B [T]his seems to be a RAID5 now, right? Well, that's what I want, but the command was: btrfs balance start -dprofiles=single -mprofiles=raid1 /mnt/__Complete_Disk/ So, we would expect raid1 here, no? No. The behavior might be a bit counterintuitive on first glance, but once the logic is understood, it makes sense. 1) You had tried the initial raid5 convert using an earlier kernel that had incomplete raid5 support, as evidenced by the lack of the global- reserve line in btrfs fi df, on a new enough userspace that it should have had it. 2) That initial attempt ran out of space, possibly because it was keeping the single and raid1 chunks around due to fragmentation (Hugo's guess), or due to a now fixed raid5 conversion bugs in the old kernel[1] (my guess), or possibly due to some other bug that's apparently fixed in newer kernels, thus the successful completion of the conversion below. 3) But that initial attempt still did one critical thing -- set the default new-chunk type to raid5, for both data and metadata. 4) So when the second btrfs balance attempt came along, this one primarily intended to clean up that fragmentation that Hugo expected, and thus targeted at those old single data and raid1 metadata chunks, when it rewrote those chunks it used the new chunk default, rewriting them into raid5. That was a result that Hugo obviously didn't predict as his instructions suggested following up with another balance command to complete the conversion. And neither Chris (apparently) nor I (definitely!) foresaw it either. But the behavior does make sense, once you take into account the default chunk type, and that a balance-convert does normally change it. And FWIW, the precise behavior of this default chunk type selector and when it falls back to single data and raid1 or dup metadata (as it will in some instances with a degraded filesystem), has both been problematic before, and is being debated in a current thread, due to the implications for writable mounts of degraded single-device raid1s, for instance. It's behavior in corner-cases like these that is much of the reason most regulars on this list don't consider btrfs fully stable and mature, just yet, because sometimes that corner-case behavior can mean the filesystem doing the wrong thing, going read-only, without any way to correct the problem even tho things are generally still fine, because correcting the problem would require a writable filesystem, thus creating a chicken and egg situation where correcting the problem requires a writable filesystem, but a writable filesystem isn't allowed until the problem is corrected, for instance. (As of now, in that situation a user has little choice but to copy the data on that read-only filesystem elsewhere, do a mkfs to wipe away the problem, and restore to the fresh filesystem. Technically, that shouldn't be required.) --- [1] FWIW, for online tasks like btrfs balance, the btrfs-progs userspace simply issues the commands to the kernel, which does the real work. For offline tasks such as btrfs check or btrfs restore, userspace is the real brains and the kernel simply relays the commands at the device level, without much involvement by the kernel's btrfs code at all. So while you had a current userspace, the old kernel was the critical part since btrfs balance is an online command in which it's the kernel's btrfs code that does the real work. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Data single *and* raid?
On Fri, Aug 07, 2015 at 07:16:04AM +0200, Hendrik Friedel wrote: Hello Quo, thanks for your reply. But then: root@homeserver:/mnt/__Complete_Disk# btrfs fi df /mnt/__Complete_Disk/ Data, RAID5: total=3.83TiB, used=3.78TiB System, RAID5: total=32.00MiB, used=576.00KiB Metadata, RAID5: total=6.46GiB, used=4.84GiB GlobalReserve, single: total=512.00MiB, used=0.00B GlobalReserve is not a chunk type, it just means a range of metadata reserved for overcommiting. And it's always single. Personally, I don't think it should be output in fi df command, as it's in a higher level than chunk. At least for your case, nothing is needed to worry about. But this seems to be a RAID5 now, right? Well, that's what I want, but the command was: btrfs balance start -dprofiles=single -mprofiles=raid1 /mnt/__Complete_Disk/ So, we would expect raid1 here, no? No, profiles= selects chunks of that profile. If you wanted to change the RAID level, you should use convert=. Hugo. Greetings, Hendrik On 01.08.2015 22:44, Chris Murphy wrote: On Sat, Aug 1, 2015 at 2:32 PM, Hugo Mills h...@carfax.org.uk wrote: On Sat, Aug 01, 2015 at 10:09:35PM +0200, Hendrik Friedel wrote: Hello, I converted an array to raid5 by btrfs device add /dev/sdd /mnt/new_storage btrfs device add /dev/sdc /mnt/new_storage btrfs balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/ The Balance went through. But now: Label: none uuid: a8af3832-48c7-4568-861f-e80380dd7e0b Total devices 3 FS bytes used 5.28TiB devid1 size 2.73TiB used 2.57TiB path /dev/sde devid2 size 2.73TiB used 2.73TiB path /dev/sdc devid3 size 2.73TiB used 2.73TiB path /dev/sdd btrfs-progs v4.1.1 Already the 2.57TiB is a bit surprising: root@homeserver:/mnt# btrfs fi df /mnt/new_storage/ Data, single: total=2.55TiB, used=2.55TiB Data, RAID5: total=2.73TiB, used=2.72TiB System, RAID5: total=32.00MiB, used=736.00KiB Metadata, RAID1: total=6.00GiB, used=5.33GiB Metadata, RAID5: total=3.00GiB, used=2.99GiB Looking at the btrfs fi show output, you've probably run out of space during the conversion, probably due to an uneven distribution of the original single chunks. I think I would suggest balancing the single chunks, and trying the conversion (of the unconverted parts) again: # btrfs balance start -dprofiles=single -mprofile=raid1 /mnt/new_storage/ # btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft /mnt/new_storage/ Yep I bet that's it also. btrfs fi usage might be better at exposing this case. -- Hugo Mills | There is no dark side to the Moon, really. As a hugo@... carfax.org.uk | matter of fact, it's all dark. http://carfax.org.uk/ | PGP: E2AB1DE4 |Pink Floyd signature.asc Description: Digital signature
Re: Data single *and* raid?
Hello Quo, thanks for your reply. But then: root@homeserver:/mnt/__Complete_Disk# btrfs fi df /mnt/__Complete_Disk/ Data, RAID5: total=3.83TiB, used=3.78TiB System, RAID5: total=32.00MiB, used=576.00KiB Metadata, RAID5: total=6.46GiB, used=4.84GiB GlobalReserve, single: total=512.00MiB, used=0.00B GlobalReserve is not a chunk type, it just means a range of metadata reserved for overcommiting. And it's always single. Personally, I don't think it should be output in fi df command, as it's in a higher level than chunk. At least for your case, nothing is needed to worry about. But this seems to be a RAID5 now, right? Well, that's what I want, but the command was: btrfs balance start -dprofiles=single -mprofiles=raid1 /mnt/__Complete_Disk/ So, we would expect raid1 here, no? Greetings, Hendrik On 01.08.2015 22:44, Chris Murphy wrote: On Sat, Aug 1, 2015 at 2:32 PM, Hugo Mills h...@carfax.org.uk wrote: On Sat, Aug 01, 2015 at 10:09:35PM +0200, Hendrik Friedel wrote: Hello, I converted an array to raid5 by btrfs device add /dev/sdd /mnt/new_storage btrfs device add /dev/sdc /mnt/new_storage btrfs balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/ The Balance went through. But now: Label: none uuid: a8af3832-48c7-4568-861f-e80380dd7e0b Total devices 3 FS bytes used 5.28TiB devid1 size 2.73TiB used 2.57TiB path /dev/sde devid2 size 2.73TiB used 2.73TiB path /dev/sdc devid3 size 2.73TiB used 2.73TiB path /dev/sdd btrfs-progs v4.1.1 Already the 2.57TiB is a bit surprising: root@homeserver:/mnt# btrfs fi df /mnt/new_storage/ Data, single: total=2.55TiB, used=2.55TiB Data, RAID5: total=2.73TiB, used=2.72TiB System, RAID5: total=32.00MiB, used=736.00KiB Metadata, RAID1: total=6.00GiB, used=5.33GiB Metadata, RAID5: total=3.00GiB, used=2.99GiB Looking at the btrfs fi show output, you've probably run out of space during the conversion, probably due to an uneven distribution of the original single chunks. I think I would suggest balancing the single chunks, and trying the conversion (of the unconverted parts) again: # btrfs balance start -dprofiles=single -mprofile=raid1 /mnt/new_storage/ # btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft /mnt/new_storage/ Yep I bet that's it also. btrfs fi usage might be better at exposing this case. -- Hendrik Friedel Auf dem Brink 12 28844 Weyhe Tel. 04203 8394854 Mobil 0178 1874363 --- Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft. https://www.avast.com/antivirus -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Data single *and* raid?
Hendrik Friedel wrote on 2015/08/06 20:57 +0200: Hello Hugo, hello Chris, thanks for your advice. Now I am here: btrfs balance start -dprofiles=single -mprofiles=raid1 /mnt/__Complete_Disk/ Done, had to relocate 0 out of 3939 chunks root@homeserver:/mnt/__Complete_Disk# btrfs fi show Label: none uuid: a8af3832-48c7-4568-861f-e80380dd7e0b Total devices 3 FS bytes used 3.78TiB devid1 size 2.73TiB used 2.72TiB path /dev/sde devid2 size 2.73TiB used 2.23TiB path /dev/sdc devid3 size 2.73TiB used 2.73TiB path /dev/sdd btrfs-progs v4.1.1 So, that looks good. But then: root@homeserver:/mnt/__Complete_Disk# btrfs fi df /mnt/__Complete_Disk/ Data, RAID5: total=3.83TiB, used=3.78TiB System, RAID5: total=32.00MiB, used=576.00KiB Metadata, RAID5: total=6.46GiB, used=4.84GiB GlobalReserve, single: total=512.00MiB, used=0.00B GlobalReserve is not a chunk type, it just means a range of metadata reserved for overcommiting. And it's always single. Personally, I don't think it should be output in fi df command, as it's in a higher level than chunk. At least for your case, nothing is needed to worry about. Thanks, Qu Is the RAID5 expected here? I did not yet run: btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft /mnt/new_storage/ Regards, Hendrik On 01.08.2015 22:44, Chris Murphy wrote: On Sat, Aug 1, 2015 at 2:32 PM, Hugo Mills h...@carfax.org.uk wrote: On Sat, Aug 01, 2015 at 10:09:35PM +0200, Hendrik Friedel wrote: Hello, I converted an array to raid5 by btrfs device add /dev/sdd /mnt/new_storage btrfs device add /dev/sdc /mnt/new_storage btrfs balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/ The Balance went through. But now: Label: none uuid: a8af3832-48c7-4568-861f-e80380dd7e0b Total devices 3 FS bytes used 5.28TiB devid1 size 2.73TiB used 2.57TiB path /dev/sde devid2 size 2.73TiB used 2.73TiB path /dev/sdc devid3 size 2.73TiB used 2.73TiB path /dev/sdd btrfs-progs v4.1.1 Already the 2.57TiB is a bit surprising: root@homeserver:/mnt# btrfs fi df /mnt/new_storage/ Data, single: total=2.55TiB, used=2.55TiB Data, RAID5: total=2.73TiB, used=2.72TiB System, RAID5: total=32.00MiB, used=736.00KiB Metadata, RAID1: total=6.00GiB, used=5.33GiB Metadata, RAID5: total=3.00GiB, used=2.99GiB Looking at the btrfs fi show output, you've probably run out of space during the conversion, probably due to an uneven distribution of the original single chunks. I think I would suggest balancing the single chunks, and trying the conversion (of the unconverted parts) again: # btrfs balance start -dprofiles=single -mprofile=raid1 /mnt/new_storage/ # btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft /mnt/new_storage/ Yep I bet that's it also. btrfs fi usage might be better at exposing this case. -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Data single *and* raid?
Hello Hugo, hello Chris, thanks for your advice. Now I am here: btrfs balance start -dprofiles=single -mprofiles=raid1 /mnt/__Complete_Disk/ Done, had to relocate 0 out of 3939 chunks root@homeserver:/mnt/__Complete_Disk# btrfs fi show Label: none uuid: a8af3832-48c7-4568-861f-e80380dd7e0b Total devices 3 FS bytes used 3.78TiB devid1 size 2.73TiB used 2.72TiB path /dev/sde devid2 size 2.73TiB used 2.23TiB path /dev/sdc devid3 size 2.73TiB used 2.73TiB path /dev/sdd btrfs-progs v4.1.1 So, that looks good. But then: root@homeserver:/mnt/__Complete_Disk# btrfs fi df /mnt/__Complete_Disk/ Data, RAID5: total=3.83TiB, used=3.78TiB System, RAID5: total=32.00MiB, used=576.00KiB Metadata, RAID5: total=6.46GiB, used=4.84GiB GlobalReserve, single: total=512.00MiB, used=0.00B Is the RAID5 expected here? I did not yet run: btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft /mnt/new_storage/ Regards, Hendrik On 01.08.2015 22:44, Chris Murphy wrote: On Sat, Aug 1, 2015 at 2:32 PM, Hugo Mills h...@carfax.org.uk wrote: On Sat, Aug 01, 2015 at 10:09:35PM +0200, Hendrik Friedel wrote: Hello, I converted an array to raid5 by btrfs device add /dev/sdd /mnt/new_storage btrfs device add /dev/sdc /mnt/new_storage btrfs balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/ The Balance went through. But now: Label: none uuid: a8af3832-48c7-4568-861f-e80380dd7e0b Total devices 3 FS bytes used 5.28TiB devid1 size 2.73TiB used 2.57TiB path /dev/sde devid2 size 2.73TiB used 2.73TiB path /dev/sdc devid3 size 2.73TiB used 2.73TiB path /dev/sdd btrfs-progs v4.1.1 Already the 2.57TiB is a bit surprising: root@homeserver:/mnt# btrfs fi df /mnt/new_storage/ Data, single: total=2.55TiB, used=2.55TiB Data, RAID5: total=2.73TiB, used=2.72TiB System, RAID5: total=32.00MiB, used=736.00KiB Metadata, RAID1: total=6.00GiB, used=5.33GiB Metadata, RAID5: total=3.00GiB, used=2.99GiB Looking at the btrfs fi show output, you've probably run out of space during the conversion, probably due to an uneven distribution of the original single chunks. I think I would suggest balancing the single chunks, and trying the conversion (of the unconverted parts) again: # btrfs balance start -dprofiles=single -mprofile=raid1 /mnt/new_storage/ # btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft /mnt/new_storage/ Yep I bet that's it also. btrfs fi usage might be better at exposing this case. -- Hendrik Friedel Auf dem Brink 12 28844 Weyhe Tel. 04203 8394854 Mobil 0178 1874363 --- Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft. https://www.avast.com/antivirus -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Data single *and* raid?
Hello, Looking at the btrfs fi show output, you've probably run out of space during the conversion, probably due to an uneven distribution of the original single chunks. I think I would suggest balancing the single chunks, and trying the conversion (of the unconverted parts) again: # btrfs balance start -dprofiles=single -mprofile=raid1 /mnt/new_storage/ # btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft /mnt/new_storage/ Yep I bet that's it also. btrfs fi usage might be better at exposing this case. Thanks for your hints. The balance is running now for about 11h: The status is a bit surprising to me: 0 out of about 2619 chunks balanced (4165 considered), 100% left btrfs fi usage is also surprising: Overall: Device size: 8.19TiB Device allocated: 2.56TiB Device unallocated:5.62TiB Device missing: 0.00B Used: 2.56TiB Free (estimated): 11.65TiB (min: 2.81TiB) Data ratio: 0.48 Metadata ratio: 1.33 Global reserve: 512.00MiB (used: 0.00B) Data,single: Size:2.55TiB, Used:2.55TiB /dev/sdc1.60TiB /dev/sde 975.44GiB Data,RAID5: Size:2.73TiB, Used:2.72TiB /dev/sdc1.12TiB /dev/sdd2.73TiB /dev/sde1.61TiB Metadata,RAID1: Size:6.00GiB, Used:5.33GiB /dev/sdc5.00GiB /dev/sdd1.00GiB /dev/sde6.00GiB Metadata,RAID5: Size:3.00GiB, Used:2.99GiB /dev/sdc2.00GiB /dev/sdd1.00GiB /dev/sde3.00GiB System,RAID5: Size:32.00MiB, Used:736.00KiB /dev/sdd 32.00MiB /dev/sde 32.00MiB Unallocated: /dev/sdc1.02MiB /dev/sdd1.02MiB /dev/sde 164.59GiB I hope, that is because Raid5 is not implemented yet? Regards, Hendrik --- Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft. https://www.avast.com/antivirus -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Data single *and* raid?
On Sun, Aug 02, 2015 at 12:31:13PM -0600, Chris Murphy wrote: On Sat, Aug 1, 2015 at 9:46 PM, Duncan 1i5t5.dun...@cox.net wrote: If it was setup with something earlier (not sure about 4.1.0, was it affected? No. but 4.0.x and earlier should be fine for setup), however, once on a new kernel the usual ENOSPC workarounds can be given a try. That would include a first balance start -dusage=0 -musage=0, and if that didn't free up at least a gig on a second device, If I'm following this correctly, the reproduce steps would be to create a single device Btrfs that's ~94% full, add two devices, then convert to raid5. I think what's going on here is empty single profile data chunks aren't being deallocated, and it's effectively a 2 device raid5. This isn't supported by the btrfs fi df output: all of the allocated space is used. Hugo. So actually, you're right, either -dusage=0 might fix it, or better, newer kernel that automatically deallocated empty/converted single profile data chunks. But right now it will take another balance in the end because it looks like this is effectively a 2 device raid5, with the 3rd drive full of single only chunks (which might be empty?). -- Hugo Mills | I'll take your bet, but make it ten thousand francs. hugo@... carfax.org.uk | I'm only a _poor_ corrupt official. http://carfax.org.uk/ | PGP: E2AB1DE4 | Capt. Renaud, Casablanca signature.asc Description: Digital signature
Re: Data single *and* raid?
On Sat, Aug 1, 2015 at 9:46 PM, Duncan 1i5t5.dun...@cox.net wrote: If it was setup with something earlier (not sure about 4.1.0, was it affected? No. but 4.0.x and earlier should be fine for setup), however, once on a new kernel the usual ENOSPC workarounds can be given a try. That would include a first balance start -dusage=0 -musage=0, and if that didn't free up at least a gig on a second device, If I'm following this correctly, the reproduce steps would be to create a single device Btrfs that's ~94% full, add two devices, then convert to raid5. I think what's going on here is empty single profile data chunks aren't being deallocated, and it's effectively a 2 device raid5. So actually, you're right, either -dusage=0 might fix it, or better, newer kernel that automatically deallocated empty/converted single profile data chunks. But right now it will take another balance in the end because it looks like this is effectively a 2 device raid5, with the 3rd drive full of single only chunks (which might be empty?). -- Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Data single *and* raid?
Chris Murphy posted on Sat, 01 Aug 2015 19:14:13 -0600 as excerpted: On Sat, Aug 1, 2015 at 6:27 PM, Duncan 1i5t5.dun...@cox.net wrote: 1) If this fs was created with btrfs-progs v4.1.1, get what you need to retrieve off of it immediately, then blow it away and start over, as the thing isn't stable and all data is at risk. Agreed. But I'd go so far as to say at this point it looks like it's wedged itself into a kind of self-induced faux-ENOSPC state because there isn't room to allocate more raid5 chunks. So, I think it's stuck in any case. Well, yes and no. If it was setup with progs v4.1.1, save what you can and blow it away as it's not stable enough to try anything else. If it was setup with something earlier (not sure about 4.1.0, was it affected? but 4.0.x and earlier should be fine for setup), however, once on a new kernel the usual ENOSPC workarounds can be given a try. That would include a first balance start -dusage=0 -musage=0, and if that didn't free up at least a gig on a second device, I'd try the old add-a- device trick and see what happens. (A few GiB thumb drive should work in a pinch, or even a ramdrive if you're willing to risk loss in the event of a crash vaporizing everything in memory including the ramdrive.) After all, even if he didn't know the risk of the still very new raid56 mode before, he does after reading my earlier message, and anything of value should be backed up before he attempts anything, so at that point, there's nothing to lose and it's upto him whether he wants to simply blow away the current setup and start over with either raid1 or (with another device) raid10, avoiding the current risk of raid56, or blow away and start over with raid56 again, knowing the risks now, or try to recover what's there, viewing it not as the easiest way but as practice in disaster recovery, again, with anything of value backed up so there's nothing to lose besides the time of fiddling with it and ending up having to blow away and restart from backups anyway, regardless of how it goes. It'd be great to reproduce this with a current kernel and see if it's still happening. Absolutely. Tho at this point I believe the chances are pretty good it's simply either that bad 4.1.1 mkfs.btrfs, or an older pre-full-raid56-support kernel that didn't handle balance to raid56 so well, yet, and that on the latest userspace and kernel the problem shouldn't reoccur. But it'd be nice to *KNOW* that, by trying to reproduce, absolutely. He may well have stumbled upon yet another confirmation of my recommendation to wait on raid56 unless you're deliberately testing it, and confirmation thereof would be halfway to getting it fixed, so those who /do/ choose to wait won't be dealing with it. =:^) -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Data single *and* raid?
On Sat, Aug 01, 2015 at 10:09:35PM +0200, Hendrik Friedel wrote: Hello, I converted an array to raid5 by btrfs device add /dev/sdd /mnt/new_storage btrfs device add /dev/sdc /mnt/new_storage btrfs balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/ The Balance went through. But now: Label: none uuid: a8af3832-48c7-4568-861f-e80380dd7e0b Total devices 3 FS bytes used 5.28TiB devid1 size 2.73TiB used 2.57TiB path /dev/sde devid2 size 2.73TiB used 2.73TiB path /dev/sdc devid3 size 2.73TiB used 2.73TiB path /dev/sdd btrfs-progs v4.1.1 Already the 2.57TiB is a bit surprising: root@homeserver:/mnt# btrfs fi df /mnt/new_storage/ Data, single: total=2.55TiB, used=2.55TiB Data, RAID5: total=2.73TiB, used=2.72TiB System, RAID5: total=32.00MiB, used=736.00KiB Metadata, RAID1: total=6.00GiB, used=5.33GiB Metadata, RAID5: total=3.00GiB, used=2.99GiB Looking at the btrfs fi show output, you've probably run out of space during the conversion, probably due to an uneven distribution of the original single chunks. I think I would suggest balancing the single chunks, and trying the conversion (of the unconverted parts) again: # btrfs balance start -dprofiles=single -mprofile=raid1 /mnt/new_storage/ # btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft /mnt/new_storage/ You may have to do this more than once. Hugo. Why is there Data single and Raid? Why is Metadata RAID1 and Raid5? A scrub is currently running and showed no errors yet. -- Hugo Mills | You've read the project plan. Forget that. We're hugo@... carfax.org.uk | going to Do Stuff and Have Fun doing it. http://carfax.org.uk/ | PGP: E2AB1DE4 | Jeremy Frey signature.asc Description: Digital signature
Re: Data single *and* raid?
On Sat, Aug 1, 2015 at 2:32 PM, Hugo Mills h...@carfax.org.uk wrote: On Sat, Aug 01, 2015 at 10:09:35PM +0200, Hendrik Friedel wrote: Hello, I converted an array to raid5 by btrfs device add /dev/sdd /mnt/new_storage btrfs device add /dev/sdc /mnt/new_storage btrfs balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/ The Balance went through. But now: Label: none uuid: a8af3832-48c7-4568-861f-e80380dd7e0b Total devices 3 FS bytes used 5.28TiB devid1 size 2.73TiB used 2.57TiB path /dev/sde devid2 size 2.73TiB used 2.73TiB path /dev/sdc devid3 size 2.73TiB used 2.73TiB path /dev/sdd btrfs-progs v4.1.1 Already the 2.57TiB is a bit surprising: root@homeserver:/mnt# btrfs fi df /mnt/new_storage/ Data, single: total=2.55TiB, used=2.55TiB Data, RAID5: total=2.73TiB, used=2.72TiB System, RAID5: total=32.00MiB, used=736.00KiB Metadata, RAID1: total=6.00GiB, used=5.33GiB Metadata, RAID5: total=3.00GiB, used=2.99GiB Looking at the btrfs fi show output, you've probably run out of space during the conversion, probably due to an uneven distribution of the original single chunks. I think I would suggest balancing the single chunks, and trying the conversion (of the unconverted parts) again: # btrfs balance start -dprofiles=single -mprofile=raid1 /mnt/new_storage/ # btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft /mnt/new_storage/ Yep I bet that's it also. btrfs fi usage might be better at exposing this case. -- Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Data single *and* raid?
Hello, I converted an array to raid5 by btrfs device add /dev/sdd /mnt/new_storage btrfs device add /dev/sdc /mnt/new_storage btrfs balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/ The Balance went through. But now: Label: none uuid: a8af3832-48c7-4568-861f-e80380dd7e0b Total devices 3 FS bytes used 5.28TiB devid1 size 2.73TiB used 2.57TiB path /dev/sde devid2 size 2.73TiB used 2.73TiB path /dev/sdc devid3 size 2.73TiB used 2.73TiB path /dev/sdd btrfs-progs v4.1.1 Already the 2.57TiB is a bit surprising: root@homeserver:/mnt# btrfs fi df /mnt/new_storage/ Data, single: total=2.55TiB, used=2.55TiB Data, RAID5: total=2.73TiB, used=2.72TiB System, RAID5: total=32.00MiB, used=736.00KiB Metadata, RAID1: total=6.00GiB, used=5.33GiB Metadata, RAID5: total=3.00GiB, used=2.99GiB Why is there Data single and Raid? Why is Metadata RAID1 and Raid5? A scrub is currently running and showed no errors yet. Greetings, Hendrik --- Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft. https://www.avast.com/antivirus -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Data single *and* raid?
Chris Murphy posted on Sat, 01 Aug 2015 14:44:52 -0600 as excerpted: On Sat, Aug 1, 2015 at 2:32 PM, Hugo Mills h...@carfax.org.uk wrote: On Sat, Aug 01, 2015 at 10:09:35PM +0200, Hendrik Friedel wrote: Hello, I converted an array to raid5 by btrfs device add /dev/sdd /mnt/new_storage btrfs device add /dev/sdc /mnt/new_storage btrfs balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/ The Balance went through. But now: Label: none uuid: a8af3832-48c7-4568-861f-e80380dd7e0b Total devices 3 FS bytes used 5.28TiB devid1 size 2.73TiB used 2.57TiB path /dev/sde devid2 size 2.73TiB used 2.73TiB path /dev/sdc devid3 size 2.73TiB used 2.73TiB path /dev/sdd btrfs-progs v4.1.1 Already the 2.57TiB is a bit surprising: root@homeserver:/mnt# btrfs fi df /mnt/new_storage/ Data, single: total=2.55TiB, used=2.55TiB Data, RAID5: total=2.73TiB, used=2.72TiB System, RAID5: total=32.00MiB, used=736.00KiB Metadata, RAID1: total=6.00GiB, used=5.33GiB Metadata, RAID5: total=3.00GiB, used=2.99GiB Looking at the btrfs fi show output, you've probably run out of space during the conversion, probably due to an uneven distribution of the original single chunks. I think I would suggest balancing the single chunks, and trying the conversion (of the unconverted parts) again: # btrfs balance start -dprofiles=single -mprofile=raid1 /mnt/new_storage/ # btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft /mnt/new_storage/ Yep I bet that's it also. btrfs fi usage might be better at exposing this case. Does fi usage deal with raid5 yet? Last I knew it didn't deal with raid56 (which I'm not using) or with mixed-bg (which I am using on one btrfs). On the mixed-bg btrfs it still warns it doesn't handle that, reporting unallocated of 16 EiB on a 256 MiB filesystem, so clearly the warning is valid. 16 EiB unallocated on a 256 MiB btrfs, I wish! progs v4.1.2, latest as of yesterday, I believe. Meanwhile, three devices the same size. He just added two of them and didn't do a rebalance after that until the raid5 conversion, so except for anything added since, all data/metadata should have started on a single device, with single data and dup metadata. The conversion was specifically to raid5 for both data/metadata, so that's what he should have ended up with. But, somewhere along the line he got 6 GiB of raid1 metadata. Either he added/rewrote a *BUNCH* of files after adding at least one device before the conversion that he didn't tell us about, or the conversion screwed up, because that's a lot of raid1 metadata coming out of nowhere! But I'm *strongly* suspecting a pre-full-raid56-support kernel, because btrfs-progs is certainly reasonably new (v4.1.1, as of yesterday 4.1.2 is the newest as I mention above), but the fi df doesn't report global reserve. The only way I know of that it wouldn't report that with a new userspace is if the kernelspace is too old. And AFAIK, the kernel was reporting global reserve (with fi df listing it as unknown if it was too old) _well_ before full raid56 support. So it's gotta be an old kernel, with only partial raid56 support, which might explain the weird to-raid56 conversion results. Finally, that's btrfs-progs 4.1.1, the one that's blacklisted due to a buggy mkfs.btrfs. If he created the filesystem with that mkfs.btrfs... maybe that explains the funky results, as well. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Data single *and* raid?
Hugo Mills posted on Sat, 01 Aug 2015 22:34:13 + as excerpted: Yes and that also puts it in the realm of kernels that weren't releasing/deallocating empty chunks; although I don't know if that's a factor, if dconvert forcibly deals with this.. It does -- you only have to look at the btrfs fi df output to see that there's no empty block groups (to within 0.1% or so) Exactly. The allocations are all full. And the fi show says there's little to no room to allocate more, as well. There's room on one device, but that's not going to help except with single, which shouldn't be allocated any more. I'd say... 1) If this fs was created with btrfs-progs v4.1.1, get what you need to retrieve off of it immediately, then blow it away and start over, as the thing isn't stable and all data is at risk. 2) If it wasn't created with progs v4.1.1, the next issue is that kernel, since it's obviously from before raid56 was fully functional (well either that or there's a more serious bug going on). Existing data should at least be reasonably stable, but with raid56 mode being so new, the newer the kernel you're using to work with it, the better. 4.1.x at LEAST, if not 4.2-rc, as we're nearing the end of the 4.2 development cycle. And plan on keeping even better than normal backups and on current on kernels for at least another several kernel cycles, if you're going to use raid56 mode, as while it's complete now, it's going to take a bit to stabilize to the level of the rest of btrfs itself, which of course is stabilizing now, but not really fully stable and mature yet, so the sysadmin's rule that data with any value is backed up, or by definition it's throw-away data, despite any claims to the contrary, continues to apply double on btrfs, compared to more mature and stable filesystems. So definitely upgrade the kernel. Then see where things stand. 3) Meanwhile, based on raid56 mode's newness, I've been recommending that people stay off it until 4.5-ish or so, basically a year after initial nominal full support, unless of course their intent is to be a leading/ bleeding edge testing and report deployment. Otherwise, use raid1 or raid10 mode until then, and evaluate raid56 mode stability around 4.5, before deploying. And if you're one of the brave doing current raid56 deployment, testing and bug reporting in full knowledge of its newness and lack of current stability and maturity, THANKS, it's your work that's helping to stabilize it for others, when they do switch to it. =:^) -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Data single *and* raid?
On Sat, Aug 1, 2015 at 2:09 PM, Hendrik Friedel hend...@friedels.name wrote: Hello, I converted an array to raid5 by btrfs device add /dev/sdd /mnt/new_storage btrfs device add /dev/sdc /mnt/new_storage btrfs balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/ The Balance went through. But now: Label: none uuid: a8af3832-48c7-4568-861f-e80380dd7e0b Total devices 3 FS bytes used 5.28TiB devid1 size 2.73TiB used 2.57TiB path /dev/sde devid2 size 2.73TiB used 2.73TiB path /dev/sdc devid3 size 2.73TiB used 2.73TiB path /dev/sdd btrfs-progs v4.1.1 Already the 2.57TiB is a bit surprising: root@homeserver:/mnt# btrfs fi df /mnt/new_storage/ Data, single: total=2.55TiB, used=2.55TiB Data, RAID5: total=2.73TiB, used=2.72TiB System, RAID5: total=32.00MiB, used=736.00KiB Metadata, RAID1: total=6.00GiB, used=5.33GiB Metadata, RAID5: total=3.00GiB, used=2.99GiB Why is there Data single and Raid? Why is Metadata RAID1 and Raid5? What kernel version? -- Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Data single *and* raid?
On Sat, Aug 1, 2015 at 3:45 PM, Duncan 1i5t5.dun...@cox.net wrote: Does fi usage deal with raid5 yet? Now that you mention it, I think it doesn't. But if it did, it would show this problem better than df I think. But, somewhere along the line he got 6 GiB of raid1 metadata. Either he added/rewrote a *BUNCH* of files after adding at least one device before the conversion that he didn't tell us about, or the conversion screwed up, because that's a lot of raid1 metadata coming out of nowhere! Yeah it's a little confusing, that's why I asked about the kernel version. But I'm *strongly* suspecting a pre-full-raid56-support kernel, because btrfs-progs is certainly reasonably new (v4.1.1, as of yesterday 4.1.2 is the newest as I mention above), but the fi df doesn't report global reserve. The only way I know of that it wouldn't report that with a new userspace is if the kernelspace is too old. Yes and that also puts it in the realm of kernels that weren't releasing/deallocating empty chunks; although I don't know if that's a factor, if dconvert forcibly deals with this.. Finally, that's btrfs-progs 4.1.1, the one that's blacklisted due to a buggy mkfs.btrfs. If he created the filesystem with that mkfs.btrfs... maybe that explains the funky results, as well. Good catch. That really ought to be filed as a bug with that distro to flat out remove 4.1.1 from the repos. -- Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Data single *and* raid?
On Sat, Aug 01, 2015 at 04:26:25PM -0600, Chris Murphy wrote: On Sat, Aug 1, 2015 at 3:45 PM, Duncan 1i5t5.dun...@cox.net wrote: Does fi usage deal with raid5 yet? Now that you mention it, I think it doesn't. But if it did, it would show this problem better than df I think. But, somewhere along the line he got 6 GiB of raid1 metadata. Either he added/rewrote a *BUNCH* of files after adding at least one device before the conversion that he didn't tell us about, or the conversion screwed up, because that's a lot of raid1 metadata coming out of nowhere! Yeah it's a little confusing, that's why I asked about the kernel version. It's 6 GiB of metadata for 5.7 GIB of data, or thereabouts. 0.1% is about the expected size. Now, given the original description, it's not clear at all why the data has suddenly doubled in size -- unless there's some snapshots, and the OP did a defrag as well. But I'm *strongly* suspecting a pre-full-raid56-support kernel, because btrfs-progs is certainly reasonably new (v4.1.1, as of yesterday 4.1.2 is the newest as I mention above), but the fi df doesn't report global reserve. The only way I know of that it wouldn't report that with a new userspace is if the kernelspace is too old. Yes and that also puts it in the realm of kernels that weren't releasing/deallocating empty chunks; although I don't know if that's a factor, if dconvert forcibly deals with this.. It does -- you only have to look at the btrfs fi df output to see that there's no empty block groups (to within 0.1% or so) Finally, that's btrfs-progs 4.1.1, the one that's blacklisted due to a buggy mkfs.btrfs. If he created the filesystem with that mkfs.btrfs... maybe that explains the funky results, as well. Good catch. That really ought to be filed as a bug with that distro to flat out remove 4.1.1 from the repos. If they picked up 4.1.1 fast enough, they should pick up 4.1.2 just as quickly... Hugo. -- Hugo Mills | What are we going to do tonight? hugo@... carfax.org.uk | The same thing we do every night, Pinky. Try to http://carfax.org.uk/ | take over the world! PGP: E2AB1DE4 | signature.asc Description: Digital signature
Re: Data single *and* raid?
On Sat, Aug 1, 2015 at 6:27 PM, Duncan 1i5t5.dun...@cox.net wrote: 1) If this fs was created with btrfs-progs v4.1.1, get what you need to retrieve off of it immediately, then blow it away and start over, as the thing isn't stable and all data is at risk. Agreed. But I'd go so far as to say at this point it looks like it's wedged itself into a kind of self-induced faux-ENOSPC state because there isn't room to allocate more raid5 chunks. So, I think it's stuck in any case. It'd be great to reproduce this with a current kernel and see if it's still happening. -- Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
