Re: Confusion with newly converted filesystem
Thank you very much, Duncan and Chris. Especially to Duncan, for his detailed reply and further suggestions. I will try to follow your advice as best I can. I am old, but I'm still learning! Tim -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Confusion with newly converted filesystem
I will try to explain what I have done, but also try to keep it fairly short. I installed a Linux distro that does not support installing to btrfs to an ext4. I ran dist-upgrade to ensure that I have the latest btrfs-tools. I upgraded the Debian kernel from 3.13 to 3.16.3. When all this was completed, there was something like 900 MB used on a 40 GB partition. Next, I booted to another distro (Arch Linux) which also has the latest kernel and btrfs-progs. I ran btrfs-convert /dev/sda6. When I rebooted to the new Debian system, the btrfs was mounted read-only. btrfs fi show / showed all 40 GB as used. I did some internet research, then I remounted the filesystem as rw and added another 40 GB partition on a separate disk drive. Then I ran btrfs balance start -dusage=30. This seemed to stabilize the filesystem to the point that it is usable. I proceeded with my original plan, which was to make it a two-drive RAID filesystem, using -dconvert =raid0 -mconvert=raid1. This succeeded, but the data and metadata usage stats still look all out of whack. After several rebalance attempts, my usage stats look like the following: btrfs fi show / shows a total usage of 1.76 GB, with 40 GB allocated and 14.03 GB used on each device. btrfs fi df / shows total data of 2 GB allocated with 1.69 GB used and metadata of 13 GB total with 72.41 MB used. Why is 13 GB needed for 72 MB of metadata? Is there any understandable way to fix this? I am not a newbie, but am by no means an expert with btrfs Thank you, Tim -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Fwd: Confusion with newly converted filesystem
Never mind. I have stumbled my way into a solution. I ran btrfs subv delete /ext2_saved. Then I ran btrfs balance start /. That relocated 15 of 15 chunks. Now fi show shows 2.03 GB used on each device and fi df shows 1 GB of metadata total. Apparently, that saved ext4 subvolume was a real mess. Tim -- Forwarded message -- From: Tim Cuthbertson ratch...@gmail.com Date: Thu, Oct 9, 2014 at 11:41 AM Subject: Confusion with newly converted filesystem To: linux-btrfs@vger.kernel.org I will try to explain what I have done, but also try to keep it fairly short. I installed a Linux distro that does not support installing to btrfs to an ext4. I ran dist-upgrade to ensure that I have the latest btrfs-tools. I upgraded the Debian kernel from 3.13 to 3.16.3. When all this was completed, there was something like 900 MB used on a 40 GB partition. Next, I booted to another distro (Arch Linux) which also has the latest kernel and btrfs-progs. I ran btrfs-convert /dev/sda6. When I rebooted to the new Debian system, the btrfs was mounted read-only. btrfs fi show / showed all 40 GB as used. I did some internet research, then I remounted the filesystem as rw and added another 40 GB partition on a separate disk drive. Then I ran btrfs balance start -dusage=30. This seemed to stabilize the filesystem to the point that it is usable. I proceeded with my original plan, which was to make it a two-drive RAID filesystem, using -dconvert =raid0 -mconvert=raid1. This succeeded, but the data and metadata usage stats still look all out of whack. After several rebalance attempts, my usage stats look like the following: btrfs fi show / shows a total usage of 1.76 GB, with 40 GB allocated and 14.03 GB used on each device. btrfs fi df / shows total data of 2 GB allocated with 1.69 GB used and metadata of 13 GB total with 72.41 MB used. Why is 13 GB needed for 72 MB of metadata? Is there any understandable way to fix this? I am not a newbie, but am by no means an expert with btrfs Thank you, Tim -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Fwd: Confusion with newly converted filesystem
Tim Cuthbertson posted on Thu, 09 Oct 2014 13:58:58 -0500 as excerpted: I ran btrfs subv delete /ext2_saved. Then I ran btrfs balance start /. That relocated 15 of 15 chunks. Now fi show shows 2.03 GB used on each device and fi df shows 1 GB of metadata total. Apparently, that saved ext4 subvolume was a real mess. Yes and no. The problem is that ext4 and btrfs work rather differently from each other, and btrfs can't manage the saved ext4 subvolume as it would normal btrfs subvolumes because doing so would break the ext4 side, killing the ability to roll-back to ext4 that's the whole point of keeping that dedicated subvolume. So once you are sure you aren't going to roll-back, deleting the ext4 saved subvolume, thus allowing btrfs to manage the entire filesystem without the previously ext4 stuff getting in the way, is high priority. IOW the conversion, like many conversions, is a compromise. It serves a certain purpose, but until the legacy stuff is gone, the new stuff is hobbled and can't be used to full effect. So yes, any btrfs converted from ext4 is going to be a real mess, in btrfs terms, until that ext4 saved subvolume is deleted, because it simply can't manage it like it can native btrfs since doing so would break the ability to roll back to the ext4. But it's an expected mess, and it's only a mess because the native formats differ. The ext4 image can be just fine in ext4, and when it is removed, btrfs is normally just fine as well. It's just the btrfs with the ext4 image still there that's a problem, and that only because the ext4 image isn't really playing by btrfs native rules, so btrfs can't properly manage it. BTW, if it was letting you balance without an error than you probably didn't run into this particular problem that often happens with ext* conversions, likely because the filesystem was new and basically all relatively small (under 1 GiB) distro files, but it's worth knowing about and doing the one additional step, just to be sure, plus for possible future conversions. With ext4, extent size is effectively unlimited. A full 4.7 GiB DVD ISO image file, for instance, properly defragged, can appear as a single 4.7 GiB extent. No problem on ext4 and in fact that'd be the ideal. On btrfs, by contrast, data chunk size, and thus largest possible extent size, is 1 GiB. That 4.7 GiB DVD ISO image would have to be broken up into at least five extents, four of a full GiB each plus the sub-GiB remainder of the file. In practice it'd likely be six extents, the beginning of the file using what was left of the current data chunk, four complete 1 GiB data chunks, and whatever was left beginning a sixth data chunk, that would eventually be filled with other file data as well. Of course the same thing applies to other large files, whatever their content and size. Big VM images are one example, big database files another, big multi-gig archive files yet another, big non-ISO media files again another. As a result, people with these sorts of large files on their originating ext4 filesystem tend to run into problems with btrfs balance, etc, after the conversion, because btrfs balance expects to see extents no larger than the btrfs-native 1 GiB data chunk, and doesn't know what to do with these 1 GiB super-extents. On converted btrfs with this sort of file, balance will simply error out while the ext4 saved subvolume remains, and normally even after it is gone, until a btrfs filesystem defrag is run on the former ext4 content to break up these super-extents into 1 GiB maximum native btrfs data chunks that btrfs in general and btrfs balance in particular can actually handle. Since you didn't run into this problem, you evidently either didn't have any of these 1 GiB files, not surprising on a fresh install, or if you did, they were already fragmented enough for btrfs balance to handle. However, I'd still recommend doing a proper btrfs filesystem defrag and then another balance, the combination of which should ensure that every last bit of what remains of the ext4 formatting is properly converted to btrfs native. Given that you already completed a balance the defrag and rebalance may not matter, but better to do it unnecessarily now and be sure, than to run into problems and /wish/ you had done so later. Additionally, doing it now, before you add too many additional files to the filesystem, will be easier and take less time than doing it later. One more tip while we're talking about defrag: If you don't have any big ( half a GiB) files to deal with, or if you do but they're all essentially static files (like already written media files that aren't going to be edited in-place), I'd strongly recommend using btrfs' autodefrag mount option, which I use on all my btrfs here. OTOH, for large internal rewrite pattern files such as active VM image files, big database files, even big torrented files until they're fully downloaded
Re: Confusion with newly converted filesystem
On Oct 9, 2014, at 2:58 PM, Tim Cuthbertson ratch...@gmail.com wrote: Never mind. I have stumbled my way into a solution. I ran btrfs subv delete /ext2_saved. Then I ran btrfs balance start /. That relocated 15 of 15 chunks. Now fi show shows 2.03 GB used on each device and fi df shows 1 GB of metadata total. Apparently, that saved ext4 subvolume was a real mess. Not a mess, it's just a side effect of the conversion while it's still reversible. It's kinda both ext3/4 and Btrfs at the same time after conversion. You can even still mount the snapshot as ext3/4. Once you're ready to commit to Btrfs and not use the ext rollback snapshot, deleting it and balancing completes the conversion. It's more of a metadata duplication via in-line migration. Chris Murphy-- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
