Re: Small fs
Chris Murphy posted on Mon, 12 Sep 2016 08:48:49 -0600 as excerpted: > On Sun, Sep 11, 2016 at 10:54 PM, Duncan <1i5t5.dun...@cox.net> wrote: > >> On the bright side, the double-whammy of being under such tight >> filesystem size constraints, coupled with finding out you have less >> than half the space of the filesystem actually available due to >> default-mixed-mode AND default dup-metadata (thus dup everything), > > I'm not following what you mean by default dup everything. You mean > that's how it used to work? Because on a real USB stick, 2GiB: Yes. What I meant is that now that --mixed isn't default, it avoids auto- halving the space due to duping the data as well (which AFAIK it used to do by default since mixed-mode treated everything as metadata, and metadata defaults are dup except on ssd), in mixed-mode. Tho now that I think of it, it's possible I'm mistaken, since I always specify single/dup/raid1/whatever, as well as mixed if I want it, choosing not to rely on the defaults, myself. Which seems a pretty wise choice, with the defaults changing over time, sometimes, as with removal of the old default mixed under a GiB, to strongly dis-recommended defaults. =:^( -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
On 12 September 2016 at 19:55, Austin S. Hemmelgarnwrote: > I'm not sure about gparted, but the default behavior for mkfs is as follows: > 1. Is the device rotational? (check /sys/block//rotational). If > not, do some extra stuff to try and ID it as an SSD. If it is an SSD, use > SINGLE mode for metadata, otherwise use DUP mode for metadata. > 2. Is the FS set for mixed-bg? If so, use the same profile for data as > metadata, otherwise use SINGLE mode for data. > > It would not surprise me if gparted switches to single metadata mode for a > small enough FS, but I'm not certain. I do think that they just use the > default selection for mixed-bg though, which means not using it in current > btrfs-progs versions. GParted always just takes the mkfs.btrfs defaults. https://git.gnome.org/browse/gparted/tree/src/btrfs.cc?h=GPARTED_0_26_1#n154 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
On 2016-09-12 14:46, Imran Geriskovan wrote: Wait wait wait a second: This is 256 MB SINGLE created by GPARTED, which is the replacement of MANUALLY CREATED 127MB DUP which is now non-existant.. Which I was not aware it was a DUP at the time.. Peeww... Small btrfs is full of surprises.. ;) What's more, I also have another 128MB SINGLE which I've been using for some years and did not bother with its DUP/SINGLENESS. And I compared them all to draw some conclusions. Heh.. That's the story. Verdict is: DUP/SINGLE is a very serious fun stuff when used unknowingly. Small btrfs is such a case. And third party tools (ex: gparted) plays with it. Let's warn users with some documentation, together with "A formal small fs" behaviour.. I'm not sure about gparted, but the default behavior for mkfs is as follows: 1. Is the device rotational? (check /sys/block//rotational). If not, do some extra stuff to try and ID it as an SSD. If it is an SSD, use SINGLE mode for metadata, otherwise use DUP mode for metadata. 2. Is the FS set for mixed-bg? If so, use the same profile for data as metadata, otherwise use SINGLE mode for data. It would not surprise me if gparted switches to single metadata mode for a small enough FS, but I'm not certain. I do think that they just use the default selection for mixed-bg though, which means not using it in current btrfs-progs versions. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
> Wait wait wait a second: > This is 256 MB SINGLE created > by GPARTED, which is the replacement of MANUALLY > CREATED 127MB DUP which is now non-existant.. > Which I was not aware it was a DUP at the time.. > Peeww... Small btrfs is full of surprises.. ;) What's more, I also have another 128MB SINGLE which I've been using for some years and did not bother with its DUP/SINGLENESS. And I compared them all to draw some conclusions. Heh.. That's the story. Verdict is: DUP/SINGLE is a very serious fun stuff when used unknowingly. Small btrfs is such a case. And third party tools (ex: gparted) plays with it. Let's warn users with some documentation, together with "A formal small fs" behaviour.. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
> btrfs filesystem df /mnt/back/boot > Data, single: total=8.00MiB, used=0.00B > System, DUP: total=8.00MiB, used=16.00KiB > Metadata, DUP: total=32.00MiB, used=112.00KiB > GlobalReserve, single: total=16.00MiB, used=0.00B > IT IS DUP!! Wait wait wait a second: This is 256 MB SINGLE created by GPARTED, which is the replacement of MANUALLY CREATED 127MB DUP which is now non-existant.. Which I was not aware it was a DUP at the time.. Peeww... Small btrfs is full of surprises.. ;) -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
On 2016-09-12 10:51, Chris Murphy wrote: On Mon, Sep 12, 2016 at 8:09 AM, Henk Slagerwrote: FWIW, I use BTRFS for /boot, but it's not for snapshotting or even the COW, it's for DUP mode and the error recovery it provides. Most people don't think about this if it hasn't happened to them, but if you get a bad read from /boot when loading the kernel or initrd, it can essentially nuke your whole system. I run BTRFS for /boot in DUP mode with mixed-bg (because I only use 512MB for boot) to mitigate the chance that a failed read has any impact, and ensure that if it does, it will refuse to boot instead of booting with a corrupted kernel or initrd. Suppose kernel and initrd are on a BTRFS fs with data, metadata and system all single profile. Will a bootloader then just continue booting up a system even when there are csum errors in kernel and/or initrd files? Suppose the bootloader is grub2. I"m wondering the same thing. I don't know if GRUB's Btrfs code checks for csum matches, and on error whether it knows to retry from some other block group. I can test this and report results (I've got GRUB built with the FUSE based mount tool, which lets you use GRUB's FS modules from regular Linux userspace), but it may be a while before I can get things set up to properly test it. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
On Mon, Sep 12, 2016 at 8:09 AM, Henk Slagerwrote: >> FWIW, I use BTRFS for /boot, but it's not for snapshotting or even the COW, >> it's for DUP mode and the error recovery it provides. Most people don't >> think about this if it hasn't happened to them, but if you get a bad read >> from /boot when loading the kernel or initrd, it can essentially nuke your >> whole system. I run BTRFS for /boot in DUP mode with mixed-bg (because I >> only use 512MB for boot) to mitigate the chance that a failed read has any >> impact, and ensure that if it does, it will refuse to boot instead of >> booting with a corrupted kernel or initrd. > > Suppose kernel and initrd are on a BTRFS fs with data, metadata and > system all single profile. Will a bootloader then just continue > booting up a system even when there are csum errors in kernel and/or > initrd files? Suppose the bootloader is grub2. I"m wondering the same thing. I don't know if GRUB's Btrfs code checks for csum matches, and on error whether it knows to retry from some other block group. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
On Sun, Sep 11, 2016 at 10:54 PM, Duncan <1i5t5.dun...@cox.net> wrote: > On the bright side, the double-whammy of being under such tight > filesystem size constraints, coupled with finding out you have less than > half the space of the filesystem actually available due to default-mixed- > mode AND default dup-metadata (thus dup everything), I'm not following what you mean by default dup everything. You mean that's how it used to work? Because on a real USB stick, 2GiB: [chris@f24m ~]$ mkfs.btrfs -M /dev/sdb btrfs-progs v4.7.2 See http://btrfs.wiki.kernel.org for more information. probe of /dev/sdb failed, cannot detect existing filesystem. ERROR: use the -f option to force overwrite of /dev/sdb [chris@f24m ~]$ sudo mkfs.btrfs -M /dev/sdb btrfs-progs v4.7.2 See http://btrfs.wiki.kernel.org for more information. Label: (null) UUID: 237d8ab9-b22f-435c-bbd8-2117afa1760c Node size: 4096 Sector size:4096 Filesystem size:1.87GiB Block group profiles: Data+Metadata:single8.00MiB System: single4.00MiB SSD detected: no Incompat features: mixed-bg, extref, skinny-metadata Number of devices: 1 Devices: IDSIZE PATH 1 1.87GiB /dev/sdb Whereas without the -M [chris@f24m ~]$ sudo mkfs.btrfs -f /dev/sdb [sudo] password for chris: btrfs-progs v4.7.2 See http://btrfs.wiki.kernel.org for more information. Label: (null) UUID: a7c79550-10b8-4a4c-9b23-9c9110e30121 Node size: 16384 Sector size:4096 Filesystem size:1.87GiB Block group profiles: Data: single8.00MiB Metadata: DUP 95.56MiB System: DUP 8.00MiB SSD detected: no Incompat features: extref, skinny-metadata Number of devices: 1 Devices: IDSIZE PATH 1 1.87GiB /dev/sdb So with -M, it's single by default.. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
On 2016-09-12 10:09, Henk Slager wrote: FWIW, I use BTRFS for /boot, but it's not for snapshotting or even the COW, it's for DUP mode and the error recovery it provides. Most people don't think about this if it hasn't happened to them, but if you get a bad read from /boot when loading the kernel or initrd, it can essentially nuke your whole system. I run BTRFS for /boot in DUP mode with mixed-bg (because I only use 512MB for boot) to mitigate the chance that a failed read has any impact, and ensure that if it does, it will refuse to boot instead of booting with a corrupted kernel or initrd. Suppose kernel and initrd are on a BTRFS fs with data, metadata and system all single profile. Will a bootloader then just continue booting up a system even when there are csum errors in kernel and/or initrd files? Suppose the bootloader is grub2. Assuming I understand how they implemented their BTRFS support in GRUB, it should still be checking csums just like the kernel implementation, and thus should throw a read error when a block is corrupted. For a stock upstream version of GRUB2, this would mean that it will refuse to boot that entry, and may try to auto-boot a fallback entry (which will probably fail given how most distros set up their fallback entries). -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
> FWIW, I use BTRFS for /boot, but it's not for snapshotting or even the COW, > it's for DUP mode and the error recovery it provides. Most people don't > think about this if it hasn't happened to them, but if you get a bad read > from /boot when loading the kernel or initrd, it can essentially nuke your > whole system. I run BTRFS for /boot in DUP mode with mixed-bg (because I > only use 512MB for boot) to mitigate the chance that a failed read has any > impact, and ensure that if it does, it will refuse to boot instead of > booting with a corrupted kernel or initrd. Suppose kernel and initrd are on a BTRFS fs with data, metadata and system all single profile. Will a bootloader then just continue booting up a system even when there are csum errors in kernel and/or initrd files? Suppose the bootloader is grub2. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
>> Just to note again: >> Ordinary 127MB btrfs gives "Out of space" around 64MB payload. 128MB is >> usable to the end. > Thanks, and just to clarify for others possibly following along or > googling it up later, that's single mode (as opposed to dup mode) for at > least data, if in normal separate data/metadata mode, and single for the > combined mixed-mode chunks if in mixed-bg mode, correct? > > Because if the data is dup mode as well, as it would be by default in > mixed-bg mode (unless on ssd), 128 MiB should allow storing only 64 MiB > (and that's not accounting for the system chunk or global reserve > metadata, so it'd be less than that) data. That's /boot on my laptop. Its fairly old fs which is created about 3-4 years. May be 5. I'm not so sure.. btrfs filesystem df /boot System, single: total=4.00MiB, used=4.00KiB Data+Metadata, single: total=124.00MiB, used=103.88MiB GlobalReserve, single: total=4.00MiB, used=0.00B IT IS SINGLE!! The 128/64Mb thing happened when I created a backup usb drive for mirroring it. Then make it 256MB. btrfs filesystem df /mnt/back/boot Data, single: total=8.00MiB, used=0.00B System, DUP: total=8.00MiB, used=16.00KiB Metadata, DUP: total=32.00MiB, used=112.00KiB GlobalReserve, single: total=16.00MiB, used=0.00B IT IS DUP!! So Duncan you got it: I'm comparing the SINGLE to DUP... I'm not sure how I got that SINGLE though... >> I'm experimenting with an extracted (and customized) initrd on /boot. >> That is, /boot is a "minimal root on its own" which can switch to real >> root or do other things. Kernel and modules at /boot will not support >> any fs other than btrfs. (Or it may) >> It seems a minimally usable root around 10MB is possible. >> And it is free of udev and systemd.. > > You don't specifically mention the distro, but given that gentoo's one of > the only general-purpose distros that hasn't switched to systemd yet (tho > it offers the choice for those who want it, and I've taken that choice > here), there's a fair chance that's what you're running, as both I and > Martin Steigerwald (based on earlier threads) are. Now the offtopic: - I'm on Arch. Considering to switch to Void. - Using syslinux as bootmanager. It is rock solid and compact. Just edit syslinux.cfg and reboot to any configuration without any hassle. - Also using runit side stepping systemd. It is also rock solid and compact. - Know your hardware and thus the Kernel. And get rid of and middleware (all udev derivatives) playing guess games on your machine. - Two custom kernel configurations: All built-in (Was using until now.) All modules. (Now switching to this.) Load all your modules on exactly known locations when booting. To do this customize your runit start up scripts. I'm switching to "All modules" approach after I determined and understand the key modules. Don't affaid. Its accually very doable. You need at most 8-9 modprobes to get them all. If you know your HW and kernel it is no big deal. - Configuring kernel teaches alot. - Compiling a kernel with only required modules takes about 15 minutes if you got your .config file right. Play with "make xconfig". Alot... - Keep all your previous .config files as record. When something goes wrong diff them for throubleshooting. - Dracut is a dirty hack. mkinitcpio is great. - Compressed initrd is bad. It hides what is going on. Extract it. Play with it. Than make it the root fs of your /boot. (I'm now here) - Starting X without udev is simple. Disable hotplug. 5-6 lines of Manual entries for keyboard, mouse and touchpad is sufficient in your /etc/X11/xorg.conf.d -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
On 2016-09-12 08:54, Imran Geriskovan wrote: On 9/11/16, Chris Murphywrote: Something else that's screwy in that bug that I just realized, why is it not defaulting to mixed-block groups on a 100MiB fallocated file? I thought mixed-bg was the default below a certain size like 2GiB or whatever? With an ordinary partition on a single disk, fs created with just "mkfs.btrfs /dev/sdxx": - 128MB works fine. - 127MB works but as if it is 64MB. Can we say size should be in multiples of 64MB? Why should it be in multiples? I think what you're describing is part of the bug above that just needs to be fixed. Btrfs itself internally uses bytes, so multiples of 64MiB is OK but I wouldn't use the word "should" with it. I'm not suggesting anything. I'm just describing the behaviour we've seen. If it is (or will be) something different its all ok for me. But, what is that "formal behaviour" at the low end? That is the discussion.. Formally, with mixed-bg and otherwise default options, you should be able to get to about 48-49% of the total size of the FS worth of data stored. In practice, I see such filesystems start having issues at about 45% of the total size. Note that this is not the percentages reported by regular df (which should say about 90% full in this case), b ut a comparison based on du -sh and the size of the partition. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
On 9/11/16, Chris Murphywrote: > Something else that's screwy in that bug that I just realized, why is > it not defaulting to mixed-block groups on a 100MiB fallocated file? I > thought mixed-bg was the default below a certain size like 2GiB or > whatever? >> With an ordinary partition on a single disk, >> fs created with just "mkfs.btrfs /dev/sdxx": >> - 128MB works fine. >> - 127MB works but as if it is 64MB. >> Can we say size should be in multiples of 64MB? > Why should it be in multiples? I think what you're describing is part > of the bug above that just needs to be fixed. Btrfs itself internally > uses bytes, so multiples of 64MiB is OK but I wouldn't use the word > "should" with it. I'm not suggesting anything. I'm just describing the behaviour we've seen. If it is (or will be) something different its all ok for me. But, what is that "formal behaviour" at the low end? That is the discussion.. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
On 2016-09-11 15:51, Martin Steigerwald wrote: Am Sonntag, 11. September 2016, 19:46:32 CEST schrieb Hugo Mills: On Sun, Sep 11, 2016 at 09:13:28PM +0200, Martin Steigerwald wrote: Am Sonntag, 11. September 2016, 16:44:23 CEST schrieb Duncan: * Metadata, and thus mixed-bg, defaults to DUP mode on a single-device filesystem (except on ssd where I actually still use it myself, and recommend it except for ssds that do firmware dedupe). In mixed-mode this means two copies of data as well, which halves the usable space. IOW, when using mixed-mode, which is recommended under a gig, and dup replication which is then the single-device default, effective usable space is **HALVED**, so 256 MiB btrfs size becomes 128 MiB usable. (!!) I don´t get this part. That is just *metadata* being duplicated, not the actual *data* inside the files. Or am I missing something here? In mixed mode, there's no distinction: Data and metadata both use the same chunks. If those chunks are DUP, then both data and metadata are duplicated, and you get half the space available. In german I´d say "autsch", in english according to pda.leo.org "ouch", to this. Okay, I just erased using mixed mode as an idea from my mind altogether :). Until recently (when DUP mode for data on a single device got added), it did have advantages. If you're running on a single disk and need replication, it's a lot more efficient to just run a single partition in DUP mode than two in RAID1 mode. It also makes it a bit less likely to hit ENOSPC issues on small filesystems. Just like I think I will never use a BTRFS below 5 GiB. Well, with one exception, maybe on the eMMC flash of the new Omnia Turris router that I hope will arrive soon at my place. FWIW, I don't use anything smaller than 8G unless I have to (with the notable exception of /boot), and when I do need to, I usually just use mixed mode because once you get that small, you need all the help you can get to keep from the whole FS being allocated to chunks. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
On 2016-09-11 15:21, Martin Steigerwald wrote: Am Sonntag, 11. September 2016, 21:56:07 CEST schrieb Imran Geriskovan: On 9/11/16, Duncan <1i5t5.dun...@cox.net> wrote: Martin Steigerwald posted on Sun, 11 Sep 2016 17:32:44 +0200 as excerpted: What is the smallest recommended fs size for btrfs? Can we say size should be in multiples of 64MB? Do you want to know the smalled *recommended* or the smallest *possible* size? In fact both. I'm reconsidering my options for /boot Well my stance on boot still is: Ext4. Done. :) It just does not bother me. It practically makes no difference at all. It has no visible effect on my user experience and I never saw the need to snapshot / boot. FWIW, I use BTRFS for /boot, but it's not for snapshotting or even the COW, it's for DUP mode and the error recovery it provides. Most people don't think about this if it hasn't happened to them, but if you get a bad read from /boot when loading the kernel or initrd, it can essentially nuke your whole system. I run BTRFS for /boot in DUP mode with mixed-bg (because I only use 512MB for boot) to mitigate the chance that a failed read has any impact, and ensure that if it does, it will refuse to boot instead of booting with a corrupted kernel or initrd. But another approach in case you want to use BTRFS for /boot is to use a subvolume. Thats IMHO the SLES 12 default setup. They basically create subvolumes for /boot, /var, /var/lib/mysql – you name it. Big advantage: You have one big FS and do not need to plan space for partitions or LVs. Disadvantage: If it breaks, it breaks. That said, I think at a new installation I may do this for /boot. Just put it inside a subvolume. From my experiences at work with customer systems and even some systems I setup myself, I often do not use little partitions anymore. I did so for a CentOS 7 training VM, just 2 GiB XFS for /var. Guess what happened? Last update was too long ago, so… yum tried to download a ton of packages and then complained it has not enough space in /var. Luckily I used LVM, so I enlarged partition LVM resides on, enlarged PV and then enlarged /var. There may be valid reasons to split things up, and I am quite comfortable with splitting / boot out, cause its, well, plannable easily enough. And it may make sense to split /var or /var/log out. But on BTRFS I would likely use subvolumes. Only thing I may separate would be /home to make it easier on a re-installation of the OS to keep it around. That said, I never ever reinstalled the Debian on this ThinkPad T520 since I initially installed it. And on previous laptops I even copied the Debian on the older laptop onto the newer laptop. With the T520 I reinstalled, cause I wanted to switch to 64 bit cleanly. Yeah, small partitions make sense in some cases, but in a lot of places they're typically used, there are perfectly legitimate reasons to suddenly need multiple gigabytes of extra space for a short period of time. In my case, the only partition that I have split out that's less than a few GB is /boot, but I usually split out /usr/src and /usr/portage, because I need them to be fast, and run /tmp and /var/tmp on tmpfs (so technically split out). I personally see little value in splitting out /home on a single user system unless you're multi-booting or planning to switch distros at some point (or are running a distro that needs special handling to do release upgrades). -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
Chris Murphy posted on Sun, 11 Sep 2016 21:03:04 -0600 as excerpted: > The man page says: > "The recommended size for the mixed mode is for filesystems less than > 1GiB." But in this case recommended !=default which requires some mental > gymnastics to rectify. If mixed-bg becomes obsolete upon enospc being no > more likely with isolated block groups, then OK fine, but in the > meantime... On the bright side, the double-whammy of being under such tight filesystem size constraints, coupled with finding out you have less than half the space of the filesystem actually available due to default-mixed- mode AND default dup-metadata (thus dup everything), gets eliminated, and barring problems with unbalanced chunk-sizes, you actually get to use most of capacity of the filesystem for actual files, instead of less than half of it. =:^) But I remain unconvinced that benefit outweighs the serious administrative headaches trying to run without mixed-mode on such small btrfs is likely to generate. And what's worse, it's the same folks that are likely to have problems coping with either issue, but fixing the under-half-available-for-use (at the cost of filesystem resiliency) is a one-time thing, while the administrative issue of unbalanced chunks is likely to come back to bite them again and again. But still, having people find they can fit only ~110 MiB in a 256 MiB btrfs, while with ext*, they can fit say 240 MiB in the same size filesystem, could be a bit more to try to explain to the technically under-literate, than the devs decided they were willing to deal with. Just saying it'd divided in chunks and another chunk won't fit is arguably easier than trying to explain why the filesystem will fit less than half of what the size of it suggests it should fit, IOW. And I think that argument /was/ made, to some extent. But the whole thing only came up because they found testing with small filesystems inconvenient due to the mixed-bg default, so rather than fix that by fixing the tests, they broke the previously sane defaults, instead. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
Imran Geriskovan posted on Sun, 11 Sep 2016 21:56:07 +0300 as excerpted: > On 9/11/16, Duncan <1i5t5.dun...@cox.net> wrote: >> Martin Steigerwald posted on Sun, 11 Sep 2016 17:32:44 +0200 as >> excerpted: What is the smallest recommended fs size for btrfs? Can we say size should be in multiples of 64MB? >>> Do you want to know the smalled *recommended* or the smallest >>> *possible* >>> size? > > In fact both. > I'm reconsidering my options for /boot >> [Snip my detail points, leaving just the summary...] >> So of my 256 MiB btrfs mixed-mode /boot, 31+4=35 MiB is overhead, >> leaving 221 MiB for actual data and metadata. But due to dup mode >> that's halved, to 110.5 MiB usable space. > > That's quite an info.. Thanks a lot.. > > Just to note again: > Ordinary 127MB btrfs gives "Out of space" around 64MB payload. 128MB is > usable to the end. Thanks, and just to clarify for others possibly following along or googling it up later, that's single mode (as opposed to dup mode) for at least data, if in normal separate data/metadata mode, and single for the combined mixed-mode chunks if in mixed-bg mode, correct? Because if the data is dup mode as well, as it would be by default in mixed-bg mode (unless on ssd), 128 MiB should allow storing only 64 MiB (and that's not accounting for the system chunk or global reserve metadata, so it'd be less than that) data. > I'm experimenting with an extracted (and customized) initrd on /boot. > That is, /boot is a "minimal root on its own" which can switch to real > root or do other things. Kernel and modules at /boot will not support > any fs other than btrfs. (Or it may) > > It seems a minimally usable root around 10MB is possible. > And it is free of udev and systemd.. You don't specifically mention the distro, but given that gentoo's one of the only general-purpose distros that hasn't switched to systemd yet (tho it offers the choice for those who want it, and I've taken that choice here), there's a fair chance that's what you're running, as both I and Martin Steigerwald (based on earlier threads) are. FWIW, a bit more about my /boot (and kernel/initr*) layout, then, since it may be apropos... both because you're dealing with /boot, and because you might be on gentoo, as well, in which case you may find even more useful hints in the below. Of course you can set it up how you like, but this is what I've found works very well for me. =:^) 1) Since I custom-configure and build my own kernels anyway, I build monolithic, no kernel modules, and indeed, the kernel options allowing loading modules are turned off -- more secure that way since I don't load modules anyway. That simplifies a lot of stuff, including no longer needing module-init- tools or whatever installed at all. So it's not. (I negated the @system listing for it; actually I negated @system entirely and added what I actually needed to my @world via world-sets, but that's beside the point.) 2) Since a multi-device btrfs / effectively requires an initr*, I do run one, using dracut to generate it, but I don't boot it as a separate file. Instead I have the kernel configured to pull it in and append it as an initramfs to each built kernel. The idea here is that I keep tested-bootable kernels along with their tested-bootable initr*s around, so whatever I may be upgrading, kernel, dracut, the initr* itself, or some package with a binary in the initr*, I always have a fallback kernel and its attached initramfs that's tested bootable. That way, should some kernel/initr* component fail, I know I can simply pick an older/tested kernel/initramfs file, since they're a single file, and boot with it, using that boot to investigate and fix the problem with the kernel/initramfs. And I also know that once I've tested a kernel/initramfs to work, if it suddenly fails, it can't be something in the initramfs or kernel itself, as that is known to work, it must instead be some problem with the hardware or in the boot after handover to the main /. One additional detail here regarding dracut: Even if you've configured dracut to not worry about modules as you run a monolithic kernel, for some reason (bug?) it still requires (as opposed to uses if installed) rmmod for an initr* build. Why it requires rmmod but not modprobe is beyond me, but it does. So now I have a no-rmmod.patch in /etc/portage/patches/sys-kernel/dracut , and I can use dracut setup for a monolithic kernel initr*, without having to have the useless rmmod from module-init-tools around for it to pull in when I build a new initr*. 3) Due to btrfs dup-mode halving my usable space on /boot, something I should have anticipated but didn't, I have only ~110 MiB of space on / boot, and the grub2 installation eats up some of it. Of course that leaves me way less room for kernels than I expected, particularly when I had the uncompressed initr* (that the kernel build compresses and appends as an initramfs
Re: Small fs
On Sun, Sep 11, 2016 at 8:00 PM, Duncan <1i5t5.dun...@cox.net> wrote: > Chris Murphy posted on Sun, 11 Sep 2016 14:33:18 -0600 as excerpted: > >> Something else that's screwy in that bug that I just realized, why is it >> not defaulting to mixed-block groups on a 100MiB fallocated file? I >> thought mixed-bg was the default below a certain size like 2GiB or >> whatever? > > You apparently missed the memo... I got the memo right after I clicked send and read the thread. > Newer btrfs-progs mkfs.btrfs no longer defaults under-1-GiB to mixed-bg > mode, tho it remains very strongly recommended below 1 GiB, and soft- > recommended to somewhere between 4 and 32 GiB (I believe the wiki says 5 > GiB at this point but don't know how it arrived at that, but the numbers > I've seen suggested on-list range between 4 and 32 GiB, as above). > > The explanation of why, based on the thread where I remember it coming > up, was because defaulting to mixed-mode was making testing more > complex. Don't ask me to agree with that because I most certainly don't; > IMO sane defaults for normal use, which everyone seems to agree mixed- > mode for under a GiB is, should apply, and if testing needs special- > cased, well, special-case it. But none-the-less, that's the context in > which it was agreed to do away with the mixed-mode default, despite it > still being extremely strongly recommended for under a GiB. The man page says: "The recommended size for the mixed mode is for filesystems less than 1GiB." But in this case recommended !=default which requires some mental gymnastics to rectify. If mixed-bg becomes obsolete upon enospc being no more likely with isolated block groups, then OK fine, but in the meantime... -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
Chris Murphy posted on Sun, 11 Sep 2016 14:33:18 -0600 as excerpted: > Something else that's screwy in that bug that I just realized, why is it > not defaulting to mixed-block groups on a 100MiB fallocated file? I > thought mixed-bg was the default below a certain size like 2GiB or > whatever? You apparently missed the memo... Newer btrfs-progs mkfs.btrfs no longer defaults under-1-GiB to mixed-bg mode, tho it remains very strongly recommended below 1 GiB, and soft- recommended to somewhere between 4 and 32 GiB (I believe the wiki says 5 GiB at this point but don't know how it arrived at that, but the numbers I've seen suggested on-list range between 4 and 32 GiB, as above). The explanation of why, based on the thread where I remember it coming up, was because defaulting to mixed-mode was making testing more complex. Don't ask me to agree with that because I most certainly don't; IMO sane defaults for normal use, which everyone seems to agree mixed- mode for under a GiB is, should apply, and if testing needs special- cased, well, special-case it. But none-the-less, that's the context in which it was agreed to do away with the mixed-mode default, despite it still being extremely strongly recommended for under a GiB. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
On Sun, Sep 11, 2016 at 9:27 AM, Imran Geriskovanwrote: > What is the smallest recommended fs size for btrfs? It depends on the layout. And there is some confusion about the mkfs command message it returns when it doesn't work out. https://bugzilla.kernel.org/show_bug.cgi?id=112461 Something else that's screwy in that bug that I just realized, why is it not defaulting to mixed-block groups on a 100MiB fallocated file? I thought mixed-bg was the default below a certain size like 2GiB or whatever? > > - There are mentions of 256MB around the net. > - Gparted reserves minimum of 256MB for btrfs. > > With an ordinary partition on a single disk, > fs created with just "mkfs.btrfs /dev/sdxx": > - 128MB works fine. > - 127MB works but as if it is 64MB. > > Can we say size should be in multiples of 64MB? Why should it be in multiples? I think what you're describing is part of the bug above that just needs to be fixed. Btrfs itself internally uses bytes, so multiples of 64MiB is OK but I wouldn't use the word "should" with it. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
Am Sonntag, 11. September 2016, 19:46:32 CEST schrieb Hugo Mills: > On Sun, Sep 11, 2016 at 09:13:28PM +0200, Martin Steigerwald wrote: > > Am Sonntag, 11. September 2016, 16:44:23 CEST schrieb Duncan: > > > * Metadata, and thus mixed-bg, defaults to DUP mode on a single-device > > > filesystem (except on ssd where I actually still use it myself, and > > > recommend it except for ssds that do firmware dedupe). In mixed-mode > > > this means two copies of data as well, which halves the usable space. > > > > > > IOW, when using mixed-mode, which is recommended under a gig, and dup > > > replication which is then the single-device default, effective usable > > > space is **HALVED**, so 256 MiB btrfs size becomes 128 MiB usable. (!!) > > > > I don´t get this part. That is just *metadata* being duplicated, not the > > actual *data* inside the files. Or am I missing something here? > >In mixed mode, there's no distinction: Data and metadata both use > the same chunks. If those chunks are DUP, then both data and metadata > are duplicated, and you get half the space available. In german I´d say "autsch", in english according to pda.leo.org "ouch", to this. Okay, I just erased using mixed mode as an idea from my mind altogether :). Just like I think I will never use a BTRFS below 5 GiB. Well, with one exception, maybe on the eMMC flash of the new Omnia Turris router that I hope will arrive soon at my place. -- Martin -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
On Sun, Sep 11, 2016 at 09:13:28PM +0200, Martin Steigerwald wrote: > Am Sonntag, 11. September 2016, 16:44:23 CEST schrieb Duncan: > > * Metadata, and thus mixed-bg, defaults to DUP mode on a single-device > > filesystem (except on ssd where I actually still use it myself, and > > recommend it except for ssds that do firmware dedupe). In mixed-mode > > this means two copies of data as well, which halves the usable space. > > > > IOW, when using mixed-mode, which is recommended under a gig, and dup > > replication which is then the single-device default, effective usable > > space is **HALVED**, so 256 MiB btrfs size becomes 128 MiB usable. (!!) > > I don´t get this part. That is just *metadata* being duplicated, not the > actual *data* inside the files. Or am I missing something here? In mixed mode, there's no distinction: Data and metadata both use the same chunks. If those chunks are DUP, then both data and metadata are duplicated, and you get half the space available. Hugo. -- Hugo Mills | Questions are a burden, and answers a prison for hugo@... carfax.org.uk | oneself http://carfax.org.uk/ | PGP: E2AB1DE4 | The Prisoner signature.asc Description: Digital signature
Re: Small fs
Am Sonntag, 11. September 2016, 21:56:07 CEST schrieb Imran Geriskovan: > On 9/11/16, Duncan <1i5t5.dun...@cox.net> wrote: > > Martin Steigerwald posted on Sun, 11 Sep 2016 17:32:44 +0200 as excerpted: > >>> What is the smallest recommended fs size for btrfs? > >>> Can we say size should be in multiples of 64MB? > >> > >> Do you want to know the smalled *recommended* or the smallest *possible* > >> size? > > In fact both. > I'm reconsidering my options for /boot Well my stance on boot still is: Ext4. Done. :) It just does not bother me. It practically makes no difference at all. It has no visible effect on my user experience and I never saw the need to snapshot / boot. But another approach in case you want to use BTRFS for /boot is to use a subvolume. Thats IMHO the SLES 12 default setup. They basically create subvolumes for /boot, /var, /var/lib/mysql – you name it. Big advantage: You have one big FS and do not need to plan space for partitions or LVs. Disadvantage: If it breaks, it breaks. That said, I think at a new installation I may do this for /boot. Just put it inside a subvolume. >From my experiences at work with customer systems and even some systems I setup myself, I often do not use little partitions anymore. I did so for a CentOS 7 training VM, just 2 GiB XFS for /var. Guess what happened? Last update was too long ago, so… yum tried to download a ton of packages and then complained it has not enough space in /var. Luckily I used LVM, so I enlarged partition LVM resides on, enlarged PV and then enlarged /var. There may be valid reasons to split things up, and I am quite comfortable with splitting / boot out, cause its, well, plannable easily enough. And it may make sense to split /var or /var/log out. But on BTRFS I would likely use subvolumes. Only thing I may separate would be /home to make it easier on a re-installation of the OS to keep it around. That said, I never ever reinstalled the Debian on this ThinkPad T520 since I initially installed it. And on previous laptops I even copied the Debian on the older laptop onto the newer laptop. With the T520 I reinstalled, cause I wanted to switch to 64 bit cleanly. -- Martin -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
Am Sonntag, 11. September 2016, 16:44:23 CEST schrieb Duncan: > * Metadata, and thus mixed-bg, defaults to DUP mode on a single-device > filesystem (except on ssd where I actually still use it myself, and > recommend it except for ssds that do firmware dedupe). In mixed-mode > this means two copies of data as well, which halves the usable space. > > IOW, when using mixed-mode, which is recommended under a gig, and dup > replication which is then the single-device default, effective usable > space is **HALVED**, so 256 MiB btrfs size becomes 128 MiB usable. (!!) I don´t get this part. That is just *metadata* being duplicated, not the actual *data* inside the files. Or am I missing something here? -- Martin -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
On 9/11/16, Duncan <1i5t5.dun...@cox.net> wrote: > Martin Steigerwald posted on Sun, 11 Sep 2016 17:32:44 +0200 as excerpted: >>> What is the smallest recommended fs size for btrfs? >>> Can we say size should be in multiples of 64MB? >> Do you want to know the smalled *recommended* or the smallest *possible* >> size? In fact both. I'm reconsidering my options for /boot > * Metadata, and thus mixed-bg, defaults to DUP mode on a single-device > filesystem (except on ssd where I actually still use it myself, and > recommend it except for ssds that do firmware dedupe). In mixed-mode > this means two copies of data as well, which halves the usable space. > IOW, when using mixed-mode, which is recommended under a gig, and dup > replication which is then the single-device default, effective usable > space is **HALVED**, so 256 MiB btrfs size becomes 128 MiB usable. (!!) > * There's also a system chunk to consider. This too is normally dup mode > on single device, raid1 on multi. While it shrinks to some extent with > size of filesystem, my 256 MiB /boot still has a 15.5 MiB system chunk, > doubled due to dup mode to 31 MiB. This is overhead you won't be able to > use for anything else. > > * There's also global reserve. This is a part of metadata (and thus > mixed-mode) that cannot be used under normal circumstances either. > However, unlike normal metadata, the accounting here is single -- it's X > space reserved no matter the replication type. On my 256 MiB /boot, it's > 4 MiB. > > So of my 256 MiB btrfs mixed-mode /boot, 31+4=35 MiB is overhead, leaving > 221 MiB for actual data and metadata. But due to dup mode that's halved, > to 110.5 MiB usable space. That's quite an info.. Thanks a lot.. Just to note again: Ordinary 127MB btrfs gives "Out of space" around 64MB payload. 128MB is usable to the end. I'm experimenting with an extracted (and customized) initrd on /boot. That is, /boot is a "minimal root on its own" which can switch to real root or do other things. Kernel and modules at /boot will not support any fs other than btrfs. (Or it may) It seems a minimally usable root around 10MB is possible. And it is free of udev and systemd.. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
Martin Steigerwald posted on Sun, 11 Sep 2016 17:32:44 +0200 as excerpted: > Am Sonntag, 11. September 2016, 18:27:30 CEST schrieben Sie: >> What is the smallest recommended fs size for btrfs? >> >> - There are mentions of 256MB around the net. >> - Gparted reserves minimum of 256MB for btrfs. >> >> With an ordinary partition on a single disk, >> fs created with just "mkfs.btrfs /dev/sdxx": >> - 128MB works fine. >> - 127MB works but as if it is 64MB. >> >> Can we say size should be in multiples of 64MB? > > Do you want to know the smalled *recommended* or the smallest *possible* > size? > > I personally wouldn´t go below one or two GiB or or so with BTRFS. On > small filesystems, I don´t know the treshold right now it uses a mixed > metadata/data format. And I think using smaller BTRFS filesystem invited > any left over "filesystem is full while it isn´t" issues. I agree with the general recommendations and *strongly* recommend (and use) mixed-mode below say 2 GiB myself, but I don't necessarily agree that the practical bottom limit is 1 GiB, as I partition heavily and have a couple smaller btrfs myself. There are some limits that should be kept in mind, however. * Metadata, and thus mixed-bg, defaults to DUP mode on a single-device filesystem (except on ssd where I actually still use it myself, and recommend it except for ssds that do firmware dedupe). In mixed-mode this means two copies of data as well, which halves the usable space. IOW, when using mixed-mode, which is recommended under a gig, and dup replication which is then the single-device default, effective usable space is **HALVED**, so 256 MiB btrfs size becomes 128 MiB usable. (!!) When the filesystem is that small, space tends to be at a premium in any case, and halving the available space can take a big bite out of things, so one considering such a btrfs should consider carefully, and possibly specify single mode if the loss of the data on the filesystem won't be a big deal, or double the filesystem size to account for dup data, if it would be more appropriate. Or do pair-device which defaults to raid1 mode for metadata and mixed. Or choose a more traditional filesystem, perhaps ext* based, perhaps even without a journal if size is that constrained, because the journal takes space as well, because fscking that small a filesystem in the event of a crash shouldn't take that long anyway. * There's also a system chunk to consider. This too is normally dup mode on single device, raid1 on multi. While it shrinks to some extent with size of filesystem, my 256 MiB /boot still has a 15.5 MiB system chunk, doubled due to dup mode to 31 MiB. This is overhead you won't be able to use for anything else. * There's also global reserve. This is a part of metadata (and thus mixed-mode) that cannot be used under normal circumstances either. However, unlike normal metadata, the accounting here is single -- it's X space reserved no matter the replication type. On my 256 MiB /boot, it's 4 MiB. So of my 256 MiB btrfs mixed-mode /boot, 31+4=35 MiB is overhead, leaving 221 MiB for actual data and metadata. But due to dup mode that's halved, to 110.5 MiB usable space. For something like a dedicated /boot that may be acceptable. For many applications it wouldn't be, and either the target size would need doubled, or filesystem robustness would need compromised by switching to single mode instead of dup. Or use a different filesystem. Tho to compare apples to apples, most other filesystems are only single mode anyway, and if it's an acceptable usage for them, it may well be an acceptable usage for btrfs as well. It's just that btrfs likes a bit more metadata robustness than that. But with such limited sizes, backing up the data if it's worth the hassle shouldn't be horribly time or resource consuming, and single mode can then make a bit more sense. FWIW, in addition to my previously mentioned 256 MiB /boot, and a backup /boot (selectable via bios) on another device, I have a 640 MiB btrfs pair-device raid1 /var/log. It's mixed-mode too, 640 MiB per device, but pair-device raid1, so I don't have to worry about the 2X data factor on a single device. All my other "system" partitions are btrfs raid1 as well, including /, /home, and the packages/build partition. But those are all over 2 GiB each. All my btrfs are on ssd, with only my media partitions and backups on spinning rust (reiserfs). -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Small fs
Am Sonntag, 11. September 2016, 18:27:30 CEST schrieben Sie: > What is the smallest recommended fs size for btrfs? > > - There are mentions of 256MB around the net. > - Gparted reserves minimum of 256MB for btrfs. > > With an ordinary partition on a single disk, > fs created with just "mkfs.btrfs /dev/sdxx": > - 128MB works fine. > - 127MB works but as if it is 64MB. > > Can we say size should be in multiples of 64MB? Do you want to know the smalled *recommended* or the smallest *possible* size? I personally wouldn´t go below one or two GiB or or so with BTRFS. On small filesystems, I don´t know the treshold right now it uses a mixed metadata/data format. And I think using smaller BTRFS filesystem invited any left over "filesystem is full while it isn´t" issues. Well there we go. Excerpt from mkfs.btrfs(8) manpage: -M|--mixed Normally the data and metadata block groups are isolated. The mixed mode will remove the isolation and store both types in the same block group type. This helps to utilize the free space regardless of the purpose and is suitable for small devices. The separate allocation of block groups leads to a situation where the space is reserved for the other block group type, is not available for allocation and can lead to ENOSPC state. The recommended size for the mixed mode is for filesystems less than 1GiB. The soft recommendation is to use it for filesystems smaller than 5GiB. The mixed mode may lead to degraded performance on larger filesystems, but is otherwise usable, even on multiple devices. The nodesize and sectorsize must be equal, and the block group types must match. Note versions up to 4.2.x forced the mixed mode for devices smaller than 1GiB. This has been removed in 4.3+ as it caused some usability issues. Thanks -- Martin -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html