Re: subvols and parents - how?
Christoph Anton Mitterer posted on Sat, 12 Dec 2015 03:32:57 +0100 as excerpted: > [B]etter said I largely rewrote: > https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Subvolumes > which I think should rather be its own wiki article. > > One of the devs/experts... please double check it and pick/drop what > you like/dislike. I'd hesitate to call myself a real "expert" on subvolumes because I don't run them here (tho I did play with them a bit, before deciding the additional benefit for my use-case wasn't worth the additional complexity), so nearly all of what I post on the subject is simply restatement of other people's posts on the subject[1], but... Looks quite reasonable to me... and /much/ more filled out than the last time I read it. =:^) > Especially notice that I've changed what was in the wiki, namely > subvolid=0 would mount the toplevel subovl,... I changed that to 5. On that particular point, subvolid=0 is a special-case alias to subvolid=5, for those for which 0 as reference to the toplevel subvolume makes more sense than 5. So 0 or 5, doesn't particularly matter in practice. Tho 5 is the actual tree ID (AFAIK) with 0 simply an alias to it, so 5 is arguably technically slightly more correct, while 0 could be argued to be humanly slightly more correct as the logical/expected ID for the toplevel subvolume. You might put that there too, somewhere, if you have a mind to do it... > Also I assumed the manpage would be correct and subvol= is always > relative to the top level subvol, thus subvol=/ should mount that. Haven't the foggiest... > What's still missing now, IMHO, is: > - the snapshots subchapter itself is not complete (especially ro vs. rw > snapshots)... and implications like not mounting snapshots noatime, and > write amplification effects Yes. > - a guide when one should make subvole (e.g. keeping things on the root > fs together, unless it's separate like /var/www is usually, but > /var/lib typically "corresponds" to a state of /etc and /usr. I see you added that (as mentioned in a followup). Looks good. > - what I've asked in another mail,.. about subvols and mountopts. > > That rework also contains your security idea... shamelessly not quoting > you there O:-) That looks good too... tho to be fair that too was simply reposting someone else's idea... which was both *entirely* new to me when I read it as I'd simply not considered that before, and equally immediately obvious in hindsight, which is why it stuck with me so well, because the combination of entirely new viewpoint on something and "Duh, he's absolutely right and I should have seen it myself" tends to /be/ pretty memorable! =:^) Anyway, glad it's mentioned on the wiki, now. =:^) --- [1] No claim of subvolume expert: And also because I've a bit of the sysadmin's healthy superstition of ever claiming to be an expert until it's so clearly the case it's becoming ridiculous to try to claim otherwise, since if there's the slightest doubt, Murphy will surely see/ hear and assert his law shortly thereafter, demonstrating the fallacy of the claim, likely with a brown-bag level error! =:^\ Of course, without anything but the toplevel subvolumes here, that might make the unwary sysadmin unwisely bold, but lest I be unpleasantly reminded of the fact that the top-level's a subvolume too, I'm still wary of tempting either Murphy or fate with such a claim of expertise! =:^\ -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: subvols and parents - how?
On Wed, 2015-12-09 at 10:53 +, Duncan wrote: > If you use the recipe (subvol create, cp with reflink) it suggests > there, > you'll end up with the reflinked copy in a subvol. > > You can then mount that subvol over top of the existing dir, and > *new* > file opens will access the new subvol, tho existing open files will > of > course continue to access the files/reflinks to which they have a > reference, "underneath" the new mount. Sure, which would mean however that a downtime is still necessary. > For some services it's possible to signal them to reload their > files. > Where this is possible, you can do the overmount trick and then > signal > them to reload, and they should keep running, otherwise undisturbed > (altho > any changes between the reflink and processing the signal will still > go > to the existing open files, I don't believe there's a way around > that). Yep,.. but as you say,... it doesn't really help to avoid the downtime,... rather lead to data corruption (not on the filesystem level, of course, but within the application). > But AFAIK that's the closest it gets, and nothing more along that > line is > planned. I've been so free to add that idea to the project ideas: https://btrfs.wiki.kernel.org/index.php/Project_ideas#.28Atomically.29_convert_directories_into_subvolumes_and_vice_versa any developer or more experienced user than me is of course free to remove that again, if it seems not so important or isn't possible to be implemented. > In general, keep in mind that subvolumes work in most respects very > much > like normal directories do, except where they don't. =:^) :-P :-P Cheers, Chris. smime.p7s Description: S/MIME cryptographic signature
Re: subvols and parents - how?
On Sat, 2015-12-12 at 03:32 +0100, Christoph Anton Mitterer wrote: > What's still missing now, IMHO, is: > - a guide when one should make subvole (e.g. keeping things on the > root > fs together, unless it's separate like /var/www is usually, but > /var/lib typically "corresponds" to a state of /etc and /usr. I just added that: https://btrfs.wiki.kernel.org/index.php/SysadminGuide#When_To_Make_Subvolumes Chris. smime.p7s Description: S/MIME cryptographic signature
Re: subvols and parents - how?
On 2015-12-09 22:56, Duncan wrote: > Austin S Hemmelgarn posted on Wed, 09 Dec 2015 14:04:06 -0500 as > excerpted: > >> Agreed. It's not too bad fixing a Gentoo system (as long as >> /var/lib/portage/world is still correct, you can just nuke the installed >> package database and emerge world, it'll take time, but it will get your >> system in a guaranteed consistent state). > > For sufficiently loose values of "consistent", yes, as I found out by > experience. But it can be done, and I do have the experience to prove it. > > What happens in practice is that while yes, as long as @world is correct > you can install to current and have all those files tracked again as > appropriate, if your package installation database is missing or out of > sync with what's actually on your filesystem(s), where the new version of > various packages will replace older files as they come across them during > the install process (subject to CONFIG_PROTECT of course, this part isn't > the problem), the problem is actually where the files of the actually > installed but untracked version differ from those of the version you're > installing. Oh, definately, it's a useable system short term, but not something you should be depending on. The other big difference though is that it's then trivial to bootstrap a clean install on the same system if you have the space for it (which is what I usually end up doing). smime.p7s Description: S/MIME Cryptographic Signature
Re: subvols and parents - how?
Christoph Anton Mitterer posted on Wed, 09 Dec 2015 05:36:37 +0100 as excerpted: > On Fri, 2015-11-27 at 01:02 +, Duncan wrote: > [snip snap] >> #2 The point I was trying to make, now, to mount it you'll mount not a >> native nested subvol, and not a directly available sibling >> 5/subvols/home, but you'll actually be reaching into an entirely >> different nesting structure to grab something down inside, mounting >> 5/subvols/root/home subvolume nesting down inside the direct >> 5/subvols/root sibling subvol. > > Okay so your main point was basically "keeping things administrable"... =:^) >> one of which was that everything that the package manager installs >> should be on the same partition with the installed-package database, so >> if it has to be restored from backup, at least if it's all old, at >> least it's all equally old, and the package database actually matches >> what's on the system because it's in the same backup! > I basically agree, though I'd allow few exceptions, like database-like > data that is stored in /var/ sometimes and that doesn't need to be > consistent with anything but iself... e.g. static web pages > (/var/www)... postgresl DB, or sks keyserver DB... and so on. Of course. But such data isn't usually managed by the package manager, so splitting off a partition or subvolume or whatever, is fair game. IOW, /var can and should still be on /, but /var/www and the like can be on some other subvolume or filesystem. > > btw: What's the proper way for merging / splitting into subvols. > E.g. consider I have: > 5 > | > +--root (subvol) > | > +-- var (no subvol) > > And say I would want to split of var/www into a subvol. > Well one obvious way would be with mv (and AFAIU that would keep my > ref-links with clones, if any) but that also means that anything that > accesses /var/www probably needs a downtime. > Is it planned to have a special function that basically says: > "make dir foo and anything below (except nested subvols) a subvol named > foo, immediately and atomically"? AFAIK, no. There's a similar, but not identical, question covered on the wiki, use- cases page (link very likely wrapped as I'm not bothering to jump thru the hoops to unwrap while keeping everything else formatted, ATM). https://btrfs.wiki.kernel.org/index.php/ UseCases#Can_I_take_a_snapshot_of_a_directory.3F If you use the recipe (subvol create, cp with reflink) it suggests there, you'll end up with the reflinked copy in a subvol. You can then mount that subvol over top of the existing dir, and *new* file opens will access the new subvol, tho existing open files will of course continue to access the files/reflinks to which they have a reference, "underneath" the new mount. For some services it's possible to signal them to reload their files. Where this is possible, you can do the overmount trick and then signal them to reload, and they should keep running, otherwise undisturbed (altho any changes between the reflink and processing the signal will still go to the existing open files, I don't believe there's a way around that). But AFAIK that's the closest it gets, and nothing more along that line is planned. In general, keep in mind that subvolumes work in most respects very much like normal directories do, except where they don't. =:^) If it can't be done with normal directories and it's not specifically part of subvol/ snapshot functionality, it very likely can't be done with subvols either. From my read, that's the general rule that applies here. > And similar vice-versa... a special function that says: > "make subvol foo and anything below (except nested subvols) a dir of the > parent subvol named foo, immediately and atomically"? > > Could be handy for real world administration, especially when one want's > to avoid downtimes. > > btw: Few days ago, either Hugo or your thought that mv'ing a subvol > would change it's UUID, but my try (which was with coreutils 8.3 -> no > reflinked mv) seemed to show it wouldn't but there was no further reply > then... so am I right that the UUID wouldn't change? That was Hugo. I read your reply then with interest, but my own use-case doesn't involve subvols, so I'd be unlikely to know that level of specific detail, and even more unlikely to post about it unless I was quite sure on it (as I am about the above due to the wiki reference and previous list discussion). >> The same idea applies here. Once you start reaching into nested >> subvols to get the deeper nested subvols you're trying to mount, it's >> too much and you're just begging to get it wrong under the extreme >> pressures of a disaster recovery. > Well apparently you oversaw the extremely simple and reliable solution: > leaving a tiny little note on your desk saying something like: "dear > boss, things are screwed up, I'm on vacation now..." ;-) =:^) -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and
Re: subvols and parents - how?
Christoph Anton Mitterer posted on Wed, 09 Dec 2015 05:38:02 +0100 as excerpted: > On Fri, 2015-11-27 at 02:02 +, Duncan wrote: >> Consider a setuid-root binary with a recently publicized but patched on >> your system vuln. But if you have root snapshots from before the patch >> and those snapshots are nested below root, then they're always >> accessible. If the path to the vulnerable setuid is as user accessible >> as it likely was in its original location, then anyone with login >> access to the system is likely to be able to run it from the >> snapshot... and will be able to get root due to the vuln. > > Hmm good point... I think it would be great if you could add that > scenario somewhere to the documentation. :-) FWIW, I (personally, not sure if that "you" was singular or plural) am much more comfortable on the lists than on wikis, which I tend to treat much as I did printed encyclopedias back in the day -- as great references, but read-only from my perspective. So while I know they can be edited in theory, I just don't tend to get around to it, and don't believe I've ever even gotten a wiki login, on /any/ wiki, let alone the btrfs wiki. But as you've likely noticed, I do find them great references to read and to post links to. =:^) So while I'm not opposed in principle to editing the wiki, in practice I'm never going to do it. I'll reply here with the info, and if it gets to the wiki, it's almost certainly because someone else posted it there, based on my post here, or not. > Based on that one can easily think about more/similar examples... device > file that had too permissive modes set, and where snapshotted like > that... and so on. Exactly... > I think that's another example why it would be nice if btrfs had > something (per subvolume) like ext4's default mount options (I mean the > ones stored in the superblock). > > Not only would it allow the userland tools to do things like "adding > notatime" [...] it would also allow to set things like nodev, noexec, > nosuid and that like on subvols... and again it would make the whole > thing practically usable with nested subvols. > > Where would be the appropriate place to record that as a feature > request? > Simply here on the list? While the appropriate place would be on the wiki, where there's a page for that (here, the request is likely to be lost to history, while on the wiki it's in a nice list that both devs and other requesters can look at, a year, five years, a decade... into the future), in this case... There's already a framework for it and a limited implementation, tho only a small subset of possible options are yet available. The UI exposure is as btrfs property (see the btrfs-property manpage), with the properties, in general, implemented as extented attributes. You can browse what's currently available, but as I said, it's pretty limited ATM, pretty much just enough to prove the concept and serve as a placeholder for more to be added, later. Of course other informational and sometimes configurable bits are exposed in the developing btrfs sysfs tree, which is where most of the work of that sort has been focused recently. But it too is somewhat immature and developing, tho at this point it seems to be further along than properties, partly because btrfs' own tools are using it at times. Basically, both properties and the sysfs interface demonstrate btrfs' status as stabilizING and maturING, as that sort of thing really can't be implemented until the basics have settled down far enough to form a stable reference point, but not yet fully stable or mature, as both interfaces are as yet not fully developed, with more to be added. But in both cases there's enough there to browse and get a reasonable idea how things are being exposed and how they'll work once they are. =:^) -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: subvols and parents - how?
On 2015-11-27 03:02, Duncan wrote: >>> Then there's the security angle to consider. With the (basically, >>> possibly modified as I suggested) flat layout, mounting something >>> doesn't automatically give people in-tree access to nested subvolumes >>> (subject to normal file permissions, of course), like nested layout >>> does. And with (possibly modified) flat layout, the whole subvolume >>> tree doesn't need to be mounted all the time either, only when you're >>> actually working with subvolumes. > >> Uhm, I don't get the big security advantage here... whether nested or >> manually mounted to a subdir,... if the permissions are insecure I'll >> have a problem... if they're secure, than not. > > Consider a setuid-root binary with a recently publicized but patched on > your system vuln. But if you have root snapshots from before the patch > and those snapshots are nested below root, then they're always > accessible. If the path to the vulnerable setuid is as user accessible > as it likely was in its original location, then anyone with login access > to the system is likely to be able to run it from the snapshot... and > will be able to get root due to the vuln. > > On a flat layout, a snapshot with the vuln would have to be mounted > before it could be accessed, as otherwise it'd be outside the mounted > tree. Talking about security: If you split the "var" parts from the root, you get a nice side-effect that the generation of the root subvolume does not change until you update your system (e.g. by installing/updating packages). Consider this setup: 5 | +--root (subvol) | | | +--var (subvol, or mounted) | +--tmp (subvol, or tmpfs) | +--run (subvol, or tmpfs) | +--home (subvol, or mounted) | +--snapshots (dir, uid=root,mode=700) | +--root.20150101 +--root.20150102 +--... Now if you take a look at the output of "btrfs sub list -c /", you can see exactly when your root subvolume has changed. This means if you always create a new snapshot of "root" after you update your system, you are able to spot changes to the system easily: # btrfs sub list -c / ID 100 gen 4242 cgen 50 top level 5 path root ID 101 gen 4500 cgen 53 top level 5 path var [...] ID 5000 gen 4111 cgen 4111 top level 5 path snapshots/root.20151203 ID 5001 gen 4242 cgen 4242 top level 5 path snapshots/root.20151205 [...] In this example, I can see that "root" has not been altered since 20151205, as "cgen" of the snapshot matches the "gen" of the root subvolume. This is the last time I updated this system. If the "gen" of the root subvolume would be greater than 4242, I know some changes were made to the system root, and I can start tracking it down using "btrfs subvol find-new". -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: subvols and parents - how?
On 2015-12-09 05:53, Duncan wrote: > Christoph Anton Mitterer posted on Wed, 09 Dec 2015 05:36:37 +0100 as > excerpted: > >> On Fri, 2015-11-27 at 01:02 +, Duncan wrote: >> [snip snap] >>> #2 The point I was trying to make, now, to mount it you'll mount not a >>> native nested subvol, and not a directly available sibling >>> 5/subvols/home, but you'll actually be reaching into an entirely >>> different nesting structure to grab something down inside, mounting >>> 5/subvols/root/home subvolume nesting down inside the direct >>> 5/subvols/root sibling subvol. >> >> Okay so your main point was basically "keeping things administrable"... > > =:^) > >>> one of which was that everything that the package manager installs >>> should be on the same partition with the installed-package database, so >>> if it has to be restored from backup, at least if it's all old, at >>> least it's all equally old, and the package database actually matches >>> what's on the system because it's in the same backup! > >> I basically agree, though I'd allow few exceptions, like database-like >> data that is stored in /var/ sometimes and that doesn't need to be >> consistent with anything but iself... e.g. static web pages >> (/var/www)... postgresl DB, or sks keyserver DB... and so on. > > Of course. But such data isn't usually managed by the package manager, > so splitting off a partition or subvolume or whatever, is fair game. > > IOW, /var can and should still be on /, but /var/www and the like can be > on some other subvolume or filesystem. > Agreed. It's not too bad fixing a Gentoo system (as long as /var/lib/portage/world is still correct, you can just nuke the installed package database and emerge world, it'll take time, but it will get your system in a guaranteed consistent state). On something using dpkg or rpm though, it's got the potential to be almost impossible to recover from without a significant amount of low-level knowledge of the package manager's installation database. smime.p7s Description: S/MIME Cryptographic Signature
Re: subvols and parents - how?
On Fri, 2015-11-27 at 02:02 +, Duncan wrote: > Uhm, I don't get the big security advantage here... whether nested > > or > > manually mounted to a subdir,... if the permissions are insecure > > I'll > > have a problem... if they're secure, than not. > Consider a setuid-root binary with a recently publicized but patched > on > your system vuln. But if you have root snapshots from before the > patch > and those snapshots are nested below root, then they're always > accessible. If the path to the vulnerable setuid is as user > accessible > as it likely was in its original location, then anyone with login > access > to the system is likely to be able to run it from the snapshot... and > will be able to get root due to the vuln. Hmm good point... I think it would be great if you could add that scenario somewhere to the documentation. :-) Based on that one can easily think about more/similar examples... device file that had too permissive modes set, and where snapshotted like that... and so on. I think that's another example why it would be nice if btrfs had something (per subvolume) like ext4's default mount options (I mean the ones stored in the superblock). Not only would it allow the userland tools to do things like "adding notatime" per default on snapshots (at least ro snapshot), so that one can have them nested and still doesn't suffer from the previously discussed writes-on-read-amplifications... it would also allow to set things like nodev, noexec, nosuid and that like on subvols... and again it would make the whole thing practically usable with nested subvols. Where would be the appropriate place to record that as a feature request? Simply here on the list? Cheers, Chris. smime.p7s Description: S/MIME cryptographic signature
Re: subvols and parents - how?
On Fri, 2015-11-27 at 01:02 +, Duncan wrote: [snip snap] > #1 could be a pain to setup if you weren't actually mounting it > previously, just relying on the nested tree, AND... > > #2 The point I was trying to make, now, to mount it you'll mount not > a > native nested subvol, and not a directly available sibling > 5/subvols/home, but you'll actually be reaching into an entirely > different nesting structure to grab something down inside, mounting > 5/subvols/root/home subvolume nesting down inside the direct > 5/subvols/root sibling subvol. Okay so your main point was basically "keeping things administrable"... > one of which was that everything > that the package manager installs should be on the same partition > with > the installed-package database, so if it has to be restored from > backup, > at least if it's all old, at least it's all equally old, and the > package > database actually matches what's on the system because it's in the > same > backup! I basically agree, though I'd allow few exceptions, like database-like data that is stored in /var/ sometimes and that doesn't need to be consistent with anything but iself... e.g. static web pages (/var/www)... postgresl DB, or sks keyserver DB... and so on. btw: What's the proper way for merging / splitting into subvols. E.g. consider I have: 5 | +--root (subvol) | +-- var (no subvol) And say I would want to split of var/www into a subvol. Well one obvious way would be with mv (and AFAIU that would keep my ref-links with clones, if any) but that also means that anything that accesses /var/www probably needs a downtime. Is it planned to have a special function that basically says: "make dir foo and anything below (except nested subvols) a subvol named foo, immediately and atomically"? And similar vice-versa... a special function that says: "make subvol foo and anything below (except nested subvols) a dir of the parent subvol named foo, immediately and atomically"? Could be handy for real world administration, especially when one want's to avoid downtimes. btw: Few days ago, either Hugo or your thought that mv'ing a subvol would change it's UUID, but my try (which was with coreutils 8.3 -> no reflinked mv) seemed to show it wouldn't but there was no further reply then... so am I right that the UUID wouldn't change? > The same idea applies here. Once you start reaching into nested > subvols > to get the deeper nested subvols you're trying to mount, it's too > much > and you're just begging to get it wrong under the extreme pressures > of a > disaster recovery. Well apparently you oversaw the extremely simple and reliable solution: leaving a tiny little note on your desk saying something like: "dear boss, things are screwed up, I'm on vacation now..." ;-) Thanks, Chris. smime.p7s Description: S/MIME cryptographic signature
Re: subvols and parents - how?
Christoph Anton Mitterer posted on Tue, 24 Nov 2015 22:25:50 +0100 as excerpted: >> Suppose you only want to rollback /, because some update screwed you >> up, >> but not /home, which is fine. If /home is a nested subvolume, then >> you're now mounting the nested home subvolume from some other nesting >> tree entirely, > That's a bit unclear to me,... I thought when I make a snapshot, any > nested subvols wouldn't be snapshotted and thus be empty dirs. > So I'd have rather that if I would simply have no /home (if I didn't > move it to the rolled-back subvol manually) What I was intending to convey but apparently failed to be quite clear enough, suppose: 5 | +-+ subvols (dir) | +-+ root (subvol) | | | + home (nested subvol) | +-+ snaps-2015.0901 (dir) | +-+ root-2015.0901 (subvol) As long as you're on the working /, then /home is a nested subvol, and you don't have to mount it to access, tho you can if you want. But now, you roll back to snaps-2015.0901/root-2015.0901. It won't have /home nested underneath, as you correctly pointed out, but in ordered to access it, you now MUST mount /home, which... #1 could be a pain to setup if you weren't actually mounting it previously, just relying on the nested tree, AND... #2 The point I was trying to make, now, to mount it you'll mount not a native nested subvol, and not a directly available sibling 5/subvols/home, but you'll actually be reaching into an entirely different nesting structure to grab something down inside, mounting 5/subvols/root/home subvolume nesting down inside the direct 5/subvols/root sibling subvol. With just one level of nesting and one additional mount, it's not too hard to keep track of, but if you're dealing with four or five levels of subvol nesting and some of them you're mounting the working head copy while others you're rolling back, it could get difficult to keep straight in your head what's going on. Consider a layout like this: 5 +-+ subvols (dir) | +-+ root (subvol) | | | +-+ home (subvol) | | | | | +-+ henry (dir, no subvol for henry) | | | | | +-+ fred (subvol) | | | | | | | +-+ vms (subvol) | | | | | +-+ betty (subvol) | | | +-+ svr (subvol) | | | +-+ vms (subvol) | +-+ snaps-2015.0901 (dir) | +-+ root-2015.0901 (subvol here and below) | +-+ home-2015.0901 | +-+ fred-2015.0901 | +-+ fred-vms-2015.0901 | +-+ betty-2015.0901 | +-+ svr-2015.0901 | +-+ svr-vms-2015.0901 Now, you were hacked and they encrypted a bunch of stuff, but you were lucky and caught them before they got everything. You need to roll back root but not home, fred is fine, but his vms subvol needs rolled back, betty needs rolled back, svr needs rolled back, but svr's vms are fine. Try to sort THAT out along with the nesting, and keep it all straight while under the severe pressure of trying to recover from a hack in time for those svr things to go live for Black Friday in a few hours, where in a single day you expect to make as much as you normally do in a month, the rest of the year! The pressure is on! Oh, and you weren't actually doing the mounts as you were depending on the nested tree, so you have to actually setup the mounts as well, not just switch them to point to the appropriate location. OK, so that's a bit contrived, but server encryption for ransom hacks are in the news, black Friday starts in a few hours here, and I think the point should be obvious! =:^) (Some years ago, before btrfs, I had something similar setup but with partitions. Disaster struck and I ended up with / from one backup, /usr from another, and /var, with the package database of what was installed on the other two, from current, or something like that. Needless to say I learned quite some lessons from that, one of which was that everything that the package manager installs should be on the same partition with the installed-package database, so if it has to be restored from backup, at least if it's all old, at least it's all equally old, and the package database actually matches what's on the system because it's in the same backup! That partition and btrfs, along with each of its various backups, are now 8 GiB each, so it's not like I'll run out of room with several levels of backup. I went mdraid after that, but after an initial experiment with lvm on top of the raid, I decided that was too complex to deal with in the pressure of a disaster and redid it to multiple raids on parallel partitioned hardware. In a disaster the raid would be bad enough to deal with but tolerable, but I did NOT need the complexity of lvm on top of raid, and after dealing with the parts of three different installs mess, I had the hard-earned wisdom to realize it! The same idea applies here. Once you start reaching into nested subvols to get the deeper nested subvols you're trying to mount, it's too much and you're just begging
Re: subvols and parents - how?
Christoph Anton Mitterer posted on Tue, 24 Nov 2015 22:25:50 +0100 as excerpted: >> Then there's the security angle to consider. With the (basically, >> possibly modified as I suggested) flat layout, mounting something >> doesn't automatically give people in-tree access to nested subvolumes >> (subject to normal file permissions, of course), like nested layout >> does. And with (possibly modified) flat layout, the whole subvolume >> tree doesn't need to be mounted all the time either, only when you're >> actually working with subvolumes. > Uhm, I don't get the big security advantage here... whether nested or > manually mounted to a subdir,... if the permissions are insecure I'll > have a problem... if they're secure, than not. Consider a setuid-root binary with a recently publicized but patched on your system vuln. But if you have root snapshots from before the patch and those snapshots are nested below root, then they're always accessible. If the path to the vulnerable setuid is as user accessible as it likely was in its original location, then anyone with login access to the system is likely to be able to run it from the snapshot... and will be able to get root due to the vuln. On a flat layout, a snapshot with the vuln would have to be mounted before it could be accessed, as otherwise it'd be outside the mounted tree. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: subvols and parents - how?
On Tue, 2015-11-24 at 21:55 +, Hugo Mills wrote: > In practice, new content is checked by a number of people when > it's > put in, so the case of someone putting random poorly-thought-out crap > in the wiki isn't particularly likely to happen. Well... it may work in 99% cases... but there could something slip through, which isn't as easy the case in manpages, which also tend to be less messy than the huge pile of wiki pages where similar/related things are described on different pages. Imagine a case, a non-experienced user update the wiki saying that -- repair should be used, he may not even doing it in bad faith, perhaps he had success with it and now writes a recipe. It may take a while until someone of the more experienced guys notices that and corrects it. But if ", in the meantime had some fs corruptions,... I may experience already severe problems by following that suggestion... (and while I do have many backups of all my data, others may not, and if their life's data is concerned, they'd be screwed). So even if it takes you just a few hours to correct such rubbish, you know that Murphy's law may still hit n people during that time ;-) > Please feel free to add the things you'd like to see. As I said > above, we do check the docs on the wiki as they're changed, so if > you're wrong on some details, it won't be a major issue for very > long. If you want to discuss details before you write something, > start > a conversation -- either on here, or on IRC (or even on the Talk > pages > of the wiki). Well I can write a list together of things which I think should be part of some more general documentation (i.e. less documentation about options of the tools)... questions a complete newcomer to btrfs may have who needs however more than "just a filesystem". > Note that the "parent" of send -p and of snapshots is not the same > relationship as the "parent" (containing subvol) of the tree > structure. This is an awkward nomenclature problem, and I'm not sure > how to fix it. Yeah, that was clear... :-) Maybe call the "parent" from send -p "base" or something like that... IMHO that would fit more as the parent there is more like a "fundament". Anyway, it's still not as bad as the usage of "RAID1" ;-) > because > you can't rename a subvol across another subvol boundary. That's not quite clear to me... I had subvols like that: /top/root/below-root /top/below-top and was able to move that to: /top/root/below-top /top/below-root i.e. not just changing names but swapping as in: mv /top/root/below-top /top/tmp mv /top/below-root /top/root/below-root mv /top/tmp /top/below-top with top, root, below-top and below-root all being the same subvols Thanks a lot for your explanations :) Chris. smime.p7s Description: S/MIME cryptographic signature
Re: subvols and parents - how?
On Tue, 2015-11-24 at 23:30 +, Hugo Mills wrote: > Yes, that makes sense. Feel free to shamelessly use my idea (as well as the one to call btrfs' RAID1 replica2 or something else) :-O > With a recent mv root@heisenberg:/mnt# mv --version mv (GNU coreutils) 8.23 => not recent enough... > but I think you'll find that the UUID of the subvols changes. (At > least, I hope it does. If it doesn't, then my mental model of what > the FS is doing is *really* screwed up). Well... see below: root@heisenberg:~# truncate -s 2G image root@heisenberg:~# losetup -f image root@heisenberg:~# mkfs.btrfs /dev/loop0 btrfs-progs v4.3 See http://btrfs.wiki.kernel.org for more information. Performing full device TRIM (2.00GiB) ... Label: (null) UUID: 10e1a55c-448a-4f37-ae5c-6a7801a7f202 Node size: 16384 Sector size:4096 Filesystem size:2.00GiB Block group profiles: Data: single8.00MiB Metadata: DUP 110.38MiB System: DUP 12.00MiB SSD detected: no Incompat features: extref, skinny-metadata Number of devices: 1 Devices: IDSIZE PATH 1 2.00GiB /dev/loop0 root@heisenberg:~# mount /dev/loop0 /mnt/ root@heisenberg:/mnt# btrfs subvolume create root Create subvolume './root' root@heisenberg:/mnt# btrfs subvolume create below-top Create subvolume './below-top' root@heisenberg:/mnt# cd root/ root@heisenberg:/mnt/root# btrfs subvolume create below-root Create subvolume './below-root' root@heisenberg:/mnt# btrfs subvolume list /mnt/ -pacguqt ID gen cgenparent top level parent_uuid uuidpath -- --- -- - --- 257 9 7 5 5 - 8fbf521e-77f9-0d49-9891-87767f98c655root 258 8 8 5 5 - b49131e9-4207-aa42-8195-c50de5f06136below-top 259 9 9 257 257 - 20c042be-ead8-204a-a684-94c1a770e739/root/below-root root@heisenberg:/mnt# mv root/below-root/ tmp root@heisenberg:/mnt# mv below-top/ root/ root@heisenberg:/mnt# mv tmp/ below-root root@heisenberg:/mnt# btrfs subvolume list /mnt/ -pacguqt ID gen cgenparent top level parent_uuid uuidpath -- --- -- - --- 257 9 7 5 5 - 8fbf521e-77f9-0d49-9891-87767f98c655root 258 8 8 257 257 - b49131e9-4207-aa42-8195-c50de5f06136/root/below-top 259 9 9 5 5 - 20c042be-ead8-204a-a684-94c1a770e739below-root root@heisenberg:/mnt# So the UUIDs seem to stay the same (or are these other UUIDs?) Hope I haven't ruined your day now ;-) Cheers, Chris. smime.p7s Description: S/MIME cryptographic signature
Re: subvols and parents - how?
On Wed, Nov 25, 2015 at 12:20:09AM +0100, Christoph Anton Mitterer wrote: > On Tue, 2015-11-24 at 21:55 +, Hugo Mills wrote: > > In practice, new content is checked by a number of people when > > it's > > put in, so the case of someone putting random poorly-thought-out crap > > in the wiki isn't particularly likely to happen. > Well... it may work in 99% cases... but there could something slip > through, which isn't as easy the case in manpages, which also tend to > be less messy than the huge pile of wiki pages where similar/related > things are described on different pages. > > Imagine a case, a non-experienced user update the wiki saying that -- > repair should be used, he may not even doing it in bad faith, perhaps > he had success with it and now writes a recipe. > It may take a while until someone of the more experienced guys notices > that and corrects it. You can get update notifications, with the diffs for each page, via RSS. (At least, I do, and I think David and a few others monitor it in the same way). The window of failure is fairly small, particularly in view of the number of such "dangerous" changes made. The total vulnerability is measured in hours per year... > But if ", in the meantime had some fs corruptions,... I may experience > already severe problems by following that suggestion... (and while I do > have many backups of all my data, others may not, and if their life's > data is concerned, they'd be screwed). > > So even if it takes you just a few hours to correct such rubbish, you > know that Murphy's law may still hit n people during that time ;-) > > > > Please feel free to add the things you'd like to see. As I said > > above, we do check the docs on the wiki as they're changed, so if > > you're wrong on some details, it won't be a major issue for very > > long. If you want to discuss details before you write something, > > start > > a conversation -- either on here, or on IRC (or even on the Talk > > pages > > of the wiki). > Well I can write a list together of things which I think should be part > of some more general documentation (i.e. less documentation about > options of the tools)... questions a complete newcomer to btrfs may > have who needs however more than "just a filesystem". > > > > Note that the "parent" of send -p and of snapshots is not the same > > relationship as the "parent" (containing subvol) of the tree > > structure. This is an awkward nomenclature problem, and I'm not sure > > how to fix it. > Yeah, that was clear... :-) > Maybe call the "parent" from send -p "base" or something like that... > IMHO that would fit more as the parent there is more like a > "fundament". Yes, that makes sense. > Anyway, it's still not as bad as the usage of "RAID1" ;-) > > > > because > > you can't rename a subvol across another subvol boundary. > That's not quite clear to me... I had subvols like that: > /top/root/below-root > /top/below-top > and was able to move that to: > /top/root/below-top > /top/below-root > > i.e. not just changing names but swapping as in: > mv /top/root/below-top /top/tmp > mv /top/below-root /top/root/below-root > mv /top/tmp /top/below-top > > with top, root, below-top and below-root all being the same subvols With a recent mv, it'll be doing doing reflink copies followed by delete for all of the contents, which makes it pretty efficient, but I think you'll find that the UUID of the subvols changes. (At least, I hope it does. If it doesn't, then my mental model of what the FS is doing is *really* screwed up). Hugo. > Thanks a lot for your explanations :) > > Chris. -- Hugo Mills | Never underestimate the bandwidth of a Volvo filled hugo@... carfax.org.uk | with backup tapes. http://carfax.org.uk/ | PGP: E2AB1DE4 | signature.asc Description: Digital signature
Re: subvols and parents - how?
On Tue, 2015-11-24 at 08:29 +, Duncan wrote: > OK, found it on the wiki. It wasn't under use-cases, where I > initially > thought to look, but under sysadmin guide. Specifically, see section > 4.2, managing snapshots, but I'd suggest reading the entire > subvolumes > discussion, section 4, or even most/all of the page. > > https://btrfs.wiki.kernel.org/index.php/SysadminGuide Well I've had read that, but it's pretty vague and especially doesn't mentioned any of the filesystem internal implications (if there are any)... like I wondered before, whether this has effects on ref-links not being used when e.g. sending/recieving ... or on future planned extensions like recursive snapshots. > > Suppose you only want to rollback /, because some update screwed you > up, > but not /home, which is fine. If /home is a nested subvolume, then > you're now mounting the nested home subvolume from some other nesting > tree entirely, That's a bit unclear to me,... I thought when I make a snapshot, any nested subvols wouldn't be snapshotted and thus be empty dirs. So I'd have rather that if I would simply have no /home (if I didn't move it to the rolled-back subvol manually) > 5 > > > +-roots (dir not subvol, note the s, rootS, plural) > > +-root (subvol, mountpoint /) > > > +-boot/ (dir) > > > +-root/ (dir) > > > +-lib/ (dir) > > > +-home/ (empty dir as mountpoint) > > +-root-snapshot-2015.0301 (dated snapshot of root) > > +-root-snapshot-2015.0601 (dated snapshot of root) > > +-root-snapshot-2015.0901 (dated snapshot of root) > +-homes (dir not subvol) > > +-home (subvol, mountpoint /home) > > +-home-snapshot-2015.0301 (dated snapshot of home) > ... That's more what I've had in mind... Actually something like this: 5 | +-root (=subvol) | +-boot | +-home (subvo=/home being mounted heron) | +-lib +-home (subvol, the current version) +-snapshots (=dir) +-root | +-2015-01-14 (subvol, snapshot) | +-2015-09-30 (subvol, snapshot) +-home +-2015-06-04 (subvol, snapshot) +-2015-09-01 (subvol, snapshot) And it once more points to the problem of the wiki... anyone can write (I think even I) and it's totally unclear at the first glance how "[non-]official" and outdated something may be. Apart from the problem that many important questions (from the PoV of an more advanced admin that doesn't just do mkfs.btrfs and then never again thinks about it) :-( > Meanwhile, if the intention for a subvolume is simply to exclude that > subtree from snapshotting of the parent, as might be the case for > example that is in fact also use case.. so in practise I'll probably have a mix of (a) and (b). > if you have a VMs subvol, with the VM image files set NOCOW to avoid > fragmentation, since snapshotting nocow files forces cow1 (a cow at > the > first write of that block, before returning to nocow, because a > snapshot > locks the existing extents in place for the snapshot, so initial > writes > to a block after a snapshot /can't be nocow or it'd change the > snapshot > too...) Ah that's good to know how that works (or better said, that it works at all)... I've already wondered myself several times what happens when I snapshot NOCOW files, ... something that's I guess also missing from the (missing ;-) ) btrfs-end-user overview and details documentation. > OTOH, if there's a chance you might want to mount that subvolume in a > roll-back scenario, under the snapshot you're rolling back to, then > it > makes sense to put it directly under ID 5 again, and mount it in any > case. Yes. > Then there's the security angle to consider. With the (basically, > possibly modified as I suggested) flat layout, mounting something > doesn't > automatically give people in-tree access to nested subvolumes > (subject to > normal file permissions, of course), like nested layout does. And > with > (possibly modified) flat layout, the whole subvolume tree doesn't > need to > be mounted all the time either, only when you're actually working > with > subvolumes. Uhm, I don't get the big security advantage here... whether nested or manually mounted to a subdir,... if the permissions are insecure I'll have a problem... if they're secure, than not. Of course I use insecure permissions and don't mount the subvols then I'd be safe in flat setup, but not in a nested setup...(where they subvol is "auto-mounted")... But that seems pretty awkward. Mhh I think my main question turns out to be whether the different layouts would have any technical (i.e. not administrative) effects... like the aforementioned stuff of recursive snapshots (should such thing ever come to life). But at least from the userspace tools it seems that I can move subvols arbitrarily and they adapt their parent IDs accordingly... So I guess whatever setup one uses nested/flat/mixed... doesn't rule out any functionalities for the future?! thx, Chris. smime.p7s Description: S/MIME cryptographic signature
Re: subvols and parents - how?
On Tue, Nov 24, 2015 at 10:25:50PM +0100, Christoph Anton Mitterer wrote: > On Tue, 2015-11-24 at 08:29 +, Duncan wrote: > > OK, found it on the wiki. It wasn't under use-cases, where I > > initially > > thought to look, but under sysadmin guide. Specifically, see section > > 4.2, managing snapshots, but I'd suggest reading the entire > > subvolumes > > discussion, section 4, or even most/all of the page. > > > > https://btrfs.wiki.kernel.org/index.php/SysadminGuide > Well I've had read that, but it's pretty vague and especially doesn't > mentioned any of the filesystem internal implications (if there are > any)... like I wondered before, whether this has effects on ref-links > not being used when e.g. sending/recieving ... or on future planned > extensions like recursive snapshots. No, there's no particular implications one way or the other in terms of reflinks. Obviously, if recursive snapshots get implemented, it'll remove some of the current awkwardness with nested subvols, but it won't invalidate any existing setups that use the recommendations in the Sysadmin Guide. > > Suppose you only want to rollback /, because some update screwed you > > up, > > but not /home, which is fine. If /home is a nested subvolume, then > > you're now mounting the nested home subvolume from some other nesting > > tree entirely, > That's a bit unclear to me,... I thought when I make a snapshot, any > nested subvols wouldn't be snapshotted and thus be empty dirs. Correct. This is actually a use-case for nested subvols, and one which snapper uses -- the target for snapshots of /foo is /foo/.snapshots/$date, where /foo/.snapshots is a subvol in its own right. So, if you have a subdir which you won't want to include in snapshots of a subvol, make it a subvol itself. > So I'd have rather that if I would simply have no /home (if I didn't > move it to the rolled-back subvol manually) Yup, this is one of the main reasons for not nesting subvols. > > 5 > > > > > +-roots (dir not subvol, note the s, rootS, plural) > > > +-root (subvol, mountpoint /) > > > > +-boot/ (dir) > > > > +-root/ (dir) > > > > +-lib/ (dir) > > > > +-home/ (empty dir as mountpoint) > > > +-root-snapshot-2015.0301 (dated snapshot of root) > > > +-root-snapshot-2015.0601 (dated snapshot of root) > > > +-root-snapshot-2015.0901 (dated snapshot of root) > > +-homes (dir not subvol) > > > +-home (subvol, mountpoint /home) > > > +-home-snapshot-2015.0301 (dated snapshot of home) > > ... > That's more what I've had in mind... > Actually something like this: > 5 > | > +-root (=subvol) > | +-boot > | +-home (subvo=/home being mounted heron) > | +-lib > +-home (subvol, the current version) > +-snapshots (=dir) > +-root > | +-2015-01-14 (subvol, snapshot) > | +-2015-09-30 (subvol, snapshot) > +-home > +-2015-06-04 (subvol, snapshot) > +-2015-09-01 (subvol, snapshot) > > > And it once more points to the problem of the wiki... anyone can write > (I think even I) and it's totally unclear at the first glance how > "[non-]official" and outdated something may be. > Apart from the problem that many important questions (from the PoV of > an more advanced admin that doesn't just do mkfs.btrfs and then never > again thinks about it) :-( In practice, new content is checked by a number of people when it's put in, so the case of someone putting random poorly-thought-out crap in the wiki isn't particularly likely to happen. > > Meanwhile, if the intention for a subvolume is simply to exclude that > > subtree from snapshotting of the parent, as might be the case for > > example > that is in fact also use case.. so in practise I'll probably have a mix > of (a) and (b). > > > > if you have a VMs subvol, with the VM image files set NOCOW to avoid > > fragmentation, since snapshotting nocow files forces cow1 (a cow at > > the > > first write of that block, before returning to nocow, because a > > snapshot > > locks the existing extents in place for the snapshot, so initial > > writes > > to a block after a snapshot /can't be nocow or it'd change the > > snapshot > > too...) > Ah that's good to know how that works (or better said, that it works at > all)... I've already wondered myself several times what happens when I > snapshot NOCOW files, ... something that's I guess also missing from > the (missing ;-) ) btrfs-end-user overview and details documentation. Please feel free to add the things you'd like to see. As I said above, we do check the docs on the wiki as they're changed, so if you're wrong on some details, it won't be a major issue for very long. If you want to discuss details before you write something, start a conversation -- either on here, or on IRC (or even on the Talk pages of the wiki). > > OTOH, if there's a chance you might want to mount that subvolume in a > > roll-back scenario, under the snapshot you're rolling back to, then > > it > > makes sense to put it directly under ID 5 again,
Re: subvols and parents - how?
Christoph Anton Mitterer posted on Tue, 24 Nov 2015 05:56:00 +0100 as excerpted: > When I use subvolumes than these have always a parent subvolume (except > ID5), so I can basically decide between two ways: > > a) make child subvolumes, e.g. > 5 > | > +-root (=subvol, mountpoint /) > +-boot/ > +-root/ > +-lib/ > +-home/ (=subvolume) > and soon on... perhaps the whole thing without the dedicated root- > subovlume (although that's probably not so smart, I guess). > > b) place at least some of the subvolumes directly below the top-level > and mount them e.g. via /etc/fstab, e.g. > 5 > | > +-root (=subvol, mountpoint /) > | +-boot/ > | +-root/ > | +-lib/ > +-home/ (=subvolume, mountpoint /home) > > > Now I wondered whether this has any technical implications, but neither > the wiki, nor the manpages seem to explain a lot here. Very astute question! =:^) Somewhere on the wiki I believe there's a recommendation to use (b) layout, but to some extent it depends on why you're actually doing subvolumes. OK, found it on the wiki. It wasn't under use-cases, where I initially thought to look, but under sysadmin guide. Specifically, see section 4.2, managing snapshots, but I'd suggest reading the entire subvolumes discussion, section 4, or even most/all of the page. https://btrfs.wiki.kernel.org/index.php/SysadminGuide (More below.) > The "differences", AFAIU, are the follows: > - When I mount a given subvolume,.. it's childs are automatically > "there". > Whereas when I don't have them as childs (as in (b)) I must of course > mount them somehow manually. > - Analogously for umounting. > - I can move existing subvols to higher/lower levels, and the parent > IDs will change accordingly. > > So basically it makes no difference, right? Or is there anything more > technical going on? E.g. with the ref-links or so? > Right now, there are, AFAIK, neither recursive snapshots (and especially > not atomic ones) nor recursive send/receive, right? > If that should ever be implemented, would I perhaps have problems with > (a) or (b)? If you're doing subvolumes for snapshotting and potential rollback purposes, layout (b) can be preferable as it allows a more direct mix and match rollback. Suppose you only want to rollback /, because some update screwed you up, but not /home, which is fine. If /home is a nested subvolume, then you're now mounting the nested home subvolume from some other nesting tree entirely, whereas if they're all under top-level, you simply mount the /home subvolume under whatever snapshot of / you are currently booting. Of course the reverse applies as well, if / is fine but you want to rollback /home. Again, with nesting you're reaching into some other nesting to mount what you want, and it can get a bit unintuitive and difficult to track, particularly if you go more than the two levels deep, but if all the snapshots are direct children of the top level ID 5, it's a lot easier. Tho I'd actually suggest a variant of the flat layout they suggest in the sysadmin's guide. What I'd do is something like this (using your tree drawing style): 5 | +-roots (dir not subvol, note the s, rootS, plural) | +-root (subvol, mountpoint /) | | +-boot/ (dir) | | +-root/ (dir) | | +-lib/ (dir) | | +-home/ (empty dir as mountpoint) | +-root-snapshot-2015.0301 (dated snapshot of root) | +-root-snapshot-2015.0601 (dated snapshot of root) | +-root-snapshot-2015.0901 (dated snapshot of root) +-homes (dir not subvol) | +-home (subvol, mountpoint /home) | +-home-snapshot-2015.0301 (dated snapshot of home) ... Of course, you might also organize by date instead of subvol... 5 | +- heads (dir, headS plural) | +-root (subvol) | +-home (subvol) | +-whatever (subvol) +-snapshots-2015.0301 (dir, snapshotS plural) | +-root-2015.0301 (snapshot of heads/root) | +-home-2015.0301 (snapshot of heads/home) | ... +-snapshots-2015.0601 (dir) | +-root-2015.0601 (snapshot) | ... +-snapshots-2015.0901 (dir) | +-root-2015.0901 (snapshot) ... Either of these would make finding a desired snapshot to rollback to much easier than a pure flat subvols/snapshots layout, with the preferred one depending on whether you want subvols/snapshots grouped by date or by snapshotted mountpoint. The dates organization would make cleaning up old snapshots by date, and visually checking that the snapshot cleanup script (if automated) is working as intended, somewhat easier, however. Meanwhile, if the intention for a subvolume is simply to exclude that subtree from snapshotting of the parent, as might be the case for example if you have a VMs subvol, with the VM image files set NOCOW to avoid fragmentation, since snapshotting nocow files forces cow1 (a cow at the first write of that block, before returning to nocow, because a snapshot locks the existing extents in place for the snapshot, so initial writes to a block after a snapshot /can't be nocow or it'd change the snapshot too...), and it's
