Re: Adding an ioctl to check for disklabel existence
On Thu 03 Oct 2019 at 10:17:47 -0700, John Nemeth wrote: > On Oct 3, 2:42pm, Rhialto wrote: > } I was thinking the other day that it might be useful if gpt had a > } subcommand to spit out a script to duplicate the partitioning of a disk, > } but without the "unique" parts. The script would of course be > } hand-editable for any changes one might want to make. > > By "unique" parts, do you mean just the UUIDs, or do you mean > other parts as well? What would the output look like? Just the UUIDs, those are the things I'm aware of. The sort of script I was thinking of would be something like this: gpt destroy sd1 gpt create sd1 gpt add -a 2m -l efi -s 128m -t efi sd1 gpt add -a 2m -l root -s 768m -t ffs sd1 gpt add -a 2m -l var -s3g -t ffs sd1 gpt add -a 2m -l swap -s 8200m -t swap sd1 gpt add -a 2m -l usr -s 20g -t ffs sd1 gpt add -a 2m -l tmp -s 1024m -t ffs sd1 gpt add -a 2m -l home -s 128g -t ffs sd1 It would leave out all/most things that are defaulted. (probably when automatically generated it would use the -a alignment only on the first partition, or if there is unused space between them) (maybe it would even use some shell variables for things like the disk device) > The bit about not modifying it is more of a caution. You can > modify it if you're careful (checksums are recomputed during > restore). Obviously if you make an error, it may not be usable > for restore, so you should work with a copy. Yes, when I wanted to clone the partitioning of my disk, I was indeed thinking of modifying the output from 'gpt backup', but it seemed a bit easier to create a little script such as the above, in case I want to make the same partitions again later. (In fact the version that I really used had some "are you sure" type prompts in it) -Olaf. -- Olaf 'Rhialto' Seibert -- rhialto at falu dot nl ___ Anyone who is capable of getting themselves made President should on \X/ no account be allowed to do the job. --Douglas Adams, "THGTTG" signature.asc Description: PGP signature
Re: Adding an ioctl to check for disklabel existence
On Oct 3, 2:42pm, Rhialto wrote: } On Wed 02 Oct 2019 at 19:40:01 -0700, John Nemeth wrote: } > Cloning disks always presents issues. However, gpt(8) has } > grown a "uuid" command to generate new UUIDs. This was primarily } > done to help with the cloning problem. Cloning a disk and then } > putting two disks with the same UUIDs on the same system is an } > operator error. } } I was thinking the other day that it might be useful if gpt had a } subcommand to spit out a script to duplicate the partitioning of a disk, } but without the "unique" parts. The script would of course be } hand-editable for any changes one might want to make. By "unique" parts, do you mean just the UUIDs, or do you mean other parts as well? What would the output look like? } Such functionality would be the equivalent of using disklabel to get the } editable text-version of a disklabel from one disk, and applying it to a } different disk. } } I use this sort of thing if I want to create a backup disk which should } have the same layout as the original. } } There is "gpt backup", but the manual page mentions "It should not be } modified" and it contains the guids of the partition which in many cases } should NOT be duplicated. It also isn't very readable. The bit about not modifying it is more of a caution. You can modify it if you're careful (checksums are recomputed during restore). Obviously if you make an error, it may not be usable for restore, so you should work with a copy. }-- End of excerpt from Rhialto
Re: Adding an ioctl to check for disklabel existence
On Thu, Oct 03, 2019 at 09:51:40AM -0600, Warner Losh wrote: > NetBSD is, of course, free to do what it likes. My semi-outsider's view > suggests, though, that the FreeBSD experience is relevant and timely. So this discussion wandered off topic a bit. Of course NetBSD already supports gpt and there is no plan to change (legacy) disklabels. Actually the whole original issue was triggered by my changes to the installer being too strict with disklabels on legacy devices on certain machines, where it now would reject a native disk and instead ask for the prefered partitioning scheme - which is not a good idea when performing an update. I have a good solution for that in userland now w/o any kernel support required. Martin
Re: Adding an ioctl to check for disklabel existence
On Thu, Oct 3, 2019 at 9:19 AM Robert Elz wrote: > Now it makes no sense at all. > FreeBSD made the explicit decision when disks were sneaking up on 2TB to move to GPT labels. Why invent a new scheme that interoperates poorly with other things? GPT, for better or worse, won. disklabel64 would add no value over GPT, require a lot of extra code and be an ongoing source of confusion and difficulty for our users. This is why UFS2 didn't bring in a 64-bit disklabel format... NetBSD is, of course, free to do what it likes. My semi-outsider's view suggests, though, that the FreeBSD experience is relevant and timely. Warner
Re: Adding an ioctl to check for disklabel existence
Date:Wed, 2 Oct 2019 21:47:38 -0400 (EDT) From:Mouse Message-ID: <201910030147.vaa03...@stone.rodents-montreal.org> John Nemeth said pretty much exactly what I would have said, but there are a few points on which I would like to expand... | - Partitions are huge. 128 bytes, Who cares? These days I (and may others) align partitions on fairly large boundaries (I use 1MB) for (some) future proofing of drive technology - that means that the space between 0 and 1MB is unused for any filesystem purpose, we may as well use all of it for labels (though even the biggest GPT label doesn't come near to that). Back in the 1970's when labels (MBR and disklabel) were invented, trying to compress them into the smallest possible space made sense. Now it is simply false ecomomy, saving a few meaningless sectors is irrelevant, losing flexibility is a killer. | - Partition types are UUIDs. Why use 16 bytes when you have, in the |information-theoretic sense, maybe three bits of information? We know it isn't 3, the 8 bits that MBRs allowed is almost all used up. You might only need a few, but other people need a different few (you omitted LFS, ZFS, HFS, ...) But what really matters is that any small number scheme needs a central registry, and someone to maintain it - and then some mechanism to extend it when it starts to run out. Any new partition scheme designed now really needs to be planned to last for at least as long, hopefully 10 times as long, as disklabels and MBR have lasted (40 years) - with who knows what kinds of uses of it into the future. Anyone ought to be able to design a new filesystem and give it a type which won't conflict with others (MBR types already failed at that, disklabels are used in such limited systems that it hasn't been such a problem). And they ought to be able to do that without having to deal with a central bureaucracy which might say "no" or "use one of the private use types we have defined until there are millions of users, then we'll allocate a real number, and all those millions can be changed", and "by the way, where do we send the invoice for this service?" | - Partition names(!) are character strings, rather than octet strings The thing that matters is actually a UUID, what you're referring to is the one thing that is missing from MBR and disklabels that I miss most. I can see from the label that there are N partitions on the drive, and what generic type they are (7 FFS and 1 swap) - but which one is which? Currently we do that (for FFS anyway) by checking the "last mounted" in the superblock, but that is unreliable (doesn't work at all for read only mounted filesystems which might never have been mounted RW, and if they were might have been on /mnt when they were) and isn't available at all in many other filesystem types (which of these swap partitions had dumps enabled on it?) A human readable name, in the label (ie: associated with the partition, not the filesystem) can help with that it - it even assists during planning when no filesystems have yet been created, just to make it easier to keep track of which have had space allocated. They're not required (by anything) to be unique, or really usable for much, but they're an incredibly useful thing to have. |It most certainly is |not appropriate for the partition table format to mandate/demand the |presence of Unicode support in the operating system. Nothing of the kind is required, the OS doesn't need to use the label (name) at all if it doesn't want (for a long time, NetBSD used GPT without doing so in the kernel - only recently has some of this been added.) | It does have one nice thing, a (supposedly-)unique partition | identifier. That's the real name (identifier) of the partition, not the name. | (Of course, nothing can ensure this actually _is_ unique; No, but statistically, cloning aside (where sometimes being not unique is exactly what is required, a clone should be identical) they are very likely unique, and if generater properly, I don't imagine I'll even see a (non-clone) duplicate. I don't imagine you will either, unless you force it. | No, not to be different. To be better. | | Except, of course, that (I assume) you don't think it _would_ be | better. It is entirely possible that one might design a partitioning scheme that is better than GPT. But by your argument I couldn't use that one, as it is just as possible to design one even better still, and even that one isn't usable for the same reason. At some point we (the world, not NetBSD) needs to simply say "good enough" and stop arguing. GPT is certainly good enough. Much better than disklabels or MBR (aside from anything else it has no fixed maximum number of partitions ... a particular label can only handle some number, but you get to decide when you create the label how many you need). Further everything you don't like about it, I consider a step in
Re: Adding an ioctl to check for disklabel existence
On Thu, Oct 03, 2019 at 02:42:10PM +0200, Rhialto wrote: > I was thinking the other day that it might be useful if gpt had a > subcommand to spit out a script to duplicate the partitioning of a disk, > but without the "unique" parts. The script would of course be > hand-editable for any changes one might want to make. Just create a backup of the gpt (gpt backup [-o outfile]), restore it on the other disk (gpt restore [-F] [-i infile]) and swap the IDs (gpt uuid [-a]). In a few days you will also be able to use sysinst to do this (optionally including the binary content of the partitions or just the partitions so you can apply more efficient options for the content), this is a side effect of some improvements I am currently working on. Martin
Re: Adding an ioctl to check for disklabel existence
On Wed 02 Oct 2019 at 19:40:01 -0700, John Nemeth wrote: > Cloning disks always presents issues. However, gpt(8) has > grown a "uuid" command to generate new UUIDs. This was primarily > done to help with the cloning problem. Cloning a disk and then > putting two disks with the same UUIDs on the same system is an > operator error. I was thinking the other day that it might be useful if gpt had a subcommand to spit out a script to duplicate the partitioning of a disk, but without the "unique" parts. The script would of course be hand-editable for any changes one might want to make. Such functionality would be the equivalent of using disklabel to get the editable text-version of a disklabel from one disk, and applying it to a different disk. I use this sort of thing if I want to create a backup disk which should have the same layout as the original. There is "gpt backup", but the manual page mentions "It should not be modified" and it contains the guids of the partition which in many cases should NOT be duplicated. It also isn't very readable. The intermediate solution I have used so far is to partition my disks in all cases using a simple script, and keep the script corresponding to each disk, so that I can use it again later. But if you didn't originally partition a disk this way, it doesn't help. -Olaf. -- Olaf 'Rhialto' Seibert -- rhialto at falu dot nl ___ Anyone who is capable of getting themselves made President should on \X/ no account be allowed to do the job. --Douglas Adams, "THGTTG" signature.asc Description: PGP signature
Re: Adding an ioctl to check for disklabel existence
While I agree NetBSD needs to support and work well with GPT in order to interoperate with other systems, there is also prior art in extending disklabel to 64bits - OpenBSD did this back in 2007 (though there were a fair few follow up commits to cleanup the fallout :) https://github.com/openbsd/src/commit/ddfcbf38c8ab6225a6b172d829aa957007d2587f#diff-192d23728acf9d8a70ab7259784d4162 David
Re: Adding an ioctl to check for disklabel existence
On Oct 2, 9:47pm, Mouse wrote: } } >> If _that_'s what you're concerned about, then just grow the relevant } >> fields (and, presumably, change the magic number). } > Any change to the label format or semantics would make it be a } > completely different object, no longer compatible with anything. } } Of course. } } > If we were going to invent something new that way, we may as well } > make it lots better - and ideally compatible with other systems so we } > can read one another's drives. } } > That's what GPT is, and it is already supported. } } Except for the "better" part, which is a matter of opinion. I don't } care for GPT. } } - Partitions are huge. 128 bytes, when you actually need maybe 18 (and }about a quarter of even that is pure future-proofing paranoia). I assume that you're talking about the size of a partition entry here. The size might be somewhat wasteful, but on a multi-terabyte drive isn't a particularly large concern. } - Partition types are UUIDs. Why use 16 bytes when you have, in the }information-theoretic sense, maybe three bits of information? }(At least, I can't recall the last time I used a partition type }other than FFS, unused, swap, MSDOS, or NTFS - and NTFS only for }trying to figure out what's on Windows disks. Six bits, maybe, }based on doubling the number of types I see in disklabel_gpt.h. My }"18" above assumes 16 bits.) I have 38 types in my list, which I'm sure is incomplete. That's a bit more than three bits. When the space is small compared to the utilisation, collisions are highly likely. Anybody can generate an UUID and if done properly the likelihood of collisions is very small. This would be the obvious reason for using an UUID. Remember that GPT is a standard meant to be used by all operating systems on multiple platforms } - Partition names(!) are character strings, rather than octet strings }or integers (an encoding is specified). This mandates a lot of }character-handling crap that does not belong in boot blocks and }arguably does not belong in operating systems. It most certainly is }not appropriate for the partition table format to mandate/demand the }presence of Unicode support in the operating system. } } It does have one nice thing, a (supposedly-)unique partition } identifier. (Of course, nothing can ensure this actually _is_ unique; } if nothing else, cloning an entire disk will perforce clone these } `unique' IDs.) But it doesn't need to be 16 bytes long! Even adding } 14 bytes of partition ID to the 18 bytes above still comes out to only } a quarter of the space GPT burns on each partition. Cloning disks always presents issues. However, gpt(8) has grown a "uuid" command to generate new UUIDs. This was primarily done to help with the cloning problem. Cloning a disk and then putting two disks with the same UUIDs on the same system is an operator error. } > Why would we want to invent something new, just to be different? } } No, not to be different. To be better. Ah yes, https://xkcd.org/927/ } Except, of course, that (I assume) you don't think it _would_ be } better. Better or not better is not really the point, being incompatible for specious reasons is. }-- End of excerpt from Mouse
Re: Adding an ioctl to check for disklabel existence
>> If _that_'s what you're concerned about, then just grow the relevant >> fields (and, presumably, change the magic number). > Any change to the label format or semantics would make it be a > completely different object, no longer compatible with anything. Of course. > If we were going to invent something new that way, we may as well > make it lots better - and ideally compatible with other systems so we > can read one another's drives. > That's what GPT is, and it is already supported. Except for the "better" part, which is a matter of opinion. I don't care for GPT. - Partitions are huge. 128 bytes, when you actually need maybe 18 (and about a quarter of even that is pure future-proofing paranoia). - Partition types are UUIDs. Why use 16 bytes when you have, in the information-theoretic sense, maybe three bits of information? (At least, I can't recall the last time I used a partition type other than FFS, unused, swap, MSDOS, or NTFS - and NTFS only for trying to figure out what's on Windows disks. Six bits, maybe, based on doubling the number of types I see in disklabel_gpt.h. My "18" above assumes 16 bits.) - Partition names(!) are character strings, rather than octet strings or integers (an encoding is specified). This mandates a lot of character-handling crap that does not belong in boot blocks and arguably does not belong in operating systems. It most certainly is not appropriate for the partition table format to mandate/demand the presence of Unicode support in the operating system. It does have one nice thing, a (supposedly-)unique partition identifier. (Of course, nothing can ensure this actually _is_ unique; if nothing else, cloning an entire disk will perforce clone these `unique' IDs.) But it doesn't need to be 16 bytes long! Even adding 14 bytes of partition ID to the 18 bytes above still comes out to only a quarter of the space GPT burns on each partition. > Why would we want to invent something new, just to be different? No, not to be different. To be better. Except, of course, that (I assume) you don't think it _would_ be better. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: Adding an ioctl to check for disklabel existence
Date:Wed, 2 Oct 2019 10:48:26 -0400 (EDT) From:Mouse Message-ID: <201910021448.kaa25...@stone.rodents-montreal.org> | If _that_'s what you're concerned about, then just grow the relevant | fields (and, presumably, change the magic number). Any change to the label format or semantics would make it be a completely different object, no longer compatible with anything. If we were going to invent something new that way, we may as well make it lots better - and ideally compatible with other systems so we can read one another's drives. That's what GPT is, and it is already supported. Why would we want to invent something new, just to be different? | Or fix sector sizes other than 512 bytes. disklabels work fine with drives with different sector sizes, and scale the size limit proportionally, but only on drives with different sector sizes. There was a fad for making drives export bigger sector sizes for a while, mostly, I suspect, because MBR has the same limitation as disklabel does. Now GPT is widely available, everything seems to be back advertising 512 byte sectors again (regardless of whatever happens to be the way the physical drive is used - which with caching, and likely full track reads most of the time, is difficult to observe externally). kre
Re: Adding an ioctl to check for disklabel existence
> however, disklabel fails at >2TiB for 512 byte sector, If _that_'s what you're concerned about, then just grow the relevant fields (and, presumably, change the magic number). Or fix sector sizes other than 512 bytes. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: Adding an ioctl to check for disklabel existence
Date:Mon, 30 Sep 2019 06:43:10 - (UTC) From:mlel...@serpens.de (Michael van Elst) Message-ID: | It also says 'fictitious' when you then write that label and read it | back (and that magic word is also used by the wedge naming code to | be treated like an empty label). Yes, the idea (the one change that would be needed) was to delete (blank/zero out) the label in the label read from disc routine(s) if it is seen to contain "fictitious" - then the wedge code wouldn't need to deal with it. | It might be easier to use a different, more obscure field in the | disklabel instead of d_packname (the "label" field) for such magic | values. It makes little difference, whatever is used when the kernel invents one, disklabel (or at least old versions of it) will simply write it to disc when making (unrelated) changes. The kernel is already using this field - inserting its magic value, we don't really want that in a from disc label, so fixing the label read routine(s) to delete it is probably worth doing whether or not Martin decides to use this, or another method, to deal with non disklabel labels that have been translated. kre
Re: Adding an ioctl to check for disklabel existence
m...@eterna.com.au (matthew green) writes: >however, disklabel fails at >2TiB for 512 byte sector, so i'm >now thinking that fixing this doesn't really solve the problem >for the future properly -- disklabel doesn't return a true >label here anyway... so it seems that we should be retiring >DIOCGDINFO usage as much as possible, rather than figuring out >how to enhance it.. The problem is not with default labels (i.e. no disklabel yet written) but with generated ones (converted from some native label). All such labels are (so far) as limited as disklabels and couldn't be used for larger disks. So I don't think that this is a problem. Tagging such labels with a magic value or flag bit has also a very limited scope. If we have native labels that support larger disks we can easily write a wedge autodiscover method for these. Of course we could do this also without the need for larger disks and phase out the fake label generation somewhen. -- -- Michael van Elst Internet: mlel...@serpens.de "A potential Snark may lurk in every tree."
Re: Adding an ioctl to check for disklabel existence
k...@munnari.oz.au (Robert Elz) writes: >Yes, we will - but can't we make that something detectable? If the >kernel invents a lael, it says "fictitious" in the label field. It also says 'fictitious' when you then write that label and read it back (and that magic word is also used by the wedge naming code to be treated like an empty label). It might be easier to use a different, more obscure field in the disklabel instead of d_packname (the "label" field) for such magic values. -- -- Michael van Elst Internet: mlel...@serpens.de "A potential Snark may lurk in every tree."
Re: Adding an ioctl to check for disklabel existence
On Mon, Sep 30, 2019 at 11:15:02AM +1000, matthew green wrote: > however, disklabel fails at >2TiB for 512 byte sector, so i'm > now thinking that fixing this doesn't really solve the problem > for the future properly -- disklabel doesn't return a true > label here anyway... so it seems that we should be retiring > DIOCGDINFO usage as much as possible, rather than figuring out > how to enhance it.. Thanks for all comments. Overall I agree - let's leave it alone for now. I have a slightly-evil-but-not-too-hackish idea what to do in sysinst to avoid special casing it at all. Martin
re: Adding an ioctl to check for disklabel existence
> Yes, we will - but can't we make that something detectable? If the > kernel invents a lael, it says "fictitious" in the label field. i had a similar idea, and i like your "converted" idea too. i was looking at struct disklabel itself, and while i realised we could probably abuse d_subtype to mark this? it seems to be entirely unused in our code base. however, disklabel fails at >2TiB for 512 byte sector, so i'm now thinking that fixing this doesn't really solve the problem for the future properly -- disklabel doesn't return a true label here anyway... so it seems that we should be retiring DIOCGDINFO usage as much as possible, rather than figuring out how to enhance it.. .mrg.
Re: Adding an ioctl to check for disklabel existence
Date:Sun, 29 Sep 2019 13:32:51 +0200
From:Martin Husemann
Message-ID: <20190929113251.ga12...@mail.duskware.de>
| If you just ask for the label
|
| disklabel $mydisk
|
| you will always get something.
Yes, we will - but can't we make that something detectable? If the
kernel invents a lael, it says "fictitious" in the label field.
All we need to do is make sure that no label read from a disc (into
the kernel, if read directly by userland, then userland knows what it
got and from where, already) contains that value, right? Like if
someone has actually written a label with that string ("fictitious")
to a drive, then when read back we just change it to "unnamed" (or even
just empty) -- it is a bit hard to believe that anyone wants their drive
actuially named "fictitious" (they'd still be able to use "fictional"
if they wanted).
| Now sometimes it is important to know the difference.
If we do that, we will know the difference, and what's more:
| some architectures have native ways to store their partitions [...]
| The kernel converts internally from the native partitions to disklabel when
| trying to read a disklabel,
when that happens, it could set the label to "converted" or if
the native format has a name that we include, put that name in brackets
("[name]") or something so it can be reconised (and then remove the
brackets if this is one of the cases where the kernel coverts back).
| Userland, however, has no way (that I know of) to learn about
| such conversions.
After that it would have, without need for extra ioctls (or for that
matter, any extra system call or work beyond a strcmp or two) - and what's
more it would work just fine with existing tools, as only the kernel, and
then utilities that desire to make use of the extra info (sysinst I presume)
needs to be changed.
We could have the disklabel utility refuse to allow the user to write
back a label containing "fictitious" (or "converted") as a name sometime
later (or now, bnut that isn't really crucial, and people using existing
tools wouldn't see such a change anyway).
| Or just extend and version DIOCGDINFO to provide that flag word in addition
| to the disklabel?
Do we actually need something new, or just this interpretation of the
label field?
kre
Re: Adding an ioctl to check for disklabel existence
mar...@duskware.de (Martin Husemann) writes: >So far all good. But there is a catch: some architectures have native >ways to store their partitions and no real userland support for those schemes. Some drivers may also have their own idea about what disklabels are, in particular cd(4). I wouldn't change anything here, in particular not add new functionality. There is already too much magic. If we really need to distinguish between read and faked labels we could tag the faked labels. There are unused words, even spare ones that could hold flags or magic numbers. Reading a label from disk would then need to verify and strip such information. The real problem however is then to write a label. Just writing a normal label that would override the faked label after a reboot might be confusing when native OS and NetBSD don't agree in their partitioning scheme. It might be simpler to just let writing a disklabel fail when a faked one is used. Then userland doesn't have to be aware of the difference, just handle the error. -- -- Michael van Elst Internet: mlel...@serpens.de "A potential Snark may lurk in every tree."
Re: Adding an ioctl to check for disklabel existence
> So far all good. But there is a catch: some architectures have native > ways to store their partitions and no real userland support for those > schemes. [...] > Userland, however, has no way (that I know of) to learn about such > conversions. The above recipe with "disklabel -r" will fail on such > disks, as -r makes disklabel check for a *real* disklabel and fail if > there is none (and besides arcon32, the userland disklabel binary > does not know of any such magic kernel conversion). [ side note: that > acorn32 magic userland code should be removed ] Userland has fdisk for MBR parititoning, disklabel -r for BSD disklabels, and sunlabel for Sun labels. (Are there other such tools? I'm not aware of any.) It seems to me that the rightest ways to go, depending on which way NetBSD wants to jump here, are (1) Eliminate such knowledge from userland entirely. (2) Merge tools such as fdisk and sunlabel into disklabel -r (though MBR partitioning is a bit of an odd duck in practice, because MBR partitions are typically carved up further, which in my experience is not true of either BSD or Sun labels). (3) Split acorn32 userland code out into a separate userland tool a la sunlabel; ideally, write similar tools for any other formats of interest (IIRC mac68k has one, though that memory is fuzzy; there doubtless are others I know nothing about). (4) Something else I haven't thought of. That is, pick a direction and then go there. The current state is a rather ad-hoc mishmash of all three of the above, with all the issues that inconsistency brings. Personally, I don't like (1), because I find such tools too useful; I'm not sure whether I prefer (2) or (3). /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
