Re: partman, growlight, discoverable partitions, and fun

[nick black]

Steve McIntyre left as an exercise for the reader:
> 100% true - expect people are busy, rather than hostile! :-)

i must have come across as far more disappointed than i felt --
i meant "hoped" in the literal sense, of "did not expect, but
thought plausible and welcome" =]. no, this has been an A-, A
interaction in my book--i half-expected flat rejection.

so thanks, debian! i will return to my codehole and bring
back something more tangible. but i found this quite promising.

-- 
nick black -=- https://www.nick-black.com
to make an apple pie from scratch,
you need first invent a universe.


signature.asc
Description: PGP signature

Re: partman, growlight, discoverable partitions, and fun

[Steve McIntyre]

Hi Nick!

On Tue, Sep 28, 2021 at 12:47:11PM +0200, Cyril Brulebois wrote:
>Hi nick,
>
>[ cc += debian-boot@ ]
>
>nick black  (2021-09-27):
>> Marc Haber left as an exercise for the reader:
>> > But maybe an alternative? I find the partitioning step one of the
>> > most error-prone and hard-to-use parts of non-trivial Debian
>> > installations.
>> 
>> so overall, i've got to say the feedback i heard here was a lot more
>> positive than i was expecting, though there was a bit less than i had
>> hoped for. but perhaps something that can be touched would see more
>> interest?
>
>FWIW I've followed the answers to your mail over the last few days,
>but I haven't had a chance to look at either the video or the slides
>(only 4 days before your initial mail and the one I'm replying to…).

Amen to that!

>At first glance, it seems fine to be experimenting with a different
>partitioner; of course all people are somewhere on the love/hate
>spectrum regarding partman and the zillions of partman-* packages, but
>it's indeed a shell maze and it *probably* could use some heavy lifting.
>I keep hoping that simple use cases are made simple(r)… maybe growlight
>could help there (e.g. “I'd like soft RAID1 with encrypted LVM, all that
>with a UEFI boot — therefore ESP —, without being a storage wizard”).

ACK. I've done a *bit* of hacking around partman over the last few
years - it's heavy going and probably the least well understood part
of d-i... :-/

>I suppose some step (once you've experimented on a growlight-only
>d-i) would be to have both partman and growlight, and let people
>choose (maybe with a flag at boot time, or by entering expert mode or
>whatever), until we have a better idea what works, what doesn't, what
>can be fixed, what cannot, and until a decision is made for the next
>Debian stable release.
>
>Leaving the technical integration aside for a moment, one question
>that comes to mind is whether this would be a one-shot contribution or
>some kind of longer commitment to maintain that different partitioner.
>I seem to remember earlier attempts at revamping the partitioning
>step, which stalled eventually.
>
>(Your recent mail to debian-newmaint@ is a hint; your apparent
>steadiness of those packages maintenance-wise is another; and your
>apparent interest in adding support to possibly missing features yet
>another.)
>
>Of course, one might object that the question is moot as there isn't
>much active development on partman components (even if some patches have
>been submitted over the last few days), but at least that's a known
>(imperfect, but…) beast.

Nod!

>> given that no one seemed to reject the idea out of hand, i'm going to
>> go ahead and rebase my work from 2012 (or more likely look at it once
>> and redo it) in a Salsa fork of d-i, and make some installation media
>> available. forgive me for likely only having x86 available at first.
>> i'll try to have this done within a week or so, and put it up on my
>> server. people can then give it a whirl.
>
>Feel free to touch base with debian-boot@ and/or debian-cd@ once you
>have a working proof of concept that some people have toyed with; we can
>discuss how the “alternative” part could be implemented (different
>images, or both partitioners ship together, with some option/selection —
>people might remember GRUB vs. LILO). I cannot guarantee a timely answer
>(life tends to keep me busy), but you shouldn't see a lack of answer
>after just a few days as if people were not interested.

100% true - expect people are busy, rather than hostile! :-)

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
Google-bait:   https://www.debian.org/CD/free-linux-cd
  Debian does NOT ship free CDs. Please do NOT contact the mailing
  lists asking us to send them to you.

Re: partman, growlight, discoverable partitions, and fun

[Wouter Verhelst]

On Tue, Sep 28, 2021 at 12:22:18PM +0200, John Paul Adrian Glaubitz wrote:
> On 9/28/21 12:13, Wouter Verhelst wrote:
> > IOW, chill out, nobody's going to kill off partman unless there's
> > something that's *actually* better than partman.
> 
> Just some comments after reading after this [1] because I honestly find it
> unfair the way I am being cornered here.

Yes, and what I'm saying is, I don't think anyone is trying to corner
you here, Nick are just saying "hey here's some cool new tech, what do
people think". The consensus in my reading is that people like it enough
that we might want to see some proof of concept, and then, assuming it
doesn't lack functionality that we don't want to lose and gives us
something new that we do like (e.g., better usability, more features,
etc etc) then we can perhaps replace partman one or more releases down
the line as the default partitioner.

You have some very good arguments against growlight in its *current*
state. That's fine. Just stay on top of things, and perhaps help Nick
out with testing a few of the more exotic features that partman
currently supports? Either through emulators or by giving him access to
real hardware if you can provide it (and given your collection, I
suspect you can ;-) ).

partman is currently being maintained by the d-i team very much in a
drive-by patches manner, but it's not actually seeing development of new
features. If Nick is offering to take over that work by replacing the
parted bits in d-i by his pet project, *and* he's offering to make sure
that we don't lose functionality we actually care about, then I think
that's a good thing? It's just a matter of making sure Nick *knows*
about the important bits, and that they get implemented in whatever we
end up replacing partman with before we actually do so...

-- 
 w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

Re: partman, growlight, discoverable partitions, and fun

[nick black]

Wouter Verhelst left as an exercise for the reader:
> iSCSI works very differently and is way more complex, but I remember
> from when I last played with it (which is a while ago, so the details
> are hazy) that it's not possible to set up in a non-persistent manner
> (i.e., all iSCSI connections survive reboot unless explicitly deleted,
> although obviously partman-iscsi has to do some dark magic to ensure the
> configuration is migrated from the d-i environment to the live system).

i've actually worked pretty extensively with iSCSI--i presented
on it at LPC2015 [0] =]. as far as i understand, iSCSI
connections are initiated and managed by the iscsid userspace
daemon (aside from root-on-iSCSI, which uses iscsistart, or at
least did. UEFI/BIOS iSCSI can also server here).

> There's also ATA-over-Ethernet, Fibrechannel-over-ethernet, multipath,
> and a whole slew of other things, if you want to configure this from
> growlight.

as you note, most of this is not stuff i want to slap a UI on,
but i'd certainly want to hit full partman feature parity...in
time. if it's best early on, i feel no shame punting more
esoteric setups to partman; as i've said, i would expect partman
to remain present on the installation media for at least some
significant time.

> I could be wrong though, haven't looked at growlight in much detail, and
> in the end it's your call, not mine :-D

nope, pretty much totally correct.

--nick

[0]
https://nick-black.com/dankwiki/images/e/ea/Public_LPC2015_-_Dynamic_iSCSI_at_Scale-_Remote_paging_at_Google.pdf

-- 
nick black -=- https://www.nick-black.com
to make an apple pie from scratch,
you need first invent a universe.


signature.asc
Description: PGP signature

Re: partman, growlight, discoverable partitions, and fun

[Cyril Brulebois]

Hi nick,

[ cc += debian-boot@ ]

nick black  (2021-09-27):
> Marc Haber left as an exercise for the reader:
> > But maybe an alternative? I find the partitioning step one of the
> > most error-prone and hard-to-use parts of non-trivial Debian
> > installations.
> 
> so overall, i've got to say the feedback i heard here was a lot more
> positive than i was expecting, though there was a bit less than i had
> hoped for. but perhaps something that can be touched would see more
> interest?

FWIW I've followed the answers to your mail over the last few days,
but I haven't had a chance to look at either the video or the slides
(only 4 days before your initial mail and the one I'm replying to…).


At first glance, it seems fine to be experimenting with a different
partitioner; of course all people are somewhere on the love/hate
spectrum regarding partman and the zillions of partman-* packages, but
it's indeed a shell maze and it *probably* could use some heavy lifting.
I keep hoping that simple use cases are made simple(r)… maybe growlight
could help there (e.g. “I'd like soft RAID1 with encrypted LVM, all that
with a UEFI boot — therefore ESP —, without being a storage wizard”).

I suppose some step (once you've experimented on a growlight-only
d-i) would be to have both partman and growlight, and let people
choose (maybe with a flag at boot time, or by entering expert mode or
whatever), until we have a better idea what works, what doesn't, what
can be fixed, what cannot, and until a decision is made for the next
Debian stable release.


Leaving the technical integration aside for a moment, one question
that comes to mind is whether this would be a one-shot contribution or
some kind of longer commitment to maintain that different partitioner.
I seem to remember earlier attempts at revamping the partitioning
step, which stalled eventually.

(Your recent mail to debian-newmaint@ is a hint; your apparent
steadiness of those packages maintenance-wise is another; and your
apparent interest in adding support to possibly missing features yet
another.)

Of course, one might object that the question is moot as there isn't
much active development on partman components (even if some patches have
been submitted over the last few days), but at least that's a known
(imperfect, but…) beast.

> given that no one seemed to reject the idea out of hand, i'm going to
> go ahead and rebase my work from 2012 (or more likely look at it once
> and redo it) in a Salsa fork of d-i, and make some installation media
> available. forgive me for likely only having x86 available at first.
> i'll try to have this done within a week or so, and put it up on my
> server. people can then give it a whirl.

Feel free to touch base with debian-boot@ and/or debian-cd@ once you
have a working proof of concept that some people have toyed with; we can
discuss how the “alternative” part could be implemented (different
images, or both partitioners ship together, with some option/selection —
people might remember GRUB vs. LILO). I cannot guarantee a timely answer
(life tends to keep me busy), but you shouldn't see a lack of answer
after just a few days as if people were not interested.

> note that there would still be some questions where i'd need input
> from Project members (as noted in my Debconf [0] presentation),
> particularly regarding translation (even if i can lift significant
> portions from partman, i'd need it looked over) and facilitating
> accessibility technology.

I won't delve into specifics (I did mention I didn't do any research,
right?) but as long as your application can be controlled via debconf,
it should fit right in within all our installation images (text based
installer, graphical installer, network-controller installer, etc.), and
things like localization and accessibility support should be automatic
(of course you'd still need to get the translation process bootstrapped
for your own strings at some point, but the inner workings should all be
there already).


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Re: partman, growlight, discoverable partitions, and fun

[Wouter Verhelst]

On Tue, Sep 28, 2021 at 06:19:28AM -0400, nick black wrote:
> Wouter Verhelst left as an exercise for the reader:
> > One thing that partman does is "support plug-ins", to allow for
> > configuring block devices before being able to partition them, where
> > needed. This can be useful for iSCSI, multipath, or (the one I care most
> > about) NBD. I wrote a "partman-nbd" a few years back to do just that:
> > https://salsa.debian.org/installer-team/partman-nbd
> 
> Thanks a lot for pointing this out, Wouter -- this is *exactly*
> the kind of feedback I was hoping for! I allow loopback devices
> to be set up, but not in any reboot-crossing manner, and I have
> no NBD nor iSCSI functionality.
> 
>  https://github.com/dankamongmen/growlight/issues/150

NBD is fairly easy to set up (well, it is for me, but then I'm biased).
For the server side, you install nbd-server, you create a (probably
sparse) large file somewhere, and then edit /etc/nbd-server/config to
point to that. For the client side, you install nbd-client and read "man
5 nbdtab" if you want to persist things, or you read "man 8 nbd-client"
if you just want to connect now and not care about reboot.

iSCSI works very differently and is way more complex, but I remember
from when I last played with it (which is a while ago, so the details
are hazy) that it's not possible to set up in a non-persistent manner
(i.e., all iSCSI connections survive reboot unless explicitly deleted,
although obviously partman-iscsi has to do some dark magic to ensure the
configuration is migrated from the d-i environment to the live system).

There's also ATA-over-Ethernet, Fibrechannel-over-ethernet, multipath,
and a whole slew of other things, if you want to configure this from
growlight.

That said though, I would personally recommend against doing that. From
where I'm standing, it seems to be out of scope for growlight? If you're
replacing partman in d-i, you'd still need to add *some* d-i
integration, and I'd imagine that integration is where this type of
device configuration would go. As far as running growlight outside of
d-i goes, there I'd think you'd just tell the user to configure the
device before starting growlight (or you'd give them the ability to
re-scan for new devices after they'd set up the device in a (different)
terminal), and then that'll be all? Otherwise you'll end up with a
neverending story of "oh here's another type of device that needs to be
added", and I don't think that's a great rabbit hole to go down.

I could be wrong though, haven't looked at growlight in much detail, and
in the end it's your call, not mine :-D

-- 
 w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

Re: partman, growlight, discoverable partitions, and fun

[John Paul Adrian Glaubitz]

On 9/28/21 12:13, Wouter Verhelst wrote:
> IOW, chill out, nobody's going to kill off partman unless there's
> something that's *actually* better than partman.

Just some comments after reading after this [1] because I honestly find it
unfair the way I am being cornered here.

First of all, neither Fedora or openSUSE seem to use glowlight as their
primary partitioning tool nor are they using discoverable partitions.

Secondly, discoverable partitions are primarily intended for containers
and systemd is using mkosi [2] for generating such images with discoverable
partitions.

Thirdly, Luca Boccassi considers himself to be systemd upstream [3] without
disclosing that information here. This is an information that he should
at least have disclosed before starting to voice his support for discoverable
partitions and insisting that the feature is important enough to kill off
parted.

Thanks,
Adrian

> [1] https://lwn.net/Articles/859240/
> [2] http://0pointer.net/blog/mkosi-a-tool-for-generating-os-images.html
> [3] https://lwn.net/Articles/859769/

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaub...@debian.org
`. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
  `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: partman, growlight, discoverable partitions, and fun

[nick black]

Wouter Verhelst left as an exercise for the reader:
> One thing that partman does is "support plug-ins", to allow for
> configuring block devices before being able to partition them, where
> needed. This can be useful for iSCSI, multipath, or (the one I care most
> about) NBD. I wrote a "partman-nbd" a few years back to do just that:
> https://salsa.debian.org/installer-team/partman-nbd

Thanks a lot for pointing this out, Wouter -- this is *exactly*
the kind of feedback I was hoping for! I allow loopback devices
to be set up, but not in any reboot-crossing manner, and I have
no NBD nor iSCSI functionality.

 https://github.com/dankamongmen/growlight/issues/150

-- 
nick black -=- https://www.nick-black.com
to make an apple pie from scratch,
you need first invent a universe.


signature.asc
Description: PGP signature

Re: partman, growlight, discoverable partitions, and fun

[Wouter Verhelst]

On Mon, Sep 27, 2021 at 03:18:48PM +0200, John Paul Adrian Glaubitz wrote:
> 
> 
> > On Sep 27, 2021, at 2:25 PM, Luca Boccassi  wrote:
> > 
> > Even if that interpretation would work as an excuse to never do
> > anything, and I'm not really sure it does, this specification has been
> > published in 2014 [0] so even by Debian standard it's old stuff.
> 
> That’s not what I said so. You’re trying to dismiss my opinion as completely 
> invalid now by trying to frame it such that I am against progress. I am not.
> 
> > It's
> > older than Debian Jessie, which was EOL'd last year. If libparted can't
> > keep up with 7 years old stuff that in the meantime was implemented in
> > util-linux's (which is a truly universal tool) in 2014, gdisk in 2019,
> > and so on, then to me it sounds like a tool in maintenance mode:
> > perfectly fine and adequate for existing tools and programs, but not
> > quite the best choice for new tools developed from scratch.
> 
> Whether a tool that was developed new from scratch is automatically better is
> not a given. The burden of proof is on the person trying to introduce the new
> software, not on the people maintaining the current set of software.

I think you're reading too much into the question here. The whole
*point* of Nick asking whether people would be opposed to that is
precisely *because* he wants to provide proof that his solution is
better than parted.

You've shown some things that are missing, and his immediate answer is
"ah, right, I'll need to add that then, but would need some assistance
to test that properly".

What's the problem with that?

Nobody is proposing to replace partman tomorrow.

Nobody is proposing to replace partman without testing the replacement.

It's also perfectly possible to imagine a transition period where both
partitioners are supported. That's the whole point of making d-i
modular: you can replace one component with another, and it adds new
features that support more use cases without killing off the older ones
because you can always ask for the other module. In fact, if memory
serves well, partman is the *second* partitioner that was written for
d-i, the first one having been replaced after just such a transition.

IOW, chill out, nobody's going to kill off partman unless there's
something that's *actually* better than partman.

-- 
 w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

Re: partman, growlight, discoverable partitions, and fun

[Wouter Verhelst]

Hi Nick,

On Mon, Sep 27, 2021 at 06:25:03PM -0400, nick black wrote:
> Marc Haber left as an exercise for the reader:
> > But maybe an alternative? I find the partitioning step one of the most
> > error-prone and hard-to-use parts of non-trivial Debian installations.
> 
> so overall, i've got to say the feedback i heard here was a lot
> more positive than i was expecting, though there was a bit less
> than i had hoped for. but perhaps something that can be touched
> would see more interest?
> 
> given that no one seemed to reject the idea out of hand, i'm
> going to go ahead and rebase my work from 2012 (or more likely
> look at it once and redo it) in a Salsa fork of d-i, and make
> some installation media available. forgive me for likely only
> having x86 available at first. i'll try to have this done within
> a week or so, and put it up on my server. people can then give
> it a whirl.

I hadn't replied to it *yet*, but had fully intended to do so (just
didn't get around to that yet).

One thing that partman does is "support plug-ins", to allow for
configuring block devices before being able to partition them, where
needed. This can be useful for iSCSI, multipath, or (the one I care most
about) NBD. I wrote a "partman-nbd" a few years back to do just that:

https://salsa.debian.org/installer-team/partman-nbd

This adds a few options to the partitioner menu to allow you to
add or remove an NBD device, and then if used makes sure the resulting
system is configured properly so that the NBD device(s) configured in
the installer will work after the system has been booted.

It would be nice if whatever component you write to replace partman has
support for things along these lines too. I don't mind having to rewrite
partman-nbd if that ends up being necessary (it's trivial enough), but
others might have different ideas about, say, partman-iscsi (just using
that as an example though, no idea about the details there).

Thanks,

-- 
 w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

Re: partman, growlight, discoverable partitions, and fun

[nick black]

Marc Haber left as an exercise for the reader:
> But maybe an alternative? I find the partitioning step one of the most
> error-prone and hard-to-use parts of non-trivial Debian installations.

so overall, i've got to say the feedback i heard here was a lot
more positive than i was expecting, though there was a bit less
than i had hoped for. but perhaps something that can be touched
would see more interest?

given that no one seemed to reject the idea out of hand, i'm
going to go ahead and rebase my work from 2012 (or more likely
look at it once and redo it) in a Salsa fork of d-i, and make
some installation media available. forgive me for likely only
having x86 available at first. i'll try to have this done within
a week or so, and put it up on my server. people can then give
it a whirl.

it's hard to see how exactly i proceed from that point, save in
a reactive fashion (not complaining, just saying). but we'll see
what happens when an ISO is available!

note that there would still be some questions where i'd need
input from Project members (as noted in my Debconf [0]
presentation), particularly regarding translation (even if i can
lift significant portions from partman, i'd need it looked
over) and facilitating accessibility technology.

--nick

[0] https://nick-black.com/dankwiki/images/b/b9/Parting_ways_with_partman.pdf

-- 
nick black -=- https://www.nick-black.com
to make an apple pie from scratch,
you need first invent a universe.


signature.asc
Description: PGP signature

Re: partman, growlight, discoverable partitions, and fun

[Metztli Information Technology]

On 9/27/21 4:01 AM, John Paul Adrian Glaubitz wrote:

Hello Nick!

On 9/26/21 16:29, Nick Black wrote:

I'd be delighted to support them -- as in, I am honestly eager
to add ATARI support; that sounds awesome -- I just need some
way to test the implementations, either via someone running it
on the environment, or getting access to such a machine.

There are emulators available for Atari such as Aranym. And emulators
available for Amiga such as fs-uae. FWIW, parted should contain everything
needed to be able to implement your own support for these partition tables.


I think it makes little sense to not use libparted as it already supports
all common and less common partition types and reimplementing everything
that libparted makes little sense to me.

parted did not have ZFS support when I embarked on this project
(it appears to have it now). i would not be opposed to
leveraging libparted if it presents a definite advantage;
supporting more partition types, so long as it exposes an API i
can easily work with, would be such an advantage.

Well, using a missing feature in the past against parted that is present
there is not such a good argument against using it, to be honest.


i do note that libparted2 is 621K in the archive, whereas
growlight itself is only 555K. it is of course possible that
all that weight is desirable functionality.

I think 70k more disk space is not really a concern.


with that said, i would *still want to test on the target
environment*, to make sure i'm using libparted correctly there.
so that necessity remains.

I thought you didn't depend on libparted?


would this allay your concerns?

No, not really. I consider a partitioning tool to be too important
to be replaced by an unproven solution.

Growlight seems like an adequate tool for Debian expert option as it has 
potential to enable additional flexibility and extensibility for file 
system options which may not offered in the current set of Parted 
utilities. For instance, the current Growlight string/option 'enter 
filesystem name' could be re-implemented, with a couple lines of code, 
into something like 'enter plugin override'


< https://metztli.it/bullseye/gl/gl-reiser4-plugin-override.png >

etc., for other filesystems.


Also Growlight may reduce at least a subset of UDEB modules needed for 
each file system supported by Debian, thus reducing complexity. For 
instance, it was relatively straightforward to add support for reiser4 
-- as compared to initial efforts developing substantial code for a 
partman-[filesystem] UDEB module.


< https://metztli.it/bullseye/gl/growlight-reiser4.png >

In other words, even if there is no compelling reason to update d-i at 
the moment, Growlight has potential to decrease complexity in the long run.


Just a point of view from an outsider, of course, I do not want anyone 
'to have a cow' for refusing to potentially move away from their place 
of comfort and/or entertaining a perspective that may be considered 'taboo.'



Best Professional Regards.

--

Jose R R
http://metztli.it 
-
Download Metztli Reiser4: Debian Buster w/ Linux 5.13.14 AMD64
-
feats ZSTD compression https://sf.net/projects/metztli-reiser4/ 


-
or SFRN 5.1.3, Metztli Reiser5 https://sf.net/projects/debian-reiser4/ 


---
Official current Reiser4 resources: https://reiser4.wiki.kernel.org/ 

Re: partman, growlight, discoverable partitions, and fun

[Marco d'Itri]

On Sep 27, John Paul Adrian Glaubitz  wrote:

> Not for me, though. Debian has always followed the philosophy to be a 
> universal
> operating system, which also meant that we can't (immediately) use all the new
> technologies and features that other distributions or upstream projects 
> develop.
It never meant that.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: partman, growlight, discoverable partitions, and fun

[John Paul Adrian Glaubitz]

> On Sep 27, 2021, at 3:41 PM, Luca Boccassi  wrote:
> 
> On Mon, 2021-09-27 at 15:18 +0200, John Paul Adrian Glaubitz wrote:
>> 
 On Sep 27, 2021, at 2:25 PM, Luca Boccassi  wrote:
>>> 
>>> Even if that interpretation would work as an excuse to never do
>>> anything, and I'm not really sure it does, this specification has been
>>> published in 2014 [0] so even by Debian standard it's old stuff.
>> 
>> That’s not what I said so. You’re trying to dismiss my opinion as completely 
>> invalid now by trying to frame it such that I am against progress. I am not.
> 
> You dismissed it because it's "new technology":
> 
>> Not for me, though. Debian has always followed the philosophy to be a 
>> universal
>> operating system, which also meant that we can't (immediately) use all the 
>> new
>> technologies and features that other distributions or upstream projects 
>> develop.

You’re reducing my argument to the word “new” which is definitely not my point. 
As you can see from the rest of my message my primary point is that Debian 
follows a different philosophy meaning that we don’t always adopt technologies 
that other distributions use.

Fedora and openSUSE are much more similar to each other than Debian is to both.

> I simply pointed out that it's a 7 years old spec that saw an entire
> LTS Debian version (8, we are now at 11) being released and EOL'ed
> since the time it was published. If this is too new to consider, then
> so are all Debian releases newer than Wheezy.

Yes, but the age was never my argument. My argument was that replacing such a 
fundamental software component like the partitioning tool in an installer is 
something that has to be justified by technical merits and by weighing pros and 
cons against each other.

The fact that’s it’s newer or has the single feature X is not sufficient in my 
opinion. Especially when there is no proof yet that getting support for 
discoverable partitions justifies the loss of features that parted has.

>>> It's
>>> older than Debian Jessie, which was EOL'd last year. If libparted can't
>>> keep up with 7 years old stuff that in the meantime was implemented in
>>> util-linux's (which is a truly universal tool) in 2014, gdisk in 2019,
>>> and so on, then to me it sounds like a tool in maintenance mode:
>>> perfectly fine and adequate for existing tools and programs, but not
>>> quite the best choice for new tools developed from scratch.
>> 
>> Whether a tool that was developed new from scratch is automatically better 
>> is not a given. The burden of proof is on the person trying to introduce the 
>> new software, not on the people maintaining the current set of software.
>> 
>> And claiming that parted is in pure maintenance mode is not true either. It 
>> has a paid developer working on the project and is receiving updates and 
>> improvements.
>> 
>> Whether growlight is better and more suitable for Debian needs to be 
>> technically proven, not just by arguing that it’s the newer project.
>> 
>> Adrian
> 
> Of course. But jumping in and saying "you should use X instead of Y",
> you can't pretend that nobody asks questions such as "ok, but does libX
> support this very much related and relevant 7 years old specification
> that other comparable tools support", no matter how awkward it is for
> libX.

I was not the one that was making this request, it was Nick. I’m perfectly fine 
with the status quo.

Again, the party introducing the new solution should provide the arguments to 
convince the maintainers of the existing solution.

For example, a convincing argument would be a demonstration installation ISO 
which let’s others interested in the project test it and check whether it 
delivers the improvements that were promised.

I don’t think that this is such an extraordinary requirement to ask for.

Also, I would be interested to know what approaches are currently used in 
Fedora, Arch, Gentoo and openSUSE (I will check that later myself when I’m back 
at the computer).

Adrian 

Re: partman, growlight, discoverable partitions, and fun

[Lennart Sorensen]

On Mon, Sep 27, 2021 at 03:18:48PM +0200, John Paul Adrian Glaubitz wrote:
> Whether a tool that was developed new from scratch is automatically better is 
> not a given. The burden of proof is on the person trying to introduce the new 
> software, not on the people maintaining the current set of software.
> 
> And claiming that parted is in pure maintenance mode is not true either. It 
> has a paid developer working on the project and is receiving updates and 
> improvements.
> 
> Whether growlight is better and more suitable for Debian needs to be 
> technically proven, not just by arguing that it’s the newer project.

I would have thought that if libparted was missing 1 or 2 features,
it would make more sense to add those features than to write a new tool
duplicating most of the functionality.

Well unless working with the maintainers of libparted is imposible.
There are a few projects like that but I don't remember ever seeing
complains about libparted.  Now you have two tools both missing some
features.  Hardly an improvement.

-- 
Len Sorensen

Re: partman, growlight, discoverable partitions, and fun

[Luca Boccassi]

On Mon, 2021-09-27 at 15:18 +0200, John Paul Adrian Glaubitz wrote:
> 
> > On Sep 27, 2021, at 2:25 PM, Luca Boccassi  wrote:
> > 
> > Even if that interpretation would work as an excuse to never do
> > anything, and I'm not really sure it does, this specification has been
> > published in 2014 [0] so even by Debian standard it's old stuff.
> 
> That’s not what I said so. You’re trying to dismiss my opinion as completely 
> invalid now by trying to frame it such that I am against progress. I am not.

You dismissed it because it's "new technology":

> Not for me, though. Debian has always followed the philosophy to be a 
> universal
> operating system, which also meant that we can't (immediately) use all the new
> technologies and features that other distributions or upstream projects 
> develop.

I simply pointed out that it's a 7 years old spec that saw an entire
LTS Debian version (8, we are now at 11) being released and EOL'ed
since the time it was published. If this is too new to consider, then
so are all Debian releases newer than Wheezy.

> > It's
> > older than Debian Jessie, which was EOL'd last year. If libparted can't
> > keep up with 7 years old stuff that in the meantime was implemented in
> > util-linux's (which is a truly universal tool) in 2014, gdisk in 2019,
> > and so on, then to me it sounds like a tool in maintenance mode:
> > perfectly fine and adequate for existing tools and programs, but not
> > quite the best choice for new tools developed from scratch.
> 
> Whether a tool that was developed new from scratch is automatically better is 
> not a given. The burden of proof is on the person trying to introduce the new 
> software, not on the people maintaining the current set of software.
> 
> And claiming that parted is in pure maintenance mode is not true either. It 
> has a paid developer working on the project and is receiving updates and 
> improvements.
> 
> Whether growlight is better and more suitable for Debian needs to be 
> technically proven, not just by arguing that it’s the newer project.
> 
> Adrian

Of course. But jumping in and saying "you should use X instead of Y",
you can't pretend that nobody asks questions such as "ok, but does libX
support this very much related and relevant 7 years old specification
that other comparable tools support", no matter how awkward it is for
libX.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: partman, growlight, discoverable partitions, and fun

[John Paul Adrian Glaubitz]

> On Sep 27, 2021, at 2:25 PM, Luca Boccassi  wrote:
> 
> Even if that interpretation would work as an excuse to never do
> anything, and I'm not really sure it does, this specification has been
> published in 2014 [0] so even by Debian standard it's old stuff.

That’s not what I said so. You’re trying to dismiss my opinion as completely 
invalid now by trying to frame it such that I am against progress. I am not.

> It's
> older than Debian Jessie, which was EOL'd last year. If libparted can't
> keep up with 7 years old stuff that in the meantime was implemented in
> util-linux's (which is a truly universal tool) in 2014, gdisk in 2019,
> and so on, then to me it sounds like a tool in maintenance mode:
> perfectly fine and adequate for existing tools and programs, but not
> quite the best choice for new tools developed from scratch.

Whether a tool that was developed new from scratch is automatically better is 
not a given. The burden of proof is on the person trying to introduce the new 
software, not on the people maintaining the current set of software.

And claiming that parted is in pure maintenance mode is not true either. It has 
a paid developer working on the project and is receiving updates and 
improvements.

Whether growlight is better and more suitable for Debian needs to be 
technically proven, not just by arguing that it’s the newer project.

Adrian

Re: partman, growlight, discoverable partitions, and fun

[Holger Levsen]

On Sat, Sep 25, 2021 at 06:49:53PM -0400, Nick Black wrote:
> So the only ones covered by partman and not covered by growlight would be:
> amiga, atari, sun,
> and mac (if mac is not the same as APM). I don't see any difficulty in
> adding these four, so long
> as there's someone with an Amiga or ATARI who'd be willing to test them
> out. If there are no such
> people, is it that important?

those ancient hardwares are not important to Debian, we have ceased to support
them years ago.

some people OTOH support them on Debian Ports and some of these people are very
vocal about the need of supporting architectures there. In my opinion they 
should 
only be heard if they also offer to do the work needed to keep those ancient
architectures alive. (and so far I've not even read an offer to help *testing*
such code there and also no offer to help developing such code...)

blocking progress on main Debian architectures because of some unsupported 
ancient
hardwares seems more problematic to me than not supporting those ancient
plattform.


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

It's not about saving the climate or the planet, it's about saving us, the
children and grandchildren. The planet will survive anyway.


signature.asc
Description: PGP signature

Re: partman, growlight, discoverable partitions, and fun

[Luca Boccassi]

On Mon, 2021-09-27 at 13:06 +0200, John Paul Adrian Glaubitz wrote:
> Hello!
> 
> On 9/27/21 12:46, Luca Boccassi wrote:
> > > Also, since parted is maintained by RedHat, I would expect that this 
> > > feature
> > > would land in parted soon as well.
> > > 
> > If the question is "why does X not use libparted", "does not support
> > discoverable parts spec" is a good enough answer for me.
> 
> Not for me, though. Debian has always followed the philosophy to be a 
> universal
> operating system, which also meant that we can't (immediately) use all the new
> technologies and features that other distributions or upstream projects 
> develop.
> 
> For example, we don't use systemd-homed or systemd-firstboot either. That 
> doesn't
> mean Debian is per se worse off. It just means Debian tries to cater 
> different use
> cases and follows different concepts.
> 
> It's different for distributions like Fedora or openSUSE which focus on a very
> limited set of targets and use cases. That's why they can quickly adopt to all
> the new features and technologies that upstream projects like systemd develop.
> 
> I'm not saying that one philosophy is better than the other. I'm just saying 
> that
> Debian is not like these other distributions.
> 
> Adrian

Even if that interpretation would work as an excuse to never do
anything, and I'm not really sure it does, this specification has been
published in 2014 [0] so even by Debian standard it's old stuff. It's
older than Debian Jessie, which was EOL'd last year. If libparted can't
keep up with 7 years old stuff that in the meantime was implemented in
util-linux's (which is a truly universal tool) in 2014, gdisk in 2019,
and so on, then to me it sounds like a tool in maintenance mode:
perfectly fine and adequate for existing tools and programs, but not
quite the best choice for new tools developed from scratch.

-- 
Kind regards,
Luca Boccassi

[0]
https://cgit.freedesktop.org/wiki/www/log/Specifications/DiscoverablePartitionsSpec.mdwn


signature.asc
Description: This is a digitally signed message part

Re: partman, growlight, discoverable partitions, and fun

[Florian Lohoff]

On Sun, Sep 26, 2021 at 10:50:35AM +0200, Adam Borowski wrote:
> On Sun, Sep 26, 2021 at 01:41:18AM -0400, nick black wrote:
> > Marco d'Itri left as an exercise for the reader:
> > > And the preseeding syntax is as powerful as it is inconvenient.
> 
> > > Implementing support for more partition formats, if missing, should be 
> > > rather easy.
> > > But which ones do we need for architectures which are not actually dead?
> > 
> > So, as I responded to Adrian [0], the only missing partition
> > types appear to be amiga, atari, and sun. Adding them ought be
> > simple enough, though I'd need testers with the hardware, or
> > access to the hardware.
> 
> I'd start with asking porters of m68k and sparc64 whether today's systems
> even run anything but Linux.  I think there's little point in keeping compat
> with 80s' OSes.
> 
> At a risk of drawing ire of m68k/sparc64 folks, I'd also suggest not putting
> your tuits there until this millenium's hardware is covered well.

This might be needed for booting purposes. 80ies Workstations tend to
have ROMs/BIOSes much like UEFI today and may even be booting files from
a Filesystem on a specific partition and thus disk label type.

So you are not breaking compatibility with 80ies OSes but
the platform as a whole.

Flo
-- 
Florian Lohoff f...@zz.de
  Any sufficiently advanced technology is indistinguishable from magic.


signature.asc
Description: PGP signature

Re: partman, growlight, discoverable partitions, and fun

[John Paul Adrian Glaubitz]

Hello!

On 9/27/21 12:46, Luca Boccassi wrote:
>> Also, since parted is maintained by RedHat, I would expect that this feature
>> would land in parted soon as well.
>>
> If the question is "why does X not use libparted", "does not support
> discoverable parts spec" is a good enough answer for me.

Not for me, though. Debian has always followed the philosophy to be a universal
operating system, which also meant that we can't (immediately) use all the new
technologies and features that other distributions or upstream projects develop.

For example, we don't use systemd-homed or systemd-firstboot either. That 
doesn't
mean Debian is per se worse off. It just means Debian tries to cater different 
use
cases and follows different concepts.

It's different for distributions like Fedora or openSUSE which focus on a very
limited set of targets and use cases. That's why they can quickly adopt to all
the new features and technologies that upstream projects like systemd develop.

I'm not saying that one philosophy is better than the other. I'm just saying 
that
Debian is not like these other distributions.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaub...@debian.org
`. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
  `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: partman, growlight, discoverable partitions, and fun

[John Paul Adrian Glaubitz]

Hello Nick!

On 9/26/21 16:29, Nick Black wrote:
> I'd be delighted to support them -- as in, I am honestly eager
> to add ATARI support; that sounds awesome -- I just need some
> way to test the implementations, either via someone running it
> on the environment, or getting access to such a machine.

There are emulators available for Atari such as Aranym. And emulators
available for Amiga such as fs-uae. FWIW, parted should contain everything
needed to be able to implement your own support for these partition tables.

>> I think it makes little sense to not use libparted as it already supports
>> all common and less common partition types and reimplementing everything
>> that libparted makes little sense to me.
> 
> parted did not have ZFS support when I embarked on this project
> (it appears to have it now). i would not be opposed to
> leveraging libparted if it presents a definite advantage;
> supporting more partition types, so long as it exposes an API i
> can easily work with, would be such an advantage.

Well, using a missing feature in the past against parted that is present
there is not such a good argument against using it, to be honest.

> i do note that libparted2 is 621K in the archive, whereas
> growlight itself is only 555K. it is of course possible that
> all that weight is desirable functionality.

I think 70k more disk space is not really a concern.

> with that said, i would *still want to test on the target
> environment*, to make sure i'm using libparted correctly there.
> so that necessity remains.

I thought you didn't depend on libparted?

> would this allay your concerns?

No, not really. I consider a partitioning tool to be too important
to be replaced by an unproven solution.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaub...@debian.org
`. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
  `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: partman, growlight, discoverable partitions, and fun

[Luca Boccassi]

On Sun, 2021-09-26 at 12:45 +0200, John Paul Adrian Glaubitz wrote:
> Hello!
> 
> On 9/26/21 12:40, Luca Boccassi wrote:
> > Does libparted support the discoverable partitions spec?
> 
> I'm not sure, this thread is the first time I have heard about discoverable
> partitions. I have read up first what the motivations for these are and
> how useful they would be for Debian.
> 
> Also, since parted is maintained by RedHat, I would expect that this feature
> would land in parted soon as well.
> 
> Adrian

If the question is "why does X not use libparted", "does not support
discoverable parts spec" is a good enough answer for me.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: partman, growlight, discoverable partitions, and fun

[Philipp Kern]

On 26.09.21 10:50, Adam Borowski wrote:

My biggest worry personally (aside from the realpolitik of
getting this change through) regards the automated partitioning
language available through the preseed system. Trying to emulate
this bug-for-bug is a non-starter, I think, both from a
technical and quality-of-life standpoint. If the emulation can't
be perfectly accurate, I don't think it ought be attempted for
such a critical, delicate procedure.

I personally think that preseed is nasty enough that users who do automation
on a scale that would make learning it worthwhile already have a better way to
do such automation.  For me, d-i is for manual installs, scripted stuff
wants a partitioner + glorified debootstrap.


As someone who has tried in the past to get partitioning correct using 
preseeding on a wider variety of disk shapes, I think anything but would 
be an improvement. FAI's setup-storage is obviously better. But good 
riddance to the lack of sensible debugging of the shell script horror 
story that is the existing system. :)


Kind regards
Philipp Kern

Re: partman, growlight, discoverable partitions, and fun

[Nick Black]

John Paul Adrian Glaubitz left as an exercise for the reader:
> So, you are not using libparted then?

i am not.

> Yes, it is important as we're supporting these architectures in Debian Ports
> and I invested quite some time to get Atari partition support added [1],
> for example.

I'd be delighted to support them -- as in, I am honestly eager
to add ATARI support; that sounds awesome -- I just need some
way to test the implementations, either via someone running it
on the environment, or getting access to such a machine.

> I think it makes little sense to not use libparted as it already supports
> all common and less common partition types and reimplementing everything
> that libparted makes little sense to me.

parted did not have ZFS support when I embarked on this project
(it appears to have it now). i would not be opposed to
leveraging libparted if it presents a definite advantage;
supporting more partition types, so long as it exposes an API i
can easily work with, would be such an advantage.

i do note that libparted2 is 621K in the archive, whereas
growlight itself is only 555K. it is of course possible that
all that weight is desirable functionality.

with that said, i would *still want to test on the target
environment*, to make sure i'm using libparted correctly there.
so that necessity remains.

would this allay your concerns?

--nick

-- 
nick black -=- https://www.nick-black.com
to make an apple pie from scratch,
you need first invent a universe.


signature.asc
Description: PGP signature

Re: partman, growlight, discoverable partitions, and fun

[Luca Boccassi]

On Sun, 26 Sept 2021 at 10:05, John Paul Adrian Glaubitz
 wrote:
>
> Hello Nick!
>
> On 9/26/21 00:49, Nick Black wrote:
> > It supports MBR, GPT, and APM:
> >
> > https://github.com/dankamongmen/growlight/blob/master/src/ptable.c#L51-L123
> >
> > (sorry for the gmail-style top posting; i can't find your mail in my mbox
> > for whatever reason)
> >
> > Any other partition schemes ought be trivial to add; they've not been added
> > yet
> > simply because I don't have means with which to test them.
>
> So, you are not using libparted then?
>
> > Looking at the "partition types" section of the Linux configuration, I see:
> >  Acorn, AIX, Alpha, Amiga, Atari, Macintosh, MSDOS, BSD, Minix, Solaris
> > x86, Unixware,
> >  Windows LDM, SGI, Ultrix, Sun, Karma, EFI/GPT, and SYSV68.
> >
> > Looking at the disk-label code from partman, I see:
> >  gpt, msdos, amiga, atari, mac, sun
> >
> > So the only ones covered by partman and not covered by growlight would be:
> > amiga, atari, sun,
> > and mac (if mac is not the same as APM). I don't see any difficulty in
> > adding these four, so long
> > as there's someone with an Amiga or ATARI who'd be willing to test them
> > out. If there are no such
> > people, is it that important?
>
> Yes, it is important as we're supporting these architectures in Debian Ports
> and I invested quite some time to get Atari partition support added [1],
> for example.
>
> I think it makes little sense to not use libparted as it already supports
> all common and less common partition types and reimplementing everything
> that libparted makes little sense to me.
>
> Adrian

Does libparted support the discoverable partitions spec?

Kind Regards,
Luca Boccassi

Re: partman, growlight, discoverable partitions, and fun

[John Paul Adrian Glaubitz]

Hello!

On 9/26/21 12:40, Luca Boccassi wrote:
> Does libparted support the discoverable partitions spec?

I'm not sure, this thread is the first time I have heard about discoverable
partitions. I have read up first what the motivations for these are and
how useful they would be for Debian.

Also, since parted is maintained by RedHat, I would expect that this feature
would land in parted soon as well.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaub...@debian.org
`. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
  `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: partman, growlight, discoverable partitions, and fun

[nick black]

Adam Borowski left as an exercise for the reader:
> I do have a different wish, though.  Could you please purge any references
> to drivemakers' units (stuff like MiB = million bytes, which current
> partitioner maliciously[1] swaps around with proper MB of 1048576B)? 

this really probably belongs over in the growlight bugtracker, but:

* i actually use the "memory units" for almost all interfaces,
  because that gives you the numbers you expect. for instance,
  my Seagate Exos 12TBs are 5859442688 4GB AF physical sectors,
  addressable as 23437770752 512B logical sectors. using
  tebibytes, this is 10.914TiB, as opposed to 12.000TB.

  as you can see at [0], we see 12T.

  there's only one place off the top of my head where they're
  *not* used, which is...

> Having them in the user interface is deeply harmful: people will get
> unoptimal alignment unless they 1. know about it, and 2. are careful enough. 

* alignment =] there we absolutely want to be using MiB etc.,
  and we do.

> >From your comments before I see that you try to do proper alignment, but in
> too many cases no matter how you try, the installer won't align well enough
> because the hardware might be newer than the version of growlight, hide its
> inner workings for marketing reason (like stealth SMR drives), etc.
> On the other hand, a completely oblivious user will get good alignment if
> you show numbers measured in gigabytes rather than gillionbytes.

so i'm not sure i necessarily buy that claim. if i'm overriding
the default alignment, i typically want to specify a value
that's independent of the disk size, and i always want it to be
in a *iB unit. oh, do you mean secondary and later partitions?
iirc, i accept (in addition to absolute sector numbers) a "+"
syntax where "+1M" would mean "the first possible 1MiB alignment
within this empty space", equal to the beginning of the empty
space when it happens to start on 1MiB multiple.

i don't know, mang; if i'm explicitly supplying sectors for
alignment purposes, i'm checking units pretty carefully.
preferably, i'm never doing that -- the only reason i can see
would be if i want to leave some large (larger than the
desirable alignment) chunk of empty space between two partitions.

and again, in any kind of alignment context, that's when you
*want* to be using MiB. and in such a context, "M" by itself is
interpreted that way.

> I know of only one case of multi-GB alignment (some early versions of ipmctl
> wanted a multiple of 32GB because certain vendor BIOSes had problems with
> smaller blocks), but the required alignment there is 1GB for years.

where here i assume you mean 1GiB aka 2³⁰ bytes, not 1GB aka 10⁹
bytes, correct? you could enter that as 1G, or 1GiB, or 1024MiB,
or 1048576KiB, or 1073741824. Using 1GB or 1000MB or 100KB
or 10 would force undesirable behavior.

changes to the text/UI to gently nudge users to the correct
behavior will be cheerfully considered!

> And most importantly: thanks for this effort, it's greatly appreciated!

thank you for your kind words! i'd love to see this happen.

--nick

[0] https://nick-black.com/images/growlight-2021-09-26.png

-- 
nick black -=- https://www.nick-black.com
to make an apple pie from scratch,
you need first invent a universe.


signature.asc
Description: PGP signature

Re: partman, growlight, discoverable partitions, and fun

[John Paul Adrian Glaubitz]

Hello Nick!

On 9/26/21 00:49, Nick Black wrote:
> It supports MBR, GPT, and APM:
> 
> https://github.com/dankamongmen/growlight/blob/master/src/ptable.c#L51-L123
> 
> (sorry for the gmail-style top posting; i can't find your mail in my mbox
> for whatever reason)
> 
> Any other partition schemes ought be trivial to add; they've not been added
> yet
> simply because I don't have means with which to test them.

So, you are not using libparted then?

> Looking at the "partition types" section of the Linux configuration, I see:
>  Acorn, AIX, Alpha, Amiga, Atari, Macintosh, MSDOS, BSD, Minix, Solaris
> x86, Unixware,
>  Windows LDM, SGI, Ultrix, Sun, Karma, EFI/GPT, and SYSV68.
> 
> Looking at the disk-label code from partman, I see:
>  gpt, msdos, amiga, atari, mac, sun
> 
> So the only ones covered by partman and not covered by growlight would be:
> amiga, atari, sun,
> and mac (if mac is not the same as APM). I don't see any difficulty in
> adding these four, so long
> as there's someone with an Amiga or ATARI who'd be willing to test them
> out. If there are no such
> people, is it that important?

Yes, it is important as we're supporting these architectures in Debian Ports
and I invested quite some time to get Atari partition support added [1],
for example.

I think it makes little sense to not use libparted as it already supports
all common and less common partition types and reimplementing everything
that libparted makes little sense to me.

Adrian

> [1] 
> https://git.savannah.gnu.org/cgit/parted.git/commit/?id=9c266205416ec956d6205c828211480de3767d02

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaub...@debian.org
`. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
  `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: partman, growlight, discoverable partitions, and fun

[Adam Borowski]

On Sun, Sep 26, 2021 at 01:41:18AM -0400, nick black wrote:
> Marco d'Itri left as an exercise for the reader:
> > And the preseeding syntax is as powerful as it is inconvenient.

> > Implementing support for more partition formats, if missing, should be 
> > rather easy.
> > But which ones do we need for architectures which are not actually dead?
> 
> So, as I responded to Adrian [0], the only missing partition
> types appear to be amiga, atari, and sun. Adding them ought be
> simple enough, though I'd need testers with the hardware, or
> access to the hardware.

I'd start with asking porters of m68k and sparc64 whether today's systems
even run anything but Linux.  I think there's little point in keeping compat
with 80s' OSes.

At a risk of drawing ire of m68k/sparc64 folks, I'd also suggest not putting
your tuits there until this millenium's hardware is covered well.

> My biggest worry personally (aside from the realpolitik of
> getting this change through) regards the automated partitioning
> language available through the preseed system. Trying to emulate
> this bug-for-bug is a non-starter, I think, both from a
> technical and quality-of-life standpoint. If the emulation can't
> be perfectly accurate, I don't think it ought be attempted for
> such a critical, delicate procedure.

I personally think that preseed is nasty enough that users who do automation
on a scale that would make learning it worthwhile already have a better way to
do such automation.  For me, d-i is for manual installs, scripted stuff
wants a partitioner + glorified debootstrap.


I do have a different wish, though.  Could you please purge any references
to drivemakers' units (stuff like MiB = million bytes, which current
partitioner maliciously[1] swaps around with proper MB of 1048576B)? 
Having them in the user interface is deeply harmful: people will get
unoptimal alignment unless they 1. know about it, and 2. are careful enough. 
>From your comments before I see that you try to do proper alignment, but in
too many cases no matter how you try, the installer won't align well enough
because the hardware might be newer than the version of growlight, hide its
inner workings for marketing reason (like stealth SMR drives), etc.
On the other hand, a completely oblivious user will get good alignment if
you show numbers measured in gigabytes rather than gillionbytes.

I know of only one case of multi-GB alignment (some early versions of ipmctl
wanted a multiple of 32GB because certain vendor BIOSes had problems with
smaller blocks), but the required alignment there is 1GB for years.

And most importantly: thanks for this effort, it's greatly appreciated!


Meow.

[1]. The malice hasn't been invented by the implementor of the old
partitioner -- it was done by marketing departments of disk vendors in the
old days; they don't even do so anymore but as they tried going through
standard bodies while fighting lawsuits, some damage lingers on.  The fault
of our old partitioner is that it didn't filter out the malice.
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ A white dwarf seeks a red giant for a binary relationship.
⢿⡄⠘⠷⠚⠋⠀
⠈⠳⣄

Re: partman, growlight, discoverable partitions, and fun

[nick black]

Marco d'Itri left as an exercise for the reader:
> On Sep 24, Marc Haber  wrote:
> 
> > But maybe an alternative? I find the partitioning step one of the most
> > error-prone and hard-to-use parts of non-trivial Debian installations.
> And the preseeding syntax is as powerful as it is inconvenient.
> I had not heard of growlight before yesterday, but from what I have read 
> I think that it is very promising.
> 
> Implementing support for more partition formats, if missing, should be 
> rather easy.
> But which ones do we need for architectures which are not actually dead?

So, as I responded to Adrian [0], the only missing partition
types appear to be amiga, atari, and sun. Adding them ought be
simple enough, though I'd need testers with the hardware, or
access to the hardware.

My biggest worry personally (aside from the realpolitik of
getting this change through) regards the automated partitioning
language available through the preseed system. Trying to emulate
this bug-for-bug is a non-starter, I think, both from a
technical and quality-of-life standpoint. If the emulation can't
be perfectly accurate, I don't think it ought be attempted for
such a critical, delicate procedure.

If faithfully honoring the preseed language is seen as a hard
requirement (not that I've heard this from anyone), it's
probably not happening. How important is that?

I could do a limited implementation, perhaps, where I clearly
error out on a spec I can't handle, falling back to partman.

If, on the other hand, it seems time to revamp the automatic
partitioning specification DSL, I could probably get behind
that. Maybe even the old one; I'd need see how complex it is (I
recall some pain trying to write complicated partman specs in
the past, but it was many years ago).

So...how do I go about making this happen? fwiw, I'm but a lowly
DM, not a DD.

--nick

[0] https://lists.debian.org/debian-devel/2021/09/msg00365.html

-- 
nick black -=- https://www.nick-black.com
to make an apple pie from scratch,
you need first invent a universe.


signature.asc
Description: PGP signature

Re: partman, growlight, discoverable partitions, and fun

[Nick Black]

It supports MBR, GPT, and APM:

https://github.com/dankamongmen/growlight/blob/master/src/ptable.c#L51-L123

(sorry for the gmail-style top posting; i can't find your mail in my mbox
for whatever reason)

Any other partition schemes ought be trivial to add; they've not been added
yet
simply because I don't have means with which to test them.

Looking at the "partition types" section of the Linux configuration, I see:
 Acorn, AIX, Alpha, Amiga, Atari, Macintosh, MSDOS, BSD, Minix, Solaris
x86, Unixware,
 Windows LDM, SGI, Ultrix, Sun, Karma, EFI/GPT, and SYSV68.

Looking at the disk-label code from partman, I see:
 gpt, msdos, amiga, atari, mac, sun

So the only ones covered by partman and not covered by growlight would be:
amiga, atari, sun,
and mac (if mac is not the same as APM). I don't see any difficulty in
adding these four, so long
as there's someone with an Amiga or ATARI who'd be willing to test them
out. If there are no such
people, is it that important?

On Thu, Sep 23, 2021 at 5:29 PM John Paul Adrian Glaubitz <
glaub...@physik.fu-berlin.de> wrote:

> Hello!
>
> On 9/23/21 22:57, nick black wrote:
> > I am just finishing up the implementation of Discoverable GPT
> > Partitions [0] in my growlight tool, and it seems a good time to
> > see if I can push this discussion [1] forward.
>
> Does it support partition tables other than GPT, in particular all
> of the partition tables that parted supports?
>
> If not, I don't think it would be a viable replacement for partman.
>
> Adrian
>
> --
>  .''`.  John Paul Adrian Glaubitz
> : :' :  Debian Developer - glaub...@debian.org
> `. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
>   `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913
>
>

-- 
nick black 
to make an apple pie from scratch, one need first invent a universe.

Re: partman, growlight, discoverable partitions, and fun

[Marco d'Itri]

On Sep 24, Marc Haber  wrote:

> But maybe an alternative? I find the partitioning step one of the most
> error-prone and hard-to-use parts of non-trivial Debian installations.
And the preseeding syntax is as powerful as it is inconvenient.
I had not heard of growlight before yesterday, but from what I have read 
I think that it is very promising.

Implementing support for more partition formats, if missing, should be 
rather easy.
But which ones do we need for architectures which are not actually dead?

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: partman, growlight, discoverable partitions, and fun

[Marc Haber]

On Thu, 23 Sep 2021 23:29:14 +0200, John Paul Adrian Glaubitz
 wrote:
>If not, I don't think it would be a viable replacement for partman.

But maybe an alternative? I find the partitioning step one of the most
error-prone and hard-to-use parts of non-trivial Debian installations.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: partman, growlight, discoverable partitions, and fun

[John Paul Adrian Glaubitz]

Hello!

On 9/23/21 22:57, nick black wrote:
> I am just finishing up the implementation of Discoverable GPT
> Partitions [0] in my growlight tool, and it seems a good time to
> see if I can push this discussion [1] forward.

Does it support partition tables other than GPT, in particular all
of the partition tables that parted supports?

If not, I don't think it would be a viable replacement for partman.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaub...@debian.org
`. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
  `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

partman, growlight, discoverable partitions, and fun

[nick black]

Hello there, debian-boot!

I am just finishing up the implementation of Discoverable GPT
Partitions [0] in my growlight tool, and it seems a good time to
see if I can push this discussion [1] forward.

I'd like to sound out especially the d-i shareholders on
replacement of partman with growlight. I can already hear
hackles raising; I invite you to watch [2]/read [3] my Debconf
21 presentation, "Parting Ways with Partman", where I hope
I've made a fairly complete and not-too-incorrect assessment
of the situation.

Essentially, my argument is that a tool which can both (a) play
the system preparation role of partman in d-i and (b) function
as a mainstream post-install disk management/analysis tool is
likely to minimize time to support of new features, and doesn't
consume developer time that can't be used outside of d-i.

If any longtime developers are truly wedded to partman, I don't
want to step on any toes; I'm the new guy; I certainly don't see
any immediately compelling technical reason to make a switch.
If you enjoy volunteer hackery on partman, great. I suspect,
however, that partman is no one's great love. Growlight is a
tool I've maintained and expanded over a decade, and intend to
continue improving, outside of any system installation role.

Christian PERRIER wrote on debian-boot 2013-02-07 [4]:

  "It's quite likely that the first step is then to have
   growlight in main Debian, isn't it?"

Just so, and that was accomplished last year [5]. So let's move on
to the second step, and consider this major move. I very very
much hope you'll read through [3] with an open mind.

Thanks!

--nick

[0] https://systemd.io/DISCOVERABLE_PARTITIONS/
[1] https://lists.debian.org/debian-boot/2013/02/msg00122.html
[2] https://nick-black.com/tabpower/debconf21-growlight.mkv
[3] 
https://nick-black.com/dankwiki/index.php?title=File:Parting_ways_with_partman.pdf
[4] https://lists.debian.org/debian-boot/2013/02/msg00164.html
[5] https://tracker.debian.org/pkg/growlight

-- 
nick black -=- https://www.nick-black.com
to make an apple pie from scratch,
you need first invent a universe.


signature.asc
Description: PGP signature