Re: bts reassign 878722 partman-auto

[Ben Hildred]

On Mon, Nov 13, 2017 at 1:17 PM, Nicholas D Steeves 
wrote:

> On Mon, Nov 13, 2017 at 10:35:19AM -0700, Ben Hildred wrote:
> >On Fri, Nov 10, 2017 at 12:50 PM, Cyril Brulebois <[1]k...@debian.org
> >
> >wrote:
> >
> >  Nicholas D Steeves <[2]nstee...@gmail.com> (2017-11-10):
> >  > 1) get a list of disks
> >  > 2) identify the disk used by the installer
> >  > 3) exclude the disk found at #2
> >
> >  How do you do 2?
> >
> >  Last I touched this, nothing obvious appeared in d-i to know what
> the
> >  installer was booted from. ISTR having suggested at the time that
> >  bootloaders could set something to help d-i figure out where it
> booted
> >  from, but I don't think anything happened in this area since then.
> >
> >OK, This Is a crazy Idea, but . . . When generating Installer images,
> they
> >get various readmes and so on, and I believe one of them includes
> version
> >information, so we can parse that file for the version number,
> compare it
> >with the one in the initrd (to be added). This handles the common
> cases of
> >cd (overkill as cds are read only) and USB (presumably most
> important),
> >but fails on net-install (where it is not needed). We can have
> installer
> >loader scripts copy the version info file to mark the drives they are
> >using which should catch most of the rest of the cases.
> >A variation of this is to have a pseudo random token in the version
> file
> >which is passed on the command-line to the installer instead of
> modifying
> >the initrd. This has the advantage that we could have special case
> values
> >for net-boot to skip the scan. (ie if the token was a hexadecimal
> value
> >but the special case was the word netboot.
> >
> >both cases make identifying and protecting partitions used to store
> >archives and iso images easy by manually placing the version file.
>
> Why does net-boot need to be special cased?  By default, shouldn't the
> net-boot media be excluded as an installation target--except for the
> expert case.
>

I was not talking about the net-boot cd, (that is cd or in some cases usb)
but a tfttp boot or similar where  there is no install media at all, and
the value would be a speed hack only.

>
> Another feature that could be piggybacked on the "mount block device
> and identify if it's Debian installation media" is OS identification.
> It's been so many years since I dual-booted with another OS that I
> don't know if this functionality already exists.  If it does exist,
> I'm guessing that is where this new "Identify installation media"
> could be added.
>

If we do this we might not want to special case netboot


> Re: the identification step:
>
> I don't think it's crazy :-)  Do you know if such a magic file already
> exists (especially if the installer build was recorded)?
>

the netboot.tar.gz file has version.info which I was thinking of which is
not currently included in mini.iso, but f1.txt has version and build date
(lines 3 and 4 in the one I checked).

>
> Can busybox-provided cat, grep, cksum and cmp provide strong enough
> magic file match, or do you think we also need a
> /sys/.../something/uuid check as well?
>
> Cheers,
> Nicholas
>

standard tools should do it.

-- 
--
Ben Hildred
Automation Support Services

Re: bts reassign 878722 partman-auto

[Nicholas D Steeves]

On Mon, Nov 13, 2017 at 10:35:19AM -0700, Ben Hildred wrote:
>On Fri, Nov 10, 2017 at 12:50 PM, Cyril Brulebois <[1]k...@debian.org>
>wrote:
> 
>  Nicholas D Steeves <[2]nstee...@gmail.com> (2017-11-10):
>  > 1) get a list of disks
>  > 2) identify the disk used by the installer
>  > 3) exclude the disk found at #2
> 
>  How do you do 2?
> 
>  Last I touched this, nothing obvious appeared in d-i to know what the
>  installer was booted from. ISTR having suggested at the time that
>  bootloaders could set something to help d-i figure out where it booted
>  from, but I don't think anything happened in this area since then.
> 
>OK, This Is a crazy Idea, but . . . When generating Installer images, they
>get various readmes and so on, and I believe one of them includes version
>information, so we can parse that file for the version number, compare it
>with the one in the initrd (to be added). This handles the common cases of
>cd (overkill as cds are read only) and USB (presumably most important),
>but fails on net-install (where it is not needed). We can have installer
>loader scripts copy the version info file to mark the drives they are
>using which should catch most of the rest of the cases.
>A variation of this is to have a pseudo random token in the version file
>which is passed on the command-line to the installer instead of modifying
>the initrd. This has the advantage that we could have special case values
>for net-boot to skip the scan. (ie if the token was a hexadecimal value
>but the special case was the word netboot.
> 
>both cases make identifying and protecting partitions used to store
>archives and iso images easy by manually placing the version file.

Why does net-boot need to be special cased?  By default, shouldn't the
net-boot media be excluded as an installation target--except for the
expert case.

Another feature that could be piggybacked on the "mount block device
and identify if it's Debian installation media" is OS identification.
It's been so many years since I dual-booted with another OS that I
don't know if this functionality already exists.  If it does exist,
I'm guessing that is where this new "Identify installation media"
could be added.

Re: the identification step:

I don't think it's crazy :-)  Do you know if such a magic file already
exists (especially if the installer build was recorded)?

Can busybox-provided cat, grep, cksum and cmp provide strong enough
magic file match, or do you think we also need a
/sys/.../something/uuid check as well?

Cheers,
Nicholas


signature.asc
Description: PGP signature

Re: bts reassign 878722 partman-auto

[Nicholas Steeves]

Hi Kibi,

I see Ben Hildred send a reply before I finished this draft :-)  I'll
reply to it separately.

On 10 November 2017 at 14:50, Cyril Brulebois  wrote:
> Nicholas D Steeves  (2017-11-10):
>> 1) get a list of disks
>> 2) identify the disk used by the installer
>> 3) exclude the disk found at #2
>
> How do you do 2?
>
> Last I touched this, nothing obvious appeared in d-i to know what the
> installer was booted from. ISTR having suggested at the time that
> bootloaders could set something to help d-i figure out where it booted
> from, but I don't think anything happened in this area since then.

My first thought was to implement something like the /\ this /\ , so
that /proc/cmdline could be used.  The next idea was to tie it to the
installer build somehow.  eg:

/dev/disk/by-uuid/2017-10-10-10-09-57-00 for Debian 9.2.1.  Is that only for 
amd64?
 * is a link that points to a device name...remove that disk from the list
 * not necessary for optical media IIRC
 * how hard would it be to make the installer aware of this?
 * Is PTUUID better? (eg: 5c91ec76) Or does this depend on blkid?

Alternatively, and expensive fall-back like this:
use existing code to get list of disks and partition
for i in partitions_or_unpartitioned_disks; do
  mount $i \
&& check for something that indicates it's Debian installation media \
&& check for specific version of something (would need a hook when building 
media to embed this)
or maybe check for existence of initrd.gz in $i, unpack it \
and compare checksum of something on initrd_used_to_boot_installer and 
initrd_found_on_$i \
&& rm disk_for_$i from list
or alternatively, maybe just check for a file that indicates "This is 
Debian installation media"
done

Alternatively all disks that match something that indicates "This is
Debian installation media", unless in expert mode.  This breaks
installation from one drive that has been prepared as installation
media to another drive that has been prepared as installation media
(possibly with a different version),

Most these methods break down for custom media that uses
grub-imageboot...that's the case I don't know how to solve, but maybe
the following would work: use /proc/cmdline and then loop-back mount
the image, check for match between file on running system and mounted
image, and remove the parent disk/partition from the list?

Cheers,
Nicholas


signature.asc
Description: PGP signature

Re: bts reassign 878722 partman-auto

[Ben Hildred]

On Fri, Nov 10, 2017 at 12:50 PM, Cyril Brulebois  wrote:

> Nicholas D Steeves  (2017-11-10):
> > 1) get a list of disks
> > 2) identify the disk used by the installer
> > 3) exclude the disk found at #2
>
> How do you do 2?
>
> Last I touched this, nothing obvious appeared in d-i to know what the
> installer was booted from. ISTR having suggested at the time that
> bootloaders could set something to help d-i figure out where it booted
> from, but I don't think anything happened in this area since then.
>
>
OK, This Is a crazy Idea, but . . . When generating Installer images, they
get various readmes and so on, and I believe one of them includes version
information, so we can parse that file for the version number, compare it
with the one in the initrd (to be added). This handles the common cases of
cd (overkill as cds are read only) and USB (presumably most important), but
fails on net-install (where it is not needed). We can have installer loader
scripts copy the version info file to mark the drives they are using which
should catch most of the rest of the cases.

A variation of this is to have a pseudo random token in the version file
which is passed on the command-line to the installer instead of modifying
the initrd. This has the advantage that we could have special case values
for net-boot to skip the scan. (ie if the token was a hexadecimal value but
the special case was the word netboot.

both cases make identifying and protecting partitions used to store
archives and iso images easy by manually placing the version file.

> 4) present modified list as target disks for installation
> > 5) unless in expert mode, where a user could use one partition of a
> > disk as the installation source and another partition as the
> > installation target.  I'm not sure how important #5 is, but maybe some
> > users want to be able to do this?
>
>
> KiBi.
>



-- 
--
Ben Hildred
Automation Support Services

Re: bts reassign 878722 partman-auto

[Cyril Brulebois]

Nicholas D Steeves  (2017-11-10):
> 1) get a list of disks
> 2) identify the disk used by the installer
> 3) exclude the disk found at #2

How do you do 2?

Last I touched this, nothing obvious appeared in d-i to know what the
installer was booted from. ISTR having suggested at the time that
bootloaders could set something to help d-i figure out where it booted
from, but I don't think anything happened in this area since then.

> 4) present modified list as target disks for installation
> 5) unless in expert mode, where a user could use one partition of a
> disk as the installation source and another partition as the
> installation target.  I'm not sure how important #5 is, but maybe some
> users want to be able to do this?


KiBi.


signature.asc
Description: PGP signature

Re: bts reassign 878722 partman-auto

[Ben Hutchings]

On Fri, 2017-11-10 at 12:32 -0500, Lennart Sorensen wrote:
> On Fri, Nov 10, 2017 at 04:19:14PM +, Ben Hutchings wrote:
> > This is true, but I don't think it's a good reason not to implement a
> > mostly-reliable heuristic.
> > 
> > If there are multiple disks, there are usually going to be just 2 of
> > them, one of which contains the installer.  In any installer build
> > other than netboot, it will look for its own disk in order to load
> > udebs.  Once it has done that, it can determine that the other disk is
> > the one to install on.  That's a pretty good heuristic.
> 
> I think more than one disk in the machine isn't that unusual.

The context of this bug is that Michael wants to partition a single
disk - not set up a RAID.  I think it *is* unusual that Debian would be
installed on a single disk of a system that contains multiple internal
disks.  If someone really wants to pre-seed partman in that case then
they would have to specify the device name just as they do now
(presumably using /dev/disk/by-path/...).

> > Aside from that, we can also make a guess based on the bus type:
> > 
> > - ATA: probably internal
> 
> eSATA is not that unusual.
>
> > - NVMe: probably internal
> > - USB: probably external
> > - MMC/SD: ambiguous (eMMC must be internal, and Linux has a notion of
> > 'non-removable' slots, but I don't think userland has this info)
> > 
> > If we could get more information about MMC/SD slots then we should be
> > able to implement an heuristic that would work for >99% of cases.
> 
> You can certainly try to make a good guess, but it certainly still needs
> to be confirmed.

That seems to defeat the point of pre-seeding.

Ben.

-- 
Ben Hutchings
Who are all these weirdos? - David Bowie, reading IRC for the first
time



signature.asc
Description: This is a digitally signed message part

Re: bts reassign 878722 partman-auto

[Nicholas D Steeves]

On Fri, Nov 10, 2017 at 12:32:59PM -0500, Lennart Sorensen wrote:
> On Fri, Nov 10, 2017 at 04:19:14PM +, Ben Hutchings wrote:
> > This is true, but I don't think it's a good reason not to implement a
> > mostly-reliable heuristic.
> > 
> > If there are multiple disks, there are usually going to be just 2 of
> > them, one of which contains the installer.  In any installer build
> > other than netboot, it will look for its own disk in order to load
> > udebs.  Once it has done that, it can determine that the other disk is
> > the one to install on.  That's a pretty good heuristic.
> 
> I think more than one disk in the machine isn't that unusual.

Is there any reason why the following method wouldn't be an
improvement?:

1) get a list of disks
2) identify the disk used by the installer
3) exclude the disk found at #2
4) present modified list as target disks for installation
5) unless in expert mode, where a user could use one partition of a
disk as the installation source and another partition as the
installation target.  I'm not sure how important #5 is, but maybe some
users want to be able to do this?

Cheers,
Nicholas


signature.asc
Description: PGP signature

Re: bts reassign 878722 partman-auto

[Lennart Sorensen]

On Fri, Nov 10, 2017 at 04:19:14PM +, Ben Hutchings wrote:
> This is true, but I don't think it's a good reason not to implement a
> mostly-reliable heuristic.
> 
> If there are multiple disks, there are usually going to be just 2 of
> them, one of which contains the installer.  In any installer build
> other than netboot, it will look for its own disk in order to load
> udebs.  Once it has done that, it can determine that the other disk is
> the one to install on.  That's a pretty good heuristic.

I think more than one disk in the machine isn't that unusual.

> Aside from that, we can also make a guess based on the bus type:
> 
> - ATA: probably internal

eSATA is not that unusual.

> - NVMe: probably internal
> - USB: probably external
> - MMC/SD: ambiguous (eMMC must be internal, and Linux has a notion of
> 'non-removable' slots, but I don't think userland has this info)
> 
> If we could get more information about MMC/SD slots then we should be
> able to implement an heuristic that would work for >99% of cases.

You can certainly try to make a good guess, but it certainly still needs
to be confirmed.

-- 
Len Sorensen

Re: bts reassign 878722 partman-auto

[Ben Hutchings]

On Fri, 2017-11-10 at 10:10 -0500, Lennart Sorensen wrote:
> On Tue, Nov 07, 2017 at 09:56:31PM +0100, Michael Kesper wrote:
> > Yes sure but why can't I correct it after the fact?
> > Even "rescanning disks" does not let you chose any other disks.
> > 
> > Is there a way of chosing "first internal disk" then?
> > Imagine I want to create one installation medium for laptops which only
> > differ whether they are set up with a NVM or a sata SSD.
> > I did not find any documentation helping me with this.
> 
> I can't think of any reliable way to determine what is an internal or
> external disk, and the concept of 'first' doesn't even have meaning.

This is true, but I don't think it's a good reason not to implement a
mostly-reliable heuristic.

If there are multiple disks, there are usually going to be just 2 of
them, one of which contains the installer.  In any installer build
other than netboot, it will look for its own disk in order to load
udebs.  Once it has done that, it can determine that the other disk is
the one to install on.  That's a pretty good heuristic.

Aside from that, we can also make a guess based on the bus type:

- ATA: probably internal
- NVMe: probably internal
- USB: probably external
- MMC/SD: ambiguous (eMMC must be internal, and Linux has a notion of
'non-removable' slots, but I don't think userland has this info)

If we could get more information about MMC/SD slots then we should be
able to implement an heuristic that would work for >99% of cases.

Ben.

-- 
Ben Hutchings
Who are all these weirdos? - David Bowie, reading IRC for the first
time



signature.asc
Description: This is a digitally signed message part

Re: bts reassign 878722 partman-auto

[Lennart Sorensen]

On Tue, Nov 07, 2017 at 09:56:31PM +0100, Michael Kesper wrote:
> Yes sure but why can't I correct it after the fact?
> Even "rescanning disks" does not let you chose any other disks.
> 
> Is there a way of chosing "first internal disk" then?
> Imagine I want to create one installation medium for laptops which only
> differ whether they are set up with a NVM or a sata SSD.
> I did not find any documentation helping me with this.

I can't think of any reliable way to determine what is an internal or
external disk, and the concept of 'first' doesn't even have meaning.

-- 
Len Sorensen

Re: bts reassign 878722 partman-auto

[Michael Kesper]

Dear Cyril,


On 07.11.2017 08:12, Cyril Brulebois wrote:
> Michael Kesper  (2017-11-06):
>> I think this bug is specific to partman-auto.
>> Partman should allow rescanning devices and recognize NVMe devices when
>> preconfigured with /dev/sda.
>> Alternatively, there should be an installation target "largest disk" or
>> something similar.
> 
> I'm not sure hardcoding /dev/sda as the target device is right when all you
> have is NVMe… Don't feed wrong info through preseed in the first place?

Yes sure but why can't I correct it after the fact?
Even "rescanning disks" does not let you chose any other disks.

> Also, “largest disk” doesn't seem too good an idea, as evidenced here:
>   https://lists.debian.org/debian-boot/2017/11/msg00028.html

Is there a way of chosing "first internal disk" then?
Imagine I want to create one installation medium for laptops which only
differ whether they are set up with a NVM or a sata SSD.
I did not find any documentation helping me with this.

Best wishes
Michael



signature.asc
Description: OpenPGP digital signature

Re: bts reassign 878722 partman-auto

[Cyril Brulebois]

Hi Michael,

Michael Kesper  (2017-11-06):
> I think this bug is specific to partman-auto.
> Partman should allow rescanning devices and recognize NVMe devices when
> preconfigured with /dev/sda.
> Alternatively, there should be an installation target "largest disk" or
> something similar.

I'm not sure hardcoding /dev/sda as the target device is right when all you
have is NVMe… Don't feed wrong info through preseed in the first place?

Also, “largest disk” doesn't seem too good an idea, as evidenced here:
  https://lists.debian.org/debian-boot/2017/11/msg00028.html


KiBi.


signature.asc
Description: PGP signature

Processed: bts reassign 878722 partman-auto

[Debian Bug Tracking System]

Processing control commands:

> reassign -1 partman-auto
Bug #878722 [debian-installer] Preseeding: Unable to correct partitioning 
manually if preseed fails
Bug reassigned from package 'debian-installer' to 'partman-auto'.
Ignoring request to alter found versions of bug #878722 to the same values 
previously set
Ignoring request to alter fixed versions of bug #878722 to the same values 
previously set

-- 
878722: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=878722
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

bts reassign 878722 partman-auto

[Michael Kesper]

control: reassign -1 partman-auto


Dear maintainers,

I think this bug is specific to partman-auto.
Partman should allow rescanning devices and recognize NVMe devices when
preconfigured with /dev/sda.
Alternatively, there should be an installation target "largest disk" or
something similar.

Best wishes
Michael





signature.asc
Description: OpenPGP digital signature