Re: setup-storage fails on blank disk

[Derek Poon]

Me too.  I have been using the following script, placed in 
hooks/partition.LINUX, as a workaround.


#!/bin/bash

##
# If setup-storage encounters a disk to be partitioned, but the disk
# contains no partition table at all, it will give up.  To prevent
# such failures, this hook creates an empty partition table of the
# type specified in the setup-storage configuration file on any disk
# whose first 512 bytes are all null.
##

. /usr/lib/fai/subroutines

# Prints the absolute path of the disk configuration file, chosen based on
# the last applicable class to have a configuration file in
# /var/lib/fai/config/disk_config.
disk_config_file() {
echo $classes | tr ' ' '\n' | tac | while read class ; do
if [ -f "$FAI/disk_config/$class" ]; then
echo "$FAI/disk_config/$class"
break
fi
done
}

# Output contains one line per disk, with the device followed by the partition
# table type that it is supposed to have:
#
# /dev/sda gpt
# /dev/sda gpt-bios
# /dev/sdc msdos
want_partition_tables() {
config_file=`disk_config_file`
if [ -z "$config_file" ]; then
# No suitable setup-storage config file
exit 1
fi
awk -v disks="`/usr/lib/fai/fai-disk-info | sort`" '
BEGIN {
split(disks, dev);
}

$1 == "disk_config" && $2 ~ "disk.*" {
sub("^disk", "", $2);
for (i = 3; i <= NF; i++) {
if (split($i, optionpair, ":")) {
if (optionpair[1] == "sameas") {
sub("^disk", "", optionpair[2]);
partition_table[$2] = partition_table[optionpair[2]];
if (dev[$2]) print "/dev/" dev[$2], partition_table[$2];
next;
} else if (optionpair[1] == "disklabel") {
partition_table[$2] = optionpair[2];
if (dev[$2]) print "/dev/" dev[$2], partition_table[$2];
next;
}
}
}

# If no disklabel option is given, setup-storage uses "msdos" as
# the default.
if (dev[$2]) print "/dev/" dev[$2], "msdos";
}
' "$config_file"
}

# Returns 0 if the first 512 bytes of the specified device are all null.
has_partition_table() {
local dev="$1"
perl -e "
open D, '<', '$dev' and
read D, \$buf, 512 and
\$buf ne \"\\000\" x 512 or
exit 1
"
}

# For every disk in the setup-storage configuration, if there is no partition
# table apparent in the first 512 bytes, then we will create one of the
# appropriate type.
ensure_partition_tables_exist() {
set -o pipefail
want_partition_tables |
while read dev table ; do
if ! has_partition_table "$dev" ; then
[ "$table" = "gpt-bios" ] && table=gpt
echo "$dev has no partition table; making one of type $table"
parted --script "$dev" mktable "$table"
fi
done
}

ensure_partition_tables_exist



> On Jan 4, 2018, at 7:12 AM, Brian Kroth  wrote:
> 
> Ages ago I recall having a similar problem to that as well.  I think it was a 
> bug with the preserver_reinstall flag not working correctly with LVM setups 
> on fresh disks.  Pretty sure I ended up writing a hack that executed early on 
> in the classes phase to figure out if the disk was lacking a label (partition 
> table) and then creating one and setting the reinstall flag to force the disk 
> layout down.  Unfortunately I don't easily have access to that code anymore, 
> else I'd share it.  Also, it may no longer be applicable.  setup-storage has 
> changed a lot since then.
> 
> Cheers,
> Brian

Re: setup-storage fails on blank disk

[Brian Kroth]

Ages ago I recall having a similar problem to that as well.  I think it was
a bug with the preserver_reinstall flag not working correctly with LVM
setups on fresh disks.  Pretty sure I ended up writing a hack that executed
early on in the classes phase to figure out if the disk was lacking a label
(partition table) and then creating one and setting the reinstall flag to
force the disk layout down.  Unfortunately I don't easily have access to
that code anymore, else I'd share it.  Also, it may no longer be
applicable.  setup-storage has changed a lot since then.

Cheers,
Brian

On Thu, Jan 4, 2018 at 8:41 AM, Justin Cattle  wrote:

> Hi,
>
>
> Are you 100% sure you tried it on a "fresh" disk that was really clean?
> It defiantly feels like there is some metadata or something remaining in
> some blocks on the disk.
>
> When FAI fails, are you able check for things like md info, dm info, lvm
> info and the like?
>
> You may have to do some dmsetup remove, vgremove or pvremove.
> wiptefs is also a good utility at clearing metadata, run it against any
> partitions before removing them.
> Then remove partitions with dd, put some zeros on the first few Mg of the
> disk.
>
>
>
>
> Cheers,
> Just
>
> On 4 January 2018 at 13:47, Andreas Heinlein  wrote:
>
>> Am 03.01.2018 um 17:28 schrieb Holger Parplies:
>> > Hi,
>> >
>> > Andreas Heinlein wrote on 2018-01-03 13:53:40 +0100 [setup-storage
>> fails on blank disk]:
>> >> [...]
>> >> I have encountered a problem with setup-storage which occurs only when
>> >> the disk is blank, i.e. wiped with nwipe/dban or brand new. It then
>> >> fails on creating the LVM; running 'pvcreate' returns 'cannot open
>> >> /dev/sda5 exclusively'.
>> > this is probably unrelated, but is there any reason to put the LVM PV
>> inside
>> > a "logical" volume? DOS extended partitions seem to be the worst hack
>> ever
>> > invented to get around a limitation in a bad design, yet they repeatedly
>> > and apparently unnecessarily pop up in quoted disk_configs:
>> >
>> >> [...]
>> >> This is your disk_config file:
>> >> # generic disk configuration for one small disk
>> >> # disk size from 500Mb up to what you can buy today
>> >> #
>> >> #   [extra
>> options]
>> >>
>> >> disk_config disk1 disklabel:msdos bootable:1 preserve_lazy:6
>> align-at:1M fstabkey:uuid
>> >> primary  /boot  300  ext4rw  createopts="-O
>> ^64bit,^metadata_csum"
>> >> logical  -  29500-3  -   -
>> >> logical  /media/daten  1024- ext4acl createopts="-O
>> ^64bit,^metadata_csum -L Daten"
>> > I count three partitions, which would work perfectly with primary
>> partitions
>> > (furthermore, you are using LVM to have an arbitrary number of named and
>> > dynamic "volumes" (i.e. partitions) anyway, so if you needed more, LVM
>> would
>> > be the superior mechanism; of course, your specific requirements may
>> vary).
>> > Ok, you are preserving a logical partition, so in this particular case
>> you'd
>> > actually need to stick with logical partitions, but the partition in
>> question
>> > is ext4, not FAT-based, so it doesn't appear to be a legacy Windoze
>> issue.
>> >
>> > My point: am I missing something, and there is some obscure benefit of
>> putting
>> > an LVM container within an extended-partition-container (such as hiding
>> it
>> > from something), or is it simply a common misconception that you for
>> some
>> > reason cannot or should not put an LVM PV (or even several individual
>> native
>> > Linux partitions - such as /, /var and /tmp) into primary partitions -
>> > assuming you only need upto four of them (and, obviously, assuming you
>> are
>> > still using MSDOS partition tables)?
>> >
>> > Or, differently: for a *blank disk*, you obviously won't be preserving
>> sda6,
>> > and you probably aren't referencing it by partition number
>> ("fstabkey:uuid"),
>> > so does using 'primary' instead of 'logical' for all three partitions
>> change
>> > anything concerning the error you are experiencing?
>> >
>> > Hope that helps someone (perhaps me ;-) ...
>> >
>> > Regards,
>> > Holger
>> Hello,
>>
>> yes, you are right - in some way, this *is* a legacy windows issue that
>> has developed over time. In fact, the preserved partition once was a FAT
>> partition as long as we had dualboot installations on these machines. We
>> finally removed the dualboot two or three years ago and chose to format
>> this partition ext4 instead. Why we didn't move to primary partitions or
>> put it within the LVM at that time - I don't know.
>>
>> On the other hand, up to now we had no problems with that, so no urge to
>> change anything. If you think it might help, I could try changing this.
>>
>> Bye,
>> Andreas
>>
>
>
> Notice:  This email is confidential and may contain copyright material of
> members of the Ocado Group. Opinions and views expressed in this message
> may not necessarily reflect the opinions and views of the members of the
> Ocado 

Re: setup-storage fails on blank disk

[Justin Cattle]

Hi,


Are you 100% sure you tried it on a "fresh" disk that was really clean?
It defiantly feels like there is some metadata or something remaining in
some blocks on the disk.

When FAI fails, are you able check for things like md info, dm info, lvm
info and the like?

You may have to do some dmsetup remove, vgremove or pvremove.
wiptefs is also a good utility at clearing metadata, run it against any
partitions before removing them.
Then remove partitions with dd, put some zeros on the first few Mg of the
disk.




Cheers,
Just

On 4 January 2018 at 13:47, Andreas Heinlein  wrote:

> Am 03.01.2018 um 17:28 schrieb Holger Parplies:
> > Hi,
> >
> > Andreas Heinlein wrote on 2018-01-03 13:53:40 +0100 [setup-storage fails
> on blank disk]:
> >> [...]
> >> I have encountered a problem with setup-storage which occurs only when
> >> the disk is blank, i.e. wiped with nwipe/dban or brand new. It then
> >> fails on creating the LVM; running 'pvcreate' returns 'cannot open
> >> /dev/sda5 exclusively'.
> > this is probably unrelated, but is there any reason to put the LVM PV
> inside
> > a "logical" volume? DOS extended partitions seem to be the worst hack
> ever
> > invented to get around a limitation in a bad design, yet they repeatedly
> > and apparently unnecessarily pop up in quoted disk_configs:
> >
> >> [...]
> >> This is your disk_config file:
> >> # generic disk configuration for one small disk
> >> # disk size from 500Mb up to what you can buy today
> >> #
> >> #   [extra
> options]
> >>
> >> disk_config disk1 disklabel:msdos bootable:1 preserve_lazy:6
> align-at:1M fstabkey:uuid
> >> primary  /boot  300  ext4rw  createopts="-O
> ^64bit,^metadata_csum"
> >> logical  -  29500-3  -   -
> >> logical  /media/daten  1024- ext4acl createopts="-O
> ^64bit,^metadata_csum -L Daten"
> > I count three partitions, which would work perfectly with primary
> partitions
> > (furthermore, you are using LVM to have an arbitrary number of named and
> > dynamic "volumes" (i.e. partitions) anyway, so if you needed more, LVM
> would
> > be the superior mechanism; of course, your specific requirements may
> vary).
> > Ok, you are preserving a logical partition, so in this particular case
> you'd
> > actually need to stick with logical partitions, but the partition in
> question
> > is ext4, not FAT-based, so it doesn't appear to be a legacy Windoze
> issue.
> >
> > My point: am I missing something, and there is some obscure benefit of
> putting
> > an LVM container within an extended-partition-container (such as hiding
> it
> > from something), or is it simply a common misconception that you for some
> > reason cannot or should not put an LVM PV (or even several individual
> native
> > Linux partitions - such as /, /var and /tmp) into primary partitions -
> > assuming you only need upto four of them (and, obviously, assuming you
> are
> > still using MSDOS partition tables)?
> >
> > Or, differently: for a *blank disk*, you obviously won't be preserving
> sda6,
> > and you probably aren't referencing it by partition number
> ("fstabkey:uuid"),
> > so does using 'primary' instead of 'logical' for all three partitions
> change
> > anything concerning the error you are experiencing?
> >
> > Hope that helps someone (perhaps me ;-) ...
> >
> > Regards,
> > Holger
> Hello,
>
> yes, you are right - in some way, this *is* a legacy windows issue that
> has developed over time. In fact, the preserved partition once was a FAT
> partition as long as we had dualboot installations on these machines. We
> finally removed the dualboot two or three years ago and chose to format
> this partition ext4 instead. Why we didn't move to primary partitions or
> put it within the LVM at that time - I don't know.
>
> On the other hand, up to now we had no problems with that, so no urge to
> change anything. If you think it might help, I could try changing this.
>
> Bye,
> Andreas
>

-- 


Notice:  This email is confidential and may contain copyright material of 
members of the Ocado Group. Opinions and views expressed in this message 
may not necessarily reflect the opinions and views of the members of the 
Ocado Group. 

 

If you are not the intended recipient, please notify us immediately and 
delete all copies of this message. Please note that it is your 
responsibility to scan this message for viruses. 

 

Fetch and Sizzle are trading names of Speciality Stores Limited and Fabled 
is a trading name of Marie Claire Beauty Limited, both members of the Ocado 
Group.

 

References to the “Ocado Group” are to Ocado Group plc (registered in 
England and Wales with number 7098618) and its subsidiary undertakings (as 
that expression is defined in the Companies Act 2006) from time to time. 
 The registered office of Ocado Group plc is Buildings One & Two, Trident 
Place, Mosquito Way, Hatfield, Hertfordshire, AL10 9UL.

Re: setup-storage fails on blank disk

[Andreas Heinlein]

Am 03.01.2018 um 17:28 schrieb Holger Parplies:
> Hi,
>
> Andreas Heinlein wrote on 2018-01-03 13:53:40 +0100 [setup-storage fails on 
> blank disk]:
>> [...]
>> I have encountered a problem with setup-storage which occurs only when
>> the disk is blank, i.e. wiped with nwipe/dban or brand new. It then
>> fails on creating the LVM; running 'pvcreate' returns 'cannot open
>> /dev/sda5 exclusively'.
> this is probably unrelated, but is there any reason to put the LVM PV inside
> a "logical" volume? DOS extended partitions seem to be the worst hack ever
> invented to get around a limitation in a bad design, yet they repeatedly
> and apparently unnecessarily pop up in quoted disk_configs:
>
>> [...]
>> This is your disk_config file:
>> # generic disk configuration for one small disk
>> # disk size from 500Mb up to what you can buy today
>> #
>> #   [extra options]
>>
>> disk_config disk1 disklabel:msdos bootable:1 preserve_lazy:6 align-at:1M 
>> fstabkey:uuid 
>> primary  /boot  300  ext4rw  createopts="-O 
>> ^64bit,^metadata_csum"
>> logical  -  29500-3  -   -
>> logical  /media/daten  1024- ext4acl createopts="-O 
>> ^64bit,^metadata_csum -L Daten"
> I count three partitions, which would work perfectly with primary partitions
> (furthermore, you are using LVM to have an arbitrary number of named and
> dynamic "volumes" (i.e. partitions) anyway, so if you needed more, LVM would
> be the superior mechanism; of course, your specific requirements may vary).
> Ok, you are preserving a logical partition, so in this particular case you'd
> actually need to stick with logical partitions, but the partition in question
> is ext4, not FAT-based, so it doesn't appear to be a legacy Windoze issue.
>
> My point: am I missing something, and there is some obscure benefit of putting
> an LVM container within an extended-partition-container (such as hiding it
> from something), or is it simply a common misconception that you for some
> reason cannot or should not put an LVM PV (or even several individual native
> Linux partitions - such as /, /var and /tmp) into primary partitions -
> assuming you only need upto four of them (and, obviously, assuming you are
> still using MSDOS partition tables)?
>
> Or, differently: for a *blank disk*, you obviously won't be preserving sda6,
> and you probably aren't referencing it by partition number ("fstabkey:uuid"),
> so does using 'primary' instead of 'logical' for all three partitions change
> anything concerning the error you are experiencing?
>
> Hope that helps someone (perhaps me ;-) ...
>
> Regards,
> Holger
Hello,

yes, you are right - in some way, this *is* a legacy windows issue that
has developed over time. In fact, the preserved partition once was a FAT
partition as long as we had dualboot installations on these machines. We
finally removed the dualboot two or three years ago and chose to format
this partition ext4 instead. Why we didn't move to primary partitions or
put it within the LVM at that time - I don't know.

On the other hand, up to now we had no problems with that, so no urge to
change anything. If you think it might help, I could try changing this.

Bye,
Andreas