Re: Lockdown adaX numbers to allow booting ?

[Kurt Jaeger]

Hi!

> > > You're probably looking for this:
> > > https://lists.freebsd.org/pipermail/freebsd-fs/2011-March/011036.html
> > 
> > Thanks, that looks like a very useful approach.
> > 
> > Will test that when I'm in the housing facility, and report back.
> 
> I tested it, no change 8-(

Update on this: I did an upgrade to 12.1p1 yesterday and the box
rebooted sucessfully without any manual intervention. The mapping
from /boot/device.hints is ignored, btw (it requested different adaX numbers).

The data pool:

NAMESTATE READ WRITE CKSUM
bck ONLINE   0 0 0
  raidz2-0  ONLINE   0 0 0
ada0ONLINE   0 0 0
ada10   ONLINE   0 0 0
ada2ONLINE   0 0 0
ada3ONLINE   0 0 0
ada13   ONLINE   0 0 0
ada12   ONLINE   0 0 0
ada11   ONLINE   0 0 0
ada1ONLINE   0 0 0

The boot pool:

NAMESTATE READ WRITE CKSUM
zroot   ONLINE   0 0 0
  mirror-0  ONLINE   0 0 0
ada5p3  ONLINE   0 0 0
ada4p3  ONLINE   0 0 0

-- 
p...@opsec.eu+49 171 3101372Now what ?
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Nenhum_de_Nos]

On Sun, October 13, 2019 13:47, Kurt Jaeger wrote:
> Hi!
>
>> > You're probably looking for this:
>> > https://lists.freebsd.org/pipermail/freebsd-fs/2011-March/011036.html
>>
>> Would glabel solve it?
>
> The disks are not gpart-formatted, they are used raw.

mine neither:

root@x:~ # gpart show da0
gpart: No such geom: da0.
root@x:~ # gpart show da1
gpart: No such geom: da1.
root@x:~ # gpart show da2
gpart: No such geom: da2.


glabel status
  Name  Status  Components
   label/160_60224 N/A  da0
label/160_0221 N/A  da1
  label/160_403112 N/A  da2

pool: z160
 state: ONLINE
 config:

NAMESTATE READ WRITE CKSUM
z160ONLINE   0 0 0
  label/160_0221ONLINE   0 0 0
  label/160_403112  ONLINE   0 0 0
  label/160_60224   ONLINE   0 0 0



att,

matheus


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[O'Connor, Daniel]

> On 14 Oct 2019, at 19:08, Kurt Jaeger  wrote:
> 
> Hi!
> 
> You're probably looking for this:
> https://lists.freebsd.org/pipermail/freebsd-fs/2011-March/011036.html
 
 Would glabel solve it?
>>> 
>>> The disks are not gpart-formatted, they are used raw.
>> 
>> What file system are they formatted with?
> 
> ZFS, see
> 
> https://lists.freebsd.org/pipermail/freebsd-current/2019-September/074398.html

Given this:
https://lists.freebsd.org/pipermail/freebsd-current/2019-September/074395.html

I think your problem is far earlier than changing ada probe order will help.

It would seem more likely to be a BIOS or BTX problem :(

I would definitely try moving ada0 and ada1 to the first SATA ports to see if 
that helps.

Does anything show up on the VGA console?

--
Daniel O'Connor
"The nice thing about standards is that there
are so many of them to choose from."
 -- Andrew Tanenbaum


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Kurt Jaeger]

Hi!

> >>> You're probably looking for this:
> >>> https://lists.freebsd.org/pipermail/freebsd-fs/2011-March/011036.html
> >> 
> >> Would glabel solve it?
> > 
> > The disks are not gpart-formatted, they are used raw.
> 
> What file system are they formatted with?

ZFS, see

https://lists.freebsd.org/pipermail/freebsd-current/2019-September/074398.html

-- 
p...@opsec.eu+49 171 3101372One year to go !
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[O'Connor, Daniel]

> On 21 Sep 2019, at 02:36, Garrett Wollman  
> wrote:
> 
> In article <20190920155304.gn3...@zxy.spb.ru>, s...@zxy.spb.ru writes:
> 
>> Location of device in multi-chassis storage system is different story.
>> I am don't know how to field engineer insert disks in chassis.
>> For me simple is find in /var/run/dmesg.boot S/N <=> daXY mapping and
>> turn ON led by sas2ircu.
> 
> sesutil does this for you!
> 
> # sesutil locate daXY on
> # sesutil locate daXY off
> 
> So long as your enclosure supports SES (all the modern ones I've seen
> do) and is enumerable by ses(4).

FWIW I find it doesn't work on the Supermicro chassis (SYS-5019S-MR) we use 
with SATA disks.
[chumphon 2:42] ~> sudo sesutil locate ada0 on
sesutil: Count not find the SES id of device 'ada0'

(Also I just noticed a spelling error in the above, "Count" should be "Could")

[chumphon 2:42] ~> sudo sesutil map
ses0:
Enclosure Name: AHCI SGPIO Enclosure
Enclosure ID:0
Element 0, Type: Array Device Slot
Status: Unsupported (0x00 0x00 0x00 0x00)
Element 1, Type: Array Device Slot
Status: Unknown (0x06 0x00 0x00 0x00)
Description: SLOT 000
Element 2, Type: Array Device Slot
Status: Unknown (0x06 0x00 0x00 0x00)
Description: SLOT 001
Element 3, Type: Array Device Slot
Status: Unknown (0x06 0x00 0x00 0x00)
Description: SLOT 002
Element 4, Type: Array Device Slot
Status: Unknown (0x06 0x00 0x00 0x00)
Description: SLOT 003
Element 5, Type: Array Device Slot
Status: Unknown (0x06 0x00 0x00 0x00)
Description: SLOT 004
Element 6, Type: Array Device Slot
Status: Unknown (0x06 0x00 0x00 0x00)
Description: SLOT 005
Element 7, Type: Array Device Slot
Status: Unknown (0x06 0x00 0x00 0x00)
Description: SLOT 006
Element 8, Type: Array Device Slot
Status: Unknown (0x06 0x00 0x00 0x00)
Description: SLOT 007

Using led(4) works:
echo 1 | sudo tee /dev/led/ahci0.X.fault 
(The X lines up with the "ahcich4" part of dmesg for adaX)
although I find you can only turn the LED on and not off which limits it's 
usefulness somewhat..

--
Daniel O'Connor
"The nice thing about standards is that there
are so many of them to choose from."
 -- Andrew Tanenbaum


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[O'Connor, Daniel]

> On 14 Oct 2019, at 03:17, Kurt Jaeger  wrote:
>>> You're probably looking for this:
>>> https://lists.freebsd.org/pipermail/freebsd-fs/2011-March/011036.html
>> 
>> Would glabel solve it?
> 
> The disks are not gpart-formatted, they are used raw.

What file system are they formatted with?
UFS has both labels and IDs which show up in /dev/ufs and /dev/ufsid 
respectively.

(see the glabel man page)

--
Daniel O'Connor
"The nice thing about standards is that there
are so many of them to choose from."
 -- Andrew Tanenbaum


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Kurt Jaeger]

Hi!

> > You're probably looking for this:
> > https://lists.freebsd.org/pipermail/freebsd-fs/2011-March/011036.html
> 
> Thanks, that looks like a very useful approach.
> 
> Will test that when I'm in the housing facility, and report back.

I tested it, no change 8-(

-- 
p...@opsec.eu+49 171 3101372One year to go !
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Kurt Jaeger]

Hi!

> > You're probably looking for this:
> > https://lists.freebsd.org/pipermail/freebsd-fs/2011-March/011036.html
> 
> Would glabel solve it?

The disks are not gpart-formatted, they are used raw.

-- 
p...@opsec.eu+49 171 3101372One year to go !
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Nenhum_de_Nos]

On Thu, September 19, 2019 17:10, Daniel Engberg wrote:
> Hi Kurt!
>
> You're probably looking for this:
> https://lists.freebsd.org/pipermail/freebsd-fs/2011-March/011036.html

Would glabel solve it?

I have almost as much disks and some zpools and it runs fine.

matheus

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Garrett Wollman]

In article <20190920155304.gn3...@zxy.spb.ru>, s...@zxy.spb.ru writes:

>Location of device in multi-chassis storage system is different story.
>I am don't know how to field engineer insert disks in chassis.
>For me simple is find in /var/run/dmesg.boot S/N <=> daXY mapping and
>turn ON led by sas2ircu.

sesutil does this for you!

# sesutil locate daXY on
# sesutil locate daXY off

So long as your enclosure supports SES (all the modern ones I've seen
do) and is enumerable by ses(4).

-GAWollman

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Slawa Olhovchenkov]

On Fri, Sep 20, 2019 at 01:06:29PM -0400, Garrett Wollman wrote:

> In article <20190920155304.gn3...@zxy.spb.ru>, s...@zxy.spb.ru writes:
> 
> >Location of device in multi-chassis storage system is different story.
> >I am don't know how to field engineer insert disks in chassis.
> >For me simple is find in /var/run/dmesg.boot S/N <=> daXY mapping and
> >turn ON led by sas2ircu.
> 
> sesutil does this for you!
> 
> # sesutil locate daXY on
> # sesutil locate daXY off
> 
> So long as your enclosure supports SES (all the modern ones I've seen
> do) and is enumerable by ses(4).

In theory there is no difference between theory and practice. But, in
practice, there is.

# sesutil map
ses0:
Enclosure Name: AHCI SGPIO Enclosure 
Enclosure ID:0
Element 0, Type: Array Device Slot
Status: Unsupported (0x00 0x00 0x00 0x00)
Element 1, Type: Array Device Slot
Status: Unknown (0x06 0x00 0x00 0x00)
Description: SLOT 000
Element 2, Type: Array Device Slot
Status: Unknown (0x06 0x00 0x00 0x00)
Description: SLOT 001
Element 3, Type: Array Device Slot
Status: Unknown (0x06 0x00 0x00 0x00)
Description: SLOT 002
Element 4, Type: Array Device Slot
Status: Unknown (0x06 0x00 0x00 0x00)
Description: SLOT 003
ses1:
Enclosure Name: AHCI SGPIO Enclosure 
Enclosure ID:0
Element 0, Type: Array Device Slot
Status: Unsupported (0x00 0x00 0x00 0x00)
Element 1, Type: Array Device Slot
Status: Unknown (0x06 0x00 0x00 0x00)
Description: SLOT 000
Element 2, Type: Array Device Slot
Status: Unknown (0x06 0x00 0x00 0x00)
Description: SLOT 001
Element 3, Type: Array Device Slot
Status: Unknown (0x06 0x00 0x00 0x00)
Description: SLOT 002
Element 4, Type: Array Device Slot
Status: Unknown (0x06 0x00 0x00 0x00)
Description: SLOT 003
Element 5, Type: Array Device Slot
Status: Unknown (0x06 0x00 0x00 0x00)
Description: SLOT 004
Element 6, Type: Array Device Slot
Status: Unknown (0x06 0x00 0x00 0x00)
Description: SLOT 005

# sesutil locate ada0 on
sesutil: Count not find the SES id of device 'ada0'
# sesutil locate da0 on
sesutil: Count not find the SES id of device 'da0'
# sas2ircu 0 DISPLAY
LSI Corporation SAS2 IR Configuration Utility.
Version 20.00.00.00 (2014.09.18) 
Copyright (c) 2008-2014 LSI Corporation. All rights reserved. 

Read configuration has been initiated for controller 0

Controller information

  Controller type : SAS2308_1
  BIOS version: 7.39.02.00
  Firmware version: 20.00.07.00
  Channel description : 1 Serial Attached SCSI
  Initiator ID: 0
  Maximum physical devices: 255
  Concurrent commands supported   : 8192
  Slot: 1
  Segment : 0
  Bus : 1
  Device  : 0
  Function: 0
  RAID Support: No

IR Volume information


Physical device information

Initiator at ID #0

Device is a Hard disk
  Enclosure # : 1
  Slot #  : 0
  SAS Address : 4433221-1-0300-
  State   : Ready (RDY)
  Size (in MB)/(in sectors)   : 7630885/15628053167
  Manufacturer: ATA 
  Model Number: TOSHIBA HDWN180 
  Firmware Revision   : GX2M
  Serial No   : 79HSK0PHFAVG
  GUID: N/A
  Protocol: SATA
  Drive Type  : SATA_HDD

Device is a Hard disk
  Enclosure # : 1
  Slot #  : 1
  SAS Address : 4433221-1-0200-
  State   : Ready (RDY)
  Size (in MB)/(in sectors)   : 7630885/15628053167
  Manufacturer: ATA 

Re: Lockdown adaX numbers to allow booting ?

[Slawa Olhovchenkov]

On Fri, Sep 20, 2019 at 08:29:08AM -0700, Freddie Cash wrote:

> On Fri, Sep 20, 2019 at 7:35 AM Slawa Olhovchenkov  wrote:
> 
> > On Thu, Sep 19, 2019 at 06:04:54PM +0200, Michael Gmelin wrote:
> > > What about gpart output of the pool drives?
> > >
> > > In general you would create zpools using gptids or gpt labels, not the
> > devices, so you’re independent of device numbering. The boot loader should
> > only be installed on drives that contain the boot pool (maybe you have old
> > boot loaders on data drives?).
> >
> > ZFS work w/ ZFS labels, not w/ device names/gptids/gpt labels.
> > You don't worry about changed device names aroud reboots.
> >
> 
> Very true, from ZFS' point of view.  It writes a ZFS label to whichever
> GEOM provider you hand it (file, iSCSI device, raw device, MBR partition,
> GPT partition, etc), and it will find it's pool members based on those
> labels.  ZFS doesn't care where the device is physically connected in the
> system, just that it is connected.
> 
> But the ZFS labels aren't what it will display in "zpool list -v" or "zpool
> status" output.  That will show the GEOM provider you gave it (and,
> depending on the order that GEOM tastes the devices, and what's
> enabled/disabled in loader.conf, that output can change).  That's where
> it's useful to have human-readable, descriptive labels (like GPT partition
> labels), and to disable all the GEOM ID systems you won't be using via
> loader.conf.  So that when things go sideways, and a disk dies, you can
> find it quickly and easily.  Much easier to replace "gpt/jbod3-a6" in a
> multi-chassis storage system with 100+ drives than to figure out which bay
> corresponds to "ada73" after a couple of reboots that may or may not have
> changed the PCI bus enumeration direction, or after replacing an HBA that

Location of device in multi-chassis storage system is different story.
I am don't know how to field engineer insert disks in chassis.
For me simple is find in /var/run/dmesg.boot S/N <=> daXY mapping and
turn ON led by sas2ircu.

> enumerates drives a different way (da vs ada), or after a BIOS/EFI upgrade
> that renumbers things, or any other number of situations.  (We've run into
> most of these, and have come to rely on GPT partition labels for just this
> reason; and we stick the drive serial number on the outside of the bay,
> just in case).

> It's not a ZFS requirement.  It just makes things easier for the admin down
> the road. Especially if the admin team changes or inherits systems.  :)

This is need many manual work at setup and build time.
DC field engineer make servers for me and don't do this work for me.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Freddie Cash]

On Fri, Sep 20, 2019 at 7:35 AM Slawa Olhovchenkov  wrote:

> On Thu, Sep 19, 2019 at 06:04:54PM +0200, Michael Gmelin wrote:
> > What about gpart output of the pool drives?
> >
> > In general you would create zpools using gptids or gpt labels, not the
> devices, so you’re independent of device numbering. The boot loader should
> only be installed on drives that contain the boot pool (maybe you have old
> boot loaders on data drives?).
>
> ZFS work w/ ZFS labels, not w/ device names/gptids/gpt labels.
> You don't worry about changed device names aroud reboots.
>

Very true, from ZFS' point of view.  It writes a ZFS label to whichever
GEOM provider you hand it (file, iSCSI device, raw device, MBR partition,
GPT partition, etc), and it will find it's pool members based on those
labels.  ZFS doesn't care where the device is physically connected in the
system, just that it is connected.

But the ZFS labels aren't what it will display in "zpool list -v" or "zpool
status" output.  That will show the GEOM provider you gave it (and,
depending on the order that GEOM tastes the devices, and what's
enabled/disabled in loader.conf, that output can change).  That's where
it's useful to have human-readable, descriptive labels (like GPT partition
labels), and to disable all the GEOM ID systems you won't be using via
loader.conf.  So that when things go sideways, and a disk dies, you can
find it quickly and easily.  Much easier to replace "gpt/jbod3-a6" in a
multi-chassis storage system with 100+ drives than to figure out which bay
corresponds to "ada73" after a couple of reboots that may or may not have
changed the PCI bus enumeration direction, or after replacing an HBA that
enumerates drives a different way (da vs ada), or after a BIOS/EFI upgrade
that renumbers things, or any other number of situations.  (We've run into
most of these, and have come to rely on GPT partition labels for just this
reason; and we stick the drive serial number on the outside of the bay,
just in case).

It's not a ZFS requirement.  It just makes things easier for the admin down
the road. Especially if the admin team changes or inherits systems.  :)

-- 
Freddie Cash
fjwc...@gmail.com
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Mark Martinec]

Kurt Jaeger writes:


The problem is that if all 10 disks are connected, the system
looses track from where it should boot and fails to boot (serial boot 
log):


Consoles: internal video/keyboard  serial port
BTX loader 1.00  BTX version is 1.02
Consoles: internal video/keyboard  serial port
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS drive G: is disk4
BIOS drive H: is disk5
BIOS drive I: is disk6
BIOS drive J: is disk7
BIOS drive K: is disk8
BIOS drive L: is disk9
//
[...]
The solution right now is this to unplug all disks of the 'bck' pool,
reboot, and re-insert the data disks after the boot is finished.
[...]
No gpart on the bck pool, raw drives.



This sounds very much like my experience:

  2018-11-29, Boot loader stuck after first stage upgrading 11.2 to 
12.0-RC2

https://lists.freebsd.org/pipermail/freebsd-stable/2018-November/090129.html

https://lists.freebsd.org/pipermail/freebsd-stable/2018-December/090159.html


I now have three SuperMicro machines which are unable to boot after
upgrading 11.2 to 12.0. After unsuccessfully fiddling with boot loaders,
I have reverted two back to 11.2 (which boots and works fine again),
and the third one is now at 12.0 but needs the boot hack as described
by Kurt, i.e. pull out half the disks (of the 'data' pool), boot the
system, plug the disks back in and zfs mount the remaining pool.

Considering that the 11.2 boots and works fine on these machines,
I consider it a btx loader failure and not a BIOS issue.

What is common with these three machines is that they have one pool
on raw devices for historical reasons (not on gpt partitions).
My guess is that the new loader gets confused by these raw disks.

  Mark
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Slawa Olhovchenkov]

On Thu, Sep 19, 2019 at 06:04:54PM +0200, Michael Gmelin wrote:

> 
> 
> > On 19. Sep 2019, at 17:57, Kurt Jaeger  wrote:
> > 
> > Hi!
> > 
> >>> We have a system with 10 SATA disks. 2 disks are for the system,
> >>> 8 disks drive a data pool 'bck', configured as raidz2, for backup 
> >>> purposes:
> >>> 
> >>> bck72.8T  38.7T  34.1T- - 1%53%  1.00x  
> >>> ONLINE  -
> > 
> >>> The problem is that if all 10 disks are connected, the system
> >>> looses track from where it should boot and fails to boot (serial boot 
> >>> log):
> > 
> >> Why this order does change?  One would expect disks 0 and 1 to be OS disks 
> >> and the rest for data???
> > 
> > 0+1 are 2.5", and the initial setup was:
> > - we installed system disks as zroot 
> > - shipped the box to the housing facility
> > - booted and added the drives
> > 
> > At that time we did not do additional tests about the disk/boot sequence
> > etc.
> > 
> >> Also the question is, what you mean with ???system looses track
> > 
> > I interpret the hang during boot as 'it looses track'. So I guess
> > it tries to read the kernel from the wrong drives.
> > 
> >> disk4 becomes adaX? why it matters, are you using ufs on boot disks?
> > 
> > No, zpool only.
> > 
> > I've made a few more details available here:
> > 
> > https://people.freebsd.org/~pi/host/dmesg.txt
> > https://people.freebsd.org/~pi/host/devlist.txt
> > https://people.freebsd.org/~pi/host/gpart.txt
> > https://people.freebsd.org/~pi/host/pciconf.txt
> > https://people.freebsd.org/~pi/host/zpool.txt
> 
> What about gpart output of the pool drives?
> 
> In general you would create zpools using gptids or gpt labels, not the 
> devices, so you’re independent of device numbering. The boot loader should 
> only be installed on drives that contain the boot pool (maybe you have old 
> boot loaders on data drives?).

ZFS work w/ ZFS labels, not w/ device names/gptids/gpt labels.
You don't worry about changed device names aroud reboots.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Kurt Jaeger]

Hi!

> You're probably looking for this:
> https://lists.freebsd.org/pipermail/freebsd-fs/2011-March/011036.html

Thanks, that looks like a very useful approach.

Will test that when I'm in the housing facility, and report back.

-- 
p...@opsec.eu+49 171 3101372One year to go !
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Freddie Cash]

On Thu, Sep 19, 2019 at 12:01 PM Andreas Nilsson  wrote:

> Seems like more of a bios enumeration issue. You should be able to set a
> boot order better suited for your setup there. And if that does not work,
> just move the sata cables around seems like the most straight forward
> solution.
>
> Although I think I've heard it is bad practice to use raw devices for zfs,
> especially if need to replace a drive, which day happens to be a different
> revision, with a few fewer blocks available. Then you will not be able to
> do the replace.
>
>
Back in the good ol' days of ZFS versions where everyone was compatible
with Solaris, this was an issue.  However, the ondisk format (or ZFS label
setup?) was changed to leave 1 MB of free space at the end of the drive to
allow for this.

With ZFSv6, for example, if you used a raw device for the vdev, and that
disk died, and the replacement was 1 sector smaller, the replace would
fail.  Today, with OpenZFS, the replace would succeed.

There's also 1 MB or so of reserved space in the pool such that if you fill
the pool "100%", you can still do a "zfs destroy" of a dataset to free up
space.  Previously, this would fail, as you need space in the pool to write
the metadata for the destroy before doing the destroy.

ZFS of today is much more resilient to these kinds of niggles that bit us
all, back in the day.  :D

-- 
Freddie Cash
fjwc...@gmail.com
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Daniel Engberg]

Hi Kurt!

You're probably looking for this: 
https://lists.freebsd.org/pipermail/freebsd-fs/2011-March/011036.html


Best regards,
Daniel
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Toomas Soome]

> On 19 Sep 2019, at 21:48, Kurt Jaeger  wrote:
> 
> Hi!
> 
 Also the question is, what you mean with ???system looses track
> 
>>> I interpret the hang during boot as 'it looses track'. So I guess
>>> it tries to read the kernel from the wrong drives.
> 
>> no, loader does probe disks to see which devices make up the pool and hung 
>> system is not about reading the kernel from wrong place.
> 
>> I bet it is BIOS system?
> 
> If you mean: no UEFI boot ? Yes, it boots via BIOS, not UEFI.
> 
>> Since the raidz seems to be created partitionless, what version
>> of freebsd are you using?
> 
> FreeBSD 12.0p10, amd64.
> 
>> BIOS up to date?
> 
> The board is an X10SRi-F. dmidecode reports BIOS 3.1, supermicro
> has 3.1c, but the release notes do not mention things like that
> (or at least I don't understand them that way).
> 
> https://www.supermicro.com/en/products/motherboard/X10SRi-F
> 
>> can you test pool visibility in loader with latest current
>> usb/cd start - like press esc in menu and check lsdev -v
>> (assuming you get to menu) ? the same with uefi?
> 
> I will test this when I'm at the facility. Will take some time.
> 

Those data disks are 10TB 512e, I can not recall if loader in 12.0p10 does 
ignore partitionless disks in pools or not, if the bios is buggy about large 
disks, that does explain the hung system… in that case, the loader in current 
should be fine.

rgds,
toomas

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Andreas Nilsson]

On Thu, Sep 19, 2019, 20:56 Michael Gmelin  wrote:

>
>
> On 19. Sep 2019, at 19:15, Kurt Jaeger  wrote:
>
> >>> I've made a few more details available here:
> >
> >>> https://people.freebsd.org/~pi/host/gpart.txt
> >
> >> What about gpart output of the pool drives?
> >
> > No gpart on the bck pool, raw drives.
> >
> >> In general you would create zpools using gptids or gpt labels, not the
> devices, so you???re independent of device numbering. The boot loader
> should only be installed on drives that contain the boot pool (maybe you
> have old boot loaders on data drives?).
> >
> > I think not, because they are used as raw drives.
> >
> > Maybe that decision was an error in hindsight.
>
> Yeah, it’s not optimal that way. I made it a habit to use GPT on all pools
> and label partitions with the enclosure slots they’re in (makes it easier
> to not make mistakes in case of emergency). I also leave a bit of space at
> the beginning and end of the drive (allows adding in a boot partition later
> or more flexibility when replacing the drive).
>
> Anyway, I’m curious what the exact problem will turn out to be.
>
> Cheers,
> Michael
>
> >
> > --
> > p...@opsec.eu+49 171 3101372One year to
> go !
>
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
>

Seems like more of a bios enumeration issue. You should be able to set a
boot order better suited for your setup there. And if that does not work,
just move the sata cables around seems like the most straight forward
solution.

Although I think I've heard it is bad practice to use raw devices for zfs,
especially if need to replace a drive, which day happens to be a different
revision, with a few fewer blocks available. Then you will not be able to
do the replace.

Best regards
Andreas

>
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Michael Gmelin]

On 19. Sep 2019, at 19:15, Kurt Jaeger  wrote:

>>> I've made a few more details available here:
> 
>>> https://people.freebsd.org/~pi/host/gpart.txt
> 
>> What about gpart output of the pool drives?
> 
> No gpart on the bck pool, raw drives.
> 
>> In general you would create zpools using gptids or gpt labels, not the 
>> devices, so you???re independent of device numbering. The boot loader should 
>> only be installed on drives that contain the boot pool (maybe you have old 
>> boot loaders on data drives?).
> 
> I think not, because they are used as raw drives.
> 
> Maybe that decision was an error in hindsight.

Yeah, it’s not optimal that way. I made it a habit to use GPT on all pools and 
label partitions with the enclosure slots they’re in (makes it easier to not 
make mistakes in case of emergency). I also leave a bit of space at the 
beginning and end of the drive (allows adding in a boot partition later or more 
flexibility when replacing the drive).

Anyway, I’m curious what the exact problem will turn out to be.

Cheers,
Michael

> 
> -- 
> p...@opsec.eu+49 171 3101372One year to go !

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Kurt Jaeger]

Hi!

> >> Also the question is, what you mean with ???system looses track

> > I interpret the hang during boot as 'it looses track'. So I guess
> > it tries to read the kernel from the wrong drives.

> no, loader does probe disks to see which devices make up the pool and hung 
> system is not about reading the kernel from wrong place.

> I bet it is BIOS system?

If you mean: no UEFI boot ? Yes, it boots via BIOS, not UEFI.

> Since the raidz seems to be created partitionless, what version
> of freebsd are you using?

FreeBSD 12.0p10, amd64.

> BIOS up to date?

The board is an X10SRi-F. dmidecode reports BIOS 3.1, supermicro
has 3.1c, but the release notes do not mention things like that
(or at least I don't understand them that way).

https://www.supermicro.com/en/products/motherboard/X10SRi-F

> can you test pool visibility in loader with latest current
> usb/cd start - like press esc in menu and check lsdev -v
> (assuming you get to menu) ? the same with uefi?

I will test this when I'm at the facility. Will take some time.

Thanks for the pointers!

-- 
p...@opsec.eu+49 171 3101372One year to go !
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Kurt Jaeger]

> > I've made a few more details available here:

> > https://people.freebsd.org/~pi/host/gpart.txt

> What about gpart output of the pool drives?

No gpart on the bck pool, raw drives.

> In general you would create zpools using gptids or gpt labels, not the 
> devices, so you???re independent of device numbering. The boot loader should 
> only be installed on drives that contain the boot pool (maybe you have old 
> boot loaders on data drives?).

I think not, because they are used as raw drives.

Maybe that decision was an error in hindsight.

-- 
p...@opsec.eu+49 171 3101372One year to go !
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Guido Falsi]

On 19/09/19 18:54, Freddie Cash wrote:
> On Thu, Sep 19, 2019 at 9:47 AM Guido Falsi  > wrote:
> 
> On 19/09/19 18:04, Michael Gmelin wrote:
> >
> > What about gpart output of the pool drives?
> >
> > In general you would create zpools using gptids or gpt labels, not
> the devices, so you’re independent of device numbering. The boot
> loader should only be installed on drives that contain the boot pool
> (maybe you have old boot loaders on data drives?).
> >
> 
> Actually the installer will at least some times use the adaX device to
> create ZFS pools. At least it did for me when I recently rebuilt a
> machine after a (multiple) disk crash.
> 
> So it could not be Kurt fault if he has a pool with adaX devices in it.
> 
> I installed the system on one disk and the installed used adaX to create
> the pool. I added the second disk to the mirror a few days later.
> 
> Now I have:
> 
>   pool: zroot
>  state: ONLINE
>   scan: resilvered 24.6G in 0 days 00:09:41 with 0 errors on Tue Sep  3
> 16:10:08 2019
> config:
> 
>         NAME          STATE     READ WRITE CKSUM
>         zroot         ONLINE       0     0     0
>           mirror-0    ONLINE       0     0     0
>             ada0p4    ONLINE       0     0     0
>             gpt/zfs1  ONLINE       0     0     0
> 
> errors: No known data errors
> 
> Now, in the case of a mirror (or a zraid) this could be fixed by
> detaching adaX devices and reattaching them using the label. Disvantage
> is the cluster will need to resilver, causing some degraded time and
> extra disk load.
> 
> 
> Boot off a LiveCD/USB stick.  Export the pool.  Import the pool with -d
> /dev/gpt and it will use the GPT labels instead.  Reboot into the
> system, and it should continue to use the GPT labels.  No resilvering
> required. 

Good suggestion. I did not think about it. I'll try that once I can
reboot the machine.

-- 
Guido Falsi 
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Gary Jennejohn]

On Thu, 19 Sep 2019 18:04:54 +0200
Michael Gmelin  wrote:

> > On 19. Sep 2019, at 17:57, Kurt Jaeger  wrote:
> > 
> > Hi!
> >   
> >>> We have a system with 10 SATA disks. 2 disks are for the system,
> >>> 8 disks drive a data pool 'bck', configured as raidz2, for backup 
> >>> purposes:
> >>> 
> >>> bck72.8T  38.7T  34.1T- - 1%53%  1.00x  
> >>> ONLINE  -  
> >   
> >>> The problem is that if all 10 disks are connected, the system
> >>> looses track from where it should boot and fails to boot (serial boot 
> >>> log):  
> >   
> >> Why this order does change?  One would expect disks 0 and 1 to be OS disks 
> >> and the rest for data???  
> > 
> > 0+1 are 2.5", and the initial setup was:
> > - we installed system disks as zroot 
> > - shipped the box to the housing facility
> > - booted and added the drives
> > 
> > At that time we did not do additional tests about the disk/boot sequence
> > etc.
> >   
> >> Also the question is, what you mean with ???system looses track  
> > 
> > I interpret the hang during boot as 'it looses track'. So I guess
> > it tries to read the kernel from the wrong drives.
> >   
> >> disk4 becomes adaX? why it matters, are you using ufs on boot disks?  
> > 
> > No, zpool only.
> > 
> > I've made a few more details available here:
> > 
> > https://people.freebsd.org/~pi/host/dmesg.txt
> > https://people.freebsd.org/~pi/host/devlist.txt
> > https://people.freebsd.org/~pi/host/gpart.txt
> > https://people.freebsd.org/~pi/host/pciconf.txt
> > https://people.freebsd.org/~pi/host/zpool.txt  
> 
> What about gpart output of the pool drives?
> 
> In general you would create zpools using gptids or gpt labels,
> not the devices, so you___re independent of device numbering. 
> The boot loader should only be installed on drives that contain
> the boot pool (maybe you have old boot loaders on data
> drives?).
> 

Maybe the cam(4) hint settings could help in your case.

-- 
Gary Jennejohn
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Freddie Cash]

On Thu, Sep 19, 2019 at 9:47 AM Guido Falsi  wrote:

> On 19/09/19 18:04, Michael Gmelin wrote:
> >
> > What about gpart output of the pool drives?
> >
> > In general you would create zpools using gptids or gpt labels, not the
> devices, so you’re independent of device numbering. The boot loader should
> only be installed on drives that contain the boot pool (maybe you have old
> boot loaders on data drives?).
> >
>
> Actually the installer will at least some times use the adaX device to
> create ZFS pools. At least it did for me when I recently rebuilt a
> machine after a (multiple) disk crash.
>
> So it could not be Kurt fault if he has a pool with adaX devices in it.
>
> I installed the system on one disk and the installed used adaX to create
> the pool. I added the second disk to the mirror a few days later.
>
> Now I have:
>
>   pool: zroot
>  state: ONLINE
>   scan: resilvered 24.6G in 0 days 00:09:41 with 0 errors on Tue Sep  3
> 16:10:08 2019
> config:
>
> NAME  STATE READ WRITE CKSUM
> zroot ONLINE   0 0 0
>   mirror-0ONLINE   0 0 0
> ada0p4ONLINE   0 0 0
> gpt/zfs1  ONLINE   0 0 0
>
> errors: No known data errors
>
> Now, in the case of a mirror (or a zraid) this could be fixed by
> detaching adaX devices and reattaching them using the label. Disvantage
> is the cluster will need to resilver, causing some degraded time and
> extra disk load.
>

Boot off a LiveCD/USB stick.  Export the pool.  Import the pool with -d
/dev/gpt and it will use the GPT labels instead.  Reboot into the system,
and it should continue to use the GPT labels.  No resilvering required.

-- 
Freddie Cash
fjwc...@gmail.com
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Guido Falsi]

On 19/09/19 18:04, Michael Gmelin wrote:
> 
> What about gpart output of the pool drives?
> 
> In general you would create zpools using gptids or gpt labels, not the 
> devices, so you’re independent of device numbering. The boot loader should 
> only be installed on drives that contain the boot pool (maybe you have old 
> boot loaders on data drives?).
> 

Actually the installer will at least some times use the adaX device to
create ZFS pools. At least it did for me when I recently rebuilt a
machine after a (multiple) disk crash.

So it could not be Kurt fault if he has a pool with adaX devices in it.

I installed the system on one disk and the installed used adaX to create
the pool. I added the second disk to the mirror a few days later.

Now I have:

  pool: zroot
 state: ONLINE
  scan: resilvered 24.6G in 0 days 00:09:41 with 0 errors on Tue Sep  3
16:10:08 2019
config:

NAME  STATE READ WRITE CKSUM
zroot ONLINE   0 0 0
  mirror-0ONLINE   0 0 0
ada0p4ONLINE   0 0 0
gpt/zfs1  ONLINE   0 0 0

errors: No known data errors

Now, in the case of a mirror (or a zraid) this could be fixed by
detaching adaX devices and reattaching them using the label. Disvantage
is the cluster will need to resilver, causing some degraded time and
extra disk load.

-- 
Guido Falsi 
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Toomas Soome]

> On 19 Sep 2019, at 18:57, Kurt Jaeger  wrote:
> 
> Hi!
> 
>>> We have a system with 10 SATA disks. 2 disks are for the system,
>>> 8 disks drive a data pool 'bck', configured as raidz2, for backup purposes:
>>> 
>>> bck72.8T  38.7T  34.1T- - 1%53%  1.00x  ONLINE  
>>> -
> 
>>> The problem is that if all 10 disks are connected, the system
>>> looses track from where it should boot and fails to boot (serial boot log):
> 
>> Why this order does change?  One would expect disks 0 and 1 to be OS disks 
>> and the rest for data???
> 
> 0+1 are 2.5", and the initial setup was:
> - we installed system disks as zroot 
> - shipped the box to the housing facility
> - booted and added the drives
> 
> At that time we did not do additional tests about the disk/boot sequence
> etc.
> 
>> Also the question is, what you mean with ???system looses track
> 
> I interpret the hang during boot as 'it looses track'. So I guess
> it tries to read the kernel from the wrong drives.


no, loader does probe disks to see which devices make up the pool and hung 
system is not about reading the kernel from wrong place.

I bet it is BIOS system? Since the raidz seems to be created partitionless, 
what version of freebsd are you using? BIOS up to date? can you test pool 
visibility in loader with  latest current usb/cd start - like press esc in menu 
and check lasdev -v (assuming you get to menu)… the same with uefi?

rgds,
toomas


> 
>> disk4 becomes adaX? why it matters, are you using ufs on boot disks?
> 
> No, zpool only.
> 
> I've made a few more details available here:
> 
> https://people.freebsd.org/~pi/host/dmesg.txt
> https://people.freebsd.org/~pi/host/devlist.txt
> https://people.freebsd.org/~pi/host/gpart.txt
> https://people.freebsd.org/~pi/host/pciconf.txt
> https://people.freebsd.org/~pi/host/zpool.txt
> 
> -- 
> p...@opsec.eu+49 171 3101372One year to go !
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Michael Gmelin]

> On 19. Sep 2019, at 17:57, Kurt Jaeger  wrote:
> 
> Hi!
> 
>>> We have a system with 10 SATA disks. 2 disks are for the system,
>>> 8 disks drive a data pool 'bck', configured as raidz2, for backup purposes:
>>> 
>>> bck72.8T  38.7T  34.1T- - 1%53%  1.00x  ONLINE  
>>> -
> 
>>> The problem is that if all 10 disks are connected, the system
>>> looses track from where it should boot and fails to boot (serial boot log):
> 
>> Why this order does change?  One would expect disks 0 and 1 to be OS disks 
>> and the rest for data???
> 
> 0+1 are 2.5", and the initial setup was:
> - we installed system disks as zroot 
> - shipped the box to the housing facility
> - booted and added the drives
> 
> At that time we did not do additional tests about the disk/boot sequence
> etc.
> 
>> Also the question is, what you mean with ???system looses track
> 
> I interpret the hang during boot as 'it looses track'. So I guess
> it tries to read the kernel from the wrong drives.
> 
>> disk4 becomes adaX? why it matters, are you using ufs on boot disks?
> 
> No, zpool only.
> 
> I've made a few more details available here:
> 
> https://people.freebsd.org/~pi/host/dmesg.txt
> https://people.freebsd.org/~pi/host/devlist.txt
> https://people.freebsd.org/~pi/host/gpart.txt
> https://people.freebsd.org/~pi/host/pciconf.txt
> https://people.freebsd.org/~pi/host/zpool.txt

What about gpart output of the pool drives?

In general you would create zpools using gptids or gpt labels, not the devices, 
so you’re independent of device numbering. The boot loader should only be 
installed on drives that contain the boot pool (maybe you have old boot loaders 
on data drives?).

-m


> -- 
> p...@opsec.eu+49 171 3101372One year to go !
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Kurt Jaeger]

Hi!

> > We have a system with 10 SATA disks. 2 disks are for the system,
> > 8 disks drive a data pool 'bck', configured as raidz2, for backup purposes:
> > 
> > bck72.8T  38.7T  34.1T- - 1%53%  1.00x  ONLINE  
> > -

> > The problem is that if all 10 disks are connected, the system
> > looses track from where it should boot and fails to boot (serial boot log):

> Why this order does change?  One would expect disks 0 and 1 to be OS disks 
> and the rest for data???

0+1 are 2.5", and the initial setup was:
- we installed system disks as zroot 
- shipped the box to the housing facility
- booted and added the drives

At that time we did not do additional tests about the disk/boot sequence
etc.

> Also the question is, what you mean with ???system looses track

I interpret the hang during boot as 'it looses track'. So I guess
it tries to read the kernel from the wrong drives.

> disk4 becomes adaX? why it matters, are you using ufs on boot disks?

No, zpool only.

I've made a few more details available here:

https://people.freebsd.org/~pi/host/dmesg.txt
https://people.freebsd.org/~pi/host/devlist.txt
https://people.freebsd.org/~pi/host/gpart.txt
https://people.freebsd.org/~pi/host/pciconf.txt
https://people.freebsd.org/~pi/host/zpool.txt

-- 
p...@opsec.eu+49 171 3101372One year to go !
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Lockdown adaX numbers to allow booting ?

[Toomas Soome]

> On 19 Sep 2019, at 17:02, Kurt Jaeger  wrote:
> 
> Hi!
> 
> We have a system with 10 SATA disks. 2 disks are for the system,
> 8 disks drive a data pool 'bck', configured as raidz2, for backup purposes:
> 
> bck72.8T  38.7T  34.1T- - 1%53%  1.00x  ONLINE  -
> 
> The problem is that if all 10 disks are connected, the system
> looses track from where it should boot and fails to boot (serial boot log):
> 
> 
> /boot/config: -Dh -S115200
> 
> Consoles: internal video/keyboard  serial port
> BTX loader 1.00  BTX version is 1.02
> Consoles: internal video/keyboard  serial port
> BIOS drive C: is disk0
> BIOS drive D: is disk1
> BIOS drive E: is disk2
> BIOS drive F: is disk3
> BIOS drive G: is disk4
> BIOS drive H: is disk5
> BIOS drive I: is disk6
> BIOS drive J: is disk7
> BIOS drive K: is disk8
> BIOS drive L: is disk9
> //
> /
> 
> 
> The system disks are detected as ada4 and ada5, when all disks are
> plugged in.
> 
> The solution right now is this to unplug all disks of the 'bck' pool,
> reboot, and re-insert the data disks after the boot is finished.
> 
> I looked into the output to kenv(1), but did not find inspiration
> on how to fix this.
> 
> Now my questions:
> 
> - Shuffeling around SATA cables seems the wrong approach to fix this.
> - Can we somehow lock down the disk numbering so that the system disks
>  are detected as ada0 and ada1 ?


Why this order does change?  One would expect disks 0 and 1 to be OS disks and 
the rest for data…

Also the question is, what you mean with “system looses track”? disk4 becomes 
adaX? why it matters, are you using ufs on boot disks?

rgds,
toomas


> - Would
>  rootdev="disk4s1a"
>  in /boot/loader.conf work or is that the wrong approach ?
> - How could we configure two drivers as root devices in loader.conf ?
> 
> -- 
> p...@opsec.eu+49 171 3101372One year to go !
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Lockdown adaX numbers to allow booting ?

[Kurt Jaeger]

Hi!

We have a system with 10 SATA disks. 2 disks are for the system,
8 disks drive a data pool 'bck', configured as raidz2, for backup purposes:

bck72.8T  38.7T  34.1T- - 1%53%  1.00x  ONLINE  -

The problem is that if all 10 disks are connected, the system
looses track from where it should boot and fails to boot (serial boot log):


/boot/config: -Dh -S115200

Consoles: internal video/keyboard  serial port
BTX loader 1.00  BTX version is 1.02
Consoles: internal video/keyboard  serial port
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS drive G: is disk4
BIOS drive H: is disk5
BIOS drive I: is disk6
BIOS drive J: is disk7
BIOS drive K: is disk8
BIOS drive L: is disk9
//
/


The system disks are detected as ada4 and ada5, when all disks are
plugged in.

The solution right now is this to unplug all disks of the 'bck' pool,
reboot, and re-insert the data disks after the boot is finished.

I looked into the output to kenv(1), but did not find inspiration
on how to fix this.

Now my questions:

- Shuffeling around SATA cables seems the wrong approach to fix this.
- Can we somehow lock down the disk numbering so that the system disks
  are detected as ada0 and ada1 ?
- Would
  rootdev="disk4s1a"
  in /boot/loader.conf work or is that the wrong approach ?
- How could we configure two drivers as root devices in loader.conf ?

-- 
p...@opsec.eu+49 171 3101372One year to go !
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"