Bug#867898: debian-installer-netboot-images: debian-installer-9-netboot-amd64 scsi modules missing. Netboot image unusable

[Marc N]

Hi Robert,

After digging a bit, I think my issue lies with the APT source I used.
At work (ESXi host), netboot works because it points to a proper apt-
mirror.

On the other hand, at home I frankenbuilt mine using the DVD iso and 
specifying debian-installer/allow_unauthenticated=true. I retried after 
pointing the APT mirror to, say, deb.debian.org, and everything went back 
to normal.

So my advice is, can you check if your source http://spacewalk-
test.swr.ard/ks/dist/org/1/Debian_Stretch/ was done properly?


For reference, here were results for booting from borrowed initrd from DVD.
In those tests, /dev/sda is successfully listed.

in dmesg, after bsg has loaded:

- lsilogic: 
Fusion MPT base driver 3.04.20
Fusion MPT SPI Host driver 3.04.20
- lsisas1068:
Fusion MPT base driver 3.04.20
Fusion MPT SAS Host driver 3.04.20
- pvscsi:
VMware PVSCSI driver - version 1.0.7.0-k

/proc/modules yields a few ata modules, and depending on scsi controller in 
vmx:
- lsilogic: mptspi, scsi_transport_spi, mptscsih, mptbase, scsi_mod
- lsisas1068: mptsas, scsi_transport_sas, mptscsih, mptbase, scsi_mod
- pvscsi: vmw_pcscsi, scsi_mod
- ide0 vmdk attachment (removed SCSI controller): scsi_mod

Mit freundlichen gruessen,
Marc


On Wed, Aug 2, 2017 at 1:42 PM, Paschedag, Robert 
<paschedag.netlut...@swr.de> wrote:
> Hi Marc,
>
> which driver module has been used within your tests? In my version of 
netboot.tar.gz (initrd), the modules for LSI controllers (mptspi) were 
missing.
>
> So even on a simple VMware VM, no disks have been detected.
>
> Regards,
> Robert
>
> -Ursprüngliche Nachricht-
> Von: Marc N [mailto:de...@rasqual.e4ward.com]
> Gesendet: Mittwoch, 2. August 2017 13:38
> An: Steve McIntyre <st...@einval.com>; Cyril Brulebois <k...@debian.org>; 
867...@bugs.debian.org
> Cc: Paschedag, Robert <paschedag.netlut...@swr.de>; debian-
ker...@lists.debian.org; debian...@lists.debian.org
> Betreff: Re: Bug#867898: debian-installer-netboot-images: debian-
installer-9-netboot-amd64 scsi modules missing. Netboot image unusable
>
> Hello all,
>
> Steve, what is the suggested resolution then?
> For testing, I booted the initrd's and kernels extracted from standard 
DVD and netboot iso's via PXE. While they cannot progress further for 
obvious reasons, by switching to tty2, /dev/sda does appear in the listing. 
Those do ship with the correct modules.
>
> 1/ What would be the negative aspect of removing the scsi modules from 
the
> exclude-udeb-* lists, following the package mergers mentioned before?
>
> 2/ Also, I'm not too familiar with how the netboot is constructed, is 
there another way to have these drivers bundled in the netboot tar.gz 
(CONF.sh?)?
>
> Best regards,
> Marc
>

Bug#867898: AW: Bug#867898: debian-installer-netboot-images: debian-installer-9-netboot-amd64 scsi modules missing. Netboot image unusable

[Paschedag, Robert]

Hi Marc,

which driver module has been used within your tests? In my version of 
netboot.tar.gz (initrd), the modules for LSI controllers (mptspi) were missing.

So even on a simple VMware VM, no disks have been detected.

Regards,
Robert

-Ursprüngliche Nachricht-
Von: Marc N [mailto:de...@rasqual.e4ward.com] 
Gesendet: Mittwoch, 2. August 2017 13:38
An: Steve McIntyre <st...@einval.com>; Cyril Brulebois <k...@debian.org>; 
867...@bugs.debian.org
Cc: Paschedag, Robert <paschedag.netlut...@swr.de>; 
debian-ker...@lists.debian.org; debian...@lists.debian.org
Betreff: Re: Bug#867898: debian-installer-netboot-images: 
debian-installer-9-netboot-amd64 scsi modules missing. Netboot image unusable

Hello all,

Steve, what is the suggested resolution then?
For testing, I booted the initrd's and kernels extracted from standard DVD and 
netboot iso's via PXE. While they cannot progress further for obvious reasons, 
by switching to tty2, /dev/sda does appear in the listing. Those do ship with 
the correct modules.

1/ What would be the negative aspect of removing the scsi modules from the
exclude-udeb-* lists, following the package mergers mentioned before?

2/ Also, I'm not too familiar with how the netboot is constructed, is there 
another way to have these drivers bundled in the netboot tar.gz (CONF.sh?)?

Best regards,
Marc

Bug#867898: debian-installer-netboot-images: debian-installer-9-netboot-amd64 scsi modules missing. Netboot image unusable

[Marc N]

Hello all,

Steve, what is the suggested resolution then?
For testing, I booted the initrd's and kernels extracted from standard DVD 
and netboot iso's via PXE. While they cannot progress further for obvious 
reasons, by switching to tty2, /dev/sda does appear in the listing. Those 
do ship with the correct modules.

1/ What would be the negative aspect of removing the scsi modules from the 
exclude-udeb-* lists, following the package mergers mentioned before?

2/ Also, I'm not too familiar with how the netboot is constructed, is there 
another way to have these drivers bundled in the netboot tar.gz (CONF.sh?)?

Best regards,
Marc

Bug#867898: AW: Bug#867898: debian-installer-netboot-images: debian-installer-9-netboot-amd64 scsi modules missing. Netboot image unusable

[Paschedag, Robert]

Ah...that was a good hint the the "exclude" list.

Now I got it. In "jessie", the modules I - currently - need, have been put into 
scsi-extra-modules

[root@server debian-jessie]# head -2 README.txt
Debian GNU/Linux 8.2.0 "Jessie" - Official amd64 DVD Binary-1
   20150906-11:13
[root@server debian-jessie]# find . -iname "scsi-*.udeb" -type f
./pool/main/l/linux/scsi-extra-modules-3.16.0-4-amd64-di_3.16.7-ckt11-1+deb8u3_amd64.udeb
[root@server debian-jessie]#

root@server# ar -xf 
scsi-extra-modules-3.16.0-4-amd64-di_3.16.7-ckt11-1+deb8u3_amd64.udeb 
data.tar.xz
root@server# tar -tJf data.tar.xz |grep mpt
./lib/modules/3.16.0-4-amd64/kernel/drivers/scsi/mpt3sas/
./lib/modules/3.16.0-4-amd64/kernel/drivers/scsi/mpt3sas/mpt3sas.ko
./lib/modules/3.16.0-4-amd64/kernel/drivers/scsi/mpt2sas/
./lib/modules/3.16.0-4-amd64/kernel/drivers/scsi/mpt2sas/mpt2sas.ko
./lib/modules/3.16.0-4-amd64/kernel/drivers/message/fusion/mptspi.ko
./lib/modules/3.16.0-4-amd64/kernel/drivers/message/fusion/mptscsih.ko
./lib/modules/3.16.0-4-amd64/kernel/drivers/message/fusion/mptsas.ko
./lib/modules/3.16.0-4-amd64/kernel/drivers/message/fusion/mptfc.ko
./lib/modules/3.16.0-4-amd64/kernel/drivers/message/fusion/mptbase.ko
root@server#

Soas these modules have been "moved" to scsi-modules (see 
https://www.debian.org/devel/debian-installer/News/2016/20160521), but 
scsi-modules is "excluded" forever, this causes my error.

Isn't it?

Regards,
Robert


Von: Steve McIntyre <st...@einval.com>
Gesendet: Donnerstag, 13. Juli 2017 02:36
An: Cyril Brulebois
Cc: Paschedag, Robert; 867...@bugs.debian.org; debian-ker...@lists.debian.org; 
debian...@lists.debian.org
Betreff: Re: Bug#867898: debian-installer-netboot-images: 
debian-installer-9-netboot-amd64 scsi modules missing. Netboot image unusable

On Thu, Jul 13, 2017 at 01:09:58AM +0100, Steve McIntyre wrote:
>On Thu, Jul 13, 2017 at 01:50:34AM +0200, Cyril Brulebois wrote:
>>
>>> But this package is also missing on the "release" DVDs for "amd64".
>>> Nearly every other architecture has this package within an image
>>> (whether "netinst" oder "DVD"). See
>>> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcdimage-search.debian.org%2F%3Fsearch_area%3Drelease%26type%3Dsimple%26query%3Dscsi-modules%26Search%3DSearch%26.cgifields%3Dsearch_area%26.cgifields%3Dtype=02%7C01%7CPaschedag.Netlution%40swr.de%7C54c234bbd1d34683f51a08d4c9874eee%7Cbcca095d88d442f88260cc216b81f62d%7C0%7C0%7C636355030367786047=vxfuiB%2F8vz96VvGbWiP7SlnqMZG1EH9C8E1Z8v85qv8%3D=0
>>
>>Adding debian-cd@ accordingly.
>
>scsi-modules-* has been in the exclude-udebs list since forever... I
>can remove that if desired? Not sure of exactly why this was added -
>tracking back through debian-cd history now.

In fact, I should have thought a little longer. It's in the exclude
list as the cdrom version of the initramfs already has to include the
SCSI modules, to be able to find the CD. We dropped the separate udebs
to save space - no need for two copies on the CD.

--
Steve McIntyre, Cambridge, UK.st...@einval.com
"When C++ is your hammer, everything looks like a thumb." -- Steven M. Haflich

Bug#867898: debian-installer-netboot-images: debian-installer-9-netboot-amd64 scsi modules missing. Netboot image unusable

[Steve McIntyre]

On Thu, Jul 13, 2017 at 01:09:58AM +0100, Steve McIntyre wrote:
>On Thu, Jul 13, 2017 at 01:50:34AM +0200, Cyril Brulebois wrote:
>>
>>> But this package is also missing on the "release" DVDs for "amd64".
>>> Nearly every other architecture has this package within an image
>>> (whether "netinst" oder "DVD"). See
>>> https://cdimage-search.debian.org/?search_area=release=simple=scsi-modules=Search&.cgifields=search_area&.cgifields=type
>>
>>Adding debian-cd@ accordingly.
>
>scsi-modules-* has been in the exclude-udebs list since forever... I
>can remove that if desired? Not sure of exactly why this was added -
>tracking back through debian-cd history now.

In fact, I should have thought a little longer. It's in the exclude
list as the cdrom version of the initramfs already has to include the
SCSI modules, to be able to find the CD. We dropped the separate udebs
to save space - no need for two copies on the CD.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"When C++ is your hammer, everything looks like a thumb." -- Steven M. Haflich

Bug#867898: debian-installer-netboot-images: debian-installer-9-netboot-amd64 scsi modules missing. Netboot image unusable

[Steve McIntyre]

On Thu, Jul 13, 2017 at 01:50:34AM +0200, Cyril Brulebois wrote:
>Paschedag, Robert  (2017-07-12):
>> thank you for your hints on the packages, that *are* available for
>> amd64. The package we need is
>> 
>> scsi-modules-4.9.0-3-amd64-di_4.9.30-2_amd64.udeb
>> 
>> this one contains the needed modules for the LSI logic controller (and
>> several other)
>> 
>> ...
>> ./lib/modules/4.9.0-3-amd64/kernel/drivers/message/
>> ./lib/modules/4.9.0-3-amd64/kernel/drivers/message/fusion/
>> ./lib/modules/4.9.0-3-amd64/kernel/drivers/message/fusion/mptbase.ko
>> ./lib/modules/4.9.0-3-amd64/kernel/drivers/message/fusion/mptfc.ko
>> ./lib/modules/4.9.0-3-amd64/kernel/drivers/message/fusion/mptsas.ko
>> ./lib/modules/4.9.0-3-amd64/kernel/drivers/message/fusion/mptscsih.ko
>> ./lib/modules/4.9.0-3-amd64/kernel/drivers/message/fusion/mptspi.ko
>
>Ah, apt-file search is still a big pain as it doesn't look into udebs even
>if I have udeb sources configured… debian-kernel@, sorry for the noise.
>
>> But this package is also missing on the "release" DVDs for "amd64".
>> Nearly every other architecture has this package within an image
>> (whether "netinst" oder "DVD"). See
>> https://cdimage-search.debian.org/?search_area=release=simple=scsi-modules=Search&.cgifields=search_area&.cgifields=type
>
>Adding debian-cd@ accordingly.

scsi-modules-* has been in the exclude-udebs list since forever... I
can remove that if desired? Not sure of exactly why this was added -
tracking back through debian-cd history now.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"The problem with defending the purity of the English language is that
 English is about as pure as a cribhouse whore. We don't just borrow words; on
 occasion, English has pursued other languages down alleyways to beat them
 unconscious and rifle their pockets for new vocabulary."  -- James D. Nicoll

Bug#867898: debian-installer-netboot-images: debian-installer-9-netboot-amd64 scsi modules missing. Netboot image unusable

[Cyril Brulebois]

Hi,

Paschedag, Robert  (2017-07-12):
> thank you for your hints on the packages, that *are* available for
> amd64. The package we need is
> 
> scsi-modules-4.9.0-3-amd64-di_4.9.30-2_amd64.udeb
> 
> this one contains the needed modules for the LSI logic controller (and
> several other)
> 
> ...
> ./lib/modules/4.9.0-3-amd64/kernel/drivers/message/
> ./lib/modules/4.9.0-3-amd64/kernel/drivers/message/fusion/
> ./lib/modules/4.9.0-3-amd64/kernel/drivers/message/fusion/mptbase.ko
> ./lib/modules/4.9.0-3-amd64/kernel/drivers/message/fusion/mptfc.ko
> ./lib/modules/4.9.0-3-amd64/kernel/drivers/message/fusion/mptsas.ko
> ./lib/modules/4.9.0-3-amd64/kernel/drivers/message/fusion/mptscsih.ko
> ./lib/modules/4.9.0-3-amd64/kernel/drivers/message/fusion/mptspi.ko

Ah, apt-file search is still a big pain as it doesn't look into udebs even
if I have udeb sources configured… debian-kernel@, sorry for the noise.

> But this package is also missing on the "release" DVDs for "amd64".
> Nearly every other architecture has this package within an image
> (whether "netinst" oder "DVD"). See
> https://cdimage-search.debian.org/?search_area=release=simple=scsi-modules=Search&.cgifields=search_area&.cgifields=type

Adding debian-cd@ accordingly.

> I apologize, if the message headers are lost. Somehow, your answer did
> not yet reach my mailbox

(With or without this issue) it would be helpful to cc me in your replies.


KiBi.


signature.asc
Description: Digital signature

Bug#867898: debian-installer-netboot-images: debian-installer-9-netboot-amd64 scsi modules missing. Netboot image unusable

[Paschedag, Robert]

Hi Cyril,

thank you for your hints on the packages, that *are* available for amd64. The 
package we need is

scsi-modules-4.9.0-3-amd64-di_4.9.30-2_amd64.udeb

this one contains the needed modules for the LSI logic controller (and several 
other)

...
./lib/modules/4.9.0-3-amd64/kernel/drivers/message/
./lib/modules/4.9.0-3-amd64/kernel/drivers/message/fusion/
./lib/modules/4.9.0-3-amd64/kernel/drivers/message/fusion/mptbase.ko
./lib/modules/4.9.0-3-amd64/kernel/drivers/message/fusion/mptfc.ko
./lib/modules/4.9.0-3-amd64/kernel/drivers/message/fusion/mptsas.ko
./lib/modules/4.9.0-3-amd64/kernel/drivers/message/fusion/mptscsih.ko
./lib/modules/4.9.0-3-amd64/kernel/drivers/message/fusion/mptspi.ko
...
/lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/BusLogic.ko
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/a100u2w.ko
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/aacraid/
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/aacraid/aacraid.ko
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/advansys.ko
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/aic7xxx/
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/aic7xxx/aic79xx.ko
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/aic7xxx/aic7xxx.ko
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/aic94xx/
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/aic94xx/aic94xx.ko
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/am53c974.ko
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/arcmsr/
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/arcmsr/arcmsr.ko
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/atp870u.ko
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/be2iscsi/
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/be2iscsi/be2iscsi.ko
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/bfa/
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/bfa/bfa.ko
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/bnx2fc/
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/bnx2fc/bnx2fc.ko
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/bnx2i/
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/bnx2i/bnx2i.ko
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/ch.ko
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/csiostor/
./lib/modules/4.9.0-3-amd64/kernel/drivers/scsi/csiostor/csiostor.ko
...

But this package is also missing on the "release" DVDs for "amd64". Nearly 
every other architecture has this package within an image (whether "netinst" 
oder "DVD"). See 
https://cdimage-search.debian.org/?search_area=release=simple=scsi-modules=Search&.cgifields=search_area&.cgifields=type

Thank you.

I apologize, if the message headers are lost. Somehow, your answer did not yet 
reach my mailbox

Regards
Robert


Mit freundlichen Grüßen

Robert Paschedag
Netlution GmbH
Landteilstr. 33
68163 Mannheim

im Auftrag des
SWR
Südwestrundfunk
HA IT, Medientechnik und Programmverbreitung
Neckarstraße 230
70190 Stuttgart

Telefon +49 (0)711 /929-12654 oder
Telefon +49 (0)711 /929-13714
paschedag.netlut...@swr.de

swr.de

Bug#867898: debian-installer-netboot-images: debian-installer-9-netboot-amd64 scsi modules missing. Netboot image unusable

[Cyril Brulebois]

Control: severity -1 important

Hi Robert & kernel maintainers,

Paschedag, Robert  (2017-07-10):
> Package: debian-installer-netboot-images
> Version: 20170615
> Severity: serious
> 
> Trying to perform a PXE installation with the debian 9 (stretch)
> netboot image and a Debian 9.0.0 Release DVD 1 (amd64) fails, because
> no disks get detected by the installer.
> 
> There are no "scsi" module .udeb files downloaded by the installer.
> Also the scsi-core-modules udeb package is missing on the realease
> image disk (DVD). I also cannot find it within your cd image search
> tool (https://cdimage-search.debian.org/). Searching
> "scsi-core-modules" does not find a package for the "amd64"
> architecture.

There's a udeb for amd64 anyway:
  
https://packages.debian.org/stretch/amd64/scsi-core-modules-4.9.0-3-amd64-di/download

It seems to be used to build the cdrom initramfses on the following archs:
| kibi@armor:~/debian-installer/installer/build$ git grep scsi-core-modules|sort
| pkg-lists/cdrom/alpha.cfg:scsi-core-modules-${kernel:Version}
| pkg-lists/cdrom/amd64.cfg:scsi-core-modules-${kernel:Version}
| pkg-lists/cdrom/arm64.cfg:scsi-core-modules-${kernel:Version}
| pkg-lists/cdrom/i386.cfg:scsi-core-modules-${kernel:Version}
| pkg-lists/hd-media/armhf.cfg:scsi-core-modules-${kernel:Version}

(That's in src:debian-installer)

So it looks to me we only need to have the extra module in this udeb, and
the release images will automatically get it.


> The installation has been tried on a simple VMware VM with the LSI
> logic parallel controller. The driver that is used is the "mptspi.ko"
> module. This module (and other scsi) modules are missing.)

Kernel maintainers (in copy): maybe this wasn't added to one of the scsi
udebs because the module is below the drivers/message/fusion directory?

Feel free to steal the bug report away from us if the required change is
just about shipping extra .ko('s) into your udebs.


KiBi.


signature.asc
Description: Digital signature