Bug#725714: Partial fix using hw-setup for the missing firmware problem in d-i?

2014-10-07 Thread Petter Reinholdtsen 
Here is another draft patch for hw-detect.  This one is tested, and
find the missing firmware on my X200 test laptop.

This approach keep the non-functioning code and add two new
approaches, one looking at the meta information for loadmed modules,
and one parsing the dmesg output.  The union from all three methods
are then presented as the list of wanted firmware.

diff --git a/check-missing-firmware.sh b/check-missing-firmware.sh
index 60c6ff4..74db55b 100755
--- a/check-missing-firmware.sh
+++ b/check-missing-firmware.sh
@@ -61,6 +61,18 @@ nic_is_configured() {
return 1
 }
 
+add_if_fw_missing() {
+fwfile=$1
+if [ ! -e /lib/firmware/$fwfile ] ; then
+if grep -q ^$fwfile$ $DENIED 2/dev/null; then
+   log listed in $DENIED
+continue
+fi
+files=${files:+$files }$fwfile
+modules=$module${modules:+ $modules}
+fi
+}
+
 check_missing () {
upnics
 
@@ -117,11 +129,30 @@ check_missing () {
done
done
 
+# Workaround for bug #725714, the kernel and udev no longer
+# let us know via /dev/.udev/firmware-missing and
+# /run/udev/firmware-missing which firmware files the kernel
+# drivers look for.  This approach will only find firmware for
+# the loaded kernel modules.  Modules refusing to
+# register/load when the firmware is missing will be missed.
+for module in $(cut -d  -f1 /proc/modules); do 
+for fwfile in $(modinfo $module | grep ^firmware: | cut -d: -f2); 
do
+log looking for firmware file $fwfile needed by $module
+add_if_fw_missing $fwfile
+done
+done
+for fwfile in $(dmesg | grep 'firmware: failed' | sed 's/.*firmware: 
failed to load //' | cut -d  -f1); do
+# Dummy make sure '-n $modules' test below find something
+module=kernel
+log looking for firmware file $fwfile requested by kernel
+add_if_fw_missing $fwfile
+done
+
if [ -n $modules ]; then
log missing firmware files ($files) for $modules
return 0
else
-   log no missing firmware in $MISSING
+   log no missing firmware for any kernel module
return 1
fi
 }

-- 
Happy hacking
Petter Reinholdtsen


-- 
To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141007135247.gp10...@ulrik.uio.no

Bug#725714: Partial fix using hw-setup for the missing firmware problem in d-i?

2014-10-07 Thread Ben Hutchings 
On Tue, 2014-10-07 at 15:52 +0200, Petter Reinholdtsen wrote:
 Here is another draft patch for hw-detect.  This one is tested, and
 find the missing firmware on my X200 test laptop.
 
 This approach keep the non-functioning code

The firmware agent is never coming back, so please do remove the related
code.

 and add two new
 approaches, one looking at the meta information for loadmed modules,
 and one parsing the dmesg output.  The union from all three methods
 are then presented as the list of wanted firmware.
 
 diff --git a/check-missing-firmware.sh b/check-missing-firmware.sh
 index 60c6ff4..74db55b 100755
 --- a/check-missing-firmware.sh
 +++ b/check-missing-firmware.sh
[...]
 @@ -117,11 +129,30 @@ check_missing () {
   done
   done
  
 +# Workaround for bug #725714, the kernel and udev no longer
 +# let us know via /dev/.udev/firmware-missing and
 +# /run/udev/firmware-missing which firmware files the kernel
 +# drivers look for.

This belongs in the changelog not the code.

   This approach will only find firmware for
 +# the loaded kernel modules.  Modules refusing to
 +# register/load when the firmware is missing will be missed.
 +for module in $(cut -d  -f1 /proc/modules); do 
 +for fwfile in $(modinfo $module | grep ^firmware: | cut -d: 
 -f2); do
 +log looking for firmware file $fwfile needed by $module
 +add_if_fw_missing $fwfile
 +done
 +done
 +for fwfile in $(dmesg | grep 'firmware: failed' | sed 's/.*firmware: 
 failed to load //' | cut -d  -f1); do

Redundant use of grep; sed can do that (sed -n 's/.../.../p').

 +# Dummy make sure '-n $modules' test below find something
 +module=kernel

The driver name should appear at the start of the log line (after the
timestamp).  Use that instead of 'kernel'.

 +log looking for firmware file $fwfile requested by kernel
 +add_if_fw_missing $fwfile
 +done
 +

Indentation of the above is inconsistent with the surrounding code (4
spaces vs hard tab).

Ben.

   if [ -n $modules ]; then
   log missing firmware files ($files) for $modules
   return 0
   else
 - log no missing firmware in $MISSING
 + log no missing firmware for any kernel module
   return 1
   fi
  }
 
 -- 
 Happy hacking
 Petter Reinholdtsen
 
 

-- 
Ben Hutchings
Logic doesn't apply to the real world. - Marvin Minsky


signature.asc
Description: This is a digitally signed message part

Bug#725714: Partial fix using hw-setup for the missing firmware problem in d-i?

2014-10-07 Thread Petter Reinholdtsen 
Thank you for the code review. :)

[Ben Hutchings]
 The firmware agent is never coming back, so please do remove the related
 code.

I know and agree, but 
URL: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=725714#163 
asked for the non-working code to be kept to work with older kernels.
I do not have any strong feelings, so I left it as it was.

 This belongs in the changelog not the code.

Actually, as long as there are several blocks doing similar things, I
believe an explanation should be close to the code to explain why.  I
will wonder when I return in a few years time. :)

 The driver name should appear at the start of the log line (after
 the timestamp).  Use that instead of 'kernel'.

Yeah, but did not find a simple way to do it, and it is not affecting
the functionallity, only the user messages.  Should probably be fixed
in the final version.

 Redundant use of grep; sed can do that (sed -n 's/.../.../p').

Yeah.

 Indentation of the above is inconsistent with the surrounding code
 (4 spaces vs hard tab).

It will happen before any commit is done.

-- 
Happy hacking
Petter Reinholdtsen


-- 
To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141007150911.gq10...@ulrik.uio.no

Bug#725714: Partial fix using hw-setup for the missing firmware problem in d-i?

2014-10-07 Thread Ben Hutchings 
On Tue, 2014-10-07 at 17:09 +0200, Petter Reinholdtsen wrote:
 Thank you for the code review. :)
 
 [Ben Hutchings]
  The firmware agent is never coming back, so please do remove the related
  code.
 
 I know and agree, but 
 URL: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=725714#163 
 asked for the non-working code to be kept to work with older kernels.

Actually it is udev that removed this feature originally.  Anyway, I
shan't argue this with KiBi.

 I do not have any strong feelings, so I left it as it was.

  This belongs in the changelog not the code.
 
 Actually, as long as there are several blocks doing similar things, I
 believe an explanation should be close to the code to explain why.  I
 will wonder when I return in a few years time. :)

Perhaps this would be better placed above the old code, as that is what
should be cleaned up at some later date.

  The driver name should appear at the start of the log line (after
  the timestamp).  Use that instead of 'kernel'.
 
 Yeah, but did not find a simple way to do it, and it is not affecting
 the functionallity, only the user messages.  Should probably be fixed
 in the final version.
 
  Redundant use of grep; sed can do that (sed -n 's/.../.../p').
 
 Yeah.
 
  Indentation of the above is inconsistent with the surrounding code
  (4 spaces vs hard tab).
 
 It will happen before any commit is done.

Thanks.

Ben.

-- 
Ben Hutchings
Logic doesn't apply to the real world. - Marvin Minsky


signature.asc
Description: This is a digitally signed message part

Bug#725714: Partial fix using hw-setup for the missing firmware problem in d-i?

2014-10-06 Thread Petter Reinholdtsen 

I noticed this bug was mentioned as a regression in the Jessie
installer, and thought I should have a look at how this could be
improved from the hw-detect side.

The following patch replaces the code looking in
/dev/.udev/firmware-missing and /run/udev/firmware-missing for firmware
wanted by the kernel with code looking for firmware listed in the
meta-info of loaded modules.  It will solve the problem for at least
some kernel modules, but not the class of modules refusing to load if
their firmware files are missing.

Perhaps something do use while we wait for more information from the
kernel side?

The patch is only partly tested on the command line inside d-i, and is
based on the isenkram code, but rewritten to avoid awk, as awk is not
available in d-i.  I can brush it up if we decide to go in this
direction.

diff --git a/check-missing-firmware.sh b/check-missing-firmware.sh
index 60c6ff4..77a670e 100755
--- a/check-missing-firmware.sh
+++ b/check-missing-firmware.sh
@@ -2,7 +2,6 @@
 set -e
 . /usr/share/debconf/confmodule
 
-MISSING='/dev/.udev/firmware-missing /run/udev/firmware-missing'
 DENIED=/tmp/missing-firmware-denied
 
 if [ x$1 = x-n ]; then
@@ -67,61 +66,29 @@ check_missing () {
# Give modules some time to request firmware.
sleep 1

-   modules=
-   files=
-   for missing_dir in $MISSING
-   do
-   if [ ! -d $missing_dir ]; then
-   log $missing_dir does not exist, skipping
-   continue
-   fi
-   for file in $(find $missing_dir -type l); do
-   # decode firmware filename as encoded by
-   # udev firmware.agent
-   fwfile=$(basename $file | sed -e 's#\\x2f#/#g')
-   
-   # strip probably nonexistant firmware subdirectory
-   devpath=$(readlink $file | sed 's/\/firmware\/.*//')
-   # the symlink is supposed to point to the device in /sys
-   if ! echo $devpath | grep -q '^/sys/'; then
-   devpath=/sys$devpath
-   fi
-
-   module=$(get_module $devpath)
-   if [ -z $module ]; then
-   log failed to determine module from $devpath
-   continue
-   fi
-
-   rm -f $file
-
-   if grep -q ^$fwfile$ $DENIED 2/dev/null; then
-   continue
-   fi
-   
-   files=$fwfile${files:+ $files}
-
-   if [ $module = usbcore ]; then
-   # Special case for USB bus, which puts the
-   # real module information in a subdir of
-   # the devpath.
-   for dir in $(find $devpath -maxdepth 1 
-mindepth 1 -type d); do
-   module=$(get_module $dir)
-   if [ -n $module ]; then
-   modules=$module${modules:+ 
$modules}
-   fi
-   done
-   else
-   modules=$module${modules:+ $modules}
-   fi
-   done
-   done
-
+# Workaround for bug #725714, the kernel and udev no longer
+# let us know via /dev/.udev/firmware-missing and
+# /run/udev/firmware-missing which firmware files the kernel
+# drivers look for.  This approach will only find firmware for
+# the loaded kernel modules.  Modules refusing to
+# register/load when the firmware is missing will be missed.
+for fwfile in $(for module in $(cut -d  -f1 /proc/modules); do 
modinfo $module | grep ^firmware:|cut -d: -f2; done); do 
+if [ ! -e /lib/firmware/$fwfile ] ; then
+
+if grep -q ^$fwfile$ $DENIED 2/dev/null; then
+continue
+fi
+
+files=${files:+$files }$fwfile
+modules=$module${modules:+ $modules}
+fi
+done
+   
if [ -n $modules ]; then
log missing firmware files ($files) for $modules
return 0
else
-   log no missing firmware in $MISSING
+   log no missing firmware for any loaded kernel module
return 1
fi
 }

-- 
Happy hacking
Petter Reinholdtsen


-- 
To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/2fl8ukt1fpt@diskless.uio.no

Bug#725714: Partial fix using hw-setup for the missing firmware problem in d-i?

2014-10-06 Thread Cyril Brulebois 
Petter Reinholdtsen p...@hungry.com (2014-10-06):
 I noticed this bug was mentioned as a regression in the Jessie
 installer, and thought I should have a look at how this could be
 improved from the hw-detect side.
 
 The following patch replaces the code looking in
 /dev/.udev/firmware-missing and /run/udev/firmware-missing for firmware
 wanted by the kernel with code looking for firmware listed in the
 meta-info of loaded modules.  It will solve the problem for at least
 some kernel modules, but not the class of modules refusing to load if
 their firmware files are missing.
 
 Perhaps something do use while we wait for more information from the
 kernel side?
 
 The patch is only partly tested on the command line inside d-i, and is
 based on the isenkram code, but rewritten to avoid awk, as awk is not
 available in d-i.  I can brush it up if we decide to go in this
 direction.

The idea is to parse kernel logs.

Mraw,
KiBi.


signature.asc
Description: Digital signature

Bug#725714: Partial fix using hw-setup for the missing firmware problem in d-i?

2014-10-06 Thread Petter Reinholdtsen 
[Cyril Brulebois]
 The idea is to parse kernel logs.

OK, that might work too.  Is the idea to use the output from 'dmesg'?
Who is implementing it?  Is there a draft patch somewhere?

-- 
Happy hacking
Petter Reinholdtsen


-- 
To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141006144842.gn10...@ulrik.uio.no

Bug#725714: Partial fix using hw-setup for the missing firmware problem in d-i?

2014-10-06 Thread Cyril Brulebois 
Petter Reinholdtsen p...@hungry.com (2014-10-06):
 [Cyril Brulebois]
  The idea is to parse kernel logs.
 
 OK, that might work too.

No, it does.

 Is the idea to use the output from 'dmesg'?  Who is implementing it?
 Is there a draft patch somewhere?

I am working on it, when I'm not busy fixing or reverting regressions
people introduce in d-i packages.

KiBi.


signature.asc
Description: Digital signature

Bug#725714: Partial fix using hw-setup for the missing firmware problem in d-i?

2014-10-06 Thread Ben Hutchings 
On Mon, 2014-10-06 at 15:37 +0200, Petter Reinholdtsen wrote:
 I noticed this bug was mentioned as a regression in the Jessie
 installer, and thought I should have a look at how this could be
 improved from the hw-detect side.
 
 The following patch replaces the code looking in
 /dev/.udev/firmware-missing and /run/udev/firmware-missing for firmware
 wanted by the kernel with code looking for firmware listed in the
 meta-info of loaded modules.

That seems like a good idea.

 It will solve the problem for at least
 some kernel modules, but not the class of modules refusing to load if
 their firmware files are missing.

I'm not aware that there are any of those.

 Perhaps something do use while we wait for more information from the
 kernel side?
[...]

I'm afraid I'm unlikely to implement the proposed structured logging in
time for jessie, and the installer would need some work to make use of
structured log data.  However the firmware core code is already patched
so that it logs all failure in a consistent format that you can grep
for:

firmware: failed to load %s (%d)

The parameters are the filename and negative error code; usually that
will be -ENOENT == -2.

Ben.

-- 
Ben Hutchings
The program is absolutely right; therefore, the computer must be wrong.


signature.asc
Description: This is a digitally signed message part

Bug#725714: Partial fix using hw-setup for the missing firmware problem in d-i?

2014-10-06 Thread Cyril Brulebois 
Ben Hutchings b...@decadent.org.uk (2014-10-06):
 On Mon, 2014-10-06 at 15:37 +0200, Petter Reinholdtsen wrote:
  I noticed this bug was mentioned as a regression in the Jessie
  installer, and thought I should have a look at how this could be
  improved from the hw-detect side.
  
  The following patch replaces the code looking in
  /dev/.udev/firmware-missing and /run/udev/firmware-missing for firmware
  wanted by the kernel with code looking for firmware listed in the
  meta-info of loaded modules.
 
 That seems like a good idea.

I disagree with the code deletion. It makes it harder to combine various
d-i components across distributions to support as many use cases as
possible. Keeping a few old lines of shell code shouldn't be a problem.

  Perhaps something do use while we wait for more information from the
  kernel side?
 [...]
 
 I'm afraid I'm unlikely to implement the proposed structured logging in
 time for jessie, and the installer would need some work to make use of
 structured log data.  However the firmware core code is already patched
 so that it logs all failure in a consistent format that you can grep
 for:
 
 firmware: failed to load %s (%d)
 
 The parameters are the filename and negative error code; usually that
 will be -ENOENT == -2.

I started using this successfully but then I stumbled upon issues with
WPA2 (which turned out to be missing modules as we now know), and got
to publish Jessie Beta 2. So the current logging situation might not be
ideal yet but it seems sufficient to vastly improve the current state
of affairs.

Mraw,
KiBi.


signature.asc
Description: Digital signature