After some patching this weekend, we had a few servers go into dracut emergency
mode.
After a lot of pain and rescue system work, we found it didn't know about a
couple of the devices in the VG group that housed some needed stuff. We found
this in the dracut /run/initram/rdsosreport.txt you can get in the emergency
shell.
So looking, the missing ones seemed to not be in the rd.cio_accept line in
there.
Brought the rescue system back up and eventually figured out that those seem to
be coming out of /boot/zipl/active_devices.txt
That seems to be updated when dasd_configure is run (or now chzdev -e - see
previous email thread)
I then ran that for each of the devices needed and then ran grub2-install and
that got me past the missing devices problem.
I then ran into dracut choking on the VG with these messages
Read-only locking type set. Write locks are prohibited.
Recovery of volume group "system" failed.
Cannot process volume group system
I solved that by bringing the rescue system back up and running vgscan and it
reported:
>> 19:14:44 WARNING: Inconsistent metadata found for VG system - updating to
>> use
>> version 23
So today I go looking at more servers and pretty much all of them don't have
all the devices in active_devices.txt nor in rd.cio_accept as evidenced by the
/var/log/zypp/history.
Hmm. So maybe if I run grub2-install on one of them I can recreate the fail?
Nope. Came up just fine.
We have a case open with SUSE who has taken it to their level 3.
But I'm left with so many questions here...
Our builds are based on cloning and so I'm worried that this will strike more
servers.
SP4 now writes udev rules differently in /etc/udev/rules.d/ They start with
41 now for new disks. The SP3 and earlier generated ones start with 51.
myserver:/etc/udev/rules.d # l
total 64
drwxr-xr-x 2 root root 4096 Jun 8 19:04 ./
drwxr-xr-x 3 root root 4096 Jun 9 00:10 ../
-rw-r--r-- 1 root root 139 Jun 5 17:43 41-cio-ignore.rules
-rw-r--r-- 1 root root 396 Jun 5 17:43 41-dasd-eckd-0.0.800f.rules
-rw-r--r-- 1 root root 347 Aug 12 2016 51-dasd-0.0.0101.rules
-rw-r--r-- 1 root root 347 Aug 12 2016 51-dasd-0.0.0102.rules
-rw-r--r-- 1 root root 347 Aug 12 2016 51-dasd-0.0.0103.rules
-rw-r--r-- 1 root root 347 Aug 12 2016 51-dasd-0.0.0104.rules
-rw-r--r-- 1 root root 347 Sep 8 2016 51-dasd-0.0.8000.rules
-rw-r--r-- 1 root root 536 Dec 15 2016 51-dasd-0.0.ff00.rules
-rw-r--r-- 1 root root 536 Dec 15 2016 51-dasd-0.0.ff01.rules
-rw-r--r-- 1 root root 536 Dec 15 2016 51-dasd-0.0.ff02.rules
-rw-r--r-- 1 root root 538 Dec 15 2016 51-dasd-0.0.ff03.rules
-rw-r--r-- 1 root root 1661 Aug 12 2016 51-qeth-0.0.3000.rules
-rw-r--r-- 1 root root 1661 Aug 12 2016 51-qeth-0.0.4000.rules
-rw-r--r-- 1 root root 594 Jun 8 19:04 70-persistent-net.rules
There is also a new 41-cio-ignore.rules that looks like this
# Generated by chzdev
ACTION=="add", SUBSYSTEM=="subsystem", KERNEL=="ccw", RUN{program}+="/bin/sh -c
'echo free 800f > /proc/cio_ignore'"
Does anyone know how dracut is really supposed to be told about devices? I was
under the impression that all devices were allowed unless explicity in a
cio_ignore when running under z/VM.
At least that's the impression I get reading this
https://www.ibm.com/support/knowledgecenter/en/linuxonibm/com.ibm.linux.z.lhdd/lhdd_r_cio_ignore_cmd.html
And what's up with LVM? If it needed to upgrade itself, why didn't it prior to
this point? lvm2 was last patch a month ago. Should it have done it then? Or
did it just become inconsistent when the devices decided to go AWOL?
Marcy
This message may contain confidential and/or privileged information. If you are
not the addressee or authorized to receive this for the addressee, you must not
use, copy, disclose, or take any action based on this message or any
information herein. If you have received this message in error, please advise
the sender immediately by reply e-mail and delete this message. Thank you for
your cooperation.
----------------------------------------------------------------------
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www2.marist.edu/htbin/wlvindex?LINUX-390
