I have servers that I'm setting up with RHEL 7.8 on ECKD disk. I installed it
with root in an LVM, all of that worked fine. Now, I'm testing adding a 2nd
ECKD disk to the LVM as I know that will be required in the future, but I'm
running into a confusing problem. Everything works as I expect and the disk
space shows up in / as it should. When I reboot the server, however, it goes
to heck and ends up in emergency mode. The most prominent feature of the
console are a bunch of I/O errors on device dm-2, which is the disk I just
added. Below are the steps I followed to add the disk. Before doing this, I
had activated the disk, added it to /etc/dasd.conf and run zipl. After doing
that, I rebooted to ensure that the disk came up online, which it did. So, I'm
confident that the disk is there when the system rebooted the 2nd time.
I have a lot of colleagues who work with Linux all the time, but don't usually
have root as an LVM. They haven't seen this problem before, but they work
mainly on X. Could this be specific problem to using ECKD? I have thought of
going to an FBA LUN, but had spare ECKD to use.
Martha
[root@opncl42 ~]# cio_ignore -r 0.0.0200
[root@opncl42 ~]# chccwdev --online 0.0.0200
Setting device 0.0.0200 online
Done
[root@opncl42 ~]# lsdasd
Bus-ID Status Name Device Type BlkSz Size Blocks
==============================================================================
0.0.0150 active dasda 94:0 ECKD 4096 46068MB 11793420
0.0.0200 active dasdb 94:4 ECKD 4096 23033MB 5896620
[root@opncl42 ~]# vi /etc/dasd.conf
[root@opncl42 ~]# zipl
Using config file '/etc/zipl.conf'
Building bootmap in '/boot'
Building menu 'zipl-automatic-menu'
Adding #1: IPL section '3.10.0-1127.el7.s390x' (default)
Adding #2: IPL section '3.10.0-1127.el7.s390x_with_debugging'
Adding #3: IPL section 'linux'
Preparing boot device: dasda (0150).
Done.
[root@opncl42 ~]# reboot
---------------------------------------------------
[root@opncl42 ~]# lsdasd
Bus-ID Status Name Device Type BlkSz Size Blocks
==============================================================================
0.0.0150 active dasda 94:0 ECKD 4096 46068MB 11793420
0.0.0200 active dasdb 94:4 ECKD 4096 23033MB 5896620
[root@opncl42 ~]# dasdfmt -p -f /dev/dasdb
Please enter the blocksize of the formatting [4096]:
Drive Geometry: 32759 Cylinders * 15 Heads = 491385 Tracks
I am going to format the device /dev/dasdb in the following way:
Device number of device : 0x200
Labelling device : yes
Disk label : VOL1
Disk identifier : 0X0200
Extent start (trk no) : 0
Extent end (trk no) : 491384
Compatible Disk Layout : yes
Blocksize : 4096
Mode : Full
--->> ATTENTION! <<---
All data of that device will be lost.
Type "yes" to continue, no will leave the disk untouched: yes
Formatting the device. This may take a while (get yourself a coffee).
cyl 32759 of 32759 |#################################|100% [2m 43s]
Finished formatting the device.
Rereading the partition table... ok
[root@opncl42 ~]# pvcreate /dev/dasdb1
Device /dev/dasdb1 not found.
[root@opncl42 ~]# fdasd /dev/dasdb
reading volume label ..: VOL1
reading vtoc ..........: ok
Command action
m print this menu
p print the partition table
n add a new partition
d delete a partition
v change volume serial
t change partition type
r re-create VTOC and delete all partitions
u re-create VTOC re-using existing partition sizes
s show mapping (partition number - data set name)
q quit without saving changes
w write table to disk and exit
Command (m for help): n
First track (1 track = 48 KByte) ([2]-491384):
Using default value 2
Last track or +size[c|k|M] (2-[491384]):
Using default value 491384
Command (m for help): w
writing VTOC...
rereading partition table...
[root@opncl42 ~]# pvcreate /dev/dasdb1
Physical volume "/dev/dasdb1" successfully created.
[root@opncl42 ~]# vgdisplay
--- Volume group ---
VG Name rhel_opncl41
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size <44.50 GiB
PE Size 4.00 MiB
Total PE 11391
Alloc PE / Size 11381 / <44.46 GiB
Free PE / Size 10 / 40.00 MiB
VG UUID uEvApZ-Wrx7-7n47-f0f0-IQ6R-uU4K-JO8wAN
[root@opncl42 ~]# vgextend rhel_opncl41 /dev/dasdb1
Volume group "rhel_opncl41" successfully extended
[root@opncl42 ~]# vgdisplay
--- Volume group ---
VG Name rhel_opncl41
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 5
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 2
Act PV 2
VG Size <66.99 GiB
PE Size 4.00 MiB
Total PE 17149
Alloc PE / Size 11381 / <44.46 GiB
Free PE / Size 5768 / 22.53 GiB
VG UUID uEvApZ-Wrx7-7n47-f0f0-IQ6R-uU4K-JO8wAN
[root@opncl42 ~]# lvdisplay
--- Logical volume ---
LV Path /dev/rhel_opncl41/swap
LV Name swap
VG Name rhel_opncl41
LV UUID QzWFVP-Z5hz-gRV2-ORnb-4rR6-pdhM-d77w3n
LV Write Access read/write
LV Creation host, time opncl41, 2020-04-14 20:25:30 -0400
LV Status available
# open 2
LV Size 4.50 GiB
Current LE 1152
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:1
--- Logical volume ---
LV Path /dev/rhel_opncl41/root
LV Name root
VG Name rhel_opncl41
LV UUID G0Gy3F-lvt2-2uZy-3BKr-9yHL-uVqe-1lfycV
LV Write Access read/write
LV Creation host, time opncl41, 2020-04-14 20:25:30 -0400
LV Status available
# open 1
LV Size <39.96 GiB
Current LE 10229
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:0
[root@opncl42 ~]# lvextend -l +5768 /dev/rhel_opncl41/root
Size of logical volume rhel_opncl41/root changed from <39.96 GiB (10229
extents) to <62.49 GiB (15997 extents).
Logical volume rhel_opncl41/root successfully resized.
[root@opncl42 ~]# lvdisplay
--- Logical volume ---
LV Path /dev/rhel_opncl41/swap
LV Name swap
VG Name rhel_opncl41
LV UUID QzWFVP-Z5hz-gRV2-ORnb-4rR6-pdhM-d77w3n
LV Write Access read/write
LV Creation host, time opncl41, 2020-04-14 20:25:30 -0400
LV Status available
# open 2
LV Size 4.50 GiB
Current LE 1152
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:1
--- Logical volume ---
LV Path /dev/rhel_opncl41/root
LV Name root
VG Name rhel_opncl41
LV UUID G0Gy3F-lvt2-2uZy-3BKr-9yHL-uVqe-1lfycV
LV Write Access read/write
LV Creation host, time opncl41, 2020-04-14 20:25:30 -0400
LV Status available
# open 1
LV Size <62.49 GiB
Current LE 15997
Segments 2
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:0
[root@opncl42 ~]# df -h
Filesystem Size Used Avail Use%
Mounted on
devtmpfs 9.8G 0 9.8G 0% /dev
tmpfs 9.8G 0 9.8G 0%
/dev/shm
tmpfs 9.8G 0 9.8G 0%
/sys/fs/cgroup
/dev/mapper/rhel_opncl41-root 40G 2.0G 38G 5% /
/dev/dasda1 494M 136M 359M 28% /boot
tmpfs 2.0G 0 2.0G 0%
/run/user/1000
[root@opncl42 ~]# fsadm resize /dev/mapper/rhel_opncl41-root
meta-data=/dev/mapper/rhel_opncl41-root isize=256 agcount=4, agsize=2618624
blks
= sectsz=4096 attr=2, projid32bit=1
= crc=0 finobt=0 spinodes=0
data = bsize=4096 blocks=10474496, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal bsize=4096 blocks=5114, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 10474496 to 16380928
[root@opncl42 ~]# df -h
Filesystem Size Used Avail Use%
Mounted on
devtmpfs 9.8G 0 9.8G 0% /dev
tmpfs 9.8G 0 9.8G 0%
/dev/shm
tmpfs 9.8G 8.3M 9.8G 1%
/run
tmpfs 9.8G 0 9.8G 0%
/sys/fs/cgroup
/dev/mapper/rhel_opncl41-root 63G 2.0G 61G 4% /
/dev/dasda1 494M 136M 359M 28% /boot
tmpfs 2.0G 0 2.0G 0%
/run/user/1000
[root@opncl42 ~]# reboot
----------------------------------------------------------------------------
Last part of console as it goes into emergency mode:
13 Ý Ý32m OK Ý0m¨ Started File System Check on /dev/mapper/rhel_opncl41-root.
14 Ý Ý32m OK Ý0m¨ Started dracut initqueue hook.
15 Mounting /sysroot...
16 Ý Ý32m OK Ý0m¨ Reached target Remote File Systems (Pre).
17 Ý Ý32m OK Ý0m¨ Reached target Remote File Systems.
18 Ý 123.794007¨ Buffer I/O error on dev dm-2, logical block 16380912, async
page read
19 Ý Ý1;31mFAILED Ý0m¨ Failed to mount /sysroot.
20 See 'systemctl status sysroot.mount' for details.
21 Ý Ý1;33mDEPEND Ý0m¨ Dependency failed for Initrd Root File System.
22 Ý Ý1;33mDEPEND Ý0m¨ Dependency failed for Reload Configuration from the Real
Root.
23 Ý Ý32m OK Ý0m¨ Stopped dracut pre-udev hook.
24 Ý Ý32m OK Ý0m¨ Stopped dracut cmdline hook.
25 Ý Ý32m OK Ý0m¨ Reached target Initrd File Systems.
26 Ý Ý32m OK Ý0m¨ Stopped target Basic System.
27 Ý Ý32m OK Ý0m¨ Stopped target System Initialization.
28 Ý Ý32m OK Ý0m¨ Stopped dracut initqueue hook.
29 Ý Ý32m OK Ý0m¨ Started Emergency Shell.
30 Ý Ý32m OK Ý0m¨ Reached target Emergency Mode.
31
32 Generating "/run/initramfs/rdsosreport.txt"
33 Ý 123.845595¨ Buffer I/O error on dev dm-2, logical block 16380912, async
page read
34 Ý 123.846240¨ Buffer I/O error on dev dm-2, logical block 16380912, async
page read
35
36
37 Entering emergency mode. Exit the shell to continue.
38 Type "journalctl" to view system logs.
39 You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /
boot
40 after mounting them and attach it to a bug report.
41
42
43 :/#
------------------------------------------------
Information from Journalctl while in emergency mode:
ÝKApr 15 02:21:42 opncl42 systemdÝ1¨: Reached target Basic System. Ým
Apr 15 02:21:42 opncl42 kernel: Ý1;31mnetif_napi_add() called with weight 128 o
n devic Ým
Apr 15 02:21:42 opncl42 kernel: qeth 0.0.0700: MAC address 02:01:42:00:00:08 suc
Ým
Apr 15 02:21:42 opncl42 kernel: qeth 0.0.0700: Device is a Virtual NIC QDIO card
Ým
with link type Virt.NIC QDIO (portname: ) Ým
Apr 15 02:23:45 opncl42 dracut-initqueueÝ398¨: Warning: dracut-initqueue timeout
Ým
Apr 15 02:23:45 opncl42 kernel: Ý1;31mBuffer I/O error on dev dm-2, logical blo
ck 1638 Ým
Apr 15 02:23:45 opncl42 systemdÝ1¨: Found device /dev/mapper/rhel_opncl41-root.
Ým
Apr 15 02:23:45 opncl42 systemdÝ1¨: Starting File System Check on /dev/mapper/rh
Ým
Apr 15 02:23:45 opncl42 systemd-fsckÝ2026¨: fsck: error 2 (No such file or direc
Ým
Apr 15 02:23:45 opncl42 systemd-fsckÝ2026¨: Ý1;31mfsck failed with error code 8
. Ý0m Ým
Apr 15 02:23:45 opncl42 systemd-fsckÝ2026¨: Ý1;39mIgnoring error. Ý0m Ým
Apr 15 02:23:45 opncl42 systemdÝ1¨: Started File System Check on /dev/mapper/rhe
Ým
Apr 15 02:23:45 opncl42 kernel: Ý1;31mBuffer I/O error on dev dm-2, logical blo
ck 1638 Ým
Apr 15 02:23:45 opncl42 systemdÝ1¨: Started dracut initqueue hook. Ým
Apr 15 02:23:45 opncl42 systemdÝ1¨: Mounting /sysroot... Ým
Apr 15 02:23:45 opncl42 systemdÝ1¨: Reached target Remote File Systems (Pre). Ým
Apr 15 02:23:45 opncl42 systemdÝ1¨: Reached target Remote File Systems. Ým
Apr 15 02:23:45 opncl42 mountÝ2042¨: mount: you must specify the filesystem type
Ým
Apr 15 02:23:45 opncl42 systemdÝ1¨: Ý1;39msysroot.mount mount process exited, c
ode=exi Ým
Apr 15 02:23:45 opncl42 systemdÝ1¨: Ý1;31mFailed to mount /sysroot. Ý0m Ým
Apr 15 02:23:45 opncl42 systemdÝ1¨: Ý1;39mDependency failed for Initrd Root Fil
e Syste Ým
Apr 15 02:23:45 opncl42 systemdÝ1¨: Ý1;39mDependency failed for Reload Configur
ation f Ým
Ý7mlines 193-215 Ý27m ÝK ÝKApr 15 02:23:45 opncl42 systemdÝ1¨: Ý1;39mJob ini
Ým
Apr 15 02:23:45 opncl42 kernel: Ý1;31mBuffer I/O error on dev dm-2, logical blo
ck 1638 Ým
Apr 15 02:23:45 opncl42 systemdÝ1¨: Received SIGRTMIN+21 from PID 406 (plymouthd
Ým
Apr 15 02:23:45 opncl42 kernel: Ý1;31mBuffer I/O error on dev dm-2, logical blo
ck 1638 Ým
Apr 15 02:23:45 opncl42 kernel: Ý1;31mBuffer I/O error on dev dm-2, logical blo
ck 1638 Ým
Ý7mlines 211-233/233 (END) Ý27m ÝK ÝK ÝK Ý7mlines 211-233/233 (END) Ý27m ÝK
Martha McConaghy
Marist: System Architect/Technical Lead
SHARE Association: Vice President
Marist College IT
Poughkeepsie, NY 12601
----------------------------------------------------------------------
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www2.marist.edu/htbin/wlvindex?LINUX-390
