Duane,
> Issuing lvchange -ay /dev/xxx/xxx changes the status to available.
A bit kludgy perhaps, but could you define a service that does the lvchange
before /etc/fstab is read?
-Mike M
On Sat, Dec 5, 2020 at 6:18 PM Duane Beyer <[email protected]> wrote:
> Marcio
>
> The FCP devices are online and active after a reboot and the LUNS are
> showing up in multipath. The system is dropping into emergency mode
> because the LV's are defined in /etc/fstab. If I remove them, the fstab,
> the system IPL's with no issues. The LV's are just in Not Available
> status.
>
> Issuing lvchange -ay /dev/xxx/xxx changes the status to available.
>
> My guess is it is something in the LV's metadata that get reread/reset
> when the lvchange command is issued. The problem is that's not
> persistent, so the next reboot, we are in the same situation.
>
> Duane
>
> -----Original Message-----
> From: Linux on 390 Port <[email protected]> On Behalf Of Marcio da
> Silva Nunes
> Sent: Saturday, December 5, 2020 2:58 PM
> To: [email protected]
> Subject: Re: LVM does not come online after reboot. PVMOVE used to migrate
> data.
>
> Duane,
>
>
> A possibility that can be verified.
>
>
> When use the text-based interface yast zfcp. If cio_ignore is enabled, you
> might need to free blacklisted FCP devices before by using yast cio.
>
>
> Can this be the problem?
>
>
>
> Marcio da Silva Nunes
> Analista
> Superintendência de Produtos e Serviços-Centro de Dados Diretoria de
> Operações
> +55 (61)2021-9099
> +55 (61) 99981-0376
>
> ----- Mensagem original -----
> De: "Duane Beyer" <[email protected]>
> Para: "Linux on 390 Port" <[email protected]>
> Enviadas: Sábado, 5 de dezembro de 2020 12:28:39
> Assunto: Re: LVM does not come online after reboot. PVMOVE used to migrate
> data.
>
> Marcio,
>
> All of the disks are for data. The root file system is ECKD.
>
> The FCP LUNs were added via yast so all the configuration was done with
> yast. I do see the associated rules files and the correct fcp definitions
> in /dev/disk/....
> The system is a SLES 12 SP4: Linux lxnebnp2 4.12.14-95.16-default #1 SMP
> Mon May 6 09:58:58 UTC 2019 (1da26c7) s390x s390x s390x GNU/Linux Multipath
> Version: multipath-tools-0.7.3+129+suse.e8ca031-2.8.1.s390x
>
> /etc/udev/rules.d/41-zfcp-host-0.0.4000.rules
> /etc/udev/rules.d/41-zfcp-lun-0.0.5000.rules
> /etc/udev/rules.d/41-zfcp-lun-0.0.4000.rules
> /etc/udev/rules.d/41-zfcp-host-0.0.5000.rules
> /dev/disk/by-path/ccw-0.0.5000-zfcp-0x5005076810264e3c:0x0001000000000000
> /dev/disk/by-path/ccw-0.0.4000-zfcp-0x5005076810154da7:0x0002000000000000
> /dev/disk/by-path/ccw-0.0.4000-zfcp-0x5005076810154da7:0x0003000000000000
> .... cut to save space ....
> /dev/disk/by-path/ccw-0.0.4000-zfcp-0x5005076810154e3c:0x0001000000000000
> /dev/disk/by-path/ccw-0.0.4000-zfcp-0x5005076810154e3a:0x0000000000000000
>
> Duane
> -----Original Message-----
> From: Linux on 390 Port <[email protected]> On Behalf Of Marcio da
> Silva Nunes
> Sent: Saturday, December 5, 2020 9:27 AM
> To: [email protected]
> Subject: Re: LVM does not come online after reboot. PVMOVE used to migrate
> data.
>
> Duane,
>
> Let's see if it helps.
>
> Were the new FCP that are not associated with root included in the
> zfcp.conf file?
>
> If they are part of the root system disk then they must be in zipl.conf.
>
> Maybe it works.
>
>
> Marcio da Silva Nunes
> Analista
> Superintendência de Produtos e Serviços-Centro de Dados Diretoria de
> Operações
> +55 (61)2021-9099
> +55 (61) 99981-0376
>
> ----- Mensagem original -----
> De: "Duane Beyer" <[email protected]>
> Para: [email protected]
> Enviadas: Sexta-feira, 4 de dezembro de 2020 18:01:08
> Assunto: Re: LVM does not come online after reboot. PVMOVE used to migrate
> data.
>
> I should have included this. Status, NOT available
>
> This is what LVDISPLAY displays for the LVM disks:
> --- Logical volume ---
> LV Path /dev/u01/u01
> LV Name u01
> VG Name u01
> LV UUID 6qzW2g-xJr4-osII-ULfn-d9nb-VscF-saGdBL
> LV Write Access read/write
> LV Creation host, time lxnebnp2, 2019-06-25 13:18:38 -0400
> LV Status NOT available
> LV Size 100.00 GiB
> Current LE 25599
> Segments 1
> Allocation inherit
> Read ahead sectors auto
> Persistent major 254
> Persistent minor 123
>
>
> PVS output:
> LV VG Attr LSize Pool Origin Data% Meta% Move Log
> Cpy%Sync Convert
> home hometmp -wi-a----- 23.44g
> tmp hometmp -wi-a----- 23.44g
> u01 u01 -wim------ 100.00g
> u02 u02 -wim------ 299.99g
> u03 u034 -wim------ 50.00g
> u04 u034 -wim------ 49.00g
> u05 u056 -wim------ 49.00g
> u06 u056 -wim------ 49.00g
>
> VGDISPLAY:
> --- Volume group ---
> VG Name u01
> System ID
> Format lvm2
> Metadata Areas 1
> Metadata Sequence No 8
> VG Access read/write
> VG Status resizable
> MAX LV 0
> Cur LV 1
> Open LV 0
> Max PV 0
> Cur PV 1
> Act PV 1
> VG Size 100.00 GiB
> PE Size 4.00 MiB
> Total PE 25599
> Alloc PE / Size 25599 / 100.00 GiB
> Free PE / Size 0 / 0
> VG UUID pzRsQG-o76z-SBw2-Xu9T-T2cC-eXCZ-NU6WcQ
>
>
> Duane
>
> -----Original Message-----
> From: Duane Beyer
> Sent: Friday, December 4, 2020 3:45 PM
> To: Linux on 390 Port <[email protected]>
> Subject: LVM does not come online after reboot. PVMOVE used to migrate
> data.
>
>
> We have installed a new IBM FS7200 and are in the process of moving
> systems off the old disks and on to the new 7200. For many systems, we are
> doing this while the service is up, so we are using pvmove.
>
> All of these systems have direct attached SCSI devices that are part of
> LVM's.
>
> Since the systems are production, we used pvmove to move the data from the
> old disks to the new ones. The following process was used.
>
> pvcreate the new disk
> vgextend
> pvmove to the new pv
> vgreduce to remove the old disk
> pvremove to remove the old scsi disk definition.
>
> The old disks and new ones used different FCP devices. After the pvmove,
> we cleaned up the scsi disk infomation and verified all was good with
> multipath -ll and pvs. Everything looked good. The old fcp devices were
> taken offline with zfcp_host_configure 0.0.2000 0 followed by vmcp det
> fcpdevice.
> Everything was working fine until we rebooted one of the systems using the
> new LUNS.
>
> The system went into emergency mode.
> The following action was taken to recover:
>
> The LVM disks were commented out of fstab and the system was rebooted. The
> system came up clean without the SCSI disks.
>
> At this point, I issued a lvchange to see if that would correct the
> issue.
>
> lvchange -ay /dev/u01/u01
> lvchange -ay /dev/u02/u02
> lvchange -ay /dev/u034/u03
> lvchange -ay /dev/u034/u04
> lvchange -ay /dev/u056/u05
> lvchange -ay /dev/u056/u06
>
> It worked like a charm, but this is only dynamic temp fix. Reboot the
> system again and we are back to square one.
>
> I next tried to make the change persistant with the follow, lvchange -ay
> -M y --minor 145 u01
>
> This did not apear to do anything. I still get the NOT AVAILABLE status
> in lvdisplay.
>
> However, some of the VG's have two LV's and now I get a duplicate minor
> nunber when I run lvchange on that set.
> lvchange -ay /dev/u056/u05
> lvchange -ay /dev/u056/u06
> (For now, lets focus on the original issue)
>
> At this point I am at a loss. I am not sure why the LVM will not come
> online at boot. As a work-a-round, I have removed the LVM's volumes from
> FSTAB and manually issue the lvchange after the system comes up and then
> mount the LVM's. This is not a valid fix, but at least we are not dead in
> the water.
>
> Degug time:
> I have turned on debug for lvm in lvm.conf and have captured a LVM debug
> log.
>
> In the log, I can see each of the SCSI disks are found:
> device/dev-cache.c:714 Found dev 8:48
> /dev/disk/by-id/scsi-3600507681081026d3800000000000119 - exists.
>
> I also see this orphans message in the log that may be part of my issue.
> cache/lvmcache.c:2080 lvmcache /dev/sda: now in VG #orphans_lvm2
> (#orphans_lvm2) with 0 mda(s).
> filters/filter-signature.c:31 filter signature deferred /dev/sdao
> filters/filter-md.c:99 filter md deferred /dev/sdao
> format_text/text_label.c:423 /dev/sda: PV header extension version 2
> found
> filters/filter-persistent.c:346 filter caching good /dev/sdao
> format_text/format-text.c:331 Reading mda header sector from /dev/sda
> at 4096
>
> I found all the audit information in /etc/lvm/backup and /etc/lvm/archive
> and can see the changes that were made when I moved from the old SCSI disks
> to the new ones on the fs7200.
>
> What I don't understand is why LVM is not able to bring the logical
> volume's online when the PV's and VG's look good after reboot and simply
> issuing “lvchange -ay /dev/xxxx” dynamically fixes the issue.
>
> Any help would be greatly appriciated. Until I can resolve this, I have
> stopped the migration of any additional systems from the old disks to the
> new. In total, I have moved about 15 systems so far.
>
> I do have a backout plan, but that requires bringing down the production
> systems and doing a lot of manual work. If I can fix this dynamically, that
> would be great. If I need to issue a few commands and reboot the system to
> fix it, that would also be acceptable since the outage would be mininal.
> What I don't want to do is have to take the systems down for an extended
> amount of time to rebuild and copy to new disks.
>
> If additional information is needed, just let me know and I will provide
> it.
>
> Thanks in advance.
> Duane
>
> Duane Beyer
> Marist College
>
>
> Scanned by McAfee and confirmed virus-free.
> Find out more here: https://bit.ly/2zCJMrO
>
> ----------------------------------------------------------------------
> For LINUX-390 subscribe / signoff / archive access instructions, send
> email to [email protected] with the message: INFO LINUX-390 or visit
> http://www2.marist.edu/htbin/wlvindex?LINUX-390
>
> ----------------------------------------------------------------------
> For LINUX-390 subscribe / signoff / archive access instructions, send
> email to [email protected] with the message: INFO LINUX-390 or visit
> http://www2.marist.edu/htbin/wlvindex?LINUX-390
>
> ----------------------------------------------------------------------
> For LINUX-390 subscribe / signoff / archive access instructions, send
> email to [email protected] with the message: INFO LINUX-390 or visit
> http://www2.marist.edu/htbin/wlvindex?LINUX-390
>
> ----------------------------------------------------------------------
> For LINUX-390 subscribe / signoff / archive access instructions, send
> email to [email protected] with the message: INFO LINUX-390 or visit
> http://www2.marist.edu/htbin/wlvindex?LINUX-390
>
> ----------------------------------------------------------------------
> For LINUX-390 subscribe / signoff / archive access instructions,
> send email to [email protected] with the message: INFO LINUX-390 or
> visit
> http://www2.marist.edu/htbin/wlvindex?LINUX-390
>
--
-Mike MacIsaac
----------------------------------------------------------------------
For LINUX-390 subscribe / signoff / archive access instructions,
send email to [email protected] with the message: INFO LINUX-390 or visit
http://www2.marist.edu/htbin/wlvindex?LINUX-390
