Deric, Thanks for the call, and for helping solve my problem.
boot.multipath was not set to run at boot and was causing the vg to not be discovered. VR, Justin Sollenberger -----Original Message----- From: Linux on 390 Port [mailto:[email protected]] On Behalf Of Deric Abel Sent: Tuesday, June 16, 2009 12:07 To: [email protected] Subject: Re: Multipathing/LVM problem (was: FCP LUN multipathing / filesystem problem) It sounds as if you have a few other issues (I have sent you a private message, feel free to follow that one if needed). First you need to make sure that the following boot.* are set to run at boot up. chkconfig boot.lvm on chkconfig boot.multipath on These need to be on as well, but I suspect they already are. chkconfig lvm on chkconfig multipathd on Also, it's ok that your vg is /dev/dm-0, the major/minor device numbers should line up with the same /dev/mapper/XXXXXX, /dev/mapper/XXXXXX is an alias really. However, if you make sure that /dev/mapper/* is at the beginning in your lvm.conf filter section, that should force lvm to use /dev/mapper/* first (at least that is how I got it work on my end). Feel free to call my number that I left in my private message earlier. Deric >>> "Sollenberger, Justin W Mr CIV US DISA CDB24" <[email protected]> 6/16/2009 9:36 AM >>> Deric, Thanks for the response. I believe that is the issue, but I am unsure how to correct the mess that I'm in. I tried adjusting the lvm.conf, but because my vg is not on a /dev/mapper/xxxxxxx device it is not recognized at boot. A pvscan shows that the PV is /dev/dm-0. I added "a|/dev/dm-*|" to the filter but it still didn't recognize the vg (I'm not sure that is the route I should go anyway). Is there a way to correct the location of the vg? I created a new vg on another system using the configuration Deric provided and it worked fine with a pvscan showing a PV of /dev/mapper/xxxxxxx. How would you suggest correcting the current setup? VR, Justin Sollenberger Linux on System z Administrator Operating Systems, CDB24 Defense Information Systems Agency DECC Mechanicsburg DSN: 430-8386 Comm: 717-605-8386 [email protected] -----Original Message----- From: Linux on 390 Port [mailto:[email protected]] On Behalf Of Deric Abel Sent: Thursday, June 11, 2009 17:24 To: [email protected] Subject: Re: FCP LUN multipathing / filesystem problem Justin, I had this same problem.... What is going on is, lvm using only one path, because of the way your lvm.conf is configured. These are the changes I made to solve this problem. - scan = [ "/dev/" ] + scan = [ "/dev/", "/dev/mapper" ] - filter = ["r|/dev/.*/by-path/.*|", "r|/dev/.*/by-id/.*|", "a/.*/" ] + filter = [ "a|/dev/mapper/*|","r|/dev/sd*|","r|/dev/.*/by-path/.*|", "r|/dev/.*/by-id/.*|", "a/.*/" ] + types = [ "device-mapper", 16 ] If you install sysstat and run iostat -x -m you can see if it is in fact using all paths... as an FYI this is my multipath.conf as well. multipaths { multipath { wwid 3600507680191016e0000000000000110 alias sanvol1 path_grouping_policy multibus rr_min_io 100 rr_weight uniform failback immediate } } blacklist { devnode "^(dasd)[a-f]*" } >>> "Sollenberger, Justin W Mr CIV US DISA CDB24" <[email protected]> 6/11/2009 12:06 PM >>> We have several SLES 10 SP2 guests that are using FCP LUNS. The OS is installed on 3390's and the LUNS are used to store the customer's data. Early this morning we lost one of the paths. Some of our systems reacted as I would have expected but others did not. The ones that reacted oddly would be the ones that I would assume to be busy at that time. Those systems show I/O errors, followed by filesystem errors, and then the filesystems all remount as read-only. I've include output for both scenarios. Reacted Correctly: Jun 11 03:44:34 system_a multipathd: sdb: tur checker reports path is down Jun 11 03:44:34 system_a kernel: device-mapper: multipath: Failing path 8:16. Jun 11 03:44:39 system_a multipathd: sdb: tur checker reports path is up Reacted Oddly: Jun 11 03:44:34 system_b multipathd: sdb: tur checker reports path is down Jun 11 03:44:34 system_b kernel: end_request: I/O error, dev sdb, sector 37486992 Jun 11 03:44:34 system_b kernel: Buffer I/O error on device dm-0, logical block 4685826 Jun 11 03:44:34 system_b kernel: lost page write due to I/O error on dm-0 Jun 11 03:44:34 system_b kernel: end_request: I/O error, dev sdb, sector 42200120 Jun 11 03:44:34 system_b kernel: end_request: I/O error, dev sdb, sector 47170040 Jun 11 03:44:34 system_b kernel: Buffer I/O error on device dm-1, logical block 653327 Jun 11 03:44:34 system_b kernel: lost page write due to I/O error on dm-1 Jun 11 03:44:34 system_b kernel: Buffer I/O error on device dm-1, logical block 653328 Jun 11 03:44:34 system_b kernel: lost page write due to I/O error on dm-1 Jun 11 03:44:34 system_b kernel: Buffer I/O error on device dm-1, logical block 653329 Jun 11 03:44:34 system_b kernel: lost page write due to I/O error on dm-1 Jun 11 03:44:34 system_b kernel: Buffer I/O error on device dm-1, logical block 653330 Jun 11 03:44:35 system_b kernel: lost page write due to I/O error on dm-1 Jun 11 03:44:35 system_b kernel: end_request: I/O error, dev sdb, sector 50598296 Jun 11 03:44:35 system_b kernel: Buffer I/O error on device dm-1, logical block 1081859 Jun 11 03:44:35 system_b kernel: lost page write due to I/O error on dm-1 Jun 11 03:44:35 system_b kernel: Buffer I/O error on device dm-1, logical block 1081860 Jun 11 03:44:35 system_b kernel: lost page write due to I/O error on dm-1 Jun 11 03:44:35 system_b kernel: Buffer I/O error on device dm-1, logical block 1081861 Jun 11 03:44:35 system_b kernel: lost page write due to I/O error on dm-1 Jun 11 03:44:35 system_b kernel: Buffer I/O error on device dm-1, logical block 1081862 Jun 11 03:44:35 system_b kernel: lost page write due to I/O error on dm-1 Jun 11 03:44:35 system_b kernel: end_request: I/O error, dev sdb, sector 69213616 Jun 11 03:44:35 system_b kernel: end_request: I/O error, dev sdb, sector 58947168 Jun 11 03:44:35 system_b kernel: end_request: I/O error, dev sdb, sector 37337552 Jun 11 03:44:35 system_b kernel: Buffer I/O error on device dm-0, logical block 4667146 Jun 11 03:44:35 system_b kernel: lost page write due to I/O error on dm-0 Jun 11 03:44:35 system_b kernel: device-mapper: multipath: Failing path 8:16. Jun 11 03:44:35 system_b kernel: Aborting journal on device dm-0. Jun 11 03:44:35 system_b kernel: end_request: I/O error, dev sdb, sector 42200136 Jun 11 03:44:35 system_b kernel: end_request: I/O error, dev sdb, sector 69213640 Jun 11 03:44:35 system_b kernel: end_request: I/O error, dev sdb, sector 58947184 Jun 11 03:44:35 system_b kernel: Aborting journal on device dm-1. Jun 11 03:44:35 system_b kernel: Aborting journal on device dm-3. Jun 11 03:44:35 system_b kernel: Aborting journal on device dm-2. Jun 11 03:44:35 system_b kernel: ext3_abort called. Jun 11 03:44:35 system_b kernel: EXT3-fs error (device dm-2): ext3_journal_start_sb: Detected aborted journal Jun 11 03:44:35 system_b kernel: Remounting filesystem read-only Jun 11 03:44:35 system_b kernel: ext3_abort called. Jun 11 03:44:35 system_b kernel: EXT3-fs error (device dm-3): ext3_journal_start_sb: Detected aborted journal Jun 11 03:44:35 system_b kernel: Remounting filesystem read-only Jun 11 03:44:35 system_b kernel: ext3_abort called. Jun 11 03:44:35 system_b kernel: EXT3-fs error (device dm-0): ext3_journal_start_sb: Detected aborted journal Jun 11 03:44:35 system_b kernel: Remounting filesystem read-only Jun 11 03:44:35 system_b kernel: EXT3-fs error (device dm-1) in ext3_reserve_inode_write: Journal has aborted Jun 11 03:44:35 system_b kernel: EXT3-fs error (device dm-1) in ext3_dirty_inode: Journal has aborted Jun 11 03:44:35 system_b kernel: ext3_abort called. Jun 11 03:44:35 system_b kernel: EXT3-fs error (device dm-1): ext3_journal_start_sb: Detected aborted journal Jun 11 03:44:35 system_b kernel: Remounting filesystem read-only Jun 11 03:44:39 system_b multipathd: sdb: tur checker reports path is up Any advice or direction would be greatly appreciated. Thanks in advance. VR, Justin Sollenberger ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
