Re: [Veritas-vx] vxdisk path question/concern
The problem here is the DMP device tree is confused after reboot and that is due to the disk.info The clean up DMP device tree: 1. rm /etc/vx/disk.info 2. rm /dev/vx/dmp/* 3. rm /dev/vx/rdmp/* 4. vxconfigd -k - Original Message From: joel j...@saldino.net To: veritas-vx@mailman.eng.auburn.edu Sent: Saturday, 18 July, 2009 11:27:06 AM Subject: [Veritas-vx] vxdisk path question/concern So I had an issue the other night where a disk was bad and replaced. Not sure why this happened but for some reason it seemed that all the drives on my array dropped off the scsi bus thus taking down the entire thing and locking up the box. On reboot everything was lost as far as the disk groups but a co-worker was able to recover. The drive in question that was bad was sde in a goup of 16 disks. The new drive is installed but for some reason the path is all wonky now. [r...@p02cd01 ~]# vxdisk list DEVICE TYPEDISK GROUPSTATUS cciss/c0d0 auto:none --online invalid sda auto:cdsdisksilo2dg01silo2dg online nohotuse sdb auto:cdsdisksilo2dg02silo2dg online nohotuse sdc auto:cdsdisksilo2dg03silo2dg online nohotuse sdd auto:cdsdisksilo2dg04silo2dg online nohotuse sde auto:cdsdisksilo2dg06silo2dg online nohotuse sdf auto--error sdg auto:cdsdisksilo2dg07silo2dg online nohotuse sdh auto:cdsdisksilo2dg08silo2dg online nohotuse sdi auto:cdsdisksilo2dg09silo2dg online nohotuse sdj auto:cdsdisksilo2dg10silo2dg online nohotuse sdk auto:cdsdisksilo2dg11silo2dg online nohotuse sdl auto:cdsdisksilo2dg12silo2dg online nohotuse sdm auto:cdsdisksilo2dg13silo2dg online nohotuse sdn auto:cdsdisksilo2dg14silo2dg online nohotuse sdo auto:cdsdisksilo2dg15silo2dg online nohotuse sdp auto:cdsdisksilo2dg16silo2dg online nohotuse -- silo2dg05silo2dg removed nohotuse was:sde [r...@p02cd01 ~]# vxdisk path SUBPATH DANAME DMNAME GROUPSTATE cciss/c0d0 cciss/c0d0 - -ENABLED sda sda silo2dg01 silo2dg ENABLED sdb sdb silo2dg02 silo2dg ENABLED sdc sdc silo2dg03 silo2dg ENABLED sdd sdd silo2dg04 silo2dg ENABLED sdf sde silo2dg06 silo2dg ENABLED sde sdf - -ENABLED sdg sdg silo2dg07 silo2dg ENABLED sdh sdh silo2dg08 silo2dg ENABLED sdi sdi silo2dg09 silo2dg ENABLED sdj sdj silo2dg10 silo2dg ENABLED sdk sdk silo2dg11 silo2dg ENABLED sdl sdl silo2dg12 silo2dg ENABLED sdm sdm silo2dg13 silo2dg ENABLED sdn sdn silo2dg14 silo2dg ENABLED sdo sdo silo2dg15 silo2dg ENABLED sdp sdp silo2dg16 silo2dg ENABLED [r...@p02cd01 ~]# vxdisk list sdf Device:sdf devicetag: sdf type: auto flags: online error private autoconfig errno: Disk is not usable Multipathing information: numpaths: 1 sdestate=enabled [r...@p02cd01 ~]# vxdisk list sde Device:sde devicetag: sde type: auto hostid:opsdev1.internal.piczo.com disk: name=silo2dg06 id=1219276096.30.opsdev1.internal.piczo.com group: name=silo2dg id=1219276137.52.opsdev1.internal.piczo.com info: format=cdsdisk,privoffset=256,pubslice=3,privslice=3 flags: online ready private autoconfig nohotuse autoimport imported pubpaths: block=/dev/vx/dmp/sde3 char=/dev/vx/rdmp/sde3 version: 3.1 iosize:min=512 (bytes) max=65535 (blocks) public:slice=3 offset=2304 len=143366528 disk_offset=0 private: slice=3 offset=256 len=2048 disk_offset=0 update:time=1247816427 seqno=0.28 ssb: actual_seqno=0.0 headers: 0 240 configs: count=1 len=1280 logs: count=1 len=192 Defined regions: config priv 48-000239[000192]: copy=01 offset=00 enabled config priv 000256-001343[001088]: copy=01 offset=000192 enabled log
Re: [Veritas-vx] vxdisk path question/concern [SEC=UNCLASSIFIED]
Everything is back to normal on the box. Thanks for all the help with this I appreciate it much! ~joel Robinson, Greg wrote: UNCLASSIFIED Hi Joel, I am unsure as to why vxrecover has wedged itself like this. It does look like it is trying to abort it's current task and cannont for some reason. I would err on the side of caution and leave it until the next maintenance window and reboot the box. However, Veritas support may know better. What you should do now is recover those volumes in the NEEDSYNC state. Eg.: v mysqlData01-L01 fsgen ENABLED 13107200 -NEEDSYNC - - pl mysqlData01-P01 mysqlData01-L01 ENABLED 13107200 - ACTIVE - - sd silo2dg01-05 mysqlData01-P01 ENABLED 13107200 0 -- - pl mysqlData01-P02 mysqlData01-L01 ENABLED 13107200 - ACTIVE - - sd silo2dg09-05 mysqlData01-P02 ENABLED 13107200 0 -- - Something like: # vxrecover -b mysqlData01-L01 And then wait for it to finish re-syncing the mirror. Repeat for the rest of the volumes in that same state. You don't want to do more than 1 or 2 vxrecovers at once, all the I/O may bring the box to it's knees. I've done that before :-) Greg. -Original Message- From: veritas-vx-boun...@mailman.eng.auburn.edu [mailto:veritas-vx-boun...@mailman.eng.auburn.edu] On Behalf Of joel Sent: Tuesday, 21 July 2009 2:38 PM To: Robinson, Greg Cc: veritas-vx@mailman.eng.auburn.edu Subject: Re: [Veritas-vx] vxdisk path question/concern [SEC=UNCLASSIFIED] Nice I appreciate all your help with this. I am understanding the ins and outs a much better. I removed the disk silo2gd17 without a problem. One last thing is this task that's running. It was started by another operator a few days ago and now it just seems to be stuck: [r...@p02cd01 ~]# vxtask -l -h list Task: 162 ABORTING Type: PARENT Operation: VXRECOVER Started: Fri 17 Jul 2009 12:41:56 AM PDT Progress: 13.56% (8 of 59 jobs, 2 active) root 4489 0.0 0.0 4732 1712 ?Ss Jul17 0:00 vxrecover -g silo2dg -sb This was done I am assuming because of the bad state the system was in at the time. Any ideas on that one? I don't want to kill off the job if it's going to have repercussions. Thanks again ~joel Robinson, Greg wrote: UNCLASSIFIED Hi Joel, Well, everything is looking good. I'd run vxdiskadm again and chose option 3, to remove that disk, silo2dg17. Any volumes in the NEEDSYNC state will recover by themselves. Any others, will need a manual recover. Something like: # vxrecover -b mysqlData07-L05 For example. Greg. -Original Message- From: joel [mailto:j...@saldino.net] Sent: Tuesday, 21 July 2009 11:10 AM To: Robinson, Greg Cc: veritas-vx@mailman.eng.auburn.edu Subject: Re: [Veritas-vx] vxdisk path question/concern [SEC=UNCLASSIFIED] I was able to add sdf back in as the replacement and it seems to be recovering now. Here is the output you asked for, it's alot. silo2dg17 was my mistake when I added it back in I just picked the default name which went to the next number up. I removed that and re-added as silo2dg05 and it seemed to have worked fine. [r...@p02cd01 ~]# vxprint Disk group: silo2dg TY NAME ASSOCKSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0 dg silo2dg silo2dg ----- - dm silo2dg01sda -143366528 - NOHOTUSE - - dm silo2dg02sdb -143366528 - NOHOTUSE - - dm silo2dg03sdc -143366528 - NOHOTUSE - - dm silo2dg04sdd -143366528 - NOHOTUSE - - dm silo2dg05sdf -143366528 - NOHOTUSE - - dm silo2dg06sde -143366528 - NOHOTUSE - - dm silo2dg07sdg -143366528 - NOHOTUSE - - dm silo2dg08sdh -143366528 - NOHOTUSE - - dm silo2dg09sdi -143366528 - NOHOTUSE - - dm silo2dg10sdj -143366528 - NOHOTUSE - - dm silo2dg11sdk -143366528 - NOHOTUSE - - dm silo2dg12sdl -143366528 - NOHOTUSE - - dm silo2dg13sdm -143366528 - NOHOTUSE - - dm silo2dg14sdn -143366528 - NOHOTUSE - - dm silo2dg15sdo -143366528 - NOHOTUSE - - dm silo2dg16sdp -143366528 - NOHOTUSE - - dm silo2dg17----REMOVED - - [.. Snip ..] IMPORTANT: This email remains the property of the Australian Defence Organisation and is subject to the jurisdiction of section 70 of the Crimes Act 1914. If you have received this email in error, you are requested to contact the sender and delete the email. ___ Veritas-vx maillist - Veritas
Re: [Veritas-vx] vxdisk path question/concern [SEC=UNCLASSIFIED]
UNCLASSIFIED Hi Joel, The beauty of veritas is that it does not care about the SCSI disk names to make the disk group, and, over time, I have learned to let them go as well. What we need to concentrate on is the disk media names. The names in the DISK column below. So, according to vxdisk list output, silo2dg05 has died/been removed, and needs to be replaced. The replacement disk will be sdf (since it doesn't have a disk media name - yet) You should be able to use option 5 to do that. If you really want to disk media names to match the SCSI disk names, you can use vxedit to rename them, but it doesn't really matter. Greg. -Original Message- From: joel [mailto:j...@saldino.net] Sent: Tuesday, 21 July 2009 4:24 AM To: Robinson, Greg Cc: veritas-vx@mailman.eng.auburn.edu Subject: Re: [Veritas-vx] vxdisk path question/concern [SEC=UNCLASSIFIED] The problem is sde is the new drive and sdf is one that was already on the system. They seemed to have switched their paths somehow and I am not sure how to get around that. So right now sdf is being seen as sde in the list option. Not sure how they got into this particular state google searches haven't turned up anything relative. ~joel Robinson, Greg wrote: UNCLASSIFIED Hi Joel, I believe you can run vxdiskadm, option 5 Replace a failed or removed disk. The failed disk is sde and the replacement disk is sdf. It been so long since I've had to use that option, but I think it might also let you initalise the disk in that same option. Otherwise, option 1 to init a disk. Hope this helps, Greg. -Original Message- From: veritas-vx-boun...@mailman.eng.auburn.edu [mailto:veritas-vx-boun...@mailman.eng.auburn.edu] On Behalf Of joel Sent: Saturday, 18 July 2009 10:57 AM To: veritas-vx@mailman.eng.auburn.edu Subject: [Veritas-vx] vxdisk path question/concern So I had an issue the other night where a disk was bad and replaced. Not sure why this happened but for some reason it seemed that all the drives on my array dropped off the scsi bus thus taking down the entire thing and locking up the box. On reboot everything was lost as far as the disk groups but a co-worker was able to recover. The drive in question that was bad was sde in a goup of 16 disks. The new drive is installed but for some reason the path is all wonky now. [r...@p02cd01 ~]# vxdisk list DEVICE TYPEDISK GROUPSTATUS cciss/c0d0 auto:none --online invalid sda auto:cdsdisksilo2dg01silo2dg online nohotuse sdb auto:cdsdisksilo2dg02silo2dg online nohotuse sdc auto:cdsdisksilo2dg03silo2dg online nohotuse sdd auto:cdsdisksilo2dg04silo2dg online nohotuse sde auto:cdsdisksilo2dg06silo2dg online nohotuse sdf auto--error sdg auto:cdsdisksilo2dg07silo2dg online nohotuse sdh auto:cdsdisksilo2dg08silo2dg online nohotuse sdi auto:cdsdisksilo2dg09silo2dg online nohotuse sdj auto:cdsdisksilo2dg10silo2dg online nohotuse sdk auto:cdsdisksilo2dg11silo2dg online nohotuse sdl auto:cdsdisksilo2dg12silo2dg online nohotuse sdm auto:cdsdisksilo2dg13silo2dg online nohotuse sdn auto:cdsdisksilo2dg14silo2dg online nohotuse sdo auto:cdsdisksilo2dg15silo2dg online nohotuse sdp auto:cdsdisksilo2dg16silo2dg online nohotuse -- silo2dg05silo2dg removed nohotuse was:sde [r...@p02cd01 ~]# vxdisk path SUBPATH DANAME DMNAME GROUPSTATE cciss/c0d0 cciss/c0d0 - -ENABLED sda sda silo2dg01 silo2dg ENABLED sdb sdb silo2dg02 silo2dg ENABLED sdc sdc silo2dg03 silo2dg ENABLED sdd sdd silo2dg04 silo2dg ENABLED sdf sde silo2dg06 silo2dg ENABLED sde sdf - -ENABLED sdg sdg silo2dg07 silo2dg ENABLED sdh sdh silo2dg08 silo2dg ENABLED sdi sdi silo2dg09 silo2dg ENABLED sdj sdj silo2dg10 silo2dg ENABLED sdk sdk silo2dg11 silo2dg ENABLED sdl sdl
Re: [Veritas-vx] vxdisk path question/concern [SEC=UNCLASSIFIED]
UNCLASSIFIED Hi Joel, Well, everything is looking good. I'd run vxdiskadm again and chose option 3, to remove that disk, silo2dg17. Any volumes in the NEEDSYNC state will recover by themselves. Any others, will need a manual recover. Something like: # vxrecover -b mysqlData07-L05 For example. Greg. -Original Message- From: joel [mailto:j...@saldino.net] Sent: Tuesday, 21 July 2009 11:10 AM To: Robinson, Greg Cc: veritas-vx@mailman.eng.auburn.edu Subject: Re: [Veritas-vx] vxdisk path question/concern [SEC=UNCLASSIFIED] I was able to add sdf back in as the replacement and it seems to be recovering now. Here is the output you asked for, it's alot. silo2dg17 was my mistake when I added it back in I just picked the default name which went to the next number up. I removed that and re-added as silo2dg05 and it seemed to have worked fine. [r...@p02cd01 ~]# vxprint Disk group: silo2dg TY NAME ASSOCKSTATE LENGTH PLOFFS STATETUTIL0 PUTIL0 dg silo2dg silo2dg ----- - dm silo2dg01sda -143366528 - NOHOTUSE - - dm silo2dg02sdb -143366528 - NOHOTUSE - - dm silo2dg03sdc -143366528 - NOHOTUSE - - dm silo2dg04sdd -143366528 - NOHOTUSE - - dm silo2dg05sdf -143366528 - NOHOTUSE - - dm silo2dg06sde -143366528 - NOHOTUSE - - dm silo2dg07sdg -143366528 - NOHOTUSE - - dm silo2dg08sdh -143366528 - NOHOTUSE - - dm silo2dg09sdi -143366528 - NOHOTUSE - - dm silo2dg10sdj -143366528 - NOHOTUSE - - dm silo2dg11sdk -143366528 - NOHOTUSE - - dm silo2dg12sdl -143366528 - NOHOTUSE - - dm silo2dg13sdm -143366528 - NOHOTUSE - - dm silo2dg14sdn -143366528 - NOHOTUSE - - dm silo2dg15sdo -143366528 - NOHOTUSE - - dm silo2dg16sdp -143366528 - NOHOTUSE - - dm silo2dg17----REMOVED - - [.. Snip ..] IMPORTANT: This email remains the property of the Australian Defence Organisation and is subject to the jurisdiction of section 70 of the Crimes Act 1914. If you have received this email in error, you are requested to contact the sender and delete the email. ___ Veritas-vx maillist - Veritas-vx@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx
Re: [Veritas-vx] vxdisk path question/concern [SEC=UNCLASSIFIED]
UNCLASSIFIED Hi Joel, I am unsure as to why vxrecover has wedged itself like this. It does look like it is trying to abort it's current task and cannont for some reason. I would err on the side of caution and leave it until the next maintenance window and reboot the box. However, Veritas support may know better. What you should do now is recover those volumes in the NEEDSYNC state. Eg.: v mysqlData01-L01 fsgen ENABLED 13107200 -NEEDSYNC - - pl mysqlData01-P01 mysqlData01-L01 ENABLED 13107200 - ACTIVE - - sd silo2dg01-05 mysqlData01-P01 ENABLED 13107200 0 -- - pl mysqlData01-P02 mysqlData01-L01 ENABLED 13107200 - ACTIVE - - sd silo2dg09-05 mysqlData01-P02 ENABLED 13107200 0 -- - Something like: # vxrecover -b mysqlData01-L01 And then wait for it to finish re-syncing the mirror. Repeat for the rest of the volumes in that same state. You don't want to do more than 1 or 2 vxrecovers at once, all the I/O may bring the box to it's knees. I've done that before :-) Greg. -Original Message- From: veritas-vx-boun...@mailman.eng.auburn.edu [mailto:veritas-vx-boun...@mailman.eng.auburn.edu] On Behalf Of joel Sent: Tuesday, 21 July 2009 2:38 PM To: Robinson, Greg Cc: veritas-vx@mailman.eng.auburn.edu Subject: Re: [Veritas-vx] vxdisk path question/concern [SEC=UNCLASSIFIED] Nice I appreciate all your help with this. I am understanding the ins and outs a much better. I removed the disk silo2gd17 without a problem. One last thing is this task that's running. It was started by another operator a few days ago and now it just seems to be stuck: [r...@p02cd01 ~]# vxtask -l -h list Task: 162 ABORTING Type: PARENT Operation: VXRECOVER Started: Fri 17 Jul 2009 12:41:56 AM PDT Progress: 13.56% (8 of 59 jobs, 2 active) root 4489 0.0 0.0 4732 1712 ?Ss Jul17 0:00 vxrecover -g silo2dg -sb This was done I am assuming because of the bad state the system was in at the time. Any ideas on that one? I don't want to kill off the job if it's going to have repercussions. Thanks again ~joel Robinson, Greg wrote: UNCLASSIFIED Hi Joel, Well, everything is looking good. I'd run vxdiskadm again and chose option 3, to remove that disk, silo2dg17. Any volumes in the NEEDSYNC state will recover by themselves. Any others, will need a manual recover. Something like: # vxrecover -b mysqlData07-L05 For example. Greg. -Original Message- From: joel [mailto:j...@saldino.net] Sent: Tuesday, 21 July 2009 11:10 AM To: Robinson, Greg Cc: veritas-vx@mailman.eng.auburn.edu Subject: Re: [Veritas-vx] vxdisk path question/concern [SEC=UNCLASSIFIED] I was able to add sdf back in as the replacement and it seems to be recovering now. Here is the output you asked for, it's alot. silo2dg17 was my mistake when I added it back in I just picked the default name which went to the next number up. I removed that and re-added as silo2dg05 and it seemed to have worked fine. [r...@p02cd01 ~]# vxprint Disk group: silo2dg TY NAME ASSOCKSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0 dg silo2dg silo2dg ----- - dm silo2dg01sda -143366528 - NOHOTUSE - - dm silo2dg02sdb -143366528 - NOHOTUSE - - dm silo2dg03sdc -143366528 - NOHOTUSE - - dm silo2dg04sdd -143366528 - NOHOTUSE - - dm silo2dg05sdf -143366528 - NOHOTUSE - - dm silo2dg06sde -143366528 - NOHOTUSE - - dm silo2dg07sdg -143366528 - NOHOTUSE - - dm silo2dg08sdh -143366528 - NOHOTUSE - - dm silo2dg09sdi -143366528 - NOHOTUSE - - dm silo2dg10sdj -143366528 - NOHOTUSE - - dm silo2dg11sdk -143366528 - NOHOTUSE - - dm silo2dg12sdl -143366528 - NOHOTUSE - - dm silo2dg13sdm -143366528 - NOHOTUSE - - dm silo2dg14sdn -143366528 - NOHOTUSE - - dm silo2dg15sdo -143366528 - NOHOTUSE - - dm silo2dg16sdp -143366528 - NOHOTUSE - - dm silo2dg17----REMOVED - - [.. Snip ..] IMPORTANT: This email remains the property of the Australian Defence Organisation and is subject to the jurisdiction of section 70 of the Crimes Act 1914. If you have received this email in error, you are requested to contact the sender and delete the email. ___ Veritas-vx maillist - Veritas-vx@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx IMPORTANT: This email remains the property of the Australian Defence Organisation and is subject to the jurisdiction
