Hello all,
I have been struggled with ZFS and my data on the OpenSolaris 2009.06 and
Solaris 11. Last month, my ZFS pool tank (with RAIDz1 configured) became
unavailable and 4 out of 6 SCSI disks could not be recognized by OpenSolaris
#format command.
1) The four missing Seagate disks (1000.20GB) are c7t0d0, c7t1d0, c7t3d0,
and c7t4d0.
root@MEL-SUN-X2270:~# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c7t2d0 <ATA-ST31000528AS-CC38-931.51GB>
/pci@0,0/pci8086,340e@7/pci1000,3150@0/sd@2,0
1. c7t5d0 <ATA-ST31000528AS-CC37-931.51GB>
/pci@0,0/pci8086,340e@7/pci1000,3150@0/sd@5,0
2. c9d0 <DEFAULT cyl 60797 alt 2 hd 255 sec 63>
/pci@0,0/pci-ide@1f,2/ide@0/cmdk@0,0
3. c10d0 <DEFAULT cyl 60797 alt 2 hd 255 sec 63>
/pci@0,0/pci-ide@1f,2/ide@1/cmdk@0,0
4. c10d1 <DEFAULT cyl 3888 alt 2 hd 255 sec 63>
/pci@0,0/pci-ide@1f,2/ide@1/cmdk@1,0
root@MEL-SUN-X2270:~# iostat -E
cmdk0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Model: HITACHI HUA7250 Revision: Serial No: GTF402P6GUUS3F Size: 500.10GB
<500101152768 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0
cmdk1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Model: HITACHI HUA7250 Revision: Serial No: GTF402P6GUUGEF Size: 500.10GB
<500101152768 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0
cmdk2 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Model: SSDSA2SH032G1SB Revision: Serial No: CVEM02830008032 Size: 32.00GB
<31999500288 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0
sd1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: ATA Product: ST31000528AS Revision: CC35 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
sd2 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: ATA Product: ST31000528AS Revision: CC35 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
sd3 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: ATA Product: ST31000528AS Revision: CC38 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
sd4 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: ATA Product: ST31000528AS Revision: CC37 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
sd5 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: ATA Product: ST31000528AS Revision: CC35 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
sd6 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: ATA Product: ST31000528AS Revision: CC37 Serial No:
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
root@MEL-SUN-X2270:~# zpool status -v
pool: rpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c9d0s0 ONLINE 0 0 0
c10d0s0 ONLINE 0 0 0
errors: No known data errors
pool: tank
state: UNAVAIL
status: One or more devices could not be opened. There are insufficient
replicas for the pool to continue functioning.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-3C
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank UNAVAIL 0 0 0 insufficient replicas
raidz1 UNAVAIL 0 0 0 insufficient replicas
c7t0d0 UNAVAIL 0 0 0 cannot open
c7t1d0 UNAVAIL 0 0 0 cannot open
c7t2d0 ONLINE 0 0 0
c7t3d0 UNAVAIL 0 0 0 cannot open
c7t4d0 UNAVAIL 0 0 0 cannot open
c7t5d0 ONLINE 0 0 0
pool: temppool1
state: UNAVAIL
status: One or more devices could not be opened. There are insufficient
replicas for the pool to continue functioning.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-3C
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
temppool1 UNAVAIL 0 0 0 insufficient replicas
c13t0d0 UNAVAIL 0 0 0 cannot open
2) I did try power-cycle my SunFire X2270 server and the J4200 array. The
tank was remained faulted and the disks were not visible by the OpenSolaris.
The same result when I tried command #zpool status -x tank.
Below are the warning messages I get.
Jun 18 22:30:37 MEL-SUN-X2270 scsi: [ID 243001 kern.warning] WARNING:
/pci@0,0/pci8086,340e@7/pci1000,3150@0 (mpt0):
Jun 18 22:30:37 MEL-SUN-X2270 mpt_handle_event_sync: IOCStatus=0x8000,
IOCLogInfo=0x31110700
Jun 18 22:30:37 MEL-SUN-X2270 scsi: [ID 243001 kern.warning] WARNING:
/pci@0,0/pci8086,340e@7/pci1000,3150@0 (mpt0):
Jun 18 22:30:37 MEL-SUN-X2270 mpt_handle_event: IOCStatus=0x8000,
IOCLogInfo=0x31110700
Jun 18 22:30:39 MEL-SUN-X2270 scsi: [ID 365881 kern.info]
/pci@0,0/pci8086,340e@7/pci1000,3150@0 (mpt0):
Jun 18 22:30:39 MEL-SUN-X2270 Log info 0x31110700 received for target 1.
Jun 18 22:30:39 MEL-SUN-X2270 scsi_status=0x0, ioc_status=0x804b,
scsi_state=0xc
Jun 18 22:30:39 MEL-SUN-X2270 scsi: [ID 365881 kern.info]
/pci@0,0/pci8086,340e@7/pci1000,3150@0 (mpt0):
Jun 18 22:30:39 MEL-SUN-X2270 Log info 0x31110700 received for target 1.
Jun 18 22:30:39 MEL-SUN-X2270 scsi_status=0x0, ioc_status=0x804b,
scsi_state=0xc
Jun 18 22:30:39 MEL-SUN-X2270 scsi: [ID 365881 kern.info]
/pci@0,0/pci8086,340e@7/pci1000,3150@0 (mpt0):
Jun 18 22:30:39 MEL-SUN-X2270 Log info 0x31110700 received for target 1.
messages.1
3) I have installed Solaris 11 on another disk and tried run #format
command. I am able to see all the disks within the array now.
root@solaris11:~# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c3t0d0 <ATA-ST31000528AS-CC35-931.51GB>
/pci@0,0/pci8086,340e@7/pci1000,3150@0/sd@0,0
/dev/chassis/SUN-Storage-J4200.0939QAJ006/SCSI_Device__0/disk
1. c3t1d0 <ATA-ST31000528AS-CC35-931.51GB>
/pci@0,0/pci8086,340e@7/pci1000,3150@0/sd@1,0
/dev/chassis/SUN-Storage-J4200.0939QAJ006/SCSI_Device__1/disk
2. c3t2d0 <ATA-ST31000528AS-CC38-931.51GB>
/pci@0,0/pci8086,340e@7/pci1000,3150@0/sd@2,0
/dev/chassis/SUN-Storage-J4200.0939QAJ006/SCSI_Device__2/disk
3. c3t3d0 <ATA-ST31000528AS-CC49 cyl 60797 alt 2 hd 255 sec 126>
/pci@0,0/pci8086,340e@7/pci1000,3150@0/sd@3,0
/dev/chassis/SUN-Storage-J4200.0939QAJ006/SCSI_Device__3/disk
4. c3t4d0 <ATA-ST31000528AS-CC35-931.51GB>
/pci@0,0/pci8086,340e@7/pci1000,3150@0/sd@4,0
/dev/chassis/SUN-Storage-J4200.0939QAJ006/SCSI_Device__4/disk
5. c3t5d0 <ATA-ST31000528AS-CC37-931.51GB>
/pci@0,0/pci8086,340e@7/pci1000,3150@0/sd@5,0
/dev/chassis/SUN-Storage-J4200.0939QAJ006/SCSI_Device__5/disk
6. c5d0 <MCBQE64G- SE752X069-0001 cyl 7780 alt 2 hd 255 sec 63>
/pci@0,0/pci-ide@1f,2/ide@0/cmdk@0,0
4) I try to import the zpool from the new Solaris 11, below is the result.
root@solaris11:~# zpool import
pool: tank
id: 4902002746703797589
state: FAULTED
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
The pool may be active on another system, but can be imported using
the '-f' flag.
see: http://www.sun.com/msg/ZFS-8000-EY
config:
tank FAULTED corrupted data
raidz1-0 DEGRADED
c3t0d0 ONLINE
c3t1d0 ONLINE
c3t2d0 ONLINE
c7t3d0 UNAVAIL cannot open
c3t4d0 ONLINE
c3t5d0 ONLINE
5) Then I run the command #zpool import -f tank, it fails with error below.
root@solaris11:~# zpool import -f tank
cannot import 'tank': I/O error
Destroy and re-create the pool from
a backup source.
What can I do next to bring back my zpool as I have some of important VMs
stored on this tank?
Your input and advice given is highly appreciated!
Regards,
Lee
EMAIL DISCLAIMER This email message and its attachments are confidential and
may also contain copyright or privileged material. If you are not the intended
recipient, you may not forward the email or disclose or use the information
contained in it. If you have received this email message in error, please
advise the sender immediately by replying to this email and delete the message
and any associated attachments. Any views, opinions, conclusions, advice or
statements expressed in this email message are those of the individual sender
and should not be relied upon as the considered view, opinion, conclusions,
advice or statement of this company except where the sender expressly, and with
authority, states them to be the considered view, opinion, conclusions, advice
or statement of this company. Every care is taken but we recommend that you
scan any attachments for viruses.
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
