Re: [zfs-discuss] Does zpool clear delete corrupted files

[Jonathan Loran]

Well, I tried to clear the errors, but zpool clear didn't clear them.   
I think the errors are in the metadata in such a way that they can't  
be cleared.  I'm actually a bit scared to scrub it before I grab a  
backup, so I'm going to do that first.  After the backup, I need to  
break the mirror to pull the x4540 out, and I just hope that can  
succeed.  If not, we'll be loosing some data between the time the  
backup is taken and I roll out the new storage.


Let this be a double warning to all you zfs-ers out there:  Make sure  
you have redundancy at the zfs layer, and also do backups.   
Unfortunately for me, penny pinching has precluded both for us until  
now.


Jon

On Jun 1, 2009, at 4:19 PM, A Darren Dunham wrote:


On Mon, Jun 01, 2009 at 03:19:59PM -0700, Jonathan Loran wrote:


Kinda scary then.  Better make sure we delete all the bad files  
before

I back it up.


That shouldn't be necessary.  Clearing the error count doesn't disable
checksums.  Every read is going to verify checksums on the file data
blocks.  If it can't find at least one copy with a valid checksum,
you should just get an I/O error trying to read the file, not invalid
data.


What's odd is we've checked a few hundred files, and most of them
don't seem to have any corruption.  I'm thinking what's wrong is the
metadata for these files is corrupted somehow, yet we can read them
just fine.


Are you still getting errors?

--
Darren
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss




- _/ _/  /   - Jonathan Loran -   -
-/  /   /IT Manager   -
-  _  /   _  / / Space Sciences Laboratory, UC Berkeley
-/  / /  (510) 643-5146 jlo...@ssl.berkeley.edu
- __/__/__/   AST:7731^29u18e3




___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zpool import hangs

[Brad Reese]

Here's the output of 'zdb -e -bsvL tank' (without -c) in case it helps. I'll 
post with -c if it finishes.

Thanks,

Brad

Traversing all blocks ...
block traversal size 431585053184 != alloc 431585209344 (unreachable 156160)

bp count: 4078410
bp logical:433202894336  avg: 106218
bp physical:   431089822720  avg: 105700compression:   1.00
bp allocated:  431585053184  avg: 105821compression:   1.00
SPA allocated: 431585209344 used: 57.75%

Blocks  LSIZE   PSIZE   ASIZE avgcomp   %Total  Type
 -  -   -   -   -   --  deferred free
 1512 512  1K  1K1.00 0.00  object directory
 3  1.50K   1.50K   3.00K  1K1.00 0.00  object array
 116K 16K 32K 32K1.00 0.00  packed nvlist
 -  -   -   -   -   --  packed nvlist size
   114  13.9M   1.05M   2.11M   18.9K   13.25 0.00  bplist
 -  -   -   -   -   --  bplist header
 -  -   -   -   -   --  SPA space map header
 1.36K  6.57M   3.88M   7.94M   5.82K1.69 0.00  SPA space map
 -  -   -   -   -   --  ZIL intent log
 49.4K   791M220M442M   8.95K3.60 0.11  DMU dnode
 4 4K   2.50K   7.50K   1.88K1.60 0.00  DMU objset
 -  -   -   -   -   --  DSL directory
 2 1K  1K  2K  1K1.00 0.00  DSL directory child map
 1512 512  1K  1K1.00 0.00  DSL dataset snap map
 2 1K  1K  2K  1K1.00 0.00  DSL props
 -  -   -   -   -   --  DSL dataset
 -  -   -   -   -   --  ZFS znode
 -  -   -   -   -   --  ZFS V0 ACL
 3.77M   403G401G401G106K1.0099.87  ZFS plain file
 72.4K   114M   49.0M   98.8M   1.37K2.32 0.02  ZFS directory
 1512 512  1K  1K1.00 0.00  ZFS master node
 3  19.5K   1.50K   3.00K  1K   13.00 0.00  ZFS delete queue
 -  -   -   -   -   --  zvol object
 -  -   -   -   -   --  zvol prop
 -  -   -   -   -   --  other uint8[]
 -  -   -   -   -   --  other uint64[]
 -  -   -   -   -   --  other ZAP
 -  -   -   -   -   --  persistent error log
 1   128K   5.00K   10.0K   10.0K   25.60 0.00  SPA history
 -  -   -   -   -   --  SPA history offsets
 -  -   -   -   -   --  Pool properties
 -  -   -   -   -   --  DSL permissions
 -  -   -   -   -   --  ZFS ACL
 -  -   -   -   -   --  ZFS SYSACL
 -  -   -   -   -   --  FUID table
 -  -   -   -   -   --  FUID table size
 -  -   -   -   -   --  DSL dataset next clones
 -  -   -   -   -   --  scrub work queue
 3.89M   403G401G402G103K1.00   100.00  Total

capacity   operations   bandwidth   errors 
descriptionused avail  read write  read write  read write cksum
tank   402G  294G   463 0 1.27M 0 0 0 1
  mirror   402G  294G   463 0 1.27M 0 0 0 4
/dev/dsk/c2d0p0  69 0 4.05M 0 0 0 4
/dev/dsk/c1d0p0  67 0 3.96M 0 0 0 4
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zpool import hangs

[Brad Reese]

Hi Victor,

zdb -e -bcsvL tank
(let this go for a few hours...no output. I will let it go overnight)

zdb -e -u tank
Uberblock

magic = 00bab10c
version = 4
txg = 2435914
guid_sum = 16655261404755214374
timestamp = 1240517036 UTC = Thu Apr 23 15:03:56 2009

Thanks for your help,

Brad
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Does zpool clear delete corrupted files

[A Darren Dunham]

On Mon, Jun 01, 2009 at 03:19:59PM -0700, Jonathan Loran wrote:
> 
> Kinda scary then.  Better make sure we delete all the bad files before  
> I back it up.

That shouldn't be necessary.  Clearing the error count doesn't disable
checksums.  Every read is going to verify checksums on the file data
blocks.  If it can't find at least one copy with a valid checksum,
you should just get an I/O error trying to read the file, not invalid
data. 

> What's odd is we've checked a few hundred files, and most of them  
> don't seem to have any corruption.  I'm thinking what's wrong is the  
> metadata for these files is corrupted somehow, yet we can read them  
> just fine.

Are you still getting errors?

-- 
Darren
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Does zpool clear delete corrupted files

[Marion Hakanson]

jlo...@ssl.berkeley.edu said:
> What's odd is we've checked a few hundred files, and most of them   don't
> seem to have any corruption.  I'm thinking what's wrong is the   metadata for
> these files is corrupted somehow, yet we can read them   just fine.  I wish I
> could tell which ones are really bad, so we   wouldn't have to recreate them
> unnecessarily.  They are mirrored in   various places, or can be recreated
> via reprocessing, but recreating/  restoring that many files is no easy task.

You know, this sounds similar to what happened to me once when I did a
"zpool offline" to half of a mirror, changed a lot of stuff in the pool
(like adding 20GB of data to an 80GB pool), then "zpool online", thinking
ZFS might be smart enough to sync up the changes that had happened
since detaching.

Instead, a bunch of bad files were reported.  Since I knew nothing was
wrong with the half of the mirror that had never been offlined, I just
did a "zpool detach" of the formerly offlined drive, "zpool clear" to
clear the error counts, "zpool scrub" to check for integrity, then
"zpool attach" to cause resilver to start from scratch.

If this describes your situation, I guess the tricky part for you is to
now decide which half of your mirror is the good half.

There's always "rsync -n -v -a -c ..." to compare copies of files
that happen to reside elsewhere.  Slow but safe.

Regards,

Marion


___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Does zpool clear delete corrupted files

[Paul Choi]

If you run "zpool scrub" on the zpool, it'll do its best to identify the 
file(s) or filesystems/snapshots that have issues. Since you're on a 
single zpool, it won't self-heal any checksum errors... It'll take a 
long time, though, to scrub 30TB...


-Paul

Jonathan Loran wrote:


Kinda scary then.  Better make sure we delete all the bad files before 
I back it up.


What's odd is we've checked a few hundred files, and most of them 
don't seem to have any corruption.  I'm thinking what's wrong is the 
metadata for these files is corrupted somehow, yet we can read them 
just fine.  I wish I could tell which ones are really bad, so we 
wouldn't have to recreate them unnecessarily.  They are mirrored in 
various places, or can be recreated via reprocessing, but 
recreating/restoring that many files is no easy task.


Thanks,

Jon

On Jun 1, 2009, at 2:41 PM, Paul Choi wrote:

"zpool clear" just clears the list of errors (and # of checksum 
errors) from its stats. It does not modify the filesystem in any 
manner. You run "zpool clear" to make the zpool forget that it ever 
had any issues.


-Paul

Jonathan Loran wrote:


Hi list,

First off:
# cat /etc/release
  Solaris 10 6/06 s10x_u2wos_09a X86
 Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
  Use is subject to license terms.
   Assembled 09 June 2006

Here's an (almost) disaster scenario that came to life over the past 
week.  We have a very large zpool containing over 30TB, composed 
(foolishly) of three concatenated iSCSI SAN devices.  There's no 
redundancy in this pool at the zfs level.  We are actually in the 
process of migrating this to a x4540 + j4500 setup, but since the 
x4540 is part of the existing pool, we need to mirror it, then 
detach it so we can build out the replacement storage.
What happened was some time after I had attached the mirror to the 
x4540, the scsi_vhci/network connection went south, and the server 
panicked.  Since this system has been up, over the past 2.5 years, 
this has never happened before.  When we got the thing glued back 
together, it immediately started resilvering from the beginning, and 
reported about 1.9 million data errors.  The list from zpool status 
-v gave over 883k bad files.  This is a small percentage of the 
total number of files in this volume: over 80 million (1%).
My question is this:  When we clear the pool with zpool clear, what 
happens to all of the bad files?  Are they deleted from the pool, or 
do the error counters just get reset, leaving the bad files in 
tact?  I'm going to perform a full backup of this guy (not so easy 
on my budget), and I would rather only get the good files.


Thanks,

Jon


- _/ _/  /   - Jonathan Loran -   -
-/  /   /IT Manager   -
-  _  /   _  / / Space Sciences Laboratory, UC Berkeley
-/  / /  (510) 643-5146 
jlo...@ssl.berkeley.edu 

- __/__/__/   AST:7731^29u18e3



 



___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss







- _/ _/  /   - Jonathan Loran -   -
-/  /   /IT Manager   -
-  _  /   _  / / Space Sciences Laboratory, UC Berkeley
-/  / /  (510) 643-5146 jlo...@ssl.berkeley.edu
- __/__/__/   AST:7731^29u18e3




___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss



___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] 24x1TB ZFS system. Best practices for OS install without wasting space.

[Ray Van Dolson]

On Mon, Jun 01, 2009 at 03:30:11PM -0700, Maurice Volaski wrote:
> >So we have a 24x1TB system (from Silicon Mechanics).  It's using an LSI
> >SAS card so we don't have any hardware RAID virtual drive type options.
> >
> >Solaris 10 05/09
> >
> >I was hoping we could set up one large zpool (RAIDZ) from the installer
> >and set up a small zfs filesystem off of that for the OS leaving the
> >rest of the zpool for our data.
> >
> >However, it sounds like this isn't possible... and that a ZFS root
> >install must be done on a mirrored set of slices or disks.  So it looks
> >like either way we might be losing out on minimum 2TB (two disks).
> >
> >Obviously we could throw in a couple smaller drives internally, or
> >elsewhere... but are there any other options here?
> 
> Is this an R505? What I did was to install two 2.5" drives in one of 
> the internal bays. There's an adapter that lets you mount one atop 
> the other. You can then create a mirrored pool on these drives to 
> boot from. You can still have a DVD drive in the other bay, too.
> -- 

Yup, it is.  Sounds like a promising solution.

Where'd you get this adapter?

Thanks,
Ray
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] 24x1TB ZFS system. Best practices for OS install without wasting space.

[Maurice Volaski]

So we have a 24x1TB system (from Silicon Mechanics).  It's using an LSI
SAS card so we don't have any hardware RAID virtual drive type options.

Solaris 10 05/09

I was hoping we could set up one large zpool (RAIDZ) from the installer
and set up a small zfs filesystem off of that for the OS leaving the
rest of the zpool for our data.

However, it sounds like this isn't possible... and that a ZFS root
install must be done on a mirrored set of slices or disks.  So it looks
like either way we might be losing out on minimum 2TB (two disks).

Obviously we could throw in a couple smaller drives internally, or
elsewhere... but are there any other options here?


Is this an R505? What I did was to install two 2.5" drives in one of 
the internal bays. There's an adapter that lets you mount one atop 
the other. You can then create a mirrored pool on these drives to 
boot from. You can still have a DVD drive in the other bay, too.

--

Maurice Volaski, mvola...@aecom.yu.edu
Computing Support, Rose F. Kennedy Center
Albert Einstein College of Medicine of Yeshiva University
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Does zpool clear delete corrupted files

[Jonathan Loran]

Kinda scary then.  Better make sure we delete all the bad files before  
I back it up.


What's odd is we've checked a few hundred files, and most of them  
don't seem to have any corruption.  I'm thinking what's wrong is the  
metadata for these files is corrupted somehow, yet we can read them  
just fine.  I wish I could tell which ones are really bad, so we  
wouldn't have to recreate them unnecessarily.  They are mirrored in  
various places, or can be recreated via reprocessing, but recreating/ 
restoring that many files is no easy task.


Thanks,

Jon

On Jun 1, 2009, at 2:41 PM, Paul Choi wrote:

"zpool clear" just clears the list of errors (and # of checksum  
errors) from its stats. It does not modify the filesystem in any  
manner. You run "zpool clear" to make the zpool forget that it ever  
had any issues.


-Paul

Jonathan Loran wrote:


Hi list,

First off:
# cat /etc/release
  Solaris 10 6/06 s10x_u2wos_09a X86
 Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
  Use is subject to license terms.
   Assembled 09 June 2006

Here's an (almost) disaster scenario that came to life over the  
past week.  We have a very large zpool containing over 30TB,  
composed (foolishly) of three concatenated iSCSI SAN devices.   
There's no redundancy in this pool at the zfs level.  We are  
actually in the process of migrating this to a x4540 + j4500 setup,  
but since the x4540 is part of the existing pool, we need to mirror  
it, then detach it so we can build out the replacement storage.
What happened was some time after I had attached the mirror to the  
x4540, the scsi_vhci/network connection went south, and the server  
panicked.  Since this system has been up, over the past 2.5 years,  
this has never happened before.  When we got the thing glued back  
together, it immediately started resilvering from the beginning,  
and reported about 1.9 million data errors.  The list from zpool  
status -v gave over 883k bad files.  This is a small percentage of  
the total number of files in this volume: over 80 million (1%).
My question is this:  When we clear the pool with zpool clear, what  
happens to all of the bad files?  Are they deleted from the pool,  
or do the error counters just get reset, leaving the bad files in  
tact?  I'm going to perform a full backup of this guy (not so easy  
on my budget), and I would rather only get the good files.


Thanks,

Jon


- _/ _/  /   - Jonathan Loran  
-   -
-/  /   /IT  
Manager   -
-  _  /   _  / / Space Sciences Laboratory, UC  
Berkeley
-/  / /  (510) 643-5146 jlo...@ssl.berkeley.edu 
 

- __/__/__/   AST:7731^29u18e3





___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss







- _/ _/  /   - Jonathan Loran -   -
-/  /   /IT Manager   -
-  _  /   _  / / Space Sciences Laboratory, UC Berkeley
-/  / /  (510) 643-5146 jlo...@ssl.berkeley.edu
- __/__/__/   AST:7731^29u18e3




___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] 24x1TB ZFS system. Best practices for OS install without wasting space.

[Nicholas Lee]

IDE flash DOM?

On Tue, Jun 2, 2009 at 8:46 AM, Ray Van Dolson  wrote:

>
> Obviously we could throw in a couple smaller drives internally, or
> elsewhere... but are there any other options here?
>
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Does zpool clear delete corrupted files

[Paul Choi]

"zpool clear" just clears the list of errors (and # of checksum errors) 
from its stats. It does not modify the filesystem in any manner. You run 
"zpool clear" to make the zpool forget that it ever had any issues.


-Paul

Jonathan Loran wrote:


Hi list,

First off:  


# cat /etc/release
   Solaris 10 6/06 s10x_u2wos_09a X86
  Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
   Use is subject to license terms.
Assembled 09 June 2006

Here's an (almost) disaster scenario that came to life over the past 
week.  We have a very large zpool containing over 30TB, composed 
(foolishly) of three concatenated iSCSI SAN devices.  There's no 
redundancy in this pool at the zfs level.  We are actually in the 
process of migrating this to a x4540 + j4500 setup, but since the 
x4540 is part of the existing pool, we need to mirror it, 
then detach it so we can build out the replacement storage.  

What happened was some time after I had attached the mirror to the 
x4540, the scsi_vhci/network connection went south, and the 
server panicked.  Since this system has been up, over the past 2.5 
years, this has never happened before.  When we got the thing glued 
back together, it immediately started resilvering from the beginning, 
and reported about 1.9 million data errors.  The list from zpool 
status -v gave over 883k bad files.  This is a small percentage of the 
total number of files in this volume: over 80 million (1%).  

My question is this:  When we clear the pool with zpool clear, what 
happens to all of the bad files?  Are they deleted from the pool, or 
do the error counters just get reset, leaving the bad files in tact? 
 I'm going to perform a full backup of this guy (not so easy on my 
budget), and I would rather only get the good files.


Thanks,

Jon


- _/ _/  /   - Jonathan Loran -   -
-/  /   /IT Manager   -
-  _  /   _  / / Space Sciences Laboratory, UC Berkeley
-/  / /  (510) 643-5146 
jlo...@ssl.berkeley.edu 

- __/__/__/   AST:7731^29u18e3
 






___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
  


___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

[zfs-discuss] Does zpool clear delete corrupted files

[Jonathan Loran]

Hi list,

First off:

# cat /etc/release
   Solaris 10 6/06 s10x_u2wos_09a X86
  Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
   Use is subject to license terms.
Assembled 09 June 2006

Here's an (almost) disaster scenario that came to life over the past  
week.  We have a very large zpool containing over 30TB, composed  
(foolishly) of three concatenated iSCSI SAN devices.  There's no  
redundancy in this pool at the zfs level.  We are actually in the  
process of migrating this to a x4540 + j4500 setup, but since the  
x4540 is part of the existing pool, we need to mirror it, then detach  
it so we can build out the replacement storage.


What happened was some time after I had attached the mirror to the  
x4540, the scsi_vhci/network connection went south, and the server  
panicked.  Since this system has been up, over the past 2.5 years,  
this has never happened before.  When we got the thing glued back  
together, it immediately started resilvering from the beginning, and  
reported about 1.9 million data errors.  The list from zpool status -v  
gave over 883k bad files.  This is a small percentage of the total  
number of files in this volume: over 80 million (1%).


My question is this:  When we clear the pool with zpool clear, what  
happens to all of the bad files?  Are they deleted from the pool, or  
do the error counters just get reset, leaving the bad files in tact?   
I'm going to perform a full backup of this guy (not so easy on my  
budget), and I would rather only get the good files.


Thanks,

Jon


- _/ _/  /   - Jonathan Loran -   -
-/  /   /IT Manager   -
-  _  /   _  / / Space Sciences Laboratory, UC Berkeley
-/  / /  (510) 643-5146 jlo...@ssl.berkeley.edu
- __/__/__/   AST:7731^29u18e3




___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] 24x1TB ZFS system. Best practices for OS install without wasting space.

[Glenn Lagasse]

Hi Ray,

* Ray Van Dolson (rvandol...@esri.com) wrote:
> So we have a 24x1TB system (from Silicon Mechanics).  It's using an LSI
> SAS card so we don't have any hardware RAID virtual drive type options.
> 
> Solaris 10 05/09
> 
> I was hoping we could set up one large zpool (RAIDZ) from the installer
> and set up a small zfs filesystem off of that for the OS leaving the
> rest of the zpool for our data.
> 
> However, it sounds like this isn't possible... and that a ZFS root
> install must be done on a mirrored set of slices or disks.  So it looks
> like either way we might be losing out on minimum 2TB (two disks).
> 
> Obviously we could throw in a couple smaller drives internally, or
> elsewhere... but are there any other options here?

Currently, no.  ZFS can only boot from single or mirrord zpools.

Cheers,

-- 
Glenn
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

[zfs-discuss] 24x1TB ZFS system. Best practices for OS install without wasting space.

[Ray Van Dolson]

So we have a 24x1TB system (from Silicon Mechanics).  It's using an LSI
SAS card so we don't have any hardware RAID virtual drive type options.

Solaris 10 05/09

I was hoping we could set up one large zpool (RAIDZ) from the installer
and set up a small zfs filesystem off of that for the OS leaving the
rest of the zpool for our data.

However, it sounds like this isn't possible... and that a ZFS root
install must be done on a mirrored set of slices or disks.  So it looks
like either way we might be losing out on minimum 2TB (two disks).

Obviously we could throw in a couple smaller drives internally, or
elsewhere... but are there any other options here?

Thanks,
Ray
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] [?] - What is the recommended number of disks for a consumer PC with ZFS

[Chris Du]

I'm building my new storage server, all the parts should come in this week.

Intel XEON W3520 quad-core
12G DDR3-1333 ECC ram
2*74G 15K rpm SAS for OS
8*1T SATA disks in raiz2 or stripe 2 sets of 4-disk raidz
32G Intel X25-E SSD (may mirror it later)
2*Intel 82574L NIC
Qlogic 4Gb QLE2460 FC HBA

I have 8 SAS ports plus 6 SATA ports on motherboard, with the above config, I 
still have 2 SAS ports free to expand, I'm thinking of adding Supermicro 2.5in 
mobile rack with SAS expander later using the 2 SAS ports or add another SAS 
HBA.



You may want to replace i7-940 with XEON version and use ECC memory.

ZFS doesn't like RAID controller, at least my old SCSI raid with BBU cache. I 
had LSI megaraid 320-4x with onboard 128M BBU cache, the performance is 
horrible no matter how I tweak it. I replaced the raid controller with LSI 
21320 SCSI card, everything flies.
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] ZFS create hanging on door call?

[Stephen Green]

On Jun 1, 2009, at 4:57 AM, Darren J Moffat wrote:


Stephen Green wrote:
Hi, folks. I just built a new box and I'm running the latest  
OpenSolaris bits.  uname says:

SunOS blue 5.11 snv_111b i86pc i386 i86pc Solaris
I just did an image-update last night, but I was seeing this  
problem in 111a too.
I built myself a pool out of four 1TB disks (WD Caviar Green, if  
that matters):

 pool: tank
state: ONLINE
scrub: none requested
config:
   NAMESTATE READ WRITE CKSUM
   tankONLINE   0 0 0
 raidz1ONLINE   0 0 0
   c4d0ONLINE   0 0 0
   c4d1ONLINE   0 0 0
   c5d0ONLINE   0 0 0
   c5d1ONLINE   0 0 0
errors: No known data errors
I'm trying to create a file system in that pool to hold a mysql  
database using the command zfs create tank/mysql, but the create  
command is hanging.  Truss of the process shows:

stgr...@blue:~$ pgrep -lf zfs
7471 zfs create tank/mysql
stgr...@blue:~$ pfexec truss -p 7471
door_call(7, 0x080F7008)(sleeping...)
This is an RPC thing, right?
I had this happen a couple of times before the update yesterday as  
well.  The thing is, the file system actually got created:

stgr...@blue:~$ zfs list tank/mysql
NAME USED  AVAIL  REFER  MOUNTPOINT
tank/mysql  28.4K  2.65T  28.4K  /tank/mysql
and I can happily cd to the file system and do things there.  I've  
let zfs create sit in this state for a good long while, with  
nothing ever happening.  Control-C-ing the zfs create doesn't seem  
to cause any problems, but the whole thing is a bit worrisome.
I'm happy to do whatever crazy things you suggest to figure out why  
this is happening...


I suspect this is probably a nameservice lookup call running
'pfiles 7471' should confirm.


Thanks, I'll check that when I'm back in front of the machine. This  
might be the same thing that's causing sshes to the machine to hang too.


Steve
--
Stephen Green  //   stephen.gr...@sun.com
Principal Investigator \\   http://blogs.sun.com/searchguy
Aura Project   //   Voice: +1 781-442-0926
Sun Microsystems Labs  \\   Fax:   +1 781-442-1692



___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zpool import hangs

[Victor Latushkin]

Hi Brad,

Brad Reese wrote:

Hello,

I've run into a problem with zpool import that seems very similar to
the following thread as far as I can tell:

http://opensolaris.org/jive/thread.jspa?threadID=70205&tstart=15

The suggested solution was to use a later version of open solaris
(b99 or later) but that did not work. I've tried the following
versions of solaris without success:

Solaris 10 u4 (original system)
Solaris 10 u6
Opensolaris 2008.11
Opensolaris 2008.11 b99
SXCE b113


While booted off SXCE b113 bits (or OpenSolaris 2009.06), can you do the 
following:


zdb -e -bcsvL tank
zdb -e -u tank

Output of various commands below shows that there are some corrupted 
metadata objects - at least config object and may be some space maps. 
Option -L for zdb command turns off leak tracking so it will not try to 
load space maps, and we'll get better idea what is corrupted and how to 
proceed.


regards,
victor




Any help with this will be greatly appreciated...my last backup was
four months ago so a lot of my thesis work will be lost. I mistakenly
thought a mirrored zpool on new drives would be good enough for a
while.

So here's what happened: we had a power outage one day and as soon as
I tried to boot the server again it enters an endless reboot cycle.
So I thought the OS drive became corrupted (not mirrored) and
reinstalled the OS. Then when I try zpool import it hangs forever. I
even left it going for a couple days in case it was trying to correct
corrupted data. The same thing happens no matter what version of
solaris I use. The symptoms and diagnostic results (see below) seem
to be very similar to the post above but the solution doesn't work.

Please let me know if you need any other information.

Thanks,

Brad


bash-3.2# zpool import
  pool: tank
id: 4410438565134310480
 state: ONLINE
status: The pool is formatted using an older on-disk version.
action: The pool can be imported using its name or numeric identifier, though
some features will not be available without an explicit 'zpool upgrade'.
config:

tankONLINE
  mirrorONLINE
c2d0p0  ONLINE
c1d0p0  ONLINE
bash-3.2# zpool import tank
cannot import 'tank': pool may be in use from other system
use '-f' to import anyway
bash-3.2# zpool import -f tank
(then it hangs here forever, can't be killed)
(the following commands were performed while this was running)


bash-3.2# fmdump -eV
TIME   CLASS
May 27 2009 22:22:55.308533986 ereport.fs.zfs.checksum
nvlist version: 0
class = ereport.fs.zfs.checksum
ena = 0xd22e37db9000401
detector = (embedded nvlist)
nvlist version: 0
version = 0x0
scheme = zfs
pool = 0x3d350681ea839c50
vdev = 0x85cc302105002c5d
(end detector)

pool = tank
pool_guid = 0x3d350681ea839c50
pool_context = 0
pool_failmode = wait
vdev_guid = 0x85cc302105002c5d
vdev_type = disk
vdev_path = /dev/dsk/c2d0p0
vdev_devid = id1,c...@ast3750640as=5qd3myrh/q
parent_guid = 0x8fb729a008f16e65
parent_type = mirror
zio_err = 50
zio_offset = 0x2500407000
zio_size = 0x1000
zio_objset = 0x0
zio_object = 0x23
zio_level = 0
zio_blkid = 0x0
__ttl = 0x1
__tod = 0x4a1e038f 0x1263dae2
(and many others just like this)



bash-3.2# echo "0t3735::pid2proc|::walk thread|::findtsack -v" | mdb -k
stack pointer for thread df6ed220: e1ca8c54
  e1ca8c94 swtch+0x188()
  e1ca8ca4 cv_wait+0x53(e08442aa, e084426c, , f9c305ac)
  e1ca8ce4 txg_wait_synced+0x90(e0844100, 252b4c, 0, 0)
  e1ca8d34 spa_config_update_common+0x88(d6c429c0, 0, 0, e1ca8d68)
  e1ca8d84 spa_import_common+0x3cf()
  e1ca8db4 spa_import+0x18(dbfcf000, e3c0d018, 0, f9c65810)
  e1ca8de4 zfs_ioc_pool_import+0xcd(dbfcf000, 0, 0)
  e1ca8e14 zfsdev_ioctl+0x124()
  e1ca8e44 cdev_ioctl+0x31(2d8, 5a02, 80418d0, 13, dfde91f8, e1ca8f00)
  e1ca8e74 spec_ioctl+0x6b(d7a593c0, 5a02, 80418d0, 13, dfde91f8, e1ca8f00)
  e1ca8ec4 fop_ioctl+0x49(d7a593c0, 5a02, 80418d0, 13, dfde91f8, e1ca8f00)
  e1ca8f84 ioctl+0x171()
  e1ca8fac sys_sysenter+0x106()



bash-3.2# echo "::threadlist -v" | mdb -k
d4ed8dc0 fec1f5580   0  60 d5033604
  PC: _resume_from_idle+0xb1THREAD: txg_sync_thread()
  stack pointer for thread d4ed8dc0: d4ed8ba8
swtch+0x188()
cv_wait+0x53()
zio_wait+0x55() 	 
vdev_uberblock_sync_list+0x19e()

vdev_config_sync+0x11c()
spa_sync+0x5a5()
txg_sync_thread+0x308()
thread_start+8()
(just chose one seemingly relevant thread from the long list)



bash-3.2# zdb -e -bb tank
Traversing all blocks to verify nothing leaked ...
Assertion failed: space_map_load(&msp->ms_map, &zdb_space_map_ops, 0x0, &msp->ms_smo, spa->spa_meta_objset) == 0, file ../zdb.c, line 1420, function 
zdb_leak_init

Abort (core du

Re: [zfs-discuss] Anyone seen this panic?

[Ian Collins]

Ian Collins wrote:
I was replicating a filesystem with an application than unmounts 
filesystems before sending the snapshots (the unmount is used to check 
if the fs is busy before attempting the send).


One filesystem failed to remount and a couple of child filesystems 
where sent with their parent unmounted.
On reboot, the parent fs could not mount because the directory wasn't 
empty.


I can repeat the panic by unmounting the the parent (with force) and 
then attempting to unmount a child.


--
Ian.

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Adding a SDcard as zfs cache (L2ARC?) on a laptop?

[Daniel Carosone]

Did anyone ever have success with this?

I'm trying to add a usb flash device as rpool cache, and am hitting the same 
problem, even after working through the SMI/EFI label and other issues above.

r...@asura:~# zpool add rpool cache /dev/dsk/c6t0d0s0
invalid vdev specification
use '-f' to override the following errors:
/dev/dsk/c6t0d0s0 overlaps with /dev/dsk/c6t0d0s2
r...@asura:~# zpool add -f rpool cache /dev/dsk/c6t0d0s0
cannot add to 'rpool': invalid argument for this pool operation
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

[zfs-discuss] Anyone seen this panic?

[Ian Collins]

I was replicating a filesystem with an application than unmounts 
filesystems before sending the snapshots (the unmount is used to check 
if the fs is busy before attempting the send).


One filesystem failed to remount and a couple of child filesystems where 
sent with their parent unmounted. 


On reboot, the parent fs could not mount because the directory wasn't empty.

This appears to have cause the following panic:

Jun  1 21:11:18 arion genunix: [ID 335743 kern.notice] BAD TRAP: type=e 
(#pf Page fault) rp=fe80022959e0 addr=40 occurred in module 
"genunix" due to a NULL pointer dereference

Jun  1 21:11:18 arion unix: [ID 10 kern.notice]
Jun  1 21:11:18 arion unix: [ID 839527 kern.notice] zfs:
Jun  1 21:11:18 arion unix: [ID 753105 kern.notice] #pf Page fault
Jun  1 21:11:18 arion unix: [ID 532287 kern.notice] Bad kernel fault at 
addr=0x40
Jun  1 21:11:18 arion unix: [ID 243837 kern.notice] pid=5979, 
pc=0xfba9214e, sp=0xfe8002295ad0, eflags=0x10282
Jun  1 21:11:18 arion unix: [ID 211416 kern.notice] cr0: 
80050033 cr4: 6f8
Jun  1 21:11:18 arion unix: [ID 354241 kern.notice] cr2: 40 cr3: 
8227e2000 cr8: c
Jun  1 21:11:18 arion unix: [ID 592667 kern.notice] 
rdi:0 rsi:0 rdx:0
Jun  1 21:11:18 arion unix: [ID 592667 kern.notice] rcx: 
fea5da4e93e8  r8: fe8002295f10  r9:f
Jun  1 21:11:18 arion unix: [ID 592667 kern.notice] 
rax:0 rbx: fe8b1f39fc80 rbp: fe8002295af0
Jun  1 21:11:18 arion unix: [ID 592667 kern.notice] 
r10:0 r11:0 r12:0
Jun  1 21:11:18 arion unix: [ID 592667 kern.notice] 
r13:   2d r14:  438 r15: fbc075d0
Jun  1 21:11:18 arion unix: [ID 592667 kern.notice] fsb: 
8000 gsb: 90889800  ds:   43
Jun  1 21:11:18 arion unix: [ID 592667 kern.notice] 
es:   43  fs:0  gs:  1c3
Jun  1 21:11:18 arion unix: [ID 592667 kern.notice] 
trp:e err:0 rip: fba9214e
Jun  1 21:11:18 arion unix: [ID 592667 kern.notice] 
cs:   28 rfl:10282 rsp: fe8002295ad0
Jun  1 21:11:18 arion unix: [ID 266532 kern.notice] 
ss:   30

Jun  1 21:11:18 arion unix: [ID 10 kern.notice]
Jun  1 21:11:18 arion genunix: [ID 655072 kern.notice] fe80022958f0 
unix:die+da ()
Jun  1 21:11:18 arion genunix: [ID 655072 kern.notice] fe80022959d0 
unix:trap+5e6 ()
Jun  1 21:11:18 arion genunix: [ID 655072 kern.notice] fe80022959e0 
unix:_cmntrap+140 ()
Jun  1 21:11:18 arion genunix: [ID 655072 kern.notice] fe8002295af0 
genunix:fop_access+1e ()
Jun  1 21:11:18 arion genunix: [ID 655072 kern.notice] fe8002295c90 
nfssrv:exportfs+be ()
Jun  1 21:11:18 arion genunix: [ID 655072 kern.notice] fe8002295cd0 
unix:stubs_common_code+51 ()
Jun  1 21:11:18 arion genunix: [ID 655072 kern.notice] fe8002295d10 
nfs:nfs_export+b0 ()
Jun  1 21:11:18 arion genunix: [ID 655072 kern.notice] fe8002295d30 
zfs:zfs_ioc_share+12d ()
Jun  1 21:11:18 arion genunix: [ID 655072 kern.notice] fe8002295d80 
zfs:zfsdev_ioctl+14c ()
Jun  1 21:11:18 arion genunix: [ID 655072 kern.notice] fe8002295d90 
genunix:cdev_ioctl+1d ()
Jun  1 21:11:18 arion genunix: [ID 655072 kern.notice] fe8002295db0 
specfs:spec_ioctl+50 ()
Jun  1 21:11:18 arion genunix: [ID 655072 kern.notice] fe8002295de0 
genunix:fop_ioctl+25 ()
Jun  1 21:11:18 arion genunix: [ID 655072 kern.notice] fe8002295ec0 
genunix:ioctl+ac ()
Jun  1 21:11:18 arion genunix: [ID 655072 kern.notice] fe8002295f10 
unix:brand_sys_syscall32+1a3 ()


--
Ian.

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] ZFS create hanging on door call?

[Darren J Moffat]

Stephen Green wrote:
Hi, folks. I just built a new box and I'm running the latest OpenSolaris 
bits.  uname says:


SunOS blue 5.11 snv_111b i86pc i386 i86pc Solaris

I just did an image-update last night, but I was seeing this problem in 
111a too.


I built myself a pool out of four 1TB disks (WD Caviar Green, if that 
matters):


  pool: tank
 state: ONLINE
 scrub: none requested
config:

NAMESTATE READ WRITE CKSUM
tankONLINE   0 0 0
  raidz1ONLINE   0 0 0
c4d0ONLINE   0 0 0
c4d1ONLINE   0 0 0
c5d0ONLINE   0 0 0
c5d1ONLINE   0 0 0

errors: No known data errors

I'm trying to create a file system in that pool to hold a mysql database 
using the command zfs create tank/mysql, but the create command is 
hanging.  Truss of the process shows:


stgr...@blue:~$ pgrep -lf zfs
 7471 zfs create tank/mysql
stgr...@blue:~$ pfexec truss -p 7471
door_call(7, 0x080F7008)(sleeping...)

This is an RPC thing, right?

I had this happen a couple of times before the update yesterday as well. 
 The thing is, the file system actually got created:


stgr...@blue:~$ zfs list tank/mysql
NAME USED  AVAIL  REFER  MOUNTPOINT
tank/mysql  28.4K  2.65T  28.4K  /tank/mysql

and I can happily cd to the file system and do things there.  I've let 
zfs create sit in this state for a good long while, with nothing ever 
happening.  Control-C-ing the zfs create doesn't seem to cause any 
problems, but the whole thing is a bit worrisome.


I'm happy to do whatever crazy things you suggest to figure out why this 
is happening...


I suspect this is probably a nameservice lookup call running
'pfiles 7471' should confirm.



--
Darren J Moffat
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss