Re: [OmniOS-discuss] ZPOOL bug after upgrade to r151020

[wuffers]

I upgraded to r151020 in late Jan, and saw some strangeness with arcstat
(l2size and l2asize were huge) before I did a reboot due to some
instability a few weeks ago. I thought it was just a case of not using the
latest arcstat, and things were running fine after a reboot so didn't
pursue it.

I saw this post last week, and confirmed it was within my environment, so
did the remove/re-add of the cache devices, then a complete reboot as well.
The cache devices reported back their actual proper size (400GB) via "zpool
iostat -v". Today I checked it again and this is what I see:

# arcstat
read  hits  miss  hit%  l2read  l2hits  l2miss  l2hit%  arcsz  l2size
 l2asize
 465   4125388  53  50   3  94   230G4.4T
3.2T

# zpool iostat -v

(other info snipped for brevity)

cache  -  -  -  -  -  -
  c2t500117310015D579d0 816G  16.0E 54 23  2.32M  1.46M
  c2t50011731001631FDd0 816G  16.0E 54 23  2.32M  1.46M
  c12t500117310015D59Ed0815G  16.0E 55 23  2.35M  1.46M
  c12t500117310015D54Ed0816G  16.0E 55 23  2.36M  1.46M


I'm just waiting for the next lockup/crash..

John, were you able to compile the fix, and if so, be able send me a copy?

Thanks.


On Mon, Apr 10, 2017 at 10:00 AM, Dan McDonald  wrote:

>
> > On Apr 9, 2017, at 10:27 PM, John Barfield 
> wrote:
> >
> > Thank you Dan.
> >
> > Do you happen to have the process or know the location of a process
> document for only building ZFS?
> >
> > Ive re-built only nfs from illumos-gate in the past to resolve a bug but
> im wondering how I would build and install only zfs. (if its even possible).
> >
> > There are 2 bugs that we're suffering with at two different customer
> sites that didnt get into r151020 and Im not sure that we can make it till
> r151022 is released.
> >
> > Thanks for any advice
>
> You can build zfs the way you likely built NFS.  Build it, replace it on
> an alternate BE (in zfs's case:  /kernel/fs/amd64/zfs), and reboot.
>
> The only gotcha might be if a bugfix covers more than just ZFS itself...
> but for 7504, that's NOT the case.  :)
>
> Dan
>
> ___
> OmniOS-discuss mailing list
> OmniOS-discuss@lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] ZPOOL bug after upgrade to r151020

[Dan McDonald]

> On Apr 7, 2017, at 8:26 PM, John Barfield  wrote:
> 
> Greetings,
> 
> I just want to report that after a clean istall of r151020 I found a bug 
> whereby importing an older zpool from r151012 and running zpool upgrade 
> causes an SSD cache device size to be reported incorrectly. (only 1 out of 4 
> devices in this instance)
> 
> The cache device size is 93gb and arcstat reported it to be 680gb.
> 
> I confirmed by monitoring zpool iostat -v and saw the same size being 
> reported.
> 
> We've had a lot of weird io lockups (which is how I found the issue, we didnt 
> notice it until a month after) that brings all of our NFS mounts to a 
> screeching halt and this was the only thing I could find to be out of the 
> ordinary on the system.
> 
> CPU average @1% , 20% of ram free, no crazy processes waiting on IO. It was 
> completely invisible. At least from my testing using several dtrace scripts 
> from the net.
> 
> I can only assume that the incorrect size reporting caused the zpool to fill 
> the cache drive up beyond its physical capacity during periods of heavy load.
> 
> I removed all cache devices and then added them back to the zpool. Then all 
> disks reported correctly again. Format/diskinfo always reported correctly so 
> it was specific to zfs.
> 
> We're monitoring the NAS closely to see if the issues occur again. 

The only thing I could find that might address the symptoms you see is this:

https://illumos.org/issues/7504

Which didn't make it upstream in time to hit r151020.

You should forward this on to the illumos ZFS developers' list:  
z...@lists.illumos.org.

Dan

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss