Tuning ZFS ARC for 512GB Memory

2013-03-05 Thread Reed A. Cartwright
I have a 512GB memory system that has a 480GB (or so) L2ARC, and a
raidz2 storage pool.  No dedup or compression is enabled.  Although, I
will enable compression in the future.

By default vfs.zfs.arc_max is about 300GB and the L2ARC is hardly touched.

I've seen a couple hints recently on the mailing lists that having too
big of an ARC can be a performance hit because it takes a long time to
inspect it when something needs to integrate over the entries.

I've also had issues with running 9.1 on this machine due to zfs
deadlocks or stalls and am currently stuck using 9.0 with no issue.

So, I was wondering if shrinking the ARC would potentially help and
what should I shrink it to?

I am thinking about setting vfs.zfs.arc_max to 64GB.

Does anyone have an opinion?


--
Reed A. Cartwright, PhD
Assistant Professor of Genomics, Evolution, and Bioinformatics
School of Life Sciences
Center for Evolutionary Medicine and Informatics
The Biodesign Institute
Arizona State University
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org


Re: ZFS question

2013-03-20 Thread Reed A. Cartwright
Several people, including me, have an issue like this with 9.1.  Your
best bet is to try 9.0.

On Wed, Mar 20, 2013 at 5:49 PM, Quartz qua...@sneakertech.com wrote:
 I'm experiencing fatal issues with pools hanging my machine requiring a
 hard-reset. I'm new to freebsd and these mailing lists in particular, is
 this the place to ask for help?

 __
 it has a certain smooth-brained appeal
 ___
 freebsd-questions@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-questions
 To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org



-- 
Reed A. Cartwright, PhD
Assistant Professor of Genomics, Evolution, and Bioinformatics
School of Life Sciences
Center for Evolutionary Medicine and Informatics
The Biodesign Institute
Arizona State University
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org


Re: ZFS question

2013-03-20 Thread Reed A. Cartwright
Note that my issue seems to do with an interaction between the CAM
system and the MPS driver in 9.1.  Thus it is more than likely
different than what you are experiencing Quartz.

Now that ZFS deadman has been incorporated into stable, I'll probably
give a 9.1 (i.e. 9/stable) another try.

Jeremy, I have a question about enabling kernel dumps based on my
current swap config.

I currently have a 1TB drive split into 4 geli encrypted swap
partitions (Freebsd doesn't like swap partitions over ~250 GB and I
have lots of RAM).  These partitions are UFS-swap partitions and are
not backed by any mirroing or ZFSing.

So, how do I best enable crash dumps?  If I need to remove encryption,
I can do that.

On Wed, Mar 20, 2013 at 9:45 PM, Jeremy Chadwick j...@koitsu.org wrote:
 (Please keep me CC'd as I'm not subscribed to -questions)


 Lots to say about this.

 1. freebsd-fs is the proper list for filesystem-oriented questions of
 this sort, especially for ZFS.

 2. The issue you've described is experienced by some, and **not**
 experienced by even more/just as many, so please keep that in mind.
 Each/every person's situation/environment/issue has to be treated
 separately/as unique.

 3. You haven't provided any useful details, even in your follow-up post
 here:

 http://lists.freebsd.org/pipermail/freebsd-questions/2013-March/249958.html

 All you've provided is a general overview with no technical details,
 no actual data.  You need to provide that data verbatim.  You need to
 provide:

 - Contents of /boot/loader.conf
 - Contents of /etc/sysctl.conf
 - Output from zpool status
 - Output from zpool get all
 - Output from zfs get all
 - Output from dmesg (probably the most important)
 - Output from sysctl vfs.zfs kstat.zfs

 I particularly tend to assist with disk-level problems, so if this turns
 out to be a disk-level issue (and NOT a controller or controller driver
 issue), I can help quite a bit with that.

 4. I would **not** suggest rolling back to 9.0.  This recommendation is
 solves nothing -- if there is truly a bug/livelock issue, then that
 needs to be tracked down.  By rolling back, if there is an issue, you're
 effectively ensuring it'll never get investigated or fixed, which means
 you can probably expect to see this in 9.2, 9.3, or even 10.x onward.

 If you can't deal with the instability, or don't have the
 time/cycles/interest to help track it down, that's perfectly okay too:
 my recommendation is to go back to UFS (there's no shame in that).

 Else, as always, I strongly recommend running stable/9 (keep reading).

 5. stable/9 (a.k.a. FreeBSD 9.1-STABLE) just recently (~5 days ago)
 MFC'd an Illumos ZFS feature solely to help debug/troubleshoot this
 exact type of situation: introduction of the ZFS deadmean thread.
 Reference materials for what that is:

 http://svnweb.freebsd.org/base?view=revisionrevision=248369
 http://svnweb.freebsd.org/base?view=revisionrevision=247265
 https://www.illumos.org/issues/3246

 The purpose of this feature (enabled by default) is to induce a kernel
 panic when ZFS I/O stalls/hangs for unexpectedly long periods of time
 (configurable via vfs.zfs.deadman_synctime).

 Once the panic happens (assuming your system is configured with a slice
 dedicated to swap (ZFS-backed swap = bad bad bad) and use of
 dumpdev=auto in rc.conf), upon reboot the system should extract the
 crash dump from swap and save it into /var/crash.  At that point kernel
 developers on the -fs list can help tell you *exactly* what to do with
 kgdb(1) that can shed some light on what happened/where the issue may
 lie.

 All that's assuming that the issue truly is ZFS waiting for I/O and not
 something else (like ZFS internally spinning hard in its own code).

 Good luck, and let us know how you want to proceed.

 --
 | Jeremy Chadwick   j...@koitsu.org |
 | UNIX Systems Administratorhttp://jdc.koitsu.org/ |
 | Mountain View, CA, US|
 | Making life hard for others since 1977. PGP 4BD6C0CB |

 ___
 freebsd-questions@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-questions
 To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org



-- 
Reed A. Cartwright, PhD
Assistant Professor of Genomics, Evolution, and Bioinformatics
School of Life Sciences
Center for Evolutionary Medicine and Informatics
The Biodesign Institute
Arizona State University
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org