Re: recommended memory for zfs
On Fri, 10 May 2013 04:18:42 +0200, Benjamin Adams benjamindad...@gmail.com wrote: On 05/09/2013 10:06 PM, Jeremy Chadwick wrote: On Thu, May 09, 2013 at 09:47:27PM -0400, Benjamin Adams wrote: On 05/09/2013 08:53 PM, Shane Ambler wrote: On 09/05/2013 22:48, Benjamin Adams wrote: Hello zfs question about memory. I heard zfs is very ram hungry. Service looking to run: - nginx - postgres - php-fpm - python I have a machine with two quad core cpus but only 4 G Memory I'm looking to buy more ram now. What would be the recommend amount of memory for zfs across 6 drives on this setup? I believe I heard a calculation of 1GB cache per 1TB of disk. But basically zfs will use all free ram available if you access that much data from disk. You will want to set vfs.zfs.arc_max to allow enough ram for your apps to work in. If you consider the files for your website and the data you store you may find that you would never fill more than 500MB of cache. If you will be serving large media files that will easily use up the cache you could give them their own filesystem that only caches metadata - zfs set primarycache=metadata zroot/mediafiles Thanks for all the replies Size of DB and HD's are: Current DB Size = 23 GB HD sizes = (6) 500 GB drives Nobody is going to be able to give you a precise/accurate recommendation given the lack of detail provided, I'm sorry to say. What's the RES size of nginx (all processes combined)? What's the RES size of postgres (same)? Do you have PHP scripts that run amok for long periods of time and take up lots of RAM? Same with python? How many concurrent visitors and what sort of content are you hosting? Do you maintain/write your own PHP/Python code or are you using some crap like Wordpress? This is just a **small** list of questions -- and what may come as a shock is that I do not expect you to provide answers to any of them. They are questions that you should, for yourself, attempt to answer and work out what you need from there (teach a man to fish and all that). The advice of 1GB of RAM per 1TB of disk space is absolute nonsense on numerous levels -- whoever gave this advice to Shane either has no understanding of how filesystems/ZFS works, or does but chose to simplify to the point where they're providing half-ass information. There is no direct, or even indirect, correlation between disk capacity and ZFS ARC size -- what matter is your working set (to quote Tom). You need to have some idea of how much disk I/O you're doing, and what type of I/O (sequential or random). If you want my general advice, Benjamin, it's this: get yourself a system with *minimum* 8GB of RAM but has the physical possibility of supporting more (and only add more RAM when/if you know you need it); do not bother with ZFS on a system with 4GB. Run amd64, not i386 (I don't recommend bothering with ZFS on i386 -- I am not going to get into a discussion about this either). Run stable/9, not 9.1-RELEASE. Avoid compression and dedup. And test disk failures as well (don't get caught with your pants down later). The above advice comes from someone who did hosting (web/ssh/etc.) for almost 20 years with KISS principle applied at all levels. YMMV though, depending on what all you're doing/what you truly need. Good luck. Jeremy, Was just see if I should just get raid controller and more ram down the road. List of priorities. Main thing is I move from BSD when 9.0 came out. Was looking to see if zfs is included in the installer. Now. Sum up: upgrade ram to 16GB (not 64 like plained) and raid controller that supports level 5. Let ZFS do the RAID stuff. Do not use a RAID controller, but give the plain disks to ZFS. Some of the nice features come from ZFS doing the RAID stuff. Ronald. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: recommended memory for zfs
On May 10, 2013, at 1:08 PM, Ronald Klop ronald-freeb...@klop.yi.org wrote: On Fri, 10 May 2013 04:18:42 +0200, Benjamin Adams benjamindad...@gmail.com wrote: On 05/09/2013 10:06 PM, Jeremy Chadwick wrote: On Thu, May 09, 2013 at 09:47:27PM -0400, Benjamin Adams wrote: On 05/09/2013 08:53 PM, Shane Ambler wrote: On 09/05/2013 22:48, Benjamin Adams wrote: Hello zfs question about memory. I heard zfs is very ram hungry. Service looking to run: - nginx - postgres - php-fpm - python I have a machine with two quad core cpus but only 4 G Memory I'm looking to buy more ram now. What would be the recommend amount of memory for zfs across 6 drives on this setup? I believe I heard a calculation of 1GB cache per 1TB of disk. But basically zfs will use all free ram available if you access that much data from disk. You will want to set vfs.zfs.arc_max to allow enough ram for your apps to work in. If you consider the files for your website and the data you store you may find that you would never fill more than 500MB of cache. If you will be serving large media files that will easily use up the cache you could give them their own filesystem that only caches metadata - zfs set primarycache=metadata zroot/mediafiles Thanks for all the replies Size of DB and HD's are: Current DB Size = 23 GB HD sizes = (6) 500 GB drives Nobody is going to be able to give you a precise/accurate recommendation given the lack of detail provided, I'm sorry to say. What's the RES size of nginx (all processes combined)? What's the RES size of postgres (same)? Do you have PHP scripts that run amok for long periods of time and take up lots of RAM? Same with python? How many concurrent visitors and what sort of content are you hosting? Do you maintain/write your own PHP/Python code or are you using some crap like Wordpress? This is just a **small** list of questions -- and what may come as a shock is that I do not expect you to provide answers to any of them. They are questions that you should, for yourself, attempt to answer and work out what you need from there (teach a man to fish and all that). The advice of 1GB of RAM per 1TB of disk space is absolute nonsense on numerous levels -- whoever gave this advice to Shane either has no understanding of how filesystems/ZFS works, or does but chose to simplify to the point where they're providing half-ass information. There is no direct, or even indirect, correlation between disk capacity and ZFS ARC size -- what matter is your working set (to quote Tom). You need to have some idea of how much disk I/O you're doing, and what type of I/O (sequential or random). If you want my general advice, Benjamin, it's this: get yourself a system with *minimum* 8GB of RAM but has the physical possibility of supporting more (and only add more RAM when/if you know you need it); do not bother with ZFS on a system with 4GB. Run amd64, not i386 (I don't recommend bothering with ZFS on i386 -- I am not going to get into a discussion about this either). Run stable/9, not 9.1-RELEASE. Avoid compression and dedup. And test disk failures as well (don't get caught with your pants down later). The above advice comes from someone who did hosting (web/ssh/etc.) for almost 20 years with KISS principle applied at all levels. YMMV though, depending on what all you're doing/what you truly need. Good luck. Jeremy, Was just see if I should just get raid controller and more ram down the road. List of priorities. Main thing is I move from BSD when 9.0 came out. Was looking to see if zfs is included in the installer. Now. Sum up: upgrade ram to 16GB (not 64 like plained) and raid controller that supports level 5. Let ZFS do the RAID stuff. Do not use a RAID controller, but give the plain disks to ZFS. Some of the nice features come from ZFS doing the RAID stuff. To paraphrase this. Get yourself a nice HBA. non-RAID! For example, something based on LSI2008 with IT firmware. With few enough discs you can avoid using SAS expanders as well. RAID controllers, in addition to causing all kinds of troubles are unlikely to have sufficient bandwidth and might turn out to be the bottleneck (unless you are prepared to spend an unholy amount of money -- which are better spent for RAM and CPU). If you want performance and low latency, avoid using compression and dedup in ZFS. Set your record size appropriately for postgresql (8k) *before* you run initdb. It is best to create an separate filesystem for the database and set that property only there. If your database is heavy on updates, you might be interested to use an SSD for ZIL. In general, if you can afford it, an cheap SSD for L2ARC might do wonders -- if your data set can fit there. If you intend to use SSDs and want the best performance, use different SSDs for ZIL and L2ARC. The first needs to be fast at writing and optimised for his (for example, the
recommended memory for zfs
Hello zfs question about memory. I heard zfs is very ram hungry. Service looking to run: - nginx - postgres - php-fpm - python I have a machine with two quad core cpus but only 4 G Memory I'm looking to buy more ram now. What would be the recommend amount of memory for zfs across 6 drives on this setup? Also can 9.1 now boot to zfs from the installer? (no tricks for post install) Thanks Ben ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: recommended memory for zfs
On Thu, May 9, 2013 at 2:18 PM, Benjamin Adams benjamindad...@gmail.com wrote: Hello zfs question about memory. I heard zfs is very ram hungry. Service looking to run: - nginx - postgres - php-fpm - python I have a machine with two quad core cpus but only 4 G Memory I'm looking to buy more ram now. What would be the recommend amount of memory for zfs across 6 drives on this setup? There is no right answer to this question. ZFS does not use a lot of RAM, but it will use as much as you allow as cache (in the ARC). How much data is your working set? How much of your working set do you need to keep in cache? You need that much memory (plus whatever to run your applications). Also can 9.1 now boot to zfs from the installer? (no tricks for post install) Can you clarify? The installer still cannot setup zfs pools, you must do so manually (but then can return to the installer). Cheers Tom ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: recommended memory for zfs
Le 09/05/2013 15:18, Benjamin Adams a écrit : Hello zfs question about memory. I heard zfs is very ram hungry. Service looking to run: - nginx - postgres - php-fpm - python I have a machine with two quad core cpus but only 4 G Memory I'm looking to buy more ram now. What would be the recommend amount of memory for zfs across 6 drives on this setup? Also can 9.1 now boot to zfs from the installer? (no tricks for post install) Without cache, footprint of ZFS is larger than any other filesystem but it is really very tiny (8Mio if my memory is good). I run my server on an Intel Atom with 2GB of RAM, with Apache22, PHP, MySQL database and ejabberd. If you want to get ZFS on your system at install time, there are many scripts on the Internet, and if you want you can take mine: http://projet.beastie.eu/?p=freebsd-zfs.git;a=blob_plain;f=zfs.sh;hb=HEAD Hoping it will help you. -- Florent Peterschmitt +33 (0)6 64 33 97 92 flor...@peterschmitt.fr O ascii ribbon campaign - stop html mail - www.asciiribbon.org signature.asc Description: OpenPGP digital signature
Re: recommended memory for zfs
In the last episode (May 09), Benjamin Adams said: Hello zfs question about memory. I heard zfs is very ram hungry. Service looking to run: - nginx - postgres - php-fpm - python I have a machine with two quad core cpus but only 4 G Memory I'm looking to buy more ram now. What would be the recommend amount of memory for zfs across 6 drives on this setup? As much as is reasonable to purchase. Postgres would probably appreciate the memory more than ZFS. You can run ZFS on memory-limited machines (I've gone as far down as 256MB), but the critical part is running a 64-bit kernel. ZFS does a lot of kernel malloc/free operations, and address space fragmentation on a 32-bit system will eventually cause a panic when ZFS can't malloc a contiguous 128k chunk. -- Dan Nelson dnel...@allantgroup.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: recommended memory for zfs
On 09/05/2013 22:48, Benjamin Adams wrote: Hello zfs question about memory. I heard zfs is very ram hungry. Service looking to run: - nginx - postgres - php-fpm - python I have a machine with two quad core cpus but only 4 G Memory I'm looking to buy more ram now. What would be the recommend amount of memory for zfs across 6 drives on this setup? I believe I heard a calculation of 1GB cache per 1TB of disk. But basically zfs will use all free ram available if you access that much data from disk. You will want to set vfs.zfs.arc_max to allow enough ram for your apps to work in. If you consider the files for your website and the data you store you may find that you would never fill more than 500MB of cache. If you will be serving large media files that will easily use up the cache you could give them their own filesystem that only caches metadata - zfs set primarycache=metadata zroot/mediafiles ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: recommended memory for zfs
On 05/09/2013 08:53 PM, Shane Ambler wrote: On 09/05/2013 22:48, Benjamin Adams wrote: Hello zfs question about memory. I heard zfs is very ram hungry. Service looking to run: - nginx - postgres - php-fpm - python I have a machine with two quad core cpus but only 4 G Memory I'm looking to buy more ram now. What would be the recommend amount of memory for zfs across 6 drives on this setup? I believe I heard a calculation of 1GB cache per 1TB of disk. But basically zfs will use all free ram available if you access that much data from disk. You will want to set vfs.zfs.arc_max to allow enough ram for your apps to work in. If you consider the files for your website and the data you store you may find that you would never fill more than 500MB of cache. If you will be serving large media files that will easily use up the cache you could give them their own filesystem that only caches metadata - zfs set primarycache=metadata zroot/mediafiles Thanks for all the replies Size of DB and HD's are: Current DB Size = 23 GB HD sizes = (6) 500 GB drives ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: recommended memory for zfs
On Thu, May 09, 2013 at 09:47:27PM -0400, Benjamin Adams wrote: On 05/09/2013 08:53 PM, Shane Ambler wrote: On 09/05/2013 22:48, Benjamin Adams wrote: Hello zfs question about memory. I heard zfs is very ram hungry. Service looking to run: - nginx - postgres - php-fpm - python I have a machine with two quad core cpus but only 4 G Memory I'm looking to buy more ram now. What would be the recommend amount of memory for zfs across 6 drives on this setup? I believe I heard a calculation of 1GB cache per 1TB of disk. But basically zfs will use all free ram available if you access that much data from disk. You will want to set vfs.zfs.arc_max to allow enough ram for your apps to work in. If you consider the files for your website and the data you store you may find that you would never fill more than 500MB of cache. If you will be serving large media files that will easily use up the cache you could give them their own filesystem that only caches metadata - zfs set primarycache=metadata zroot/mediafiles Thanks for all the replies Size of DB and HD's are: Current DB Size = 23 GB HD sizes = (6) 500 GB drives Nobody is going to be able to give you a precise/accurate recommendation given the lack of detail provided, I'm sorry to say. What's the RES size of nginx (all processes combined)? What's the RES size of postgres (same)? Do you have PHP scripts that run amok for long periods of time and take up lots of RAM? Same with python? How many concurrent visitors and what sort of content are you hosting? Do you maintain/write your own PHP/Python code or are you using some crap like Wordpress? This is just a **small** list of questions -- and what may come as a shock is that I do not expect you to provide answers to any of them. They are questions that you should, for yourself, attempt to answer and work out what you need from there (teach a man to fish and all that). The advice of 1GB of RAM per 1TB of disk space is absolute nonsense on numerous levels -- whoever gave this advice to Shane either has no understanding of how filesystems/ZFS works, or does but chose to simplify to the point where they're providing half-ass information. There is no direct, or even indirect, correlation between disk capacity and ZFS ARC size -- what matter is your working set (to quote Tom). You need to have some idea of how much disk I/O you're doing, and what type of I/O (sequential or random). If you want my general advice, Benjamin, it's this: get yourself a system with *minimum* 8GB of RAM but has the physical possibility of supporting more (and only add more RAM when/if you know you need it); do not bother with ZFS on a system with 4GB. Run amd64, not i386 (I don't recommend bothering with ZFS on i386 -- I am not going to get into a discussion about this either). Run stable/9, not 9.1-RELEASE. Avoid compression and dedup. And test disk failures as well (don't get caught with your pants down later). The above advice comes from someone who did hosting (web/ssh/etc.) for almost 20 years with KISS principle applied at all levels. YMMV though, depending on what all you're doing/what you truly need. Good luck. -- | Jeremy Chadwick j...@koitsu.org | | UNIX Systems Administratorhttp://jdc.koitsu.org/ | | Mountain View, CA, US| | Making life hard for others since 1977. PGP 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: recommended memory for zfs
Probably the simplest answer is that you already have sufficient memory to run ZFS. As someone already mentioned you should use AMD64, not i386. If your setup isn't fast enough with tuning, add more if it's the bottleneck. On Thu, May 9, 2013 at 8:47 PM, Benjamin Adams benjamindad...@gmail.com wrote: On 05/09/2013 08:53 PM, Shane Ambler wrote: On 09/05/2013 22:48, Benjamin Adams wrote: Hello zfs question about memory. I heard zfs is very ram hungry. Service looking to run: - nginx - postgres - php-fpm - python I have a machine with two quad core cpus but only 4 G Memory I'm looking to buy more ram now. What would be the recommend amount of memory for zfs across 6 drives on this setup? I believe I heard a calculation of 1GB cache per 1TB of disk. But basically zfs will use all free ram available if you access that much data from disk. You will want to set vfs.zfs.arc_max to allow enough ram for your apps to work in. If you consider the files for your website and the data you store you may find that you would never fill more than 500MB of cache. If you will be serving large media files that will easily use up the cache you could give them their own filesystem that only caches metadata - zfs set primarycache=metadata zroot/mediafiles Thanks for all the replies Size of DB and HD's are: Current DB Size = 23 GB HD sizes = (6) 500 GB drives ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org -- Adam Vande More ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: recommended memory for zfs
On Thu, May 9, 2013 at 9:06 PM, Jeremy Chadwick j...@koitsu.org wrote: The advice of 1GB of RAM per 1TB of disk space is absolute nonsense on numerous levels -- whoever gave this advice to Shane either has no understanding of how filesystems/ZFS works, or does but chose to simplify to the point where they're providing half-ass information. IIRC, that used to be the guideline for memory requirements for dedup. -- Adam Vande More ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: recommended memory for zfs
On 05/09/2013 10:06 PM, Jeremy Chadwick wrote: On Thu, May 09, 2013 at 09:47:27PM -0400, Benjamin Adams wrote: On 05/09/2013 08:53 PM, Shane Ambler wrote: On 09/05/2013 22:48, Benjamin Adams wrote: Hello zfs question about memory. I heard zfs is very ram hungry. Service looking to run: - nginx - postgres - php-fpm - python I have a machine with two quad core cpus but only 4 G Memory I'm looking to buy more ram now. What would be the recommend amount of memory for zfs across 6 drives on this setup? I believe I heard a calculation of 1GB cache per 1TB of disk. But basically zfs will use all free ram available if you access that much data from disk. You will want to set vfs.zfs.arc_max to allow enough ram for your apps to work in. If you consider the files for your website and the data you store you may find that you would never fill more than 500MB of cache. If you will be serving large media files that will easily use up the cache you could give them their own filesystem that only caches metadata - zfs set primarycache=metadata zroot/mediafiles Thanks for all the replies Size of DB and HD's are: Current DB Size = 23 GB HD sizes = (6) 500 GB drives Nobody is going to be able to give you a precise/accurate recommendation given the lack of detail provided, I'm sorry to say. What's the RES size of nginx (all processes combined)? What's the RES size of postgres (same)? Do you have PHP scripts that run amok for long periods of time and take up lots of RAM? Same with python? How many concurrent visitors and what sort of content are you hosting? Do you maintain/write your own PHP/Python code or are you using some crap like Wordpress? This is just a **small** list of questions -- and what may come as a shock is that I do not expect you to provide answers to any of them. They are questions that you should, for yourself, attempt to answer and work out what you need from there (teach a man to fish and all that). The advice of 1GB of RAM per 1TB of disk space is absolute nonsense on numerous levels -- whoever gave this advice to Shane either has no understanding of how filesystems/ZFS works, or does but chose to simplify to the point where they're providing half-ass information. There is no direct, or even indirect, correlation between disk capacity and ZFS ARC size -- what matter is your working set (to quote Tom). You need to have some idea of how much disk I/O you're doing, and what type of I/O (sequential or random). If you want my general advice, Benjamin, it's this: get yourself a system with *minimum* 8GB of RAM but has the physical possibility of supporting more (and only add more RAM when/if you know you need it); do not bother with ZFS on a system with 4GB. Run amd64, not i386 (I don't recommend bothering with ZFS on i386 -- I am not going to get into a discussion about this either). Run stable/9, not 9.1-RELEASE. Avoid compression and dedup. And test disk failures as well (don't get caught with your pants down later). The above advice comes from someone who did hosting (web/ssh/etc.) for almost 20 years with KISS principle applied at all levels. YMMV though, depending on what all you're doing/what you truly need. Good luck. Jeremy, Was just see if I should just get raid controller and more ram down the road. List of priorities. Main thing is I move from BSD when 9.0 came out. Was looking to see if zfs is included in the installer. Now. Sum up: upgrade ram to 16GB (not 64 like plained) and raid controller that supports level 5. Thanks Ben ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: recommended memory for zfs
And the rule of thumb for dedupe was approx 1 GB of ARC per unique TB of data in the pool (above and beyond your normal ARC requirements). Not 1 GB of RAM per TB of disk in the pool. Very big difference between the two. :) On 2013-05-09 7:14 PM, Adam Vande More amvandem...@gmail.com wrote: On Thu, May 9, 2013 at 9:06 PM, Jeremy Chadwick j...@koitsu.org wrote: The advice of 1GB of RAM per 1TB of disk space is absolute nonsense on numerous levels -- whoever gave this advice to Shane either has no understanding of how filesystems/ZFS works, or does but chose to simplify to the point where they're providing half-ass information. IIRC, that used to be the guideline for memory requirements for dedup. -- Adam Vande More ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
