Re: Very large swap
Dennis Glatting free...@penx.com wrote: This is a proof-of-concept project ... I am doing it on the cheap ... I have committed to the project five machines. Three run over clocked Phenom II x6 processors with 16GB of RAM, 1TB disk for the OS, 1TB disk for Junk, and a 3-2TB disk RAIDz array ... These machines are liquid cooled ... A data manipulation server is running an i7 x4 with 24GB of fast RAM. It has 12 2TB disks, 2 1TB disks (OS), plus a few SSDs ... A repository server is an i7 x6 3.3GHz with 24GB of RAM, several volumes, two of which are RAIDz, SSDs, and other junk ... If that's on the cheap, I shudder to think what would be considered expensive. I've seen a whole server room with less horsepower and disk than that. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Very large swap
At 07:08 14/10/2011, Dennis Glatting wrote: This is kind of stupid question but at a minimum I thought it would be interesting to know. What is the limitations in terms of swap devices under RELENG_8 (or 9)? A single swap dev appears to be limited to 32GB (there are truncation messages on boot). I am looking at a possible need of 2-20TB (probably more) with as much main memory that is affordable. I used 40 GB swap in 8.2. I wanted to test if caching files/data on memory swap were faster than an ordinary, always the same, standard approach of read/write from/to disk when need. It was faster, not a lot faster but mesurable. Of course when app exits files/data is lost. I am working with large data sets and there are various ways of solving the problem sets but simply letting the processors swap as they work through a given problem is a possible technique. Processors, memory, storage, and other hardware are open issues but for the minute I'm doing things on the cheap, which, admittedly, has a certain intellectual amusement factor. TIA ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Very large swap
On 10/14/2011 8:08 AM, Dennis Glatting wrote: This is kind of stupid question but at a minimum I thought it would be interesting to know. What is the limitations in terms of swap devices under RELENG_8 (or 9)? A single swap dev appears to be limited to 32GB (there are truncation messages on boot). I am looking at a possible need of 2-20TB (probably more) with as much main memory that is affordable. The limit is raised to 256GB in HEAD and RELENG_8 http://svnweb.freebsd.org/base?view=revisionamp;revision=225076 I am working with large data sets and there are various ways of solving the problem sets but simply letting the processors swap as they work through a given problem is a possible technique. I would advise against this technique. Possibly, it's easier to design your program to user smaller amounts of memory and avoid swapping. After all, designing your program to use big amounts of swapped out memory *and* perform in a timely manner, can be very challenging. Nikos ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Very large swap
On 10/14/2011 11:43 AM, Nikos Vassiliadis wrote: On 10/14/2011 8:08 AM, Dennis Glatting wrote: This is kind of stupid question but at a minimum I thought it would be interesting to know. What is the limitations in terms of swap devices under RELENG_8 (or 9)? A single swap dev appears to be limited to 32GB (there are truncation messages on boot). I am looking at a possible need of 2-20TB (probably more) with as much main memory that is affordable. The limit is raised to 256GB in HEAD and RELENG_8 http://svnweb.freebsd.org/base?view=revisionamp;revision=225076 I am working with large data sets and there are various ways of solving the problem sets but simply letting the processors swap as they work through a given problem is a possible technique. I would advise against this technique. Possibly, it's easier to design your program to user smaller amounts of memory and avoid swapping. After all, designing your program to use big amounts of swapped out memory *and* perform in a timely manner, can be very challenging. Nikos Well ... I dunno how much large dataset processing you've done, but it's not that simple. Ordinarily, with modern machines and architectures, you're right. In fact, you NEVER want to swap, instead, throw memory at the problem. But when you get into really big datasets, it's a different story. You probably will not find a mobo with 20TB memory capacity :) So ... you have to do something with disk. You generally get two choices: Memory mapped files or swap. It's been some years since I considered either seriously, but they do have some tradeoffs. MM files give the programmer very fine grained control of just what might get pushed out to disk at the cost of user space context switching. Swap gets managed by the kernel which is about as efficient as disk I/O is going to get, but that means what and how things get moved on- and off disk is invisible to the application. What a lot of big data shops are moving to is SSD for such operations. SSD is VERY fast and can be RAIDed to overcome the tendency of at least the early SSD products' tendency to, um ... blow up. As always, scale is hard, and giant data problems are Really Hard (tm). That's why people like IBM, Sun/Oracle, and Teradata make lots of money building giant iron farms. 'Just my 2^1 cents worth ... -- Tim Daneliuk tun...@tundraware.com PGP Key: http://www.tundraware.com/PGP/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Very large swap
On Fri, 14 Oct 2011, Tim Daneliuk wrote: On 10/14/2011 11:43 AM, Nikos Vassiliadis wrote: On 10/14/2011 8:08 AM, Dennis Glatting wrote: This is kind of stupid question but at a minimum I thought it would be interesting to know. What is the limitations in terms of swap devices under RELENG_8 (or 9)? A single swap dev appears to be limited to 32GB (there are truncation messages on boot). I am looking at a possible need of 2-20TB (probably more) with as much main memory that is affordable. The limit is raised to 256GB in HEAD and RELENG_8 http://svnweb.freebsd.org/base?view=revisionamp;revision=225076 I am working with large data sets and there are various ways of solving the problem sets but simply letting the processors swap as they work through a given problem is a possible technique. I would advise against this technique. Possibly, it's easier to design your program to user smaller amounts of memory and avoid swapping. After all, designing your program to use big amounts of swapped out memory *and* perform in a timely manner, can be very challenging. Nikos Well ... I dunno how much large dataset processing you've done, but it's not that simple. Ordinarily, with modern machines and architectures, you're right. In fact, you NEVER want to swap, instead, throw memory at the problem. But when you get into really big datasets, it's a different story. You probably will not find a mobo with 20TB memory capacity :) So ... you have to do something with disk. You generally get two choices: Memory mapped files or swap. It's been some years since I considered either seriously, but they do have some tradeoffs. MM files give the programmer very fine grained control of just what might get pushed out to disk at the cost of user space context switching. Swap gets managed by the kernel which is about as efficient as disk I/O is going to get, but that means what and how things get moved on- and off disk is invisible to the application. What a lot of big data shops are moving to is SSD for such operations. SSD is VERY fast and can be RAIDed to overcome the tendency of at least the early SSD products' tendency to, um ... blow up. As always, scale is hard, and giant data problems are Really Hard (tm). That's why people like IBM, Sun/Oracle, and Teradata make lots of money building giant iron farms. This is a proof-of-concept project that is personally educational with a substantial amusement factor. I am doing it on the cheap, which means commercial products. I'm also doing it at home, which means expenses come out of my pocket. This project is about manipulating and creating large data sets for crypto related applications (use your imagination here). The manipulations are fairly stupid and generally I am using UNIX utilities because I don't want to re-invent existing code (I'm not that bored). I have also written some programs but they are no more than a few pages of code. I am also using MPI, OpenMP, and pthreads where supported and make sense. I have committed to the project five machines. Three (the 'attack' machines) run over clocked Phenom II x6 processors with 16GB of RAM, 1TB disk for the OS, 1TB disk for Junk, and a 3-2TB disk RAIDz array. Two of the three MBs (ASUS CROSSHAIR V FORMULA, Gigabyte GA-990FXA-UD7, and something I had lying around) are upgradable to 8150s, which I have one on order. These machines are liquid cooled because, well, for no reason in particular other than I thought it would be fun (defiantly educational). Roughly fifty percent of the parts were lying around but more importantly my wife and I keep our finances separate. :) A data manipulation server is running an i7 x4 (not over clocked but turbo is enabled) with 24GB of fast RAM. It has 12 2TB disks, 2 1TB disks (OS), plus a few SSDs configured across three volumes (OS, Junk, and a work volume). A repository server is an i7 x6 3.3GHz (not over clocked) with 24GB of RAM, several volumes, two of which are RAIDz, SSDs, and other junk. I NFS the data to the attack servers from this server. I am using an Intel Gb card, picked because it supports large MTUs. All are running RELENG_8. If this project moves beyond proof-of-concept, and therefore not my money, we're talking about 100-1,000 TB of data with real servers (SMART, iLO, a NMS, etc). I have my doubts this will happen but in the mean time it's play day. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
