Hi guys, I know how sick of me you are by now, but I've had some developments here and now I'm stuck again.
So, FINALLY I convinced the management to buy a new server. We bought an IBM x3550 with 2 Quad Core CPUs E5420 and 2GB RAM PC2-5300 667MHz. Not just that, we bought 2 of them ( we need the second one for the identical project). I really wanted to try the Yandex em driver, so I installed the *"1.2.3-RC2 * built on Wed Jun 24 10:37:51 EDT 2009 version. These are the things I've changed: /etc/sysctl.conf added: dev.em.0.rx_processing_limit=1000 dev.em.1.rx_processing_limit=1000 kern.ipc.somaxconn=1024 dev.em.0.rx_kthreads=2 dev.em.1.rx_kthreads=2 /boot/loader.conf added: hw.em.rxd="4096" hw.em.txd="4096" I wasn't sure about the kthreads, so I put "2" as a value, but I later read that people use 6. Now, for the problem: It's the same! I had 50kpps, which was 180Mbps and I had 2 CPUs go up to 100%. How is that even possible? Should I give up on RC version and go back to stable, which also means giving up on the em driver? Any other things I can adjust? By the way, I checked sysctl net.inet.ip.intr_queue_drops and it's "0". On the interfaces I see that em0(outside) has 0 errors, but on the em1(inside) there are 3666587/0. 6 of the CPUs(cores) are usually 100% idle, while the other 2 are actually stuck with emX taskq. In other words - nothing's changed. Except for 1 thing - the management spent the money and now they wanna see results. Please help! Lenny. changed: net.inet.ip.intr_queue_maxlen=4096 (old value=1000) On Thu, May 14, 2009 at 11:26 AM, Lenny <[email protected]> wrote: > Thanks for all the suggestions, guys. > Anyway, I found it very interesting that the new snapshots have yandex > driver in them, so I decided to try it. > Of course, as I don't have the new server yet, I had to try on my old IBM > x335. > > But here are a couple of things that wouldn't let me try it: > with 2 SCSI hard drives in RAID 1 (LSI controller), it would always give an > error and reboot in 15 sec. If I take out 1 drive - it would boot, but then > get stuck on "configuring wan interface" for a long time and then reboot. > It's not connected, but previous pfSense versions had no problems with it. > > Any suggestions? > > thanks, > Lenny. > > P.S. we don't have eval-systems, so I'll have to take the risk and buy a > cat in the sack. > > On Wed, May 13, 2009 at 6:13 PM, Scott Ullrich <[email protected]> wrote: > >> On Wed, May 13, 2009 at 10:21 AM, Rainer Duffner <[email protected]> >> wrote: >> > AFAIK, SUN still provides eval-systems for free. >> > I would evaluate one of the new X2270 with the Nehalem Xeons. >> > This should provide a 50% boost even on 5400-series Xeons. >> > Also, they use Intel NICs, IIRC. >> > >> > The smallest test-system already has 6 GB of RAM and costs 2000 USD, >> > which you have to pay only after 60 days. >> >> All good advice here in the last couple messages. Wanted to add that >> I would suggest trying a recent 1.2 / FreeBSD 7 pfSense snapshot from >> snapshots.pfsense.org as we added the high performance yandex driver >> located at >> http://people.yandex-team.ru/~wawa/em-6.9.6-RELENG7-yandex-1.36.2.10.tar.gz<http://people.yandex-team.ru/%7Ewawa/em-6.9.6-RELENG7-yandex-1.36.2.10.tar.gz> >> >> The README file that was included: >> >> Main features >> ------------- >> >> RX queue is being processed w/more than one thread. Use "sysctl >> dev.em.X.rx_kthreads" to alter number of threads. >> >> TX interrupts has been removed because it's not neccessary actually. >> That's why interrupt rate has been reduced twice at least. >> >> TX queue cleaning moved to seperate kthread. em_start uses mtx_trylock >> instean of mtx_lock. That's why em_start locks less. >> >> + RX queues' priority may be altered thru sysctl. System seems to be >> more stable if RX scheduled w/less priority. >> >> + RX interrupt stay masked if there is no thread ready to catch >> interrupt. The hint reduces context switching under load. >> >> You will want to experiment with 1 thread per proc and 2 threads per >> proc by setting "sysctl dev.em.X.rx_kthreads" (I think) >> >> Scott >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> Commercial support available - https://portal.pfsense.org >> >> >
