Re: [pfSense Support] throughput, haproxy
Evgeny Yurchenko wrote: I got very interesting results after moving to new kernel. From: [ ID] Interval Transfer Bandwidth [ 3] 14090.0-14100.0 sec799 MBytes670 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 14100.0-14110.0 sec795 MBytes667 Mbits/sec PID USERNAME PRI NICE SIZERES STATE C TIME WCPU COMMAND 13 root 171 ki31 0K 8K RUN1 24.3H 100.00% idle: cpu1 11 root 171 ki31 0K 8K CPU3 3 24.3H 100.00% idle: cpu3 39 root -68- 0K 8K CPU2 2 401:49 97.17% em0 taskq 40 root -68- 0K 8K CPU0 0 401:43 96.68% em1 taskq 14 root 171 ki31 0K 8K RUN0 17.7H 11.08% idle: cpu0 12 root 171 ki31 0K 8K RUN2 17.7H 10.79% idle: cpu2 To: [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 3.66 MBytes 3.07 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 10.0-20.0 sec 3.21 MBytes 2.69 Mbits/sec PID USERNAME PRI NICE SIZERES STATE C TIME WCPU COMMAND 11 root 171 ki31 0K 8K RUN3 5:40 100.00% idle: cpu3 12 root 171 ki31 0K 8K CPU2 2 5:37 100.00% idle: cpu2 13 root 171 ki31 0K 8K CPU1 1 5:41 99.17% idle: cpu1 14 root 171 ki31 0K 8K CPU0 0 5:37 98.78% idle: cpu0 495 root 40 44808K 18540K accept 1 0:01 0.00% php 41 root 43- 0K 8K WAIT 2 0:01 0.00% em0_rx_kthread_0 42 root 43- 0K 8K WAIT 1 0:01 0.00% em0_rx_kthread_1 46 root 43- 0K 8K WAIT 0 0:00 0.00% em1_rx_kthread_1 45 root 43- 0K 8K WAIT 3 0:00 0.00% em1_rx_kthread_0 Should I adjust something manually in config? Evgeny. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org interesting in fact... I guess I should wait then before trying it in production. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] throughput, haproxy
Scott Ullrich wrote: On Thu, Nov 19, 2009 at 12:35 PM, Scott Ullrich sullr...@gmail.com wrote: OK, give me a bit to get it ready. Should be back to you in a couple hours. Lenny, First of all make sure you backup your configuration and have installation media handy (just in case). Run this from a shell (option 8): fetch -o /boot/kernel/ http://cvs.pfsense.org/~sullrich/7-yandex/kernel.gz Then reboot the firewall and let me know how it goes. Scott I got very interesting results after moving to new kernel. From: [ ID] Interval Transfer Bandwidth [ 3] 14090.0-14100.0 sec799 MBytes670 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 14100.0-14110.0 sec795 MBytes667 Mbits/sec PID USERNAME PRI NICE SIZERES STATE C TIME WCPU COMMAND 13 root 171 ki31 0K 8K RUN1 24.3H 100.00% idle: cpu1 11 root 171 ki31 0K 8K CPU3 3 24.3H 100.00% idle: cpu3 39 root -68- 0K 8K CPU2 2 401:49 97.17% em0 taskq 40 root -68- 0K 8K CPU0 0 401:43 96.68% em1 taskq 14 root 171 ki31 0K 8K RUN0 17.7H 11.08% idle: cpu0 12 root 171 ki31 0K 8K RUN2 17.7H 10.79% idle: cpu2 To: [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 3.66 MBytes 3.07 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 10.0-20.0 sec 3.21 MBytes 2.69 Mbits/sec PID USERNAME PRI NICE SIZERES STATE C TIME WCPU COMMAND 11 root 171 ki31 0K 8K RUN3 5:40 100.00% idle: cpu3 12 root 171 ki31 0K 8K CPU2 2 5:37 100.00% idle: cpu2 13 root 171 ki31 0K 8K CPU1 1 5:41 99.17% idle: cpu1 14 root 171 ki31 0K 8K CPU0 0 5:37 98.78% idle: cpu0 495 root 40 44808K 18540K accept 1 0:01 0.00% php 41 root 43- 0K 8K WAIT 2 0:01 0.00% em0_rx_kthread_0 42 root 43- 0K 8K WAIT 1 0:01 0.00% em0_rx_kthread_1 46 root 43- 0K 8K WAIT 0 0:00 0.00% em1_rx_kthread_1 45 root 43- 0K 8K WAIT 3 0:00 0.00% em1_rx_kthread_0 Should I adjust something manually in config? Evgeny. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] throughput, haproxy
Scott Ullrich wrote: On Thu, Nov 19, 2009 at 12:35 PM, Scott Ullrich sullr...@gmail.com wrote: OK, give me a bit to get it ready. Should be back to you in a couple hours. Lenny, First of all make sure you backup your configuration and have installation media handy (just in case). Run this from a shell (option 8): fetch -o /boot/kernel/ http://cvs.pfsense.org/~sullrich/7-yandex/kernel.gz Then reboot the firewall and let me know how it goes. Scott - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org Scott, Does it have to be 1.2.3? Because I have 1.2.2 installed right now. Should I upgrade before that? Lenny.
Re: [pfSense Support] throughput, haproxy
On Sat, Nov 21, 2009 at 6:12 AM, Lenny five2one.le...@gmail.com wrote: Scott, Does it have to be 1.2.3? Because I have 1.2.2 installed right now. Should I upgrade before that? yes, we are moving on to 1.2.3 shortly and 1.2.2 is fading into the sunset. Scott - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] throughput, haproxy
On Thu, Nov 19, 2009 at 2:27 AM, Lenny five2one.le...@gmail.com wrote: # iperf -c 2.2.2.11 -t 1200 -i 10 -w 75000 Client connecting to 2.2.2.11, TCP port 5001 TCP window size: 73.5 KByte (WARNING: requested 73.2 KByte) [ 3] local 1.1.1.1 port 14852 connected with 2.2.2.11 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec746 MBytes626 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 10.0-20.0 sec762 MBytes639 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 20.0-30.0 sec765 MBytes642 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 30.0-40.0 sec776 MBytes651 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 40.0-50.0 sec772 MBytes648 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 50.0-60.0 sec776 MBytes651 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 60.0-70.0 sec768 MBytes644 Mbits/sec I found my old results of iperf and this was the command I executed: iperf -c server-ip -t 60 -M 500 I always got 300-400Mb/s, even with firewall off. And I could never get more than 85kpps. Unfortunately, I can't run these tests now, as the server is in production. Thanks, Lenny. Would you like to test a kernel with the Yandex driver? 1.2.3-* does not have the yandex driver included. Scott - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] throughput, haproxy
Scott Ullrich wrote: On Thu, Nov 19, 2009 at 2:27 AM, Lenny five2one.le...@gmail.com wrote: # iperf -c 2.2.2.11 -t 1200 -i 10 -w 75000 Client connecting to 2.2.2.11, TCP port 5001 TCP window size: 73.5 KByte (WARNING: requested 73.2 KByte) [ 3] local 1.1.1.1 port 14852 connected with 2.2.2.11 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec746 MBytes626 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 10.0-20.0 sec762 MBytes639 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 20.0-30.0 sec765 MBytes642 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 30.0-40.0 sec776 MBytes651 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 40.0-50.0 sec772 MBytes648 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 50.0-60.0 sec776 MBytes651 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 60.0-70.0 sec768 MBytes644 Mbits/sec I found my old results of iperf and this was the command I executed: iperf -c server-ip -t 60 -M 500 I always got 300-400Mb/s, even with firewall off. And I could never get more than 85kpps. Unfortunately, I can't run these tests now, as the server is in production. Thanks, Lenny. Would you like to test a kernel with the Yandex driver? 1.2.3-* does not have the yandex driver included. Scott - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org I sure would. Thanks. Lenny.
Re: [pfSense Support] throughput, haproxy
On Thu, Nov 19, 2009 at 12:07 PM, Lenny five2one.le...@gmail.com wrote: I sure would. Thanks. OK, give me a bit to get it ready. Should be back to you in a couple hours. Scott - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] throughput, haproxy
Lenny wrote: I always got 300-400Mb/s, even with firewall off. And I could never get more than 85kpps. Unfortunately, I can't run these tests now, as the server is in production. Thanks, Lenny. May be stupid question but.. How did you measure 85kpps and how do you measure speed and pps in production? Evgeny. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] throughput, haproxy
On Thu, Nov 19, 2009 at 12:35 PM, Scott Ullrich sullr...@gmail.com wrote: OK, give me a bit to get it ready. Should be back to you in a couple hours. Lenny, First of all make sure you backup your configuration and have installation media handy (just in case). Run this from a shell (option 8): fetch -o /boot/kernel/ http://cvs.pfsense.org/~sullrich/7-yandex/kernel.gz Then reboot the firewall and let me know how it goes. Scott - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] throughput, haproxy
Scott Ullrich wrote: On Thu, Nov 19, 2009 at 12:35 PM, Scott Ullrich sullr...@gmail.com wrote: OK, give me a bit to get it ready. Should be back to you in a couple hours. Lenny, First of all make sure you backup your configuration and have installation media handy (just in case). Run this from a shell (option 8): fetch -o /boot/kernel/ http://cvs.pfsense.org/~sullrich/7-yandex/kernel.gz Then reboot the firewall and let me know how it goes. Scott - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org ok, great, thanks a lot! But unfortunately, I'm already at home, plus I wanna see if the changes I've made to sysctl and loader.conf (the ones we talked about) going to make any difference. But I'll do it on Sunday. Thanks again, Lenny.
Re: [pfSense Support] throughput, haproxy
Lenny wrote: Evgeny Yurchenko wrote: Lenny wrote: I always got 300-400Mb/s, even with firewall off. And I could never get more than 85kpps. Unfortunately, I can't run these tests now, as the server is in production. Thanks, Lenny. May be stupid question but.. How did you measure 85kpps and how do you measure speed and pps in production? Evgeny. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org To tell you the truth I don't remember, as it was a few months ago, but I'm attaching the RRD graphs: traffic, packets and throughput. You can clearly see the peaks, although as you might know, on the graph from previous weeks the numbers actually become a bit smaller than they really were. For example, on the traffic graph it says 270Mb was a maximum outgoing, when in fact my actual maximum was about 310Mb. I would attach some newer graphs, but my next peak is in 2 days. Just to be clear: at those peaks I had my CPUs at maximum or very near that. Lenny. Ok. But looking into this http://forum.pfsense.org/index.php/topic,20624.0.html and watching my own box during tests peformed for you I see weird things: [ ID] Interval Transfer Bandwidth [ 3] 930.0-940.0 sec744 MBytes624 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 940.0-950.0 sec748 MBytes627 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 950.0-960.0 sec745 MBytes625 Mbits/sec But! So I looked into how these graphs are populated - /var/db/rrd/updaterrd.sh counter=1 while [ $counter -ne 0 ] do ... sleep 60 done So, every 60 seconds you take data by means of '/usr/bin/netstat -nbf link -I bge0' and feed it to RRD. Now let's do /usr/bin/netstat -nbf link -I bge0: NameMtu Network AddressIpkts IerrsIbytes Opkts Oerrs Obytes Coll bge0 1500 Link#1 00:0b:cd:52:5b:41 299767100 0 2605426760 299287128 0 191226159 0 Bytes Number has 9 digits so wrap will happen after receiving/transmitting 999 999 999 bytes / 60sec * 8 = 133 333 333 bits/s which is approx 130 Mb/s I believe RRD can handle wraps through 0 but at some point (speed) you'll have two(or even 3-4) wraps. What am I missing here? Evgeny. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] throughput, haproxy
Bill Marquette wrote: I'm not positive if netstat shows a 32 or 64 bit number, but it's certainly not limited to 9 digits. Your Ibytes column alone has 10 2,605,426,760. 32 bit will still wrap pretty quick however and is not suitable for gigabit links. --Bill Yes, ten digits, sorry. Anyway, we can't get true picture of bandwidth usage looking at rrd graphs and having speed 'after 500Mb/s', is it what you are saying? Thanks. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] throughput, haproxy
No you should not worry with your level of traffic. But as soon as you cross 500Mb/s you should not trust RRD any more. I was gradually increasing bandwidh usage using iperf udp -b option: 300Mb/s - ok, 400Mb/s - ok, 500Mb/s - ok, 600Mb/s - ooops -( In pfSense 2.0 we use the 64 bit counters for the data collection using the pf counters. This will prevent such wrapping. Regards, Seth - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] throughput, haproxy
On Tue, Nov 10, 2009 at 1:50 AM, Lenny five2one.le...@gmail.com wrote: At second thought, to get rid of the errors I told you about, I did 2 things: added this to /boot/loader.conf: hw.em.rxd=4096 hw.em.txd=4096 and added to /etc/sysctl.conf: dev.em.0.rx_processing_limit=1000 dev.em.1.rx_processing_limit=1000 plus, I changed net.inet.ip.intr_queue_maxlen=4096 and added kern.ipc.somaxconn=1024 These were the changes I did outside of the WebGUI. So should I still increase the dev.em.X.rx_processing_limit value? Yes, give that a try. My kernel that I have here increased em.txd and em.txr but I was unaware they where able to be set since they are hard coded in the driver? Scott - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] throughput, haproxy
On Tue, Nov 10, 2009 at 1:50 AM, Lenny five2one.le...@gmail.com wrote: Lenny wrote: Scott Ullrich wrote: On Mon, Nov 9, 2009 at 3:45 PM, Scott Ullrich sullr...@gmail.com wrote: Contact me off list. I have a kernel I need you to test. In the meantime, please try increasing these sysctl's: pfSense:~# sysctl -a | grep rx_processing_limit dev.em.0.rx_processing_limit: 100 dev.em.1.rx_processing_limit: 100 dev.em.2.rx_processing_limit: 100 dev.em.3.rx_processing_limit: 100 Try increasing each to 256, then 512, 1024, 2048, etc. If these do not help contact me for a new kernel. Scott - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org Hi Scott, Actually, I have them set on a 1000 for quite a while now. Before I did that I had errors on interfaces. Do you still want me to increase to 2048 and more? Thanks, Lenny. At second thought, to get rid of the errors I told you about, I did 2 things: added this to /boot/loader.conf: hw.em.rxd=4096 hw.em.txd=4096 and added to /etc/sysctl.conf: dev.em.0.rx_processing_limit=1000 dev.em.1.rx_processing_limit=1000 plus, I changed net.inet.ip.intr_queue_maxlen=4096 and added kern.ipc.somaxconn=1024 These were the changes I did outside of the WebGUI. So should I still increase the dev.em.X.rx_processing_limit value? Also let me know what this sysctl is showing: net.inet.ip.intr_queue_drops If it shows 0 then you might want to increase net.inet.ip.intr_queue_maxlen Scott - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] throughput, haproxy
Scott Ullrich wrote: On Tue, Nov 10, 2009 at 1:50 AM, Lenny five2one.le...@gmail.com wrote: Lenny wrote: Scott Ullrich wrote: On Mon, Nov 9, 2009 at 3:45 PM, Scott Ullrich sullr...@gmail.com wrote: Contact me off list. I have a kernel I need you to test. In the meantime, please try increasing these sysctl's: pfSense:~# sysctl -a | grep rx_processing_limit dev.em.0.rx_processing_limit: 100 dev.em.1.rx_processing_limit: 100 dev.em.2.rx_processing_limit: 100 dev.em.3.rx_processing_limit: 100 Try increasing each to 256, then 512, 1024, 2048, etc. If these do not help contact me for a new kernel. Scott - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org Hi Scott, Actually, I have them set on a 1000 for quite a while now. Before I did that I had errors on interfaces. Do you still want me to increase to 2048 and more? Thanks, Lenny. At second thought, to get rid of the errors I told you about, I did 2 things: added this to /boot/loader.conf: hw.em.rxd=4096 hw.em.txd=4096 and added to /etc/sysctl.conf: dev.em.0.rx_processing_limit=1000 dev.em.1.rx_processing_limit=1000 plus, I changed net.inet.ip.intr_queue_maxlen=4096 and added kern.ipc.somaxconn=1024 These were the changes I did outside of the WebGUI. So should I still increase the dev.em.X.rx_processing_limit value? Also let me know what this sysctl is showing: net.inet.ip.intr_queue_drops If it shows 0 then you might want to increase net.inet.ip.intr_queue_maxlen Scott - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org it's 0. Lenny.
Re: [pfSense Support] throughput, haproxy
On Mon, Nov 9, 2009 at 12:41 AM, Lenny five2one.le...@gmail.com wrote: Now I'm totally lost:( I had this long thread this year on this issue here and eventually the only thing the guys could advise me is to buy a newer server. I did. And while I do see an improvement in performance (it's about twice it was before) I'm still nowhere near what you have. I realize that your traffic is lab UDP and mine is production TCP, so let's say you'd get half of that in production, but then still - you're only on 54% CPU. By the way, how come your second NIC is only loading the CPU 4%? Shouldn't it be pretty much like the first one? It's what I have. I'm ready to show you my config/diagrams/whatever, but I need this issue resolved. Please? Contact me off list. I have a kernel I need you to test. Scott - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] throughput, haproxy
On Mon, Nov 9, 2009 at 3:45 PM, Scott Ullrich sullr...@gmail.com wrote: Contact me off list. I have a kernel I need you to test. In the meantime, please try increasing these sysctl's: pfSense:~# sysctl -a | grep rx_processing_limit dev.em.0.rx_processing_limit: 100 dev.em.1.rx_processing_limit: 100 dev.em.2.rx_processing_limit: 100 dev.em.3.rx_processing_limit: 100 Try increasing each to 256, then 512, 1024, 2048, etc. If these do not help contact me for a new kernel. Scott - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] throughput, haproxy
Scott Ullrich wrote: On Mon, Nov 9, 2009 at 3:45 PM, Scott Ullrich sullr...@gmail.com wrote: Contact me off list. I have a kernel I need you to test. In the meantime, please try increasing these sysctl's: pfSense:~# sysctl -a | grep rx_processing_limit dev.em.0.rx_processing_limit: 100 dev.em.1.rx_processing_limit: 100 dev.em.2.rx_processing_limit: 100 dev.em.3.rx_processing_limit: 100 Try increasing each to 256, then 512, 1024, 2048, etc. If these do not help contact me for a new kernel. Scott - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org Hi Scott, Actually, I have them set on a 1000 for quite a while now. Before I did that I had errors on interfaces. Do you still want me to increase to 2048 and more? Thanks, Lenny.
Re: [pfSense Support] throughput, haproxy
Lenny wrote: Scott Ullrich wrote: On Mon, Nov 9, 2009 at 3:45 PM, Scott Ullrich sullr...@gmail.com wrote: Contact me off list. I have a kernel I need you to test. In the meantime, please try increasing these sysctl's: pfSense:~# sysctl -a | grep rx_processing_limit dev.em.0.rx_processing_limit: 100 dev.em.1.rx_processing_limit: 100 dev.em.2.rx_processing_limit: 100 dev.em.3.rx_processing_limit: 100 Try increasing each to 256, then 512, 1024, 2048, etc. If these do not help contact me for a new kernel. Scott - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org Hi Scott, Actually, I have them set on a 1000 for quite a while now. Before I did that I had errors on interfaces. Do you still want me to increase to 2048 and more? Thanks, Lenny. At second thought, to get rid of the errors I told you about, I did 2 things: added this to /boot/loader.conf: hw.em.rxd=4096 hw.em.txd=4096 and added to /etc/sysctl.conf: dev.em.0.rx_processing_limit=1000 dev.em.1.rx_processing_limit=1000 plus, I changed net.inet.ip.intr_queue_maxlen=4096 and added kern.ipc.somaxconn=1024 These were the changes I did outside of the WebGUI. So should I still increase the dev.em.X.rx_processing_limit value? Lenny.
Re: [pfSense Support] throughput, haproxy
Seth Mos wrote: Lenny schreef: But I would really like to ask again, as this is very important: will replacing the PCI-X NIC with PCI-e one give some boost in performance? Unlikely, there is little reason to switch. The theoretical bandwidth cases are not too helpful. The intel dual port pci-e cards are x4 ~ (4 * 250MB/s) The intel dual port pci-x card is 64bit 133 mhz is ~ 1000MB/s So, no you are not likely to see any improvement. If any, I suspect it's more of a chipset thing. Regards, Seth - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org You're kind of taking this last hope from me:) Then what are the options for someone who has traffic more than pfSense can take? For a example, a streamer with packet length of 1840 and 50kpps, that's 700Mb. Is there a possibility of some sort of pfSense cluster? Because as far as I understand, I have one of the fastest CPUs on the market, not counting the i7 and I still can't pass more than 50kpps with a packet length of 600, and that's just image files. Thanks, Lenny. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] throughput, haproxy
From: Lenny five2one.le...@gmail.com Sent: Sunday, November 08, 2009 1:38 AM Seth Mos wrote: Lenny schreef: But I would really like to ask again, as this is very important: will replacing the PCI-X NIC with PCI-e one give some boost in performance? Unlikely, there is little reason to switch. The theoretical bandwidth cases are not too helpful. The intel dual port pci-e cards are x4 ~ (4 * 250MB/s) The intel dual port pci-x card is 64bit 133 mhz is ~ 1000MB/s So, no you are not likely to see any improvement. If any, I suspect it's more of a chipset thing. Regards, Seth - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org You're kind of taking this last hope from me:) Then what are the options for someone who has traffic more than pfSense can take? For a example, a streamer with packet length of 1840 and 50kpps, that's 700Mb. Is there a possibility of some sort of pfSense cluster? Because as far as I understand, I have one of the fastest CPUs on the market, not counting the i7 and I still can't pass more than 50kpps with a packet length of 600, and that's just image files. Thanks, Lenny. Lenny, now I am experimenting a lot trying to find out why sometimes when there is heavy load CARP-master switches to stand-by and never comes back. I know this problem is different from yours but look at the performance I get on pretty old hardware. UDP-stream generator pfSense CARP cluster on HP DL360 G3 - receiver This from receiver: [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 350.0-360.0 sec 1.05 GBytes903 Mbits/sec 0.013 ms 12/767479 (0.0016%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 360.0-370.0 sec 1.05 GBytes902 Mbits/sec 0.013 ms 334/767174 (0.044%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 370.0-380.0 sec 1.05 GBytes901 Mbits/sec 0.013 ms8/766545 (0.001%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 380.0-390.0 sec 1.05 GBytes903 Mbits/sec 0.015 ms 19/767586 (0.0025%) This is on pfSense: last pid: 44303; load averages: 0.08, 0.02, 0.00 up 3+07:30:11 23:14:56 89 processes: 6 running, 66 sleeping, 17 waiting CPU: 0.1% user, 0.0% nice, 0.2% system, 15.7% interrupt, 83.9% idle Mem: 44M Active, 10M Inact, 39M Wired, 76K Cache, 17M Buf, 1906M Free Swap: 4096M Total, 4096M Free PID USERNAME THR PRI NICE SIZERES STATE C TIME WCPU COMMAND 13 root1 171 ki31 0K 8K CPU1 1 79.3H 100.00% idle: cpu1 11 root1 171 ki31 0K 8K CPU3 3 79.3H 100.00% idle: cpu3 12 root1 171 ki31 0K 8K RUN2 79.1H 100.00% idle: cpu2 40 root1 -68- 0K 8K CPU0 0 30:17 54.20% irq30: bge1 14 root1 171 ki31 0K 8K RUN0 78.6H 41.06% idle: cpu0 39 root1 -68- 0K 8K WAIT 0 18:12 4.05% irq28: bge0 ... and it results in approximately 76kpps. And this is pretty old HP DL360 G3 with Broadcom NICs. There must be some mystery in your set up. Your system MUST perform better. Evgeny. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] throughput, haproxy
Evgeny Yurchenko wrote: Then what are the options for someone who has traffic more than pfSense can take? For a example, a streamer with packet length of 1840 and 50kpps, that's 700Mb. Is there a possibility of some sort of pfSense cluster? Because as far as I understand, I have one of the fastest CPUs on the market, not counting the i7 and I still can't pass more than 50kpps with a packet length of 600, and that's just image files. Thanks, Lenny. Lenny, now I am experimenting a lot trying to find out why sometimes when there is heavy load CARP-master switches to stand-by and never comes back. I know this problem is different from yours but look at the performance I get on pretty old hardware. UDP-stream generator pfSense CARP cluster on HP DL360 G3 - receiver This from receiver: [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 350.0-360.0 sec 1.05 GBytes903 Mbits/sec 0.013 ms 12/767479 (0.0016%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 360.0-370.0 sec 1.05 GBytes902 Mbits/sec 0.013 ms 334/767174 (0.044%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 370.0-380.0 sec 1.05 GBytes901 Mbits/sec 0.013 ms 8/766545 (0.001%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 380.0-390.0 sec 1.05 GBytes903 Mbits/sec 0.015 ms 19/767586 (0.0025%) This is on pfSense: last pid: 44303; load averages: 0.08, 0.02, 0.00 up 3+07:30:11 23:14:56 89 processes: 6 running, 66 sleeping, 17 waiting CPU: 0.1% user, 0.0% nice, 0.2% system, 15.7% interrupt, 83.9% idle Mem: 44M Active, 10M Inact, 39M Wired, 76K Cache, 17M Buf, 1906M Free Swap: 4096M Total, 4096M Free PID USERNAME THR PRI NICE SIZERES STATE C TIME WCPU COMMAND 13 root1 171 ki31 0K 8K CPU1 1 79.3H 100.00% idle: cpu1 11 root1 171 ki31 0K 8K CPU3 3 79.3H 100.00% idle: cpu3 12 root1 171 ki31 0K 8K RUN2 79.1H 100.00% idle: cpu2 40 root1 -68- 0K 8K CPU0 0 30:17 54.20% irq30: bge1 14 root1 171 ki31 0K 8K RUN0 78.6H 41.06% idle: cpu0 39 root1 -68- 0K 8K WAIT 0 18:12 4.05% irq28: bge0 ... and it results in approximately 76kpps. And this is pretty old HP DL360 G3 with Broadcom NICs. There must be some mystery in your set up. Your system MUST perform better. Evgeny. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org Evgeny, Now I'm totally lost:( I had this long thread this year on this issue here and eventually the only thing the guys could advise me is to buy a newer server. I did. And while I do see an improvement in performance (it's about twice it was before) I'm still nowhere near what you have. I realize that your traffic is lab UDP and mine is production TCP, so let's say you'd get half of that in production, but then still - you're only on 54% CPU. By the way, how come your second NIC is only loading the CPU 4%? Shouldn't it be pretty much like the first one? It's what I have. I'm ready to show you my config/diagrams/whatever, but I need this issue resolved. Please? Lenny. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] throughput, haproxy
Lenny wrote: Hi, I'm the same guy that had that long thread about not being able to push more than 15kpps. Well, this is sort of a report + some additional questions. Anyway, eventually we purchased an IBM x3550 server with 2 Quad Core CPUs (5230 I think). Now I can push 310Mb, which is about 70kpps(my average packet size grew a little bit since then and I believe it's now about 600). Lenny. Hi Lenny! I can not give you any advice but would like to share my results with HP DL360 G4 box which has two dual-core Intels 3.4.GHz running *1.2.3-RC2* built on Mon Aug 31 06:09:28 UTC 2009. It was not built for performance and has only two Broadcom NICs on motherboard. One NIC is LAN, another one is tagged with 20 VLANs though usually only one-two (max three) vlans are pushing traffic really hard simultaneously. Traffic goes up to 450Mb/s with 38kpps and CPU load is 25% during these peaks. I suspect that it is when 1CPU (core) is loaded 100% and another 3 are idling. Is this the case for you as well with 100% one CPU load and 7 others idling? Your system is much newer then mine and everybody says that Intel NICs are better than Broadcom so I would expect better performance. Your results for real traffic or you were performing tests? What kind of traffic are you pushing? I've noticed that Intel NICs deal much better with TCP than with UDP in terms of CPU usage (it can be explained only by performing some TCP functions by NIC). Please keep us posted! Evgeny. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] throughput, haproxy
Evgeny Yurchenko wrote: Lenny wrote: Hi, I'm the same guy that had that long thread about not being able to push more than 15kpps. Well, this is sort of a report + some additional questions. Anyway, eventually we purchased an IBM x3550 server with 2 Quad Core CPUs (5230 I think). Now I can push 310Mb, which is about 70kpps(my average packet size grew a little bit since then and I believe it's now about 600). Lenny. Hi Lenny! I can not give you any advice but would like to share my results with HP DL360 G4 box which has two dual-core Intels 3.4.GHz running *1.2.3-RC2* built on Mon Aug 31 06:09:28 UTC 2009. It was not built for performance and has only two Broadcom NICs on motherboard. One NIC is LAN, another one is tagged with 20 VLANs though usually only one-two (max three) vlans are pushing traffic really hard simultaneously. Traffic goes up to 450Mb/s with 38kpps and CPU load is 25% during these peaks. I suspect that it is when 1CPU (core) is loaded 100% and another 3 are idling. Is this the case for you as well with 100% one CPU load and 7 others idling? Your system is much newer then mine and everybody says that Intel NICs are better than Broadcom so I would expect better performance. Your results for real traffic or you were performing tests? What kind of traffic are you pushing? I've noticed that Intel NICs deal much better with TCP than with UDP in terms of CPU usage (it can be explained only by performing some TCP functions by NIC). Please keep us posted! Evgeny. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org Hi Evgeny, You are right about the CPU load - it is exactly what's happening, only I have 2 Cores out of 8 reaching 100%(one for each interface). My traffic is production TCP, it's a website, with mostly pictures and flash files(advertisement). But I would really like to ask again, as this is very important: will replacing the PCI-X NIC with PCI-e one give some boost in performance? Lenny. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
Re: [pfSense Support] throughput, haproxy
Lenny schreef: But I would really like to ask again, as this is very important: will replacing the PCI-X NIC with PCI-e one give some boost in performance? Unlikely, there is little reason to switch. The theoretical bandwidth cases are not too helpful. The intel dual port pci-e cards are x4 ~ (4 * 250MB/s) The intel dual port pci-x card is 64bit 133 mhz is ~ 1000MB/s So, no you are not likely to see any improvement. If any, I suspect it's more of a chipset thing. Regards, Seth - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
[pfSense Support] throughput, haproxy
Hi, I'm the same guy that had that long thread about not being able to push more than 15kpps. Well, this is sort of a report + some additional questions. Anyway, eventually we purchased an IBM x3550 server with 2 Quad Core CPUs (5230 I think). Now I can push 310Mb, which is about 70kpps(my average packet size grew a little bit since then and I believe it's now about 600). This is where I hit the CPU limit. Not sure how normal this is though. But I was thinking, will it give me some boost in performance if I use PCI-e Dual NIC from Intel instead of the PCI-X that I'm using today? (also Intel). I was thinking about this one: http://www.intel.com/products/server/adapters/pro1000pt-dualport/pro1000pt-dualport-overview.htm By the way, when testing the new server I installed the 1.2.3RC2 and I must tell you that its performance was pretty awful. Only when I replaced it with the stable 1.2.2 I got the performance I have now. I don't remember the exact numbers, but I believe the CPUs were maxed out on half the traffic I have now. I read on some DragonflyBSD forum that the new em driver is much worse than the previous version, which is used in 1.2.2. Also, I mentioned in the previous thread that while the 1.2.3 has the Yandex driver version, I could never get it to work the way it was supposed to - multithreaded. Anyway, I thought maybe you'll find this info helpful. Other than that, I have another question I wanted to ask. I saw the HAproxy package being added and since I have to replace my old Alteon now, I thought maybe it is the way to go. Will it do the job if all I need is Layer 4 load balancing? I have about 150-200k concurrent sessions at peaks. Will it survive? What about the effect on performance? I realize it will use the other cores of the CPUs, but still. I have about 1GB spare RAM on the server. And the last question. I understand that this package is only for 1.2.3 and 2.0 versions, but I installed it at home on my 1.2.2 and it seems to be OK, although I don't have much to load balance here, so I wanted to know if it will actually work with 1.2.2? Thanks a lot! Lenny. - To unsubscribe, e-mail: support-unsubscr...@pfsense.com For additional commands, e-mail: support-h...@pfsense.com Commercial support available - https://portal.pfsense.org
