I have a hard time understanding the output of /proc/interrupts since there seem to be multiple interrupts already: "eth2-TxRx-0", "eth2-TxRx-1".. but it seems to be balanced pretty poorly. Should I change smp_affinity for all these interrupts or only for the one that has the name "eth2"?
You can see the output of /proc/interrupts here -> http://i.imgur.com/ZLulmkQ.png Best Regards Philip 2013/3/22 Yongming Zhao <[email protected]> > well, it is easy to identify the irq issue here: > 1, in "top", press "1" to display all CPU details. and press "H" to > display the Traffic Server threadings, by default the process is sorted > with CPU usage desc. > you may get one of the CPU with full load but not single TS process. > > 2, "cat /proc/interrupts", and grep out your 10GE nic, check the IRQs. you > need the IRQs on different CPUs for better performance. > you may get that all the IRQs for the NIC is on one CPU, that is the CPU > with full load, typically this CPU0 > > just set the smp_affinity for each IRQ, here is a not prove to working one > line script(replace the eth1 with your NIC name): > > j=0;for i in $(grep eth1 /proc/interrupts | awk -F: "{print \$1}");do test > $j -gt $(grep processor /proc/cpuinfo | tail -n 1 | awk '{print $NF}') && > let j=0;echo $(echo -n $(python -c 'a=1<<'$(echo $j%32 | bc)'; print > "%X"%a'); echo -n $(k=$(echo $j/32 | bc);while [ $k -gt 0 ];do echo -n > ",00000000";let k=k-1;done))> /proc/irq/$i/smp_affinity;let j=j+1;done > > > FYI > > 在 2013-3-22，上午6:23，Igor Galić <[email protected]> 写道： > > This may be useful: > > http://kerneltrap.org/mailarchive/linux-netdev/2010/4/15/6274814/thread > > ------------------------------ > > Hi Yongming, > > I haven't changed the networking configuraton but I've also noticed that > once the first core is at 100% utilization the server won't answer all ping > requests anymore and has packet loss. This might be a sign that all network > traffic is handled by the first core isn't it? > > You can find a screenshot of the threading output of top here: > http://i.imgur.com/X3te2Ru.png > > Best Regards > Philip > > 2013/3/21 Yongming Zhao <[email protected]> > >> well, due to the high network traffic, have you make the 10Ge NIC irq >> balanced to multiple cpu? >> >> and can you show us the threading CPU usage in the top? >> >> thanks >> >> 在 2013-3-21，下午7:42，Philip <[email protected]> 写道： >> >> I've just upgraded to ATS 3.3.1-dev. The problem still is the same: >> http://i.imgur.com/1pHWQy7.png >> >> The load goes on one core. (The server is only running ATS) >> >> 2013/3/21 Philip <[email protected]> >> >>> Hi Igor, >>> >>> I am using ATS 3.2.4, Debian 6 (Squeeze) and a 3.2.13 Kernel. >>> >>> I was using the "traffic_line -r" command to see the number of origin >>> connections growing and htop/atop to see that only one core is 100% >>> utilized. I've already tested the following changes to the configuration: >>> >>> proxy.config.accept_threads -> 0 >>> >>> proxy.config.exec_thread.autoconfig -> 0 >>> proxy.config.exec_thread.limit -> 120 >>> >>> They had no effect there is still the one core that becomes 100% >>> utilized and turns out to be a bottleneck. >>> >>> Best Regards >>> Philip >>> >>> >>> 2013/3/21 Igor Galić <[email protected]> >>> >>>> Hi Philip, >>>> >>>> Let's start with some simple data mining: >>>> >>>> which version of ATS are you running? >>>> What OS/Distro/version are you running it on? >>>> >>>> Are you looking at stats_over_http's output to determine what's going >>>> on in ATS? >>>> >>>> -- i >>>> >>>> ------------------------------ >>>> >>>> I have noticed the following strange behavior: Once the number of >>>> origin connections start to increase and the proxying speed collapses the >>>> first core is at 100% utilization while the others are not even close to >>>> that. It seems like the origin requests are handled by the first core only. >>>> Is this expected behavior that can be changed by editing the configuration >>>> or is this a bug? >>>> >>>> >>>> >>>> 2013/3/20 Philip <[email protected]> >>>> >>>>> Hi, >>>>> >>>>> I am running ATS on a pretty large server with two physical 6 core >>>>> XEON CPUs and 22 raw device disks. I want to use that server as a frontend >>>>> for several fileservers. It is currently configured to be infront of two >>>>> file-servers. The load on the ATS server is pretty low. About 1-4% disk >>>>> utilization and 500Mbps of outgoing traffic. >>>>> >>>>> Once I direct the traffic of the third file server towards ATS >>>>> something strange happens: >>>>> >>>>> - The number of origin connection increases continually. >>>>> - Requests that hit ATS and are not cached are served really slow to >>>>> the client (about 35 kB/s) while requests that are served from the cache >>>>> are blazingly fast. >>>>> >>>>> The ATS server has a dedicated 10Gbps port that is not maxed out, no >>>>> CPU core is maxxed, there is no swapping, there are no error logs and also >>>>> the origin servers are not heavy utilized. It feels like there are not >>>>> enough workers to process the origin requests. >>>>> >>>>> Is there anything I can do to check if my theory is right and a way to >>>>> increase the number of origin workers? >>>>> >>>>> Best Regards >>>>> Philip >>>>> >>>> >>>> >>>> >>>> >>>> -- >>>> Igor Galić >>>> >>>> Tel: +43 (0) 664 886 22 883 >>>> Mail: [email protected] >>>> URL: http://brainsware.org/ >>>> GPG: 6880 4155 74BD FD7C B515 2EA5 4B1D 9E08 A097 C9AE >>>> >>>> >>> >> >> > > > > -- > Igor Galić > > Tel: +43 (0) 664 886 22 883 > Mail: [email protected] > URL: http://brainsware.org/ > GPG: 6880 4155 74BD FD7C B515 2EA5 4B1D 9E08 A097 C9AE > > >
