Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-13 Thread Ian FREISLICH
"David Christensen" wrote: > > Pyun YongHyeon wrote: > > > On Wed, Mar 10, 2010 at 02:45:47PM -0800, David Christensen wrote: > > > > The bce(4) hardware supports a linked list of pages for RX buffer=20 > > > > descriptors. The stock build supports 2 pages (RX_PAGES) with a=20 > > > > total of 511

RE: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-12 Thread David Christensen
> Pyun YongHyeon wrote: > > On Wed, Mar 10, 2010 at 02:45:47PM -0800, David Christensen wrote: > > > The bce(4) hardware supports a linked list of pages for RX buffer > > > descriptors. The stock build supports 2 pages (RX_PAGES) with a > > > total of 511 BD's per page. The hardware can support

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-10 Thread Ian FREISLICH
Pyun YongHyeon wrote: > On Wed, Mar 10, 2010 at 02:45:47PM -0800, David Christensen wrote: > > The bce(4) hardware supports a linked list of pages for RX > > buffer descriptors. The stock build supports 2 pages (RX_PAGES) > > with a total of 511 BD's per page. The hardware can support a > > maxi

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-10 Thread Ian FREISLICH
Pyun YongHyeon wrote: > I successfully reproduced the issue with netperf on BCM5709. You > can use UDP frame size 1 to trigger the issue. Now I wish I had paid closer attention ages ago. I actually saw this when I benchmarked the system post purchase, but didn't investigate further. I tested and

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-10 Thread Pyun YongHyeon
On Wed, Mar 10, 2010 at 02:45:47PM -0800, David Christensen wrote: > > > I successfully reproduced the issue with netperf on BCM5709. > > You can use UDP frame size 1 to trigger the issue. > > > > > Changing the high level design of bce_rx_intr() and > > > bce_rx_fill_chain() slightly to post a

RE: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-10 Thread David Christensen
> I successfully reproduced the issue with netperf on BCM5709. > You can use UDP frame size 1 to trigger the issue. > > > Changing the high level design of bce_rx_intr() and > > bce_rx_fill_chain() slightly to post a new buffer as each frame is > > passed to the OS would likely avoid these gaps

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-10 Thread Pyun YongHyeon
On Wed, Mar 10, 2010 at 11:11:13AM -0800, David Christensen wrote: > > > What's the traffic look like? Jumbo, standard, short > > frames? Any=20 > > > good ideas on profiling the code? I haven't figured out how to use > > > the CPU TSC but there is a free running timer on the device > > that

RE: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-10 Thread David Christensen
> > What's the traffic look like? Jumbo, standard, short > frames? Any=20 > > good ideas on profiling the code? I haven't figured out how to use > > the CPU TSC but there is a free running timer on the device > that might > > be usable to calculate where the driver's time is spent. > > It

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-10 Thread Ian FREISLICH
"David Christensen" wrote: > > Yeah, but the question is why bce(4) has no available RX buffers. > > The system has a lot of available mbufs so I don't see the=20 > > root cause here. > > What's the traffic look like? Jumbo, standard, short frames? Any=20 > good ideas on profiling the code? I h

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-09 Thread Ian FREISLICH
Pyun YongHyeon wrote: > On Tue, Mar 09, 2010 at 11:55:30PM +0200, Ian FREISLICH wrote: > > I set the RX as high as 512 in 64 quanta but it made little difference > > to the interrupt rate. At times where we experience the packet > > loss and com_no_buffers increases, the interrupt rate on between

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-09 Thread Fabien Thomas
If you are on head/stable_7/stable_8 you can also do quick test with top mode pmcstat -S unhalted-cycles -T (http://wiki.freebsd.org/PmcTools/PmcTop). For more in depth post processing with source code (c+asm) you can output to Kcachegrind (http://wiki.freebsd.org/PmcTools/PmcKcachegrind). Fabie

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-09 Thread Ryan Stone
trash_ctor and trash_dtor? You're running with INVARIANTS. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

RE: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-09 Thread David Christensen
> -Original Message- > From: Ryan Stone [mailto:ryst...@gmail.com] > Sent: Tuesday, March 09, 2010 2:31 PM > To: David Christensen > Cc: pyu...@gmail.com; Ian FREISLICH; curr...@freebsd.org > Subject: Re: dev.bce.X.com_no_buffers increasing and packet loss > &

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-09 Thread Ryan Stone
> What's the traffic look like?  Jumbo, standard, short frames?  Any > good ideas on profiling the code?  I haven't figured out how to use > the CPU TSC but there is a free running timer on the device that > might be usable to calculate where the driver's time is spent. > > Dave In my experience h

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-09 Thread Pyun YongHyeon
On Tue, Mar 09, 2010 at 11:55:30PM +0200, Ian FREISLICH wrote: > Pyun YongHyeon wrote: > > On Tue, Mar 09, 2010 at 03:31:55PM +0200, Ian FREISLICH wrote: > > > Can you explain the tunables please - I'm guessing it's these: > > I think I asked the wrong question. What is a "Quick BD Chain"? I don

RE: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-09 Thread David Christensen
> > > > It's been running for about 1:23 on the patched driver. > I'm still > > > > seeing the com_no_buffers increase: > > > > > > > > [firewall2.jnb1] ~ # sysctl dev.bce |grep com_no_buffers > > > > dev.bce.0.com_no_buffers: 5642 > > > > dev.bce.1.com_no_buffers: 497 > > > > dev.bce.2.com_no_

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-09 Thread Ian FREISLICH
Pyun YongHyeon wrote: > On Tue, Mar 09, 2010 at 03:31:55PM +0200, Ian FREISLICH wrote: > > Can you explain the tunables please - I'm guessing it's these: I think I asked the wrong question. What is a "Quick BD Chain"? What relation should this number have to traffic rate. Is there a maximum and

RE: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-09 Thread David Christensen
> > > patch can fix the RX issue you're suffering from. Anyway, > would you > > > give it try the patch at the following URL? > > > http://people.freebsd.org/~yongari/bce/bce.20100305.diff > > > The patch was generated against CURRENT and you may see a message > > > like "Disabling COAL_NOW time

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-09 Thread Pyun YongHyeon
On Tue, Mar 09, 2010 at 01:31:55PM -0800, David Christensen wrote: > > > > patch can fix the RX issue you're suffering from. Anyway, > > would you > > > > give it try the patch at the following URL? > > > > http://people.freebsd.org/~yongari/bce/bce.20100305.diff > > > > The patch was generated a

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-09 Thread Pyun YongHyeon
On Tue, Mar 09, 2010 at 10:26:29AM +0200, Ian FREISLICH wrote: > Pyun YongHyeon wrote: > > patch can fix the RX issue you're suffering from. Anyway, would you > > give it try the patch at the following URL? > > http://people.freebsd.org/~yongari/bce/bce.20100305.diff > > The patch was generated aga

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-09 Thread Pyun YongHyeon
On Tue, Mar 09, 2010 at 03:31:55PM +0200, Ian FREISLICH wrote: > Pyun YongHyeon wrote: > > On Mon, Mar 08, 2010 at 04:45:20PM +0200, Ian FREISLICH wrote: > > > Pyun YongHyeon wrote: > > > > On Fri, Mar 05, 2010 at 11:16:41PM +0200, Ian FREISLICH wrote: > > > > > Pyun YongHyeon wrote: > > > > > > Th

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-09 Thread Ian FREISLICH
Pyun YongHyeon wrote: > On Mon, Mar 08, 2010 at 04:45:20PM +0200, Ian FREISLICH wrote: > > Pyun YongHyeon wrote: > > > On Fri, Mar 05, 2010 at 11:16:41PM +0200, Ian FREISLICH wrote: > > > > Pyun YongHyeon wrote: > > > > > Thanks for the info. Frankly, I have no idea how to explain the > > > > > iss

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-09 Thread Ian FREISLICH
Pyun YongHyeon wrote: > patch can fix the RX issue you're suffering from. Anyway, would you > give it try the patch at the following URL? > http://people.freebsd.org/~yongari/bce/bce.20100305.diff > The patch was generated against CURRENT and you may see a message > like "Disabling COAL_NOW timedou

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-08 Thread Pyun YongHyeon
On Mon, Mar 08, 2010 at 04:45:20PM +0200, Ian FREISLICH wrote: > Pyun YongHyeon wrote: > > On Fri, Mar 05, 2010 at 11:16:41PM +0200, Ian FREISLICH wrote: > > > Pyun YongHyeon wrote: > > > > Thanks for the info. Frankly, I have no idea how to explain the > > > > issue given that you have no heavy lo

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-08 Thread Ian FREISLICH
Pyun YongHyeon wrote: > On Fri, Mar 05, 2010 at 11:16:41PM +0200, Ian FREISLICH wrote: > > Pyun YongHyeon wrote: > > > Thanks for the info. Frankly, I have no idea how to explain the > > > issue given that you have no heavy load. > > > > How many cores would be involved in handling the traffic and

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-05 Thread Pyun YongHyeon
On Fri, Mar 05, 2010 at 11:16:41PM +0200, Ian FREISLICH wrote: > Pyun YongHyeon wrote: > > Thanks for the info. Frankly, I have no idea how to explain the > > issue given that you have no heavy load. > > How many cores would be involved in handling the traffic and runnig > PF rules on this machine

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-05 Thread Ian FREISLICH
Pyun YongHyeon wrote: > Thanks for the info. Frankly, I have no idea how to explain the > issue given that you have no heavy load. How many cores would be involved in handling the traffic and runnig PF rules on this machine? There are 4x CPU: Quad-Core AMD Opteron(tm) Processor 8354 (2194.51-MHz

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-05 Thread Pyun YongHyeon
On Fri, Mar 05, 2010 at 10:20:18PM +0200, Ian FREISLICH wrote: > Pyun YongHyeon wrote: > > > > Would you show me the output of dmesg(bce(4)/brgphy(4) only) and > > the output of "pciconf -lcbv" for the controller? > > [firewall1.jnb1] ~ # egrep "bce|brgphy" /var/run/dmesg.boot > bce0: mem > 0x

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-05 Thread Ian FREISLICH
Pyun YongHyeon wrote: > > Would you show me the output of dmesg(bce(4)/brgphy(4) only) and > the output of "pciconf -lcbv" for the controller? [firewall1.jnb1] ~ # egrep "bce|brgphy" /var/run/dmesg.boot bce0: mem 0xe600-0xe7ff irq 72 at device 0.0 on pci4 miibus0: on bce0 brgphy0: PH

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-05 Thread Pyun YongHyeon
On Fri, Mar 05, 2010 at 08:16:31PM +0200, Ian FREISLICH wrote: > Pyun YongHyeon wrote: > > On Fri, Mar 05, 2010 at 01:20:57PM +0200, Ian FREISLICH wrote: > > > Hi > > > > > > I have a system that is experiencing mild to severe packet loss. > > > The interfaces are configured as follows: > > > > >

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-05 Thread Ian FREISLICH
Pyun YongHyeon wrote: > On Fri, Mar 05, 2010 at 01:20:57PM +0200, Ian FREISLICH wrote: > > Hi > > > > I have a system that is experiencing mild to severe packet loss. > > The interfaces are configured as follows: > > > > lagg0: bce0, bce1, bce2, bce3 lagproto lacp > > > > lagg0 then is used as

Re: dev.bce.X.com_no_buffers increasing and packet loss

2010-03-05 Thread Pyun YongHyeon
On Fri, Mar 05, 2010 at 01:20:57PM +0200, Ian FREISLICH wrote: > Hi > > I have a system that is experiencing mild to severe packet loss. > The interfaces are configured as follows: > > lagg0: bce0, bce1, bce2, bce3 lagproto lacp > > lagg0 then is used as the hwdev for the vlan interfaces. > >

dev.bce.X.com_no_buffers increasing and packet loss

2010-03-05 Thread Ian FREISLICH
Hi I have a system that is experiencing mild to severe packet loss. The interfaces are configured as follows: lagg0: bce0, bce1, bce2, bce3 lagproto lacp lagg0 then is used as the hwdev for the vlan interfaces. I have pf with a few queues for bandwidth management. There isn't that much traffi