RE: Kernel 4.6.7-rt13: Intel Ethernet driver igb causes huge latencies in cyclictest

2016-10-18 Thread Koehrer Mathias (ETAS/ESW5)
Hi all, > > >> > > >> Can you continue your bisection using 'git bisect'? You've already > > >> narrowed it down between 4.0 and 4.1, so you're well on your way. > > >> > > > > > > OK - done. > > > And finally I was successful! > > > The following git commit is the one that is causing the

Re: Kernel 4.6.7-rt13: Intel Ethernet driver igb causes huge latencies in cyclictest

2016-10-17 Thread Richard Cochran
On Mon, Oct 17, 2016 at 01:36:45PM -0500, Julia Cartwright wrote: > While that is certainly the case, and would explain the most egregious > of measured latency spikes, it doesn't invalidate the test if you > consider the valuable data point(s) to be the minimum and/or median > latencies. Well,

Re: Kernel 4.6.7-rt13: Intel Ethernet driver igb causes huge latencies in cyclictest

2016-10-17 Thread Julia Cartwright
On Sat, Oct 15, 2016 at 12:06:33AM +0200, Richard Cochran wrote: > On Fri, Oct 14, 2016 at 08:58:22AM +, Koehrer Mathias (ETAS/ESW5) wrote: > > @@ -753,7 +756,9 @@ u32 igb_rd32(struct e1000_hw *hw, u32 re > > if (E1000_REMOVED(hw_addr)) > > return ~value; > > > > +

Re: Kernel 4.6.7-rt13: Intel Ethernet driver igb causes huge latencies in cyclictest

2016-10-17 Thread Julia Cartwright
+linux-pci On Mon, Oct 17, 2016 at 08:39:40AM -0700, Alexander Duyck wrote: > On Mon, Oct 17, 2016 at 8:00 AM, Koehrer Mathias (ETAS/ESW5) > wrote: > > Hi Julia! > >> > > Have you tested on a vanilla (non-RT) kernel? I doubt there is > >> > > anything RT specific about

Re: [Intel-wired-lan] Kernel 4.6.7-rt13: Intel Ethernet driver igb causes huge latencies in cyclictest

2016-10-17 Thread Alexander Duyck
On Mon, Oct 17, 2016 at 8:00 AM, Koehrer Mathias (ETAS/ESW5) wrote: > Hi Julia! >> > > Have you tested on a vanilla (non-RT) kernel? I doubt there is >> > > anything RT specific about what you are seeing, but it might be nice >> > > to get confirmation. Also, bisection

RE: Kernel 4.6.7-rt13: Intel Ethernet driver igb causes huge latencies in cyclictest

2016-10-17 Thread Koehrer Mathias (ETAS/ESW5)
Hi Julia! > > > Have you tested on a vanilla (non-RT) kernel? I doubt there is > > > anything RT specific about what you are seeing, but it might be nice > > > to get confirmation. Also, bisection would probably be easier if you > > > confirm on a > vanilla kernel. > > > > > > I find it

Re: Kernel 4.6.7-rt13: Intel Ethernet driver igb causes huge latencies in cyclictest

2016-10-14 Thread Richard Cochran
On Fri, Oct 14, 2016 at 08:58:22AM +, Koehrer Mathias (ETAS/ESW5) wrote: > @@ -753,7 +756,9 @@ u32 igb_rd32(struct e1000_hw *hw, u32 re > if (E1000_REMOVED(hw_addr)) > return ~value; > > +trace_igb(801); > value = readl(_addr[reg]); > +

Re: Kernel 4.6.7-rt13: Intel Ethernet driver igb causes huge latencies in cyclictest

2016-10-14 Thread Julia Cartwright
On Fri, Oct 14, 2016 at 08:58:22AM +, Koehrer Mathias (ETAS/ESW5) wrote: > Hi Julia, > > > Have you tested on a vanilla (non-RT) kernel? I doubt there is anything RT > > specific > > about what you are seeing, but it might be nice to get confirmation. Also, > > bisection > > would probably

RE: Kernel 4.6.7-rt13: Intel Ethernet driver igb causes huge latencies in cyclictest

2016-10-14 Thread Koehrer Mathias (ETAS/ESW5)
Hi Julia, > Have you tested on a vanilla (non-RT) kernel? I doubt there is anything RT > specific > about what you are seeing, but it might be nice to get confirmation. Also, > bisection > would probably be easier if you confirm on a vanilla kernel. > > I find it unlikely that it's a kernel

Re: Kernel 4.6.7-rt13: Intel Ethernet driver igb causes huge latencies in cyclictest

2016-10-13 Thread Julia Cartwright
Hey Mathias- On Thu, Oct 13, 2016 at 10:57:18AM +, Koehrer Mathias (ETAS/ESW5) wrote: [..] Interesting indeed! > > Here are some places where I added traces: > > In file igb_ptp.c: > > void igb_ptp_rx_hang(struct igb_adapter *adapter) { > > struct e1000_hw *hw = >hw; > >

RE: Kernel 4.6.7-rt13: Intel Ethernet driver igb causes huge latencies in cyclictest

2016-10-13 Thread David Laight
From: Koehrer Mathias > Sent: 13 October 2016 11:57 .. > > The time between my trace points 700 and 701 is about 30us, the time > > between my > > trace points 600 and 601 is even 37us!! > > The code in between is > > tsyncrxctl = rd32(E1000_TSYNCRXCTL); resp. > > lvmmc =

RE: Kernel 4.6.7-rt13: Intel Ethernet driver igb causes huge latencies in cyclictest

2016-10-13 Thread Koehrer Mathias (ETAS/ESW5)
Hi all! > > Hi Julia, > > thanks for the detailed analysis! > > > > [...] > > Okay, we finally received our wakeup event. We were expecting to be > > woken up at 10024735653388ns, but were actually woken up at > 10024735682387ns. > > > > 10024735682387 - 10024735653388 = 28999ns > > > > Our

Re: Kernel 4.6.7-rt13: Intel Ethernet driver igb causes huge latencies in cyclictest

2016-10-13 Thread Koehrer Mathias (ETAS/ESW5)
Hi Julia, thanks for the detailed analysis! > > [...] > Okay, we finally received our wakeup event. We were expecting to be woken up > at > 10024735653388ns, but were actually woken up at 10024735682387ns. > > 10024735682387 - 10024735653388 = 28999ns > > Our timer fired ~29us late! But

Re: Kernel 4.6.7-rt13: Intel Ethernet driver igb causes huge latencies in cyclictest

2016-10-10 Thread Julia Cartwright
Hello Mathias- On Fri, Oct 07, 2016 at 08:58:08AM +, Koehrer Mathias (ETAS/ESW5) wrote: [..] > I modified the in-kernel's igb_main.c (function igb_watchdog_task) to comment > out > the access to the EICS registers: > > --- igb_main.c.orig 2016-10-07 10:43:37.855873754 +0200 > +++

Re: Kernel 4.6.7-rt13: Intel Ethernet driver igb causes huge latencies in cyclictest

2016-10-07 Thread Koehrer Mathias (ETAS/ESW5)
Hi Mitch, > > > > > > Although, to be clear, it isn't the fact that there exists 8 > > > threads, it's > > that the device is > > > firing all 8 interrupts at the same time. The time spent in hardirq > > context just waking > > > up all 8 of those threads (and the cyclictest wakeup) is enough to

RE: Kernel 4.6.7-rt13: Intel Ethernet driver igb causes huge latencies in cyclictest

2016-10-06 Thread Williams, Mitch A
.@intel.com>; Greg <gvrose8...@gmail.com> > Cc: netdev@vger.kernel.org; intel-wired-...@lists.osuosl.org; linux-rt- > us...@vger.kernel.org; Sebastian Andrzej Siewior > <sebastian.siew...@linutronix.de> > Subject: Re: [Intel-wired-lan] Kernel 4.6.7-rt13: Intel Ethernet dr

Re: Kernel 4.6.7-rt13: Intel Ethernet driver igb causes huge latencies in cyclictest

2016-10-06 Thread Henri Roosen
On 10/06/2016 09:01 AM, Koehrer Mathias (ETAS/ESW5) wrote: Hi all, Hi Mathias, Although, to be clear, it isn't the fact that there exists 8 threads, it's that the device is firing all 8 interrupts at the same time. The time spent in hardirq context just waking up all 8 of those threads

Re: Kernel 4.6.7-rt13: Intel Ethernet driver igb causes huge latencies in cyclictest

2016-10-06 Thread Koehrer Mathias (ETAS/ESW5)
Hi all, > > Although, to be clear, it isn't the fact that there exists 8 threads, it's > that the device is > firing all 8 interrupts at the same time. The time spent in hardirq context > just waking > up all 8 of those threads (and the cyclictest wakeup) is enough to cause your > regression.

Re: Kernel 4.6.7-rt13: Intel Ethernet driver igb causes huge latencies in cyclictest

2016-10-05 Thread Julia Cartwright
On Wed, Oct 05, 2016 at 07:02:21AM +, Koehrer Mathias (ETAS/ESW5) wrote: > Hi Julia, > > > > In the meanwhile I have detected another finding which might be relevant: > > > > > > With the 3.18 kernel the igb driver comes with two interrupts per > > > NIC (e.g. eth2 and eth2-TxRx0) with the