Re: Kernel panic from lagg_ioctl and lagg_port_ioctl

[Lakshmi Narasimhan Sundararajan]

Hi Hans,
Yes, we understand that piece of code.

But Panasas wants this change to MFC to 10.3. This issue is not seen in 11.
Would that happen? I have not seen acknowledgement of this issue or
confirmation of MFC to 10.3. That information will help us plan.

Regards,
LN

On Wed, Jan 13, 2016 at 6:02 PM, Hans Petter Selasky 
wrote:

> On 01/12/16 22:13, Lewis, Fred wrote:
>
>> In earlier versions of lagg_ioctl() (e.g. stable/10/r171247) all of the
>> callout
>> vectors are set to NULL which I think will prevent the problem. Similar
>> NULLing code
>> is also in stable/7. I didn't check other releases.
>>
>
> Don't forget to drain the callouts before NULL-ing them!
>
> --HPS
>

-- 


DISCLAIMER

The information in this e-mail is confidential and may be subject to legal 
privilege. It is intended solely for the addressee. Access to this e-mail 
by anyone else is unauthorized. If you have received this communication in 
error, please address with the subject heading "Received in error," send to 
i...@msystechnologies.com,  then delete the e-mail and destroy any copies of 
it. If you are not the intended recipient, any disclosure, copying, 
distribution or any action taken or omitted to be taken in reliance on it, 
is prohibited and may be unlawful. The views, opinions, conclusions and 
other information expressed in this electronic mail and any attachments are 
not given or endorsed by the company unless otherwise indicated by an 
authorized representative independent of this message.
MSys cannot guarantee that e-mail communications are secure or error-free, 
as information could be intercepted, corrupted, amended, lost, destroyed, 
arrive late or incomplete, or contain viruses, though all reasonable 
precautions have been taken to ensure no viruses are present in this e-mail. 
As our company cannot accept responsibility for any loss or damage arising 
from the use of this e-mail or attachments we recommend that you subject 
these to your virus checking procedures prior to use
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

FreeBSD LAGG port crash

[Lakshmi Narasimhan Sundararajan]

Hi Team,
I am seeing a crash with 10.1 stable which is not seen in 11-CURRENT.
This is with lagg port operations, the below simple sequence of cmds cause
system crash.

ifconfig lagg0 create
ifconfig lagg0 laggproto failover laggport le0
ifconfig lagg0 /* crash here */

See below for the backtraces.
The issue is reproducible in laggproto failover or loadbalance, but not
with lacp.

Is this a known issue?
Is there an MFC to 10.1 STABLE, that is known to fix this issue?
Is anyone looking at this currently?

Any inputs on this would be much appreciated.

Best regards,
LN
== backtrace snip ===

KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
0xfe202122c1e0
kdb_backtrace() at kdb_backtrace+0x39/frame 0xfe202122c290
panic() at panic+0x155/frame 0xfe202122c310
trap_fatal() at trap_fatal+0x38f/frame 0xfe202122c370
trap_pfault() at trap_pfault+0x308/frame 0xfe202122c410
trap() at trap+0x47a/frame 0xfe202122c620
calltrap() at calltrap+0x8/frame 0xfe202122c620
--- trap 0xc, rip = 0x80e15991, rsp = 0xfe202122c6e0, rbp =
0xfe202122c710 ---
lacp_portreq() at lacp_portreq+0x11/frame 0xfe202122c710
lagg_port2req() at lagg_port2req+0x62/frame 0xfe202122c740
lagg_ioctl() at lagg_ioctl+0x2db/frame 0xfe202122c820
ifioctl() at ifioctl+0x162b/frame 0xfe202122c8e0
kern_ioctl() at kern_ioctl+0x255/frame 0xfe202122c950
sys_ioctl() at sys_ioctl+0x13c/frame 0xfe202122c9a0
amd64_syscall() at amd64_syscall+0x351/frame 0xfe202122cab0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfe202122cab0
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8011b5b9a, rsp =
0x7fffd8e8, rbp = 0x7fffe3a0 ---


-- 


DISCLAIMER

The information in this e-mail is confidential and may be subject to legal 
privilege. It is intended solely for the addressee. Access to this e-mail 
by anyone else is unauthorized. If you have received this communication in 
error, please address with the subject heading "Received in error," send to 
i...@msystechnologies.com,  then delete the e-mail and destroy any copies of 
it. If you are not the intended recipient, any disclosure, copying, 
distribution or any action taken or omitted to be taken in reliance on it, 
is prohibited and may be unlawful. The views, opinions, conclusions and 
other information expressed in this electronic mail and any attachments are 
not given or endorsed by the company unless otherwise indicated by an 
authorized representative independent of this message.
MSys cannot guarantee that e-mail communications are secure or error-free, 
as information could be intercepted, corrupted, amended, lost, destroyed, 
arrive late or incomplete, or contain viruses, though all reasonable 
precautions have been taken to ensure no viruses are present in this e-mail. 
As our company cannot accept responsibility for any loss or damage arising 
from the use of this e-mail or attachments we recommend that you subject 
these to your virus checking procedures prior to use
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Fw: D3300: LAG LACP timeout tunable through IOCTL

[Lakshmi Narasimhan Sundararajan]

Hi FreeBSD Team,

We have been working on LACP timeout tunable to help expedite link failure 
propagation through a IOCtl interface.



The changes have been tested and review is in progress. 

The phabricator link is https://reviews.freebsd.org/D3300

We would kindly appreciate if the changes can be integrated to CURRENT at your 
earliest.


Thanks,

LN



MSYS Technologies





From: lakshmi.n_msystechnologies.com (LN)
Sent: ‎Monday‎, ‎August‎ ‎10‎, ‎2015 ‎2‎:‎52‎ ‎PM
To: lakshm...@msystechnologies.com





lakshmi.n_msystechnologies.com marked 2 inline comments as done.
lakshmi.n_msystechnologies.com added a comment.

Warren, I have addressed your comments. Please let me know how I can speed up 
the process to get this integrated to mainline.


REVISION DETAIL
  https://reviews.freebsd.org/D3300

EMAIL PREFERENCES
  https://reviews.freebsd.org/settings/panel/emailpreferences/

To: lakshmi.n_msystechnologies.com, manpages, rpokala-panasas.com
Cc: wblock, imp
___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org

Re: FreeBSD LAG LACP timeout tunable through IOCTL

[Lakshmi Narasimhan Sundararajan]

 LAGG_OPT_LACP_TIMEOUT:
+ case -LAGG_OPT_LACP_TIMEOUT:
   break;
  default:
   err(1, Invalid lagg option);
@@ -293,6 +295,8 @@
  DEF_CMD(-lacp_txtest, -LAGG_OPT_LACP_TXTEST, setlaggsetopt),
  DEF_CMD(lacp_rxtest, LAGG_OPT_LACP_RXTEST, setlaggsetopt),
  DEF_CMD(-lacp_rxtest, -LAGG_OPT_LACP_RXTEST, setlaggsetopt),
+ DEF_CMD(lacp_timeout, LAGG_OPT_LACP_TIMEOUT, setlaggsetopt),
+ DEF_CMD(-lacp_timeout, -LAGG_OPT_LACP_TIMEOUT, setlaggsetopt),
  DEF_CMD_ARG(flowid_shift, setlaggflowidshift),
 };
 static struct afswtch af_lagg = {
Index: sys/net/ieee8023ad_lacp.c
===
--- sys/net/ieee8023ad_lacp.c (revision 286151)
+++ sys/net/ieee8023ad_lacp.c (working copy)
@@ -522,7 +522,7 @@
  int error;
 
  boolean_t active = TRUE; /* XXX should be configurable */
- boolean_t fast = FALSE; /* XXX should be configurable */
+ boolean_t fast = FALSE; /* Configurable via ioctl */ 
 
  link_init_sdl(ifp, (struct sockaddr *)sdl, IFT_ETHER);
  sdl.sdl_alen = ETHER_ADDR_LEN;
Index: sys/net/ieee8023ad_lacp.h
===
--- sys/net/ieee8023ad_lacp.h (revision 286151)
+++ sys/net/ieee8023ad_lacp.h (working copy)
@@ -251,6 +251,7 @@
   u_int32_t lsc_tx_test;
  } lsc_debug;
  u_int32_t  lsc_strict_mode;
+ boolean_t  lsc_fast_timeout; /* if set, fast timeout */
 };
 
 #define LACP_TYPE_ACTORINFO 1
Index: sys/net/if_lagg.c
===
--- sys/net/if_lagg.c (revision 286151)
+++ sys/net/if_lagg.c (working copy)
@@ -1257,6 +1257,8 @@
 ro-ro_opts |= LAGG_OPT_LACP_RXTEST;
if (lsc-lsc_strict_mode != 0)
 ro-ro_opts |= LAGG_OPT_LACP_STRICT;
+   if (lsc-lsc_fast_timeout != 0)
+ro-ro_opts |= LAGG_OPT_LACP_TIMEOUT;
 
ro-ro_active = sc-sc_active;
   } else {
@@ -1292,6 +1294,8 @@
   case -LAGG_OPT_LACP_RXTEST:
   case LAGG_OPT_LACP_STRICT:
   case -LAGG_OPT_LACP_STRICT:
+  case LAGG_OPT_LACP_TIMEOUT:
+  case -LAGG_OPT_LACP_TIMEOUT:
valid = lacp = 1;
break;
   default:
@@ -1320,6 +1324,7 @@
 sc-sc_opts = ~ro-ro_opts;
   } else {
struct lacp_softc *lsc;
+   struct lacp_port *lp;
 
lsc = (struct lacp_softc *)sc-sc_psc;
 
@@ -1342,6 +1347,20 @@
case -LAGG_OPT_LACP_STRICT:
 lsc-lsc_strict_mode = 0;
 break;
+   case LAGG_OPT_LACP_TIMEOUT:
+LACP_LOCK(lsc);
+   LIST_FOREACH(lp, lsc-lsc_ports, lp_next)
+  lp-lp_state |= LACP_STATE_TIMEOUT;
+LACP_UNLOCK(lsc);
+lsc-lsc_fast_timeout = 1;
+break;
+   case -LAGG_OPT_LACP_TIMEOUT:
+LACP_LOCK(lsc);
+   LIST_FOREACH(lp, lsc-lsc_ports, lp_next)
+  lp-lp_state = ~LACP_STATE_TIMEOUT;
+LACP_UNLOCK(lsc);
+lsc-lsc_fast_timeout = 0;
+break;
}
   }
   LAGG_WUNLOCK(sc);
Index: sys/net/if_lagg.h
===
--- sys/net/if_lagg.h (revision 286151)
+++ sys/net/if_lagg.h (working copy)
@@ -150,6 +150,7 @@
 #define LAGG_OPT_LACP_STRICT  0x10  /* LACP strict mode */
 #define LAGG_OPT_LACP_TXTEST  0x20  /* LACP debug: txtest */
 #define LAGG_OPT_LACP_RXTEST  0x40  /* LACP debug: rxtest */
+#define LAGG_OPT_LACP_TIMEOUT  0x80  /* LACP timeout */
  u_int   ro_count;  /* number of ports */
  u_int   ro_active;  /* active port count */
  u_int   ro_flapping;  /* number of flapping */





Thanks,

LN






MSYS Technologies





From: Pokala, Ravi
Sent: ‎Saturday‎, ‎July‎ ‎25‎, ‎2015 ‎12‎:‎53‎ ‎AM
To: Sundararajan, Lakshmi, freebsd-net@freebsd.org
Cc: panasas-netw...@msystechnologies.com, Lewis, Fred, 'Tallam, Sreen'





Hi LN,

You also need to teach `ifconfig' how to toggle this new setting. See

sbin/ifconfig/iflagg.c:lagg_cmds[]

and how the other LACP options are handled. (Thanks to Genesys on #bsdcode
for pointing that out.)

Also, please confirm that you don't need to do any locking to walk the
list or modify any of the list elements.

Thanks,

Ravi

-Original Message-
From: Lakshmi Narasimhan Sundararajan lakshm...@msystechnologies.com
Date: 2015-07-23, Thursday at 05:25
To: freebsd-net@freebsd.org freebsd-net@freebsd.org
Cc: panasas-netw...@msystechnologies.com
panasas-netw...@msystechnologies.com, Lewis, Fred
fle...@panasas.com, Ravi Pokala rpok...@panasas.com, Tallam, Sreen
sr...@panasas.com
Subject: FreeBSD LAG LACP timeout tunable through IOCTL

Hi FreeBSD team,
In FreeBSD-10 and in Current, by default LACP supports only long timeout.
FreeBSD does not provide the way to configure LACP timeout period.
We made code changes for LACP Fast-timeout (Using IOCTL, both GET / SET)
on FreeBSD-11.

And we were able to successfully test the operation using IOCtl calls
from userland.


Initially we wanted to use sysctl, but found in FreeBSD revision history,
that sysctl in LAG results in LOR and has to be converted to IOCTL.
Please let us know your comments to take this forward.




Diffs inline:
Index: sys/net/ieee8023ad_lacp.h

FreeBSD LAG LACP timeout tunable through IOCTL

[Lakshmi Narasimhan Sundararajan]

Hi FreeBSD team,

In FreeBSD-10 and in Current, by default LACP supports only long timeout. 

FreeBSD does not provide the way to configure LACP timeout period.
We made code changes for LACP Fast-timeout (Using IOCTL, both GET / SET) on 
FreeBSD-11. 

And we were able to successfully test the operation using IOCtl calls from 
userland.


Initially we wanted to use sysctl, but found in FreeBSD revision history, that 
sysctl in LAG results in LOR and has to be converted to IOCTL.  Please let us 
know your comments to take this forward.



Diffs inline:

Index: sys/net/ieee8023ad_lacp.h
===
--- sys/net/ieee8023ad_lacp.h (revision 285195)
+++ sys/net/ieee8023ad_lacp.h (working copy)
@@ -251,6 +251,7 @@
   u_int32_t lsc_tx_test;
  } lsc_debug;
  u_int32_t  lsc_strict_mode;
+ u_int32_t  lsc_fast_timeout; /* if set, fast / short timeout */
 };
 
 #define LACP_TYPE_ACTORINFO 1
Index: sys/net/if_lagg.c
===
--- sys/net/if_lagg.c (revision 285195)
+++ sys/net/if_lagg.c (working copy)
@@ -1257,6 +1257,8 @@
 ro-ro_opts |= LAGG_OPT_LACP_RXTEST;
if (lsc-lsc_strict_mode != 0)
 ro-ro_opts |= LAGG_OPT_LACP_STRICT;
+   if (lsc-lsc_fast_timeout != 0)
+ro-ro_opts |= LAGG_OPT_LACP_TIMEOUT;
 
ro-ro_active = sc-sc_active;
   } else {
@@ -1292,6 +1294,8 @@
   case -LAGG_OPT_LACP_RXTEST:
   case LAGG_OPT_LACP_STRICT:
   case -LAGG_OPT_LACP_STRICT:
+  case LAGG_OPT_LACP_TIMEOUT:
+  case -LAGG_OPT_LACP_TIMEOUT:
valid = lacp = 1;
break;
   default:
@@ -1320,6 +1324,7 @@
 sc-sc_opts = ~ro-ro_opts;
   } else {
struct lacp_softc *lsc;
+   struct lacp_port *lp;
 
lsc = (struct lacp_softc *)sc-sc_psc;
 
@@ -1342,6 +1347,16 @@
case -LAGG_OPT_LACP_STRICT:
 lsc-lsc_strict_mode = 0;
 break;
+   case LAGG_OPT_LACP_TIMEOUT:
+   LIST_FOREACH(lp, lsc-lsc_ports, lp_next)
+  lp-lp_state |= LACP_STATE_TIMEOUT;
+lsc-lsc_fast_timeout = 1;
+break;
+   case -LAGG_OPT_LACP_TIMEOUT:
+   LIST_FOREACH(lp, lsc-lsc_ports, lp_next)
+  lp-lp_state = ~LACP_STATE_TIMEOUT;
+lsc-lsc_fast_timeout = 0;
+break;
}
   }
   LAGG_WUNLOCK(sc);
Index: sys/net/if_lagg.h
===
--- sys/net/if_lagg.h (revision 285195)
+++ sys/net/if_lagg.h (working copy)
@@ -150,6 +150,7 @@
 #define LAGG_OPT_LACP_STRICT  0x10  /* LACP strict mode */
 #define LAGG_OPT_LACP_TXTEST  0x20  /* LACP debug: txtest */
 #define LAGG_OPT_LACP_RXTEST  0x40  /* LACP debug: rxtest */
+#define LAGG_OPT_LACP_TIMEOUT  0x80  /* LACP Fast timeout */
  u_int   ro_count;  /* number of ports */
  u_int   ro_active;  /* active port count */
  u_int   ro_flapping;  /* number of flapping */



Thanks,

LN



MSYS Technologies
___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org

Reg Intel Fortville IXL driver on 11-CURRENT

[Lakshmi Narasimhan Sundararajan]

Hi FreeBSD folks,
I am part of Panasas and working on evaluating IXL over FreeBSD
[11-CURRENT] on Intel Taylor Pass platform for our next product.

In that regard, while evaluating performance, we found the Tx performance
to be very poor. We found it to be so because the tx interrupts are spread
over all the CPUs even if the traffic handling process is pinned at a
particular cpu.

And we narrowed it down to the below lines of code in the tx path within
the IXL driver. It seems to me that the logic for finding whether tx queue
is stalled might be incorrect or too aggressive. Either ways, removing the
below lines makes the tx interrupt being handled on the same cpu on which
the request was raised and the performance is very good as expected.

Filename: sys/dev/ixl/ixl_txrx.c

[lakshmis@mau-bsd-10a ~/fortville/hol/sys/dev/ixl]$ diff -c5pt ixl_txrx.c
ixl_txrx.c.mod
*** ixl_txrx.c Fri Jun 12 06:56:51 2015
--- ixl_txrx.c.mod Fri Jun 12 06:56:33 2015
*** ixl_mq_start(struct ifnet *ifp, struct m
*** 96,112 
--- 96,115 
  } else
  #endif
  i = m-m_pkthdr.flowid % vsi-num_queues;
  } else
  i = curcpu % vsi-num_queues;
+
+ #if 0
  /*
  ** This may not be perfect, but until something
  ** better comes along it will keep from scheduling
  ** on stalled queues.
  */
  if (((1  i)  vsi-active_queues) == 0)
  i = ffsl(vsi-active_queues);
+ #endif

  que = vsi-queues[i];
  txr = que-txr;

  err = drbr_enqueue(ifp, txr-br, m);
[lakshmis@mau-bsd-10a ~/fortville/hol/sys/dev/ixl]$


Would appreciate your feedback on the same.

Thanks,
LN
___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org

Re: Performance issues with Intel Fortville (XL710/ixl(4))

[Lakshmi Narasimhan Sundararajan]

Hi FreeBSD Team!

We seem to have found a problem to Tx performance. 

We found that the tx handling is spread on all CPUs causing probably cache 
trashing resulting in poor performance. 

But once we used cpuset to bind interrupt thread and iperf process to the same 
CPU, performance was close to line rate. I used userland cpuset command to 
perform this manually. I want this constrained in the kernel config/code 
through some tunables, and I am seeking your help/pointers in that regard.


My followup questions are as follows.

a) How are Tx interrupts steered from the NIC to the CPU on the transmit path? 
Would tx_complete# interrupt for packets transmitted from CPU#x, be serviced on 
the same CPU? If not, how to get this binding done?


b) I would like to use a pool of CPUs dedicated to service NIC interrupts. 
Especially on the transmit path, I would want the tx_interrupts to be handled 
on the same CPU on which request was submitted. How to get this done?


I played with the current ISR setting but I did not see any difference in how 
Interrupts are scheduled across CPU. The max interrupt threads even though set 
to one, the interrupt threads are scheduled on any CPU. Even if I set 
bindthreads to ‘1’. There is no difference in interrupt thread scheduling.


root@mau-da-27-4-1:~ # sysctl net.isr
net.isr.dispatch: direct
net.isr.maxthreads: 1
net.isr.bindthreads: 0
net.isr.maxqlimit: 10240
net.isr.defaultqlimit: 256
net.isr.maxprot: 16
net.isr.numthreads: 1


I would sincerely appreciate if you can provide some pointers on these items 
above.




Thanks

LN







From: Pokala, Ravi
Sent: ‎Wednesday‎, ‎May‎ ‎20‎, ‎2015 ‎3‎:‎34‎ ‎AM
To: freebsd-net@freebsd.org, j...@freebsd.org, e...@freebsd.org
Cc: freebsd-hack...@freebsd.org, Lewis, Fred, Sundararajan, Lakshmi





Hi folks,

At Panasas, we are working with the Intel XL710 40G NIC (aka Fortville),
and we're seeing some performance issues w/ 11-CURRENT (r282653).

Motherboard: Intel S2600KP (aka Kennedy Pass)
CPU: E5-2660 v3 @ 2.6GHz (aka Haswell Xeon)
(1 socket x 10 physical cores x 2 SMT threads) = 20 logical cores
NIC: Intel XL710, 2x40Gbps QSFP, configured in 4x10Gbps mode
RAM: 4x 16GB DDR4 DIMMs

What we've seen so far:

  - TX performance is pretty consistently lower than RX performance. All
numbers below are for unidrectional tests using `iperf':
10Gbps linksthreads/linkTX Gbps RX Gbps TX/RX
1   1   9.029.8591.57%
1   8   8.499.9185.67%
1   16  7.009.9170.63%
1   32  6.689.9267.40%

  - With multiple active links, both TX and RX performance suffer greatly;
the aggregate bandwidth tops out at about a third of the theoretical
40Gbps implied by 4x 10Gbps.
10Gbps linksthreads/linkTX Gbps RX Gbps % of 40Gbps
4   1   13.39   13.38   33.4%

  - Multi-link bidirectional throughput is absolutely terrible; the
aggregate is less than a tenth of the theoretical 40Gbps.
10Gbps linksthreads/linkTX Gbps RX Gbps % of 40Gbps
4   1   3.832.969.6% / 7.4%

  - Occasional interrupt storm messages are seen from the IRQs associated
with the NICs. Since that can impact performance, those runs were not
included in the data listed above.

Our questions:

  - How stable is ixl(4) in -CURRENT? By that, we mean both how quickly is
the driver changing, and does the driver cause any system instability?

  - What type of performance have others been getting w/ Fortville? In
40Gbps mode? In 4x10Gbps mode?

  - Does anyone have any tuning parameters they can recommend for this
card?

  - We did our testing w/ 11-CURRENT, but we will initially ship Fortville
running on 10.1-RELEASE or 10.2-RELEASE. The presence of RSS - even though
it is disabled by default - makes the driver back-port non-trivial. Is
there an estimate on when the 11-CURRENT version of the driver (1.4.1)
will get MFCed to 10-STABLE?

My colleagues Lakshmi and Fred (CCed) are working on this; please make
sure to include them if you have any comments.

Thanks,

Ravi
___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org