Am 11.05.16 um 14:50 schrieb Stephan Budach:
Am 11.05.16 um 13:36 schrieb Stephan Budach:
Am 09.05.16 um 20:43 schrieb Dale Ghent:
On May 9, 2016, at 2:04 PM, Stephan Budach <[email protected]>
wrote:
Am 09.05.16 um 16:33 schrieb Dale Ghent:
On May 9, 2016, at 8:24 AM, Stephan Budach
<[email protected]> wrote:
Hi,
I have a strange behaviour where OmniOS omnios-r151018-ae3141d
will break the LACP aggr-link on different boxes, when Intel
X540-T2s are involved. It first starts with a couple if link
downs/ups on one port and finally the link on that port
negiotates to 1GbE instead of 10GbE, which then breaks the LACP
channel on my Cisco Nexus for this connection.
I have tried swapping and interchangeing cables and thus
switchports, but to no avail.
Anyone else noticed this and even better… knows a solution to this?
Was this an issue noticed only with r151018 and not with previous
versions, or have you only tried this with 018?
By your description, I presume that the two ixgbe physical links
will stay at 10Gb and not bounce down to 1Gb if not LACP'd together?
/dale
I have noticed that on prior versions of OmniOS as well, but we
only recently started deploying 10GbE LACP bonds, when we
introduced our Nexus gear to our network. I will have to check if
both links stay at 10GbE, when not being configured as a LACP bond.
Let me check that tomorrow and report back. As we're heading for a
streched DC, we are mainly configuring 2-way LACP bonds over our
Nexus gear, so we don't actually have any single 10GbE connection,
as they will all have to be conencted to both DCs. This is achieved
by using VPCs on our Nexus switches.
Provide as much detail as you can - if you're using hw flow control,
whether both links act this way at the same time or independently,
and so-on. Problems like this often boil down to a very small and
seemingly insignificant detail.
I currently have ixgbe on the operating table for adding X550
support, so I can take a look at this; however I don't have your
type of switches available to me so LACP-specific testing is
something I can't do for you.
/dale
I checked the ixgbe.conf files on each host and they all are still at
the standard setting, which includes flow_control = 3;
So they all have flow control enabled. As for the Nexus config, all
of those ports are still on standard ethernet ports and modifications
have only been made globally to the switch.
I will now have to yank the one port on one of the hosts from the
aggr and configure it as a standalone port. Then we will see, if it
still receives the disconnects/reconnects and finally the negotiation
to 1GbE instead of 10GbE. As this only seems to happen to the same
port I never experienced other ports of the affected aggrs acting up.
I also thought to notice, that those were always the "same" physical
ports, that is the first port on the card (ixgbe0), but that might of
course be a coincidence.
Thanks,
Stephan
Ok, so we can likely rule out LACP as a generic reason for this issue…
After removing ixgbe0 from the aggr1, I plugged it into an unused port
of my Nexus FEX and low and behold, here we go:
root@tr1206902:/root# tail -f /var/adm/messages
May 11 14:37:17 tr1206902 mac: [ID 435574 kern.info] NOTICE: ixgbe0
link up, 1000 Mbps, full duplex
May 11 14:38:35 tr1206902 mac: [ID 486395 kern.info] NOTICE: ixgbe0
link down
May 11 14:38:48 tr1206902 mac: [ID 435574 kern.info] NOTICE: ixgbe0
link up, 10000 Mbps, full duplex
May 11 15:24:55 tr1206902 mac: [ID 486395 kern.info] NOTICE: ixgbe0
link down
May 11 15:25:10 tr1206902 mac: [ID 435574 kern.info] NOTICE: ixgbe0
link up, 10000 Mbps, full duplex
So, after less than an hour, we had the first link-cycle on ixgbe0,
alas on another port, which has no LACP config whatsoever. I will
monitor this for a while and see, if we will get more of those.
Thanks,
Stephan
Ehh… and sorry, I almost forgot to paste the log from the Cisco Nexus
switch:
2016 May 11 13:21:22 gh79-nx-01 %ETHPORT-5-SPEED: Interface
Ethernet141/1/9, operational speed changed to 10 Gbps
2016 May 11 13:21:22 gh79-nx-01 %ETHPORT-5-IF_DUPLEX: Interface
Ethernet141/1/9, operational duplex mode changed to Full
2016 May 11 13:21:22 gh79-nx-01 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface
Ethernet141/1/9, operational Receive Flow Control state changed to off
2016 May 11 13:21:22 gh79-nx-01 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface
Ethernet141/1/9, operational Transmit Flow Control state changed to on
2016 May 11 13:21:22 gh79-nx-01 %ETHPORT-5-IF_UP: Interface
Ethernet141/1/9 is up in mode access
2016 May 11 14:07:29 gh79-nx-01 %ETHPORT-5-IF_DOWN_LINK_FAILURE:
Interface Ethernet141/1/9 is down (Link failure)
2016 May 11 14:07:45 gh79-nx-01 last message repeated 1 time
2016 May 11 14:07:45 gh79-nx-01 %ETHPORT-5-SPEED: Interface
Ethernet141/1/9, operational speed changed to 10 Gbps
2016 May 11 14:07:45 gh79-nx-01 %ETHPORT-5-IF_DUPLEX: Interface
Ethernet141/1/9, operational duplex mode changed to Full
2016 May 11 14:07:45 gh79-nx-01 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface
Ethernet141/1/9, operational Receive Flow Control state changed to off
2016 May 11 14:07:45 gh79-nx-01 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface
Ethernet141/1/9, operational Transmit Flow Control state changed to on
2016 May 11 14:07:45 gh79-nx-01 %ETHPORT-5-IF_UP: Interface
Ethernet141/1/9 is up in mode access
Despite the clock, which seems a bit off, you can see, that this looks
to the switch as if the cable had been simply pulled and then re-plugged.
Cheers,
Stephan
_______________________________________________
OmniOS-discuss mailing list
[email protected]
http://lists.omniti.com/mailman/listinfo/omnios-discuss
