Hi Robert,
Per instructions, I took out every single reference to “ixgbe0” from my
/usbkey/config file. I then rebooted the system. Next, I issued the nictagadm
command:
- SmartOS (build: 20170720T001051Z)
[root@smartos ~]# cd /usbkey
[root@smartos /usbkey]# grep ixgbe0 config
[root@smartos /usbkey]# nictagadm add -p mtu=9000 ixgbe0 0:1b:21:bc:51:7a
MTU changes will not take effect until next reboot
[root@smartos /usbkey]# grep ixgbe0 config
ixgbe0_nic=00:1b:21:bc:51:7a
ixgbe0_mtu=9000
[root@smartos /usbkey]# reboot
Upon rebooting, I had the very same issue, with the same error being displayed
in the log file about the MTU. Once I commented out the MTU setting in
/usbkey/config, I was able to boot the system normally, but at the default MTU
of 1500.
Could this have anything to do with the fact that this is an OEM X520 instead
of Intel-branded? It doesn’t matter for Linux, but could it make a difference
for SmartOS? Here’s the output from lspci on Linux for this device:
jcroix@ubuntu-server:~$ lspci -nn -vvv | grep Ethernet
00:19.0 Ethernet controller [0200]: Intel Corporation 82579V Gigabit Network
Connection [8086:1503] (rev 06)
02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit
SFI/SFP+ Network Connection [8086:10fb] (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-1 [8086:000a]
02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit
SFI/SFP+ Network Connection [8086:10fb] (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-1 [8086:000a]
Notice that the subsystem identifier is [8086:000a], which is NOT the same
subsystem ID for Intel-branded boards. Dell uses one subsystem ID, Small Tree
another, etc. It’s still an Intel X520, and it runs perfectly as such under
Linux using the Intel driver. And, of course, it works fine under SmartOS at a
MTU of 1500. Thought I’d mention it, “just in case”.
Thanks,
John
> On Aug 18, 2017, at 10:37 AM, Robert Mustacchi <r...@joyent.com> wrote:
>
> On 8/17/17 8:41 , John Croix wrote:
>> I’ve tried to debug a little further into why the Intel X520 MTU can’t be
>> set by comparing 2 log files (/var/svc/log/network-physical:default.log)
>> from separate boots as I enabled and disabled my two 10GbE NICs. I have a
>> log file where my Myricom 10GbE adapter MTU is set to 9000 and where the
>> X520 fails to set the MTU. The Myricom has the “feature” that the MTU can
>> only be set once at the start of a boot. Here are the two relevant excerpts
>> from the log files. All lines prior in both log files are more or less
>> identical (differences being MAC addresses and NIC tag names).
>>
>> X520:
>> + link=ixgbe0
>> + [[ -z ixgbe0 ]]
>> + /usr/sbin/dladm show-linkprop -c -o value -p mtu ixgbe0
>> + curmtu=1500
>> + [[ 0 -eq 0 ]]
>> + [[ 1500 -eq 9000 ]]
>> + /usr/sbin/dladm set-linkprop -p mtu=9000 ixgbe0
>> /usr/sbin/dladm: warning: cannot set link property 'mtu' on 'ixgbe0': link
>> busy
>> + echo 'Failed to set mtu to 9000 for link ixgbe0'
>> Failed to set mtu to 9000 for link ixgbe0
>> + exit 95
>>
>> Myricom:
>> + link=myri10ge0
>> + [[ -z myri10ge0 ]]
>> + /usr/sbin/dladm show-linkprop -c -o value -p mtu myri10ge0
>> + curmtu=1500
>> + [[ 0 -eq 0 ]]
>> + [[ 1500 -eq 9000 ]]
>> + /usr/sbin/dladm set-linkprop -p mtu=9000 myri10ge0
>> + [[ true == true ]]
>>
>> Unfortunately I don’t know enough about Illumos internals to go much
>> further. Any suggestions for further experiments?
>
> Hi John,
>
> I'd probably start by taking a step back. I think that the problem here
> is likely the way that you're using the config file. What I'd do first
> is delete everything related to the ixgbe instance that you have in the
> config file. Once that comes up, I would then use nictagadm to create a
> nictag with an MTU of 9k. I would then reboot and verify that the mtu is 9k.
>
> We generally don't try to bring up the physical interface in these
> config files with the exception of the admin network, which doesn't
> support a larger mtu. Most of the GZ device tuning in SmartOS (which is
> different from normal illumos due to the live image) is driven through
> setting the nic tags up.
>
> The reason that you see the EBUSY when using dladm set-linkprop is for
> the same reason that it was seen with the myri10ge devices. lldpd has
> already started using the interface and the driver doesn't support
> changing things once it's in use.
>
> Robert
>
> Robert
>
>>> On Aug 15, 2017, at 3:06 PM, John Croix <jcr...@mac.com
>>> <mailto:jcr...@mac.com>> wrote:
>>>
>>> In my quest for better connectivity between my Mac and my iSCSI volume on
>>> my SmartOS server, I’ve picked up an Intel X520-SR2 10GbE NIC and put it
>>> into my server. The system recognized the card without an issue, and I was
>>> able to use nictagadm to add the NIC to the system. The problem is that it
>>> won’t accept a MTU parameter of 9000. I had to comment the MTU line out of
>>> /usbkey/config in order to get the server to fully boot.
>>>
>>> Here are the relevant lines in the /usbkey/config file:
>>> ixgbe0_nic=0:1b:21:bc:51:7a
>>> ixgbe0_ip=192.168.2.2
>>> ixgbe0_netmask=255.255.255.0
>>> ixgbe0_mtu=9000
>>>
>>> From “svcs -x”, I see this:
>>> svc:/network/physical:default (physical network interfaces)
>>> State: maintenance since August 15, 2017 at 07:52:33 PM UTC
>>> Reason: Start method exited with $SMF_EXIT_ERR_FATAL.
>>> See: http://illumos.org/msg/SMF-8000-KS
>>> <http://illumos.org/msg/SMF-8000-KS>
>>> See: ifconfig(1M)
>>> See: /var/svc/log/network-physical:default.log
>>> Impact: 22 dependent services are not running. (Use -v for list.)
>>>
>>> Here’s the log file content, with the error at the end:
>>> [ Aug 15 19:52:31 Executing start method ("/lib/svc/method/net-physical"). ]
>>> [ Aug 15 19:52:31 Timeout override by svc.startd. Using infinite timeout. ]
>>> + smf_configure_ip
>>> + /sbin/zonename -t
>>> + [ global = global -o shared = exclusive ]
>>> + return 0
>>> + LD_LIBRARY_PATH=/lib
>>> + export LD_LIBRARY_PATH
>>> + ADMIN_DHCP_TIMEOUT=300
>>> + ActiveAggrLinks=''
>>> + typeset -A ActiveAggrLinks
>>> + smf_netstrategy
>>> + smf_is_nonglobalzone
>>> + [ global != global ]
>>> + return 1
>>> + /sbin/netstrategy
>>> + set -- ufs none none
>>> + [ 0 -eq 0 ]
>>> + [ ufs = nfs ]
>>> + _INIT_NET_STRATEGY=none
>>> + export _INIT_NET_STRATEGY
>>> + typeset -A plumbedifs
>>> + smf_is_globalzone
>>> + [ global = global ]
>>> + return 0
>>> + /usr/sbin/dladm init-phys
>>> + log_if_state before
>>> + echo '== debug start: before =='
>>> == debug start: before ==
>>> + /usr/sbin/dladm show-phys
>>> LINK MEDIA STATE SPEED DUPLEX DEVICE
>>> myri10ge0 Ethernet unknown 10000 full myri10ge0
>>> e1000g0 Ethernet unknown 0 half e1000g0
>>> e1000g1 Ethernet unknown 0 half e1000g1
>>> ixgbe0 Ethernet down 0 unknown ixgbe0
>>> ixgbe1 Ethernet down 0 unknown ixgbe1
>>> + /sbin/ifconfig -a
>>> lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232
>>> index 1
>>> inet 127.0.0.1 netmask ff000000
>>> lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252
>>> index 1
>>> inet6 ::1/128
>>> + echo '== debug end: before =='
>>> == debug end: before ==
>>> + load_sdc_sysinfo
>>> + boot_file_config_enabled
>>> + load_sdc_config
>>> + load_sdc_bootparams
>>> + sed -e 's/,/ /g'
>>> + echo ''
>>> + create_aggrs
>>> + typeset links macs mode mtu
>>> + [[ -z '' ]]
>>> + return 0
>>> + setup_mtu
>>> + typeset tag oldifs val mac link curmtu
>>> + typeset -A mtus
>>> + typeset -A tagmap
>>> + set -o xtrace
>>> + oldifs=$' \t\n'
>>> + IFS=,
>>> + eval val='${CONFIG_secondary_mtu}'
>>> + val=''
>>> + eval mac='${CONFIG_secondary_nic}'
>>> + mac=00:25:90:35:BB:AB
>>> + [[ -z '' ]]
>>> + continue
>>> + eval val='${CONFIG_admin_mtu}'
>>> + val=''
>>> + eval mac='${CONFIG_admin_nic}'
>>> + mac=00:25:90:35:BB:AA
>>> + [[ -z '' ]]
>>> + continue
>>> + eval val='${CONFIG_ixgbe0_mtu}'
>>> + val=9000
>>> + eval mac='${CONFIG_ixgbe0_nic}'
>>> + mac=0:1b:21:bc:51:7a
>>> + [[ -z 9000 ]]
>>> + valid_mtu ixgbe0 9000
>>> + typeset tag mtu
>>> + tag=ixgbe0
>>> + mtu=9000
>>> + [[ 9000 == ~(E)[1-9][0-9][0-9][0-9] ]]
>>> + [[ 9000 -gt 9000 ]]
>>> + [[ 9000 -lt 1500 ]]
>>> + [[ -z '' ]]
>>> + tagmap[0:1b:21:bc:51:7a]=ixgbe0
>>> + [[ -z '' ]]
>>> + mtus[0:1b:21:bc:51:7a]=9000
>>> + IFS=$' \t\n'
>>> + tag=ixgbe0
>>> + eval link='${SYSINFO_NIC_ixgbe0}'
>>> + link=ixgbe0
>>> + [[ -z ixgbe0 ]]
>>> + /usr/sbin/dladm show-linkprop -c -o value -p mtu ixgbe0
>>> + curmtu=1500
>>> + [[ 0 -eq 0 ]]
>>> + [[ 1500 -eq 9000 ]]
>>> + /usr/sbin/dladm set-linkprop -p mtu=9000 ixgbe0
>>> /usr/sbin/dladm: warning: cannot set link property 'mtu' on 'ixgbe0': link
>>> busy
>>> + echo 'Failed to set mtu to 9000 for link ixgbe0'
>>> Failed to set mtu to 9000 for link ixgbe0
>>> + exit 95
>>> [ Aug 15 19:52:33 Method "start" exited with status 95. ]
>>>
>>> The diadm command shows that the card easily supports 9000:
>>> [root@smartos /opt]# dladm show-linkprop -p mtu
>>> LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
>>> myri10ge0 mtu rw 1500 1500 1500-9000
>>> e1000g0 mtu rw 1500 1500 1500-9216
>>> e1000g1 mtu rw 1500 1500 1500-9216
>>> ixgbe0 mtu rw 1500 1500 1500-15500
>>> ixgbe1 mtu rw 1500 1500 1500-15500
>>> eth0 mtu rw 1500 1500 1500
>>> eth0 mtu rw 1500 1500 1500
>>> net0 mtu rw 1500 1500 1500
>>> net0 mtu rw 1500 1500 1500
>>> eth0 mtu rw 1500 1500 1500
>>> net0 mtu rw 1500 1500 1500
>>>
>>> So I’m not sure what’s wrong at the moment. Does anybody have any
>>> experience with this card and a 9000 MTU?
>>>
>>> Regards,
>>> John
>>>
>>> smartos-discuss | Archives
>>> <https://www.listbox.com/member/archive/184463/=now>
>>> <https://www.listbox.com/member/archive/rss/184463/26313201-b436b91a> |
>>> Modify <https://www.listbox.com/member/?&;> Your Subscription
>>> <http://www.listbox.com/>
>>
>>
>
>
-------------------------------------------
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription:
https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com
