On Fri, Mar 30, 2018 at 12:34 AM, Ran Shalit <ransha...@gmail.com> wrote:
> On Thu, Mar 29, 2018 at 9:36 PM, Alexander Duyck
> <alexander.du...@gmail.com> wrote:
>> On Thu, Mar 29, 2018 at 11:18 AM, Ran Shalit <ransha...@gmail.com> wrote:
>>> On Thu, Mar 29, 2018 at 8:52 PM, Alexander Duyck
>>> <alexander.du...@gmail.com> wrote:
>>>> On Thu, Mar 29, 2018 at 7:45 AM, Ran Shalit <ransha...@gmail.com> wrote:
>>>>> On Tue, Mar 27, 2018 at 11:43 PM, Alexander Duyck
>>>>> <alexander.du...@gmail.com> wrote:
>>>>>> On Tue, Mar 27, 2018 at 1:52 AM, Ran Shalit <ransha...@gmail.com> wrote:
>>>>>>> On Thu, Mar 22, 2018 at 7:31 PM, Alexander Duyck
>>>>>>> <alexander.du...@gmail.com> wrote:
>>>>>>>> On Thu, Mar 22, 2018 at 9:11 AM, Ran Shalit <ransha...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>> On Mon, Mar 19, 2018 at 10:23 PM, Alexander Duyck
>>>>>>>>> <alexander.du...@gmail.com> wrote:
>>>>>>>>>> On Mon, Mar 19, 2018 at 12:31 PM, Ran Shalit <ransha...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>> On Mon, Mar 19, 2018 at 6:27 PM, Alexander Duyck
>>>>>>>>>>> <alexander.du...@gmail.com> wrote:
>>>>>>>>>>>> On Mon, Mar 19, 2018 at 9:07 AM, Ran Shalit <ransha...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am using igb driver:
>>>>>>>>>>>>>
>>>>>>>>>>>>> cat output/build/linux-4.10.17/.config | grep IGB
>>>>>>>>>>>>>
>>>>>>>>>>>>> CONFIG_IGB=y
>>>>>>>>>>>>>
>>>>>>>>>>>>> CONFIG_IGB_HWMON=y
>>>>>>>>>>>>>
>>>>>>>>>>>>> CONFIG_IGBVF=y
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> But every boot is takes a lot of time till ethernet is ready.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I try to disable auto negotiation, but nothing helps yet, the
>>>>>>>>>>>>> device
>>>>>>>>>>>>> resist, and keep resets phy.
>>>>>>>>>>>>>
>>>>>>>>>>>>> adapter->fc_autoneg = false;
>>>>>>>>>>>>>
>>>>>>>>>>>>> hw->mac.autoneg = false;
>>>>>>>>>>>>>
>>>>>>>>>>>>> hw->phy.autoneg_advertised = 0;
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I tried more flags, but nothing helps.
>>>>>>>>>>>>> The phy always disabled/reset at boot (led is off for 1 second
>>>>>>>>>>>>> and then on).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is there a way to disable auto-negotiation with igb driver ?
>>>>>>>>>>>>> I use buildroot with kernel 4.10.81.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you for any suggestion,
>>>>>>>>>>>>>
>>>>>>>>>>>>> ran
>>>>>>>>>>>>
>>>>>>>>>>>> Instead of trying to disable it in the driver why not just change
>>>>>>>>>>>> your
>>>>>>>>>>>> system configuration to disable it? You should be able to configure
>>>>>>>>>>>> things in either Network Manager, or via the network init scripts
>>>>>>>>>>>> so
>>>>>>>>>>>> that you instead just used a forced speed/duplex combination. If
>>>>>>>>>>>> nothing else you can go through and drop support for any other
>>>>>>>>>>>> advertised speed/duplex and that should improve the speed of
>>>>>>>>>>>> autoneg
>>>>>>>>>>>> itself.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I think I tried it and it did not work, but I shall try again.
>>>>>>>>>>> Where should I put it ? in init.d startup scripts ?
>>>>>>>>>>> I think I tried, and yet , I have seen that in reset, the leds of
>>>>>>>>>>> the
>>>>>>>>>>> phy are always turned off for a ~1.5 second and then on again.
>>>>>>>>>>> This is actually what I am trying to overcome, this strange reset of
>>>>>>>>>>> phy every powerup.
>>>>>>>>>>
>>>>>>>>>> So it sounds like what you may want to disable would not be the phy
>>>>>>>>>> autoneg, but the phy reset itself. If that is what you are looking
>>>>>>>>>> for
>>>>>>>>>> then you might try modifying igb_reset_phy, or at least when we
>>>>>>>>>> invoke
>>>>>>>>>> it in igb_power_up_link. You could look at adding a private flag to
>>>>>>>>>> the igb driver to disable it for your use case if that is the issue.
>>>>>>>>>>
>>>>>>>>>>>> You could refer to something like this for more information:
>>>>>>>>>>>> https://www.shellhacks.com/change-speed-duplex-ethernet-card-linux/
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>> - Alex
>>>>>>>>>>>
>>>>>>>>>>> Thank you very much,
>>>>>>>>>>> ran
>>>>>>>>>>
>>>>>>>>>> Now I am not so certain if this will solve you issue. What you may
>>>>>>>>>> want to do instead is take a look at the function igb_power_up_link
>>>>>>>>>> in
>>>>>>>>>> igb_main.c and possibly consider adding a flag check to allow you to
>>>>>>>>>> disable it on the systems you need to disable it on, or if this is
>>>>>>>>>> for
>>>>>>>>>> just one driver you could comment the line out and see if that will
>>>>>>>>>> solve the issue.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Using dmesg to understand what's going on , I see 2 things:
>>>>>>>>> 1. Long time interval between opening and link up (5.0 - 1.9 =
>>>>>>>>> ~3seconds !)
>>>>>>>>
>>>>>>>> For a 1G link 3 seconds isn't all that long.
>>>>>>>>
>>>>>>>>> 2. Long time interval between link up state and ping success ( 8 - 5 =
>>>>>>>>> ~3 sedconds !)
>>>>>>>>
>>>>>>>> I don't know what to tell you about that part. I suspect that may be
>>>>>>>> some delays in notifiers being processed after the interface has
>>>>>>>> reported link up. In addition I don't know what you link partner is.
>>>>>>>> Normally for a ping you have to take care of things like setting up
>>>>>>>> routes, getting an ARP out to find your target, and then sending the
>>>>>>>> ping to that address.
>>>>>>>>
>>>>>>>> What might be interesting would be to add a dump of the tx_buffer_info
>>>>>>>> data for the rings that have transmitted packets. For example we might
>>>>>>>> have reported link up, but the link partner may not have been
>>>>>>>> responding to us because it wasn't.
>>>>>>>>
>>>>>>>
>>>>>>> Hi Alexander,
>>>>>>>
>>>>>>> Is there a way to open this dump with compilation flag ? How to dump
>>>>>>> this information ?
>>>>>>> This time interval (from igb probe to link up) is now is the major
>>>>>>> delay (we solved the other time to ping).
>>>>>>>
>>>>>>> Thanks,
>>>>>>> ranran
>>>>>>
>>>>>> Actually it can be turned on dynamically to some extend. The code to
>>>>>> dump the Tx descriptor information is in a function called igb_dump in
>>>>>> the driver. To turn on the bits related to dumping Tx packets you
>>>>>> would need to use:
>>>>>> ethtool -s <ethX> msglvl hw on
>>>>>> ethtool -s <ethX> msglvl tx_done on
>>>>>> ethtool -s <ethX> msglvl pktdata on
>>>>>>
>>>>>> Then it is just a matter of triggering a call to the reset task. There
>>>>>> are a few ways to get there. If nothing else you might modify the
>>>>>> driver to call igb_dump() at the start of the igb_close() function.
>>>>>> Then all you would have to do is bring down the interface with
>>>>>> something like an ifconfig <ethX> down and that should trigger the
>>>>>> dump of the Tx descriptor data.
>>>>>>
>>>>>
>>>>>
>>>>> I've added start/stop log in main igb functions.
>>>>> I can now see things more clearly.
>>>>> please see the attached log below,
>>>>> It seems that igb_watchdog is responsible somehow for waking up and
>>>>> checking things, before deciding that link is up.
>>>>> Maybe there is a way to call it higher rate ?
>>>>
>>>> When the link state changes it is triggered via an interrupt. The time
>>>> was about 3.5s to bring up link here if I am understanding correctly.
>>>> The other issue I see in your test is I am guessing you aren't pinging
>>>> something with a static ARP entry which is another cause for delay in
>>>> your test.
>>>>
>>>> One other thing you might try doing to shave time off of link training
>>>> is to disable mdix via ethtool "ethtool -s <ethX> mdix off" and see if
>>>> that saves you any time. It might shave maybe about .5 seconds off of
>>>> the total link time.
>>>>
>>>> If you are wanting it to link faster maybe you should look at using
>>>> something besides 1000BaseT. Typically about 3 to 4 seconds is about
>>>> the limit for Gigabit copper. You may wan to look at a device with a
>>>> different PHY if you require a faster link time. For example for
>>>> automotive applications something like 1000BaseT1
>>>> (https://standards.ieee.org/events/automotive/2016/d1_09_tan_1000base_t1_learning_and_next_gen_use_cases_v6.pdf)
>>>> is supposed to be the intended way to go with link times in the
>>>> sub-200ms range.
>>>>
>>>> - Alex
>>>
>>> I will check your interesting suggestions.
>>> Do you think it might help to start igb and networking earlier ?
>>
>> No.
>>
>>> I see in dmesg log that igb starts late in kernel boot, even when I
>>> add ip=... in bootargs instead of using the init.d script (which calls
>>> ifup).
>>
>> You need enough of the basics up that 2 seconds is pretty reasonable.
I would like to add another very interesting result I got:
disabling auto-negotiation did not reduce time, and I disabled start
networking in init.d (S40network , ifup)
and started the network instead with bootargs (ip=10.0.0.2:....)
Is there any reason for this dependency of these methods in each other ?
Best Regards,
ranran
>>
>>> What about igb watchdog task, what trigger is to wake and check for
>>> Link state ? I don't understand the logic behind it. Maybe there is
>>> something I can do in driver to make it detect link up faster ?
>>
>> No there isn't. Basically after igb open is called there is an MSI-X
>> interrupt enabled that is fired as soon as the link is up. Otherwise
>> the watchdog runs on about a 2 second interval. The fact that you have
>> one watchdog at 4.7 seconds and the next at 5.5 tells me the interrupt
>> likely fired and triggered us to come in and run the watchdog.
>>
>>>
>>> [ 1.925577] ping igb_reset end
>>> [ 1.926414] pps pps0: new PPS source ptp0
>>> [ 1.926430] igb 0000:03:00.0: added PHC on eth0
>>> [ 1.926433] igb 0000:03:00.0: Intel(R) Gigabit Ethernet Network
>>> Connection
>>> [ 1.926436] igb 0000:03:00.0: eth0: (PCIe:2.5Gb/s:Width x1)
>>> 00:e0:4b:5e:d6:08
>>> [ 1.926439] igb 0000:03:00.0: eth0: PBA No: FFFFFF-0FF
>>> [ 1.926457] igb 0000:03:00.0: Using MSI-X interrupts. 4 rx
>>> queue(s), 4 tx queue(s)
>>> [ 1.926500] ping igb_probe end
>>> [ 1.928321] ping Starting network:
>>> [ 1.946220] ping __igb_open start <----------- igb open
>>
>> One thing I would be interested in seeing here is the end of igb open.
>> I wonder how long we are spending bringing up the interface.
>>
>>> [ 4.529597] ping func1 fail 1
>>> [ 4.648156] ping func1 fail 1
>>> [ 4.706496] ping igb_watchdog start
>>> [ 4.706516] ping igb_watchdog end
>>> [ 4.706556] ping igb_watchdog_task start
>>> [ 4.706562] ping igb_phy_has_link start
>>> [ 4.706711] ping igb_phy_has_link end
>>> [ 4.706716] ping igb_watchdog_task end
>>> [ 4.766559] ping func1 fail 1
>>> [ 4.885190] ping func1 fail 1
>>> [ 5.003555] ping func1 fail 1
>>> [ 5.122257] ping func1 fail 1
>>> [ 5.241001] ping func1 fail 1
>>> [ 5.359845] ping func1 fail 1
>>> [ 5.478460] ping func1 fail 1
>>> [ 5.531494] ping igb_watchdog start
>>> [ 5.531516] ping igb_watchdog end
>>> [ 5.531556] ping igb_watchdog_task start
>>> [ 5.531562] ping igb_phy_has_link start
>>> [ 5.531710] ping igb_phy_has_link end
>>> [ 5.532069] igb 0000:03:00.0 eth0: igb: eth0 NIC Link is Up 1000
>>> Mbps Full Duplex, Flow Control: RX <<----------- link is up
>>
>> Right. We are running this .8 seconds after the last instance. This
>> tells me that we were triggered by the interrupt.
>>
>>> [ 5.597265] ping func1 fail 1
>>> [ 5.715885] ping func1 fail 1
>>> [ 5.834734] ping func1 fail 1
>>> [ 5.953382] ping func1 fail 1
>>> [ 6.072240] ping func1 fail 1
>>> [ 6.190870] ping func1 fail 1
>>> [ 6.258756] ping igb_watchdog_task end
>>> [ 6.258770] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>>
>> I am kind of surprised to see it takes this long for IPv6 to fully
>> come up, but it probably has something to do with neighbor discovery.
>>
>>> [ 6.309740] ping func1 fail 1
>>> [ 6.428332] ping func1 fail 1
>>> [ 6.547061] ping func1 fail 1
>>> [ 6.665515] ping func1 fail 1
>>> [ 6.784175] ping func1 fail 1
>>> [ 6.902614] ping func1 fail 1
>>> [ 7.021295] ping func1 fail 1
>>> [ 7.139741] ping func1 fail 1
>>> [ 7.258463] ping func1 fail 1
>>> [ 7.377105] ping func1 fail 1
>>> [ 7.496115] ping func1 fail 1
>>> [ 7.508706] ping func1 ok 0 <---------- first ping success
>>
>> Right. As I said I don't know if your ping has taken care of the ARP
>> request already or not. In my experience an ARP is only sent once
>> every few seconds or so for an ongoing request, odds are it didn't go
>> out until 2 seconds after the link came up.
>>
>
> I did use static arp before ping, but it seems not too bring any time
> improvement. not sure why.
>
> start script in init.d:
>
> SO1_aa
>
> #!/bin/sh
>
> case "$1" in
> start)
>
> modprobe igb
> arp -s 10.0.0.20 50:7b:9d:a6:4c:39
> #ethtool -s eth0 autoneg off duplex full speed 100 advertise 08
> echo "ranran Starting network: " > /dev/kmsg
> /sbin/ifup -a
> .....
>
> S01_ac
>
> #!/bin/sh
>
> func1()
> {
> fping -c1 -t100 10.0.0.20
> if [ $? -eq 0 ]
> then
> echo "ranran func1 ok" $? > /dev/kmsg
> else
> echo "ranran func1 fail" $? > /dev/kmsg
> fi
> }
>
> case "$1" in
> start)
>
> func1
> func1
> func1
> ....
>
>
>> It really seems like you are getting beyond your depth. You might want
>> to spend some time researching the inner workings of Ethernet and how
>> 1000BaseT works. As I said before I don't know if it is the right
>> medium for you to be working with if you are this concerned with
>> trying to improve the link time in the way you are.
>>
>
> I understand, I wish there was more information/tutorials about this driver.
> This driver is not too easy to learn and understand.
> Thank you very very much for your helpful comments and suggestions !!
>
> Ran
>
>> - Alex
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
