On Thu, Mar 29, 2018 at 9:36 PM, Alexander Duyck
<alexander.du...@gmail.com> wrote:
> On Thu, Mar 29, 2018 at 11:18 AM, Ran Shalit <ransha...@gmail.com> wrote:
>> On Thu, Mar 29, 2018 at 8:52 PM, Alexander Duyck
>> <alexander.du...@gmail.com> wrote:
>>> On Thu, Mar 29, 2018 at 7:45 AM, Ran Shalit <ransha...@gmail.com> wrote:
>>>> On Tue, Mar 27, 2018 at 11:43 PM, Alexander Duyck
>>>> <alexander.du...@gmail.com> wrote:
>>>>> On Tue, Mar 27, 2018 at 1:52 AM, Ran Shalit <ransha...@gmail.com> wrote:
>>>>>> On Thu, Mar 22, 2018 at 7:31 PM, Alexander Duyck
>>>>>> <alexander.du...@gmail.com> wrote:
>>>>>>> On Thu, Mar 22, 2018 at 9:11 AM, Ran Shalit <ransha...@gmail.com> wrote:
>>>>>>>> On Mon, Mar 19, 2018 at 10:23 PM, Alexander Duyck
>>>>>>>> <alexander.du...@gmail.com> wrote:
>>>>>>>>> On Mon, Mar 19, 2018 at 12:31 PM, Ran Shalit <ransha...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>> On Mon, Mar 19, 2018 at 6:27 PM, Alexander Duyck
>>>>>>>>>> <alexander.du...@gmail.com> wrote:
>>>>>>>>>>> On Mon, Mar 19, 2018 at 9:07 AM, Ran Shalit <ransha...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I am using igb driver:
>>>>>>>>>>>>
>>>>>>>>>>>> cat output/build/linux-4.10.17/.config | grep IGB
>>>>>>>>>>>>
>>>>>>>>>>>> CONFIG_IGB=y
>>>>>>>>>>>>
>>>>>>>>>>>> CONFIG_IGB_HWMON=y
>>>>>>>>>>>>
>>>>>>>>>>>> CONFIG_IGBVF=y
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> But every boot is takes a lot of time till ethernet is ready.
>>>>>>>>>>>>
>>>>>>>>>>>> I try to disable auto negotiation, but nothing helps yet, the
>>>>>>>>>>>> device
>>>>>>>>>>>> resist, and keep resets phy.
>>>>>>>>>>>>
>>>>>>>>>>>> adapter->fc_autoneg = false;
>>>>>>>>>>>>
>>>>>>>>>>>> hw->mac.autoneg = false;
>>>>>>>>>>>>
>>>>>>>>>>>> hw->phy.autoneg_advertised = 0;
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I tried more flags, but nothing helps.
>>>>>>>>>>>> The phy always disabled/reset at boot (led is off for 1 second and
>>>>>>>>>>>> then on).
>>>>>>>>>>>>
>>>>>>>>>>>> Is there a way to disable auto-negotiation with igb driver ?
>>>>>>>>>>>> I use buildroot with kernel 4.10.81.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you for any suggestion,
>>>>>>>>>>>>
>>>>>>>>>>>> ran
>>>>>>>>>>>
>>>>>>>>>>> Instead of trying to disable it in the driver why not just change
>>>>>>>>>>> your
>>>>>>>>>>> system configuration to disable it? You should be able to configure
>>>>>>>>>>> things in either Network Manager, or via the network init scripts so
>>>>>>>>>>> that you instead just used a forced speed/duplex combination. If
>>>>>>>>>>> nothing else you can go through and drop support for any other
>>>>>>>>>>> advertised speed/duplex and that should improve the speed of autoneg
>>>>>>>>>>> itself.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I think I tried it and it did not work, but I shall try again.
>>>>>>>>>> Where should I put it ? in init.d startup scripts ?
>>>>>>>>>> I think I tried, and yet , I have seen that in reset, the leds of the
>>>>>>>>>> phy are always turned off for a ~1.5 second and then on again.
>>>>>>>>>> This is actually what I am trying to overcome, this strange reset of
>>>>>>>>>> phy every powerup.
>>>>>>>>>
>>>>>>>>> So it sounds like what you may want to disable would not be the phy
>>>>>>>>> autoneg, but the phy reset itself. If that is what you are looking for
>>>>>>>>> then you might try modifying igb_reset_phy, or at least when we invoke
>>>>>>>>> it in igb_power_up_link. You could look at adding a private flag to
>>>>>>>>> the igb driver to disable it for your use case if that is the issue.
>>>>>>>>>
>>>>>>>>>>> You could refer to something like this for more information:
>>>>>>>>>>> https://www.shellhacks.com/change-speed-duplex-ethernet-card-linux/
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>> - Alex
>>>>>>>>>>
>>>>>>>>>> Thank you very much,
>>>>>>>>>> ran
>>>>>>>>>
>>>>>>>>> Now I am not so certain if this will solve you issue. What you may
>>>>>>>>> want to do instead is take a look at the function igb_power_up_link in
>>>>>>>>> igb_main.c and possibly consider adding a flag check to allow you to
>>>>>>>>> disable it on the systems you need to disable it on, or if this is for
>>>>>>>>> just one driver you could comment the line out and see if that will
>>>>>>>>> solve the issue.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Using dmesg to understand what's going on , I see 2 things:
>>>>>>>> 1. Long time interval between opening and link up (5.0 - 1.9 =
>>>>>>>> ~3seconds !)
>>>>>>>
>>>>>>> For a 1G link 3 seconds isn't all that long.
>>>>>>>
>>>>>>>> 2. Long time interval between link up state and ping success ( 8 - 5 =
>>>>>>>> ~3 sedconds !)
>>>>>>>
>>>>>>> I don't know what to tell you about that part. I suspect that may be
>>>>>>> some delays in notifiers being processed after the interface has
>>>>>>> reported link up. In addition I don't know what you link partner is.
>>>>>>> Normally for a ping you have to take care of things like setting up
>>>>>>> routes, getting an ARP out to find your target, and then sending the
>>>>>>> ping to that address.
>>>>>>>
>>>>>>> What might be interesting would be to add a dump of the tx_buffer_info
>>>>>>> data for the rings that have transmitted packets. For example we might
>>>>>>> have reported link up, but the link partner may not have been
>>>>>>> responding to us because it wasn't.
>>>>>>>
>>>>>>
>>>>>> Hi Alexander,
>>>>>>
>>>>>> Is there a way to open this dump with compilation flag ? How to dump
>>>>>> this information ?
>>>>>> This time interval (from igb probe to link up) is now is the major
>>>>>> delay (we solved the other time to ping).
>>>>>>
>>>>>> Thanks,
>>>>>> ranran
>>>>>
>>>>> Actually it can be turned on dynamically to some extend. The code to
>>>>> dump the Tx descriptor information is in a function called igb_dump in
>>>>> the driver. To turn on the bits related to dumping Tx packets you
>>>>> would need to use:
>>>>> ethtool -s <ethX> msglvl hw on
>>>>> ethtool -s <ethX> msglvl tx_done on
>>>>> ethtool -s <ethX> msglvl pktdata on
>>>>>
>>>>> Then it is just a matter of triggering a call to the reset task. There
>>>>> are a few ways to get there. If nothing else you might modify the
>>>>> driver to call igb_dump() at the start of the igb_close() function.
>>>>> Then all you would have to do is bring down the interface with
>>>>> something like an ifconfig <ethX> down and that should trigger the
>>>>> dump of the Tx descriptor data.
>>>>>
>>>>
>>>>
>>>> I've added start/stop log in main igb functions.
>>>> I can now see things more clearly.
>>>> please see the attached log below,
>>>> It seems that igb_watchdog is responsible somehow for waking up and
>>>> checking things, before deciding that link is up.
>>>> Maybe there is a way to call it higher rate ?
>>>
>>> When the link state changes it is triggered via an interrupt. The time
>>> was about 3.5s to bring up link here if I am understanding correctly.
>>> The other issue I see in your test is I am guessing you aren't pinging
>>> something with a static ARP entry which is another cause for delay in
>>> your test.
>>>
>>> One other thing you might try doing to shave time off of link training
>>> is to disable mdix via ethtool "ethtool -s <ethX> mdix off" and see if
>>> that saves you any time. It might shave maybe about .5 seconds off of
>>> the total link time.
>>>
>>> If you are wanting it to link faster maybe you should look at using
>>> something besides 1000BaseT. Typically about 3 to 4 seconds is about
>>> the limit for Gigabit copper. You may wan to look at a device with a
>>> different PHY if you require a faster link time. For example for
>>> automotive applications something like 1000BaseT1
>>> (https://standards.ieee.org/events/automotive/2016/d1_09_tan_1000base_t1_learning_and_next_gen_use_cases_v6.pdf)
>>> is supposed to be the intended way to go with link times in the
>>> sub-200ms range.
>>>
>>> - Alex
>>
>> I will check your interesting suggestions.
>> Do you think it might help to start igb and networking earlier ?
>
> No.
>
>> I see in dmesg log that igb starts late in kernel boot, even when I
>> add ip=... in bootargs instead of using the init.d script (which calls
>> ifup).
>
> You need enough of the basics up that 2 seconds is pretty reasonable.
>
>> What about igb watchdog task, what trigger is to wake and check for
>> Link state ? I don't understand the logic behind it. Maybe there is
>> something I can do in driver to make it detect link up faster ?
>
> No there isn't. Basically after igb open is called there is an MSI-X
> interrupt enabled that is fired as soon as the link is up. Otherwise
> the watchdog runs on about a 2 second interval. The fact that you have
> one watchdog at 4.7 seconds and the next at 5.5 tells me the interrupt
> likely fired and triggered us to come in and run the watchdog.
>
>>
>> [ 1.925577] ping igb_reset end
>> [ 1.926414] pps pps0: new PPS source ptp0
>> [ 1.926430] igb 0000:03:00.0: added PHC on eth0
>> [ 1.926433] igb 0000:03:00.0: Intel(R) Gigabit Ethernet Network Connection
>> [ 1.926436] igb 0000:03:00.0: eth0: (PCIe:2.5Gb/s:Width x1)
>> 00:e0:4b:5e:d6:08
>> [ 1.926439] igb 0000:03:00.0: eth0: PBA No: FFFFFF-0FF
>> [ 1.926457] igb 0000:03:00.0: Using MSI-X interrupts. 4 rx
>> queue(s), 4 tx queue(s)
>> [ 1.926500] ping igb_probe end
>> [ 1.928321] ping Starting network:
>> [ 1.946220] ping __igb_open start <----------- igb open
>
> One thing I would be interested in seeing here is the end of igb open.
> I wonder how long we are spending bringing up the interface.
>
>> [ 4.529597] ping func1 fail 1
>> [ 4.648156] ping func1 fail 1
>> [ 4.706496] ping igb_watchdog start
>> [ 4.706516] ping igb_watchdog end
>> [ 4.706556] ping igb_watchdog_task start
>> [ 4.706562] ping igb_phy_has_link start
>> [ 4.706711] ping igb_phy_has_link end
>> [ 4.706716] ping igb_watchdog_task end
>> [ 4.766559] ping func1 fail 1
>> [ 4.885190] ping func1 fail 1
>> [ 5.003555] ping func1 fail 1
>> [ 5.122257] ping func1 fail 1
>> [ 5.241001] ping func1 fail 1
>> [ 5.359845] ping func1 fail 1
>> [ 5.478460] ping func1 fail 1
>> [ 5.531494] ping igb_watchdog start
>> [ 5.531516] ping igb_watchdog end
>> [ 5.531556] ping igb_watchdog_task start
>> [ 5.531562] ping igb_phy_has_link start
>> [ 5.531710] ping igb_phy_has_link end
>> [ 5.532069] igb 0000:03:00.0 eth0: igb: eth0 NIC Link is Up 1000
>> Mbps Full Duplex, Flow Control: RX <<----------- link is up
>
> Right. We are running this .8 seconds after the last instance. This
> tells me that we were triggered by the interrupt.
>
>> [ 5.597265] ping func1 fail 1
>> [ 5.715885] ping func1 fail 1
>> [ 5.834734] ping func1 fail 1
>> [ 5.953382] ping func1 fail 1
>> [ 6.072240] ping func1 fail 1
>> [ 6.190870] ping func1 fail 1
>> [ 6.258756] ping igb_watchdog_task end
>> [ 6.258770] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>
> I am kind of surprised to see it takes this long for IPv6 to fully
> come up, but it probably has something to do with neighbor discovery.
>
>> [ 6.309740] ping func1 fail 1
>> [ 6.428332] ping func1 fail 1
>> [ 6.547061] ping func1 fail 1
>> [ 6.665515] ping func1 fail 1
>> [ 6.784175] ping func1 fail 1
>> [ 6.902614] ping func1 fail 1
>> [ 7.021295] ping func1 fail 1
>> [ 7.139741] ping func1 fail 1
>> [ 7.258463] ping func1 fail 1
>> [ 7.377105] ping func1 fail 1
>> [ 7.496115] ping func1 fail 1
>> [ 7.508706] ping func1 ok 0 <---------- first ping success
>
> Right. As I said I don't know if your ping has taken care of the ARP
> request already or not. In my experience an ARP is only sent once
> every few seconds or so for an ongoing request, odds are it didn't go
> out until 2 seconds after the link came up.
>
I did use static arp before ping, but it seems not too bring any time
improvement. not sure why.
start script in init.d:
SO1_aa
#!/bin/sh
case "$1" in
start)
modprobe igb
arp -s 10.0.0.20 50:7b:9d:a6:4c:39
#ethtool -s eth0 autoneg off duplex full speed 100 advertise 08
echo "ranran Starting network: " > /dev/kmsg
/sbin/ifup -a
.....
S01_ac
#!/bin/sh
func1()
{
fping -c1 -t100 10.0.0.20
if [ $? -eq 0 ]
then
echo "ranran func1 ok" $? > /dev/kmsg
else
echo "ranran func1 fail" $? > /dev/kmsg
fi
}
case "$1" in
start)
func1
func1
func1
....
> It really seems like you are getting beyond your depth. You might want
> to spend some time researching the inner workings of Ethernet and how
> 1000BaseT works. As I said before I don't know if it is the right
> medium for you to be working with if you are this concerned with
> trying to improve the link time in the way you are.
>
I understand, I wish there was more information/tutorials about this driver.
This driver is not too easy to learn and understand.
Thank you very very much for your helpful comments and suggestions !!
Ran
> - Alex
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
