On Sat, Mar 31, 2018 at 1:16 PM, Ran Shalit <ransha...@gmail.com> wrote: > On Sat, Mar 31, 2018 at 8:24 PM, Alexander Duyck > <alexander.du...@gmail.com> wrote: >> On Sat, Mar 31, 2018 at 1:09 AM, Ran Shalit <ransha...@gmail.com> wrote: >>> On Fri, Mar 30, 2018 at 8:08 PM, Alexander Duyck >>> <alexander.du...@gmail.com> wrote: >>>> On Thu, Mar 29, 2018 at 10:35 PM, Ran Shalit <ransha...@gmail.com> wrote: >>>>> On Fri, Mar 30, 2018 at 12:34 AM, Ran Shalit <ransha...@gmail.com> wrote: >>>>>> On Thu, Mar 29, 2018 at 9:36 PM, Alexander Duyck >>>>>> <alexander.du...@gmail.com> wrote: >>>>>>> On Thu, Mar 29, 2018 at 11:18 AM, Ran Shalit <ransha...@gmail.com> >>>>>>> wrote: >>>>>>>> On Thu, Mar 29, 2018 at 8:52 PM, Alexander Duyck >>>>>>>> <alexander.du...@gmail.com> wrote: >>>>>>>>> On Thu, Mar 29, 2018 at 7:45 AM, Ran Shalit <ransha...@gmail.com> >>>>>>>>> wrote: >>>>>>>>>> On Tue, Mar 27, 2018 at 11:43 PM, Alexander Duyck >>>>>>>>>> <alexander.du...@gmail.com> wrote: >>>>>>>>>>> On Tue, Mar 27, 2018 at 1:52 AM, Ran Shalit <ransha...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>>> On Thu, Mar 22, 2018 at 7:31 PM, Alexander Duyck >>>>>>>>>>>> <alexander.du...@gmail.com> wrote: >>>>>>>>>>>>> On Thu, Mar 22, 2018 at 9:11 AM, Ran Shalit <ransha...@gmail.com> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> On Mon, Mar 19, 2018 at 10:23 PM, Alexander Duyck >>>>>>>>>>>>>> <alexander.du...@gmail.com> wrote: >>>>>>>>>>>>>>> On Mon, Mar 19, 2018 at 12:31 PM, Ran Shalit >>>>>>>>>>>>>>> <ransha...@gmail.com> wrote: >>>>>>>>>>>>>>>> On Mon, Mar 19, 2018 at 6:27 PM, Alexander Duyck >>>>>>>>>>>>>>>> <alexander.du...@gmail.com> wrote: >>>>>>>>>>>>>>>>> On Mon, Mar 19, 2018 at 9:07 AM, Ran Shalit >>>>>>>>>>>>>>>>> <ransha...@gmail.com> wrote: >>>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I am using igb driver: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> cat output/build/linux-4.10.17/.config | grep IGB >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> CONFIG_IGB=y >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> CONFIG_IGB_HWMON=y >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> CONFIG_IGBVF=y >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> But every boot is takes a lot of time till ethernet is ready. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I try to disable auto negotiation, but nothing helps yet, >>>>>>>>>>>>>>>>>> the device >>>>>>>>>>>>>>>>>> resist, and keep resets phy. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> adapter->fc_autoneg = false; >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> hw->mac.autoneg = false; >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> hw->phy.autoneg_advertised = 0; >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I tried more flags, but nothing helps. >>>>>>>>>>>>>>>>>> The phy always disabled/reset at boot (led is off for 1 >>>>>>>>>>>>>>>>>> second and then on). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Is there a way to disable auto-negotiation with igb driver ? >>>>>>>>>>>>>>>>>> I use buildroot with kernel 4.10.81. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thank you for any suggestion, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> ran >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Instead of trying to disable it in the driver why not just >>>>>>>>>>>>>>>>> change your >>>>>>>>>>>>>>>>> system configuration to disable it? You should be able to >>>>>>>>>>>>>>>>> configure >>>>>>>>>>>>>>>>> things in either Network Manager, or via the network init >>>>>>>>>>>>>>>>> scripts so >>>>>>>>>>>>>>>>> that you instead just used a forced speed/duplex combination. >>>>>>>>>>>>>>>>> If >>>>>>>>>>>>>>>>> nothing else you can go through and drop support for any other >>>>>>>>>>>>>>>>> advertised speed/duplex and that should improve the speed of >>>>>>>>>>>>>>>>> autoneg >>>>>>>>>>>>>>>>> itself. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I think I tried it and it did not work, but I shall try again. >>>>>>>>>>>>>>>> Where should I put it ? in init.d startup scripts ? >>>>>>>>>>>>>>>> I think I tried, and yet , I have seen that in reset, the leds >>>>>>>>>>>>>>>> of the >>>>>>>>>>>>>>>> phy are always turned off for a ~1.5 second and then on again. >>>>>>>>>>>>>>>> This is actually what I am trying to overcome, this strange >>>>>>>>>>>>>>>> reset of >>>>>>>>>>>>>>>> phy every powerup. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> So it sounds like what you may want to disable would not be the >>>>>>>>>>>>>>> phy >>>>>>>>>>>>>>> autoneg, but the phy reset itself. If that is what you are >>>>>>>>>>>>>>> looking for >>>>>>>>>>>>>>> then you might try modifying igb_reset_phy, or at least when we >>>>>>>>>>>>>>> invoke >>>>>>>>>>>>>>> it in igb_power_up_link. You could look at adding a private >>>>>>>>>>>>>>> flag to >>>>>>>>>>>>>>> the igb driver to disable it for your use case if that is the >>>>>>>>>>>>>>> issue. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> You could refer to something like this for more information: >>>>>>>>>>>>>>>>> https://www.shellhacks.com/change-speed-duplex-ethernet-card-linux/ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> - Alex >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thank you very much, >>>>>>>>>>>>>>>> ran >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Now I am not so certain if this will solve you issue. What you >>>>>>>>>>>>>>> may >>>>>>>>>>>>>>> want to do instead is take a look at the function >>>>>>>>>>>>>>> igb_power_up_link in >>>>>>>>>>>>>>> igb_main.c and possibly consider adding a flag check to allow >>>>>>>>>>>>>>> you to >>>>>>>>>>>>>>> disable it on the systems you need to disable it on, or if this >>>>>>>>>>>>>>> is for >>>>>>>>>>>>>>> just one driver you could comment the line out and see if that >>>>>>>>>>>>>>> will >>>>>>>>>>>>>>> solve the issue. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Using dmesg to understand what's going on , I see 2 things: >>>>>>>>>>>>>> 1. Long time interval between opening and link up (5.0 - 1.9 = >>>>>>>>>>>>>> ~3seconds !) >>>>>>>>>>>>> >>>>>>>>>>>>> For a 1G link 3 seconds isn't all that long. >>>>>>>>>>>>> >>>>>>>>>>>>>> 2. Long time interval between link up state and ping success ( 8 >>>>>>>>>>>>>> - 5 = >>>>>>>>>>>>>> ~3 sedconds !) >>>>>>>>>>>>> >>>>>>>>>>>>> I don't know what to tell you about that part. I suspect that may >>>>>>>>>>>>> be >>>>>>>>>>>>> some delays in notifiers being processed after the interface has >>>>>>>>>>>>> reported link up. In addition I don't know what you link partner >>>>>>>>>>>>> is. >>>>>>>>>>>>> Normally for a ping you have to take care of things like setting >>>>>>>>>>>>> up >>>>>>>>>>>>> routes, getting an ARP out to find your target, and then sending >>>>>>>>>>>>> the >>>>>>>>>>>>> ping to that address. >>>>>>>>>>>>> >>>>>>>>>>>>> What might be interesting would be to add a dump of the >>>>>>>>>>>>> tx_buffer_info >>>>>>>>>>>>> data for the rings that have transmitted packets. For example we >>>>>>>>>>>>> might >>>>>>>>>>>>> have reported link up, but the link partner may not have been >>>>>>>>>>>>> responding to us because it wasn't. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hi Alexander, >>>>>>>>>>>> >>>>>>>>>>>> Is there a way to open this dump with compilation flag ? How to >>>>>>>>>>>> dump >>>>>>>>>>>> this information ? >>>>>>>>>>>> This time interval (from igb probe to link up) is now is the major >>>>>>>>>>>> delay (we solved the other time to ping). >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> ranran >>>>>>>>>>> >>>>>>>>>>> Actually it can be turned on dynamically to some extend. The code to >>>>>>>>>>> dump the Tx descriptor information is in a function called igb_dump >>>>>>>>>>> in >>>>>>>>>>> the driver. To turn on the bits related to dumping Tx packets you >>>>>>>>>>> would need to use: >>>>>>>>>>> ethtool -s <ethX> msglvl hw on >>>>>>>>>>> ethtool -s <ethX> msglvl tx_done on >>>>>>>>>>> ethtool -s <ethX> msglvl pktdata on >>>>>>>>>>> >>>>>>>>>>> Then it is just a matter of triggering a call to the reset task. >>>>>>>>>>> There >>>>>>>>>>> are a few ways to get there. If nothing else you might modify the >>>>>>>>>>> driver to call igb_dump() at the start of the igb_close() function. >>>>>>>>>>> Then all you would have to do is bring down the interface with >>>>>>>>>>> something like an ifconfig <ethX> down and that should trigger the >>>>>>>>>>> dump of the Tx descriptor data. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I've added start/stop log in main igb functions. >>>>>>>>>> I can now see things more clearly. >>>>>>>>>> please see the attached log below, >>>>>>>>>> It seems that igb_watchdog is responsible somehow for waking up and >>>>>>>>>> checking things, before deciding that link is up. >>>>>>>>>> Maybe there is a way to call it higher rate ? >>>>>>>>> >>>>>>>>> When the link state changes it is triggered via an interrupt. The time >>>>>>>>> was about 3.5s to bring up link here if I am understanding correctly. >>>>>>>>> The other issue I see in your test is I am guessing you aren't pinging >>>>>>>>> something with a static ARP entry which is another cause for delay in >>>>>>>>> your test. >>>>>>>>> >>>>>>>>> One other thing you might try doing to shave time off of link training >>>>>>>>> is to disable mdix via ethtool "ethtool -s <ethX> mdix off" and see if >>>>>>>>> that saves you any time. It might shave maybe about .5 seconds off of >>>>>>>>> the total link time. >>>>>>>>> >>>>>>>>> If you are wanting it to link faster maybe you should look at using >>>>>>>>> something besides 1000BaseT. Typically about 3 to 4 seconds is about >>>>>>>>> the limit for Gigabit copper. You may wan to look at a device with a >>>>>>>>> different PHY if you require a faster link time. For example for >>>>>>>>> automotive applications something like 1000BaseT1 >>>>>>>>> (https://standards.ieee.org/events/automotive/2016/d1_09_tan_1000base_t1_learning_and_next_gen_use_cases_v6.pdf) >>>>>>>>> is supposed to be the intended way to go with link times in the >>>>>>>>> sub-200ms range. >>>>>>>>> >>>>>>>>> - Alex >>>>>>>> >>>>>>>> I will check your interesting suggestions. >>>>>>>> Do you think it might help to start igb and networking earlier ? >>>>>>> >>>>>>> No. >>>>>>> >>>>>>>> I see in dmesg log that igb starts late in kernel boot, even when I >>>>>>>> add ip=... in bootargs instead of using the init.d script (which calls >>>>>>>> ifup). >>>>>>> >>>>>>> You need enough of the basics up that 2 seconds is pretty reasonable. >>>>> >>>>> >>>>> I would like to add another very interesting result I got: >>>>> disabling auto-negotiation did not reduce time, and I disabled start >>>>> networking in init.d (S40network , ifup) >>>>> and started the network instead with bootargs (ip=10.0.0.2:....) >>>>> >>>>> Is there any reason for this dependency of these methods in each other ? >>>>> >>>>> Best Regards, >>>>> ranran >>>> >>>> So as they say "it takes two to tango". You are controlling one end of >>>> the link. What is the other end of the link configured for currently? >>>> If it is set to auto-negotiate or do stuff like automdix that will add >>>> time to link establishment. >>>> >>> >>> I understand. I did tried the autonegotiation disabling in both sides, >>> I haven't tried mdix yet, but trying to see if I have mdix >>> configuration in the windows PC NIC, >>> I don't find such configuration in the adapter in the windows adapter >>> menu, so not sure if I can change it in both sides. >> >> If you don't have that level of detail on your link partner how can >> you expect the Linux configuration to be of any help at all? I was >> concerned something like this was going on. You should have a >> symmetric setup if you want any hopes of being able to actually judge >> the changes in configuration. Otherwise you might as well just give up >> on this since you can make all the changes you want in the Linux setup >> and your Windows setup could be artificially limiting your link time >> in all cases. >> >>>> I'm not sure what you are talking about in regards to dependencies. If >>>> you are asking why the network has to be running in order to configure >>>> a network interface I would think that is kind of obvious. >>>> >>>> Thanks. >>>> >>>> - Alex >>> >>> I meant dependency by the following: >>> When using auto-negotiation (in both sides of course), I got 10 >>> seconds till ping. >>> Then if I: >>> 1. Removing ifup in startup script and used bootargs (ip=) instead - no >>> change >>> 2. Adding ifup again (and removed ip from bootargs) , and disabled >>> autonegotiation - 1.5 second improvement to time till ping. >>> 3. Now I tried again to removed ifup in startup script and use ip in >>> bootargs - got imrpovement of 1 second to time till ping. >>> >>> So, I mean that I get improvement in method described in (3) above >>> only after disabling auto-negotiation. >>> I am not sure what is the reason for this imporvement and why it >>> depends on disabling auto-negotiation. >>> I have dmesg of all trials. >>> >>> Best Regards, >>> Ran >> >> For all I know it could be noise or just a coincidence. It sounds like >> your environment has too many variables you don't have control over so >> I really have no way of knowing what is going on. My advice would be >> to look at trying to get a symmetric setup where you can setup both >> ends the same way if you are going to go down this sort of path. As is >> I don't think you ever explained why you were trying to reduce your >> time to "ping", and the fact that you are as concerned about link time >> as you are has me wondering if standard Ethernet is even the right >> medium for you or if you should be looking at something like >> 1000base-T1. >> >> - Alex > > One of the application requirement is a 4 seconds boot time from power > till Ethernet is functional. > All this effort is about it , and this is what I am trying to achieve. > All you questions and wanderings are correct, and valid. I don't have > all the answers, and check with application staff if there is doubt. > My freedom of variables is not always clear and I check them with > application staff. > > The system is composed of target and PC. > For example, The PC, which is part of the system, is not the same > hardware/software as target. > Unfortunately, I can't change the PC, for example. Another thing I > can't change is the 1G requirement, although is would have saved be > important time (if I disable auto-negotiation and use 100M).
So the problem is the minimum time that most vendors are required to meet for standard 1000Base-T to link is about 4 seconds or so. That is neglecting any software overhead. As such I would say you can't rely on this for whatever it is you plan to use it for. > All trials are valid, until I get other feedback about them from > application staff. > Though I think I first should undertand if the price worth the effort > in each method I try. > For example, you mentioned 1000base-T1, does it mean I need to change > phy on board ? Yes. That is a completely different interconnect that is meant for automotive applications. It has a required link time that is less than 200ms. > I assume that might be problematic, because I use specific board with > the igb and phy inside (kontron board). > I also think that I better understand (In general) how much time it > expected to save, to know if this (or other) method worth the effort. With some tweaking you might be able to get things down to about ~2.5 seconds for just the phy. Here is a paper on the link time from Broadcom for reference (http://www.ieee802.org/3/af/public/jan02/brown_1_0102.pdf). This all assumes of course that the other end is responding as fast if not faster which isn't likely. > About the above 3 steps in my previous post, which shows how one > change had influence on another: I did try it several times, each time > I changed only one thing in the system, doing it carefully, because as > you noticed yourself, the system has a lot of variables. Yet, I got > the same results for that experiment again and again. > I analyze kernel log (dmesg) the best I can, but don't have the full > understanding why one change had more time saving then other, or why a > change has no time saving at all. > > Thank you for the time, > ranran Best of luck to you. Really it seems like you are trying to get link times that aren't really supportable in a typical Gigabit Ethernet setup. Thanks. - Alex ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
