It works now! Thanks Alexander for your help! I really appreciate it and for me it has been very great value.
On Tue, Oct 13, 2015 at 9:47 PM, Mattias Barthel <mattiasbart...@gmail.com> wrote: > Thanks again Alex, > It might be a good idea to upgrade the driver, who knows what more there > is in store for me using this one. > > Anyways, regarding the timestamping in each rx buffer I think I found it. > > From the I350 datasheet: > ----- > When the TSAUXC.Disable systime bit is cleared and the SRRCTL[n].Timestamp > bit is set to 1, packets received to the queue will be time stamped if > they meet one of the following conditions: > — Meet the criteria defined in the TSYNCRXCTL.Type field (See Section > 8.16.1 and Section 8.16.26). > — Match the value defined in one of the ETQF registers with the 1588 time > stamp bit set (See Section 7.1.2.4) if TSYNCRXCTL.Type field defines time > stamping of L2 packets. > > — Match a 2-tuple filter with the TTQF.1588 time stamp set (See Section > 7.1.2.5) if TSYNCRXCTL.Type field defines time stamping of L4 packets. > > When detecting a receive packet that should be time stamped, the I350 > will: > • Place a 64 bit timestamp, indicating the time a packet was received by > the MAC, at the beginning of the receive buffer before the received packet. > • Set the TSIP bit in the RDESC.STATUS field of the last receive > descriptor. > • Update the RDESC.Packet Type field in the last receive descriptor. Value > in this field enables identifying that this is a PTP (Precision Time > Protocol) packet (this indication is only relevant for L2 packets). > • Update the RDESC.HDR_LEN and RDESC.PKT_LEN values to include size of > timestamp. > ----- > In my driver I found this: > > if (hw->mac.type == e1000_82580) > srrctl |= E1000_SRRCTL_TIMESTAMP; > > So again, its only being done if its 82580. I will try to set this bit > tomorrow for the I350 before trying any driver upgrades which can be quite > painful. > > Thank again! > > On Tue, Oct 13, 2015 at 8:45 PM, Alexander Duyck < > alexander.du...@gmail.com> wrote: > >> You can probably just download and use the latest driver from: >> http://sourceforge.net/projects/e1000/files/igb%20stable/ >> >> It should be able to build on a 3.0.8 kernel provided nothing has been >> back-ported that would change the APIs. >> >> - Alex >> >> On 10/13/2015 10:26 AM, Mattias Barthel wrote: >> >>> Thanks a lot Alex, there might still be some hope then. >>> Unfortunately Im bound to using the linux kernel 3.0.8. What out-of-tree >>> driver can I use with that, please? >>> >>> On Tuesday, October 13, 2015, Alexander Duyck <alexander.du...@gmail.com >>> <mailto:alexander.du...@gmail.com>> wrote: >>> >>> I don't know if it is available in that driver or not, but there >>> is an option in later igb drivers to support a time stamp being >>> written with the packet data. Based on the fact that you are >>> already seeing issues, you might want to try getting the >>> out-of-tree driver from e1000.sf.net <http://e1000.sf.net> and >>> >>> seeing if you can use that with the time stamp in packet option >>> that is enabled when you select to timestamp all incoming Rx packets. >>> >>> - Alex >>> >>> On 10/13/2015 10:01 AM, Mattias Barthel wrote: >>> >>> Yes, setting the cycles.shift to the same as on the 82580 and >>> also shifting the reads accordingly everything works. The sync >>> of shhwtstamps->syststamp too. >>> >>> My driver version is 3.0.6-k2 and linux is 3.0.8. >>> >>> Although this might not be very suited to my needs. >>> The incoming buffers wont be timestamped if you cant keep up >>> with reading the timestamping registers as I understand it. >>> This might be perfectly fine in the case of PTP where the >>> packet rate is fairly low. In my case Im looking at some >>> 80kpps that all would need to be timestamped. >>> There is a comment about this in the driver: >>> >>> *//* If this bit is set, then the RX registers contain the >>> time stamp. No/**/ other packet will be time stamped until we >>> read these registers, so/* */read the registers to make them >>> available again. Because only one/**/ packet can be time >>> stamped at a time, we know that the register/* */values must >>> belong to this one here and therefore we don't need to/**/ >>> compare any of the additional attributes stored for it./* */If >>> nothing went wrong, then it should have a shared tx_flags that >>> we/**/ can turn into a skb_shared_hwtstamps. *//* >>> *//* >>> *//* >>> >>> On Tuesday, October 13, 2015, Alexander Duyck >>> <alexander.du...@gmail.com <mailto:alexander.du...@gmail.com>> >>> wrote: >>> >>> I'm assuming you are using either an older kernel or an >>> out-of-tree driver do I have that right? I ask because >>> this code >>> doesn't resemble what is is currently in the latest Linux >>> kernel. >>> >>> Comments on what you have found are inline below. >>> >>> - Alex >>> >>> On 10/13/2015 12:53 AM, Mattias Barthel wrote: >>> >>> Hi, I think I managed to get around this problem. >>> >>> >>> in function init_hw_timer(): >>> ---- >>> case e1000_i350: >>> case e1000_82580: >>> >>> /* >>> * The 82580 timesync updates the system timer every >>> 8ns by 8ns >>> * and the value cannot be shifted. Instead we need to >>> shift >>> * the registers to generate a 64bit timer value. As a >>> result >>> * SYSTIMR/L/H, TXSTMPL/H, RXSTMPL/H all have to be >>> shifted by >>> * 24 in order to generate a larger value for >>> synchronization. >>> */ >>> adapter->cycles.shift = IGB_82580_TSYNC_SHIFT; >>> >>> ---- >>> >>> I have the I350 and I dont think this cycles shift was >>> meant >>> for my adapter. >>> IGB_82580_TSYNC_SHIFT is 24. >>> >>> Also when doing igb_read_clock, >>> this shifting is applied only for the 82580 and I guess >>> it >>> should not have been set in the first place for the I350: >>> -------- >>> /* >>> * The timestamp latches on lowest register read. For >>> the 82580 >>> * the lowest register is SYSTIMR instead of SYSTIML. >>> However >>> we never >>> * adjusted TIMINCA so SYSTIMR will just read as all 0s so >>> ignore it. >>> */ >>> if (hw->mac.type == e1000_82580) { >>> stamp = rd32(E1000_SYSTIMR) >> 8; >>> shift = IGB_82580_TSYNC_SHIFT; >>> } >>> >>> stamp |= (u64)rd32(E1000_SYSTIML) << shift; >>> stamp |= (u64)rd32(E1000_SYSTIMH) << (shift + 32); >>> ------------- >>> >>> >>> This is a bug in the code. The i350 should be shifted by >>> the same >>> value as the 82580. Then the result would be the same as >>> if you >>> had set the shift to 0. As I recall the reasoning behind the >>> change was to make use of as many bits of the time as >>> possible, >>> and on 82580 and newer the SYSTIMH portion of the clock only >>> contained 8 bits of actual clock data. >>> >>> Without that change I believe there is a risk of the >>> counter being >>> considered to have cycled too fast and causing errors as >>> the cycle >>> counter will roll over at 40 bits instead of 64. The >>> alternative >>> would be to update the cycle counter you are using so that >>> it is >>> aware that it is limited to 40 bits and then use an shift >>> of 0. >>> >>> Setting adapter->cycles.shift to 0 makes the clock run at >>> about the same speed as the system clock. >>> >>> Next problem I have is that the syststamp is not syncing >>> correctly against the system clock. >>> >>> Any help is much appreciated. >>> >>> Regards, >>> >>> Mattias >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Mon, Oct 12, 2015 at 5:04 PM, Mattias Barthel >>> <mattiasbart...@gmail.com >>> <mailto:mattiasbart...@gmail.com>> >>> wrote: >>> >>> Hi again, >>> >>> Started to try and timestamp all packets in the I350. >>> >>> Driver version: >>> >>> driver: igb >>> version: 3.0.6-k2 >>> firmware-version: 0.147-0 >>> >>> Doing an ioctl that turns on timestamping for all >>> packets. >>> >>> hwconfig.tx_type = HWTSTAMP_TX_OFF; >>> hwconfig.rx_filter = HWTSTAMP_FILTER_ALL; >>> hwconfig_requested = hwconfig; >>> ioctl(sock, SIOCSHWTSTAMP, &hwtstamp) >>> >>> The strangest thing is that although it works to >>> read the >>> timestamps the clock seems to advance very slowly >>> and/or >>> incorrectly. >>> >>> I just printout the HW timestamps alongside the >>> systemclock and >>> the hw stamping is advancing the nanoseconds when the >>> systemclock >>> is advancing us and seconds. >>> >>> Is there a know bug in this version of the driver >>> that maybe >>> initializes the clock incorrectly or maybe reads >>> incorrectly from >>> the clock? >>> >>> >>> [ 63.004116] igb_rx_hwtstamp >>> [ 63.004684] hwtstamp: 1444656871372825302, >>> syststamp: >>> 1444656871372845807 >>> [ 63.005721] system timestamp: tv_sec: 1444656793, >>> tv_usec: 683825 >>> [ 63.006631] sys hw timestamp: tstamp: >>> 1444656871372845807, >>> tv_sec: 1444656871, tv_usec: 372845 >>> [ 63.008044] hw timestamp: tstamp: >>> 1444656871372825302, >>> tv_sec: >>> 1444656871, tv_usec: 372825 >>> [ 63.009395] >>> [ 63.104111] igb_rx_hwtstamp >>> [ 63.104704] hwtstamp: 1444656871372825308, >>> syststamp: >>> 1444656871372845813 >>> [ 63.105747] system timestamp: tv_sec: 1444656793, >>> tv_usec: 783851 >>> [ 63.106669] sys hw timestamp: tstamp: >>> 1444656871372845813, >>> tv_sec: 1444656871, tv_usec: 372845 >>> [ 63.108075] hw timestamp: tstamp: >>> 1444656871372825308, >>> tv_sec: >>> 1444656871, tv_usec: 372825 >>> [ 63.109428] >>> [ 63.204109] igb_rx_hwtstamp >>> [ 63.204683] hwtstamp: 1444656871372825314, >>> syststamp: >>> 1444656871372845819 >>> [ 63.206018] system timestamp: tv_sec: 1444656793, >>> tv_usec: 884122 >>> [ 63.207242] sys hw timestamp: tstamp: >>> 1444656871372845819, >>> tv_sec: 1444656871, tv_usec: 372845 >>> [ 63.209096] hw timestamp: tstamp: >>> 1444656871372825314, >>> tv_sec: >>> 1444656871, tv_usec: 372825 >>> [ 63.210897] >>> [ 63.304068] igb_rx_hwtstamp >>> [ 63.304849] hwtstamp: 1444656871372825320, >>> syststamp: >>> 1444656871372845825 >>> [ 63.306176] system timestamp: tv_sec: 1444656793, >>> tv_usec: 984280 >>> [ 63.307391] sys hw timestamp: tstamp: >>> 1444656871372845825, >>> tv_sec: 1444656871, tv_usec: 372845 >>> [ 63.309252] hw timestamp: tstamp: >>> 1444656871372825320, >>> tv_sec: >>> 1444656871, tv_usec: 372825 >>> [ 63.311032] >>> [ 63.404107] igb_rx_hwtstamp >>> [ 63.404659] hwtstamp: 1444656871372825326, >>> syststamp: >>> 1444656871372845831 >>> [ 63.405695] system timestamp: tv_sec: 1444656794, >>> tv_usec: 83798 >>> [ 63.406635] sys hw timestamp: tstamp: >>> 1444656871372845831, >>> tv_sec: 1444656871, tv_usec: 372845 >>> [ 63.408044] hw timestamp: tstamp: >>> 1444656871372825326, >>> tv_sec: >>> 1444656871, tv_usec: 372825 >>> [ 63.409405] >>> [ 63.504084] igb_rx_hwtstamp >>> [ 63.504670] hwtstamp: 1444656871372825332, >>> syststamp: >>> 1444656871372845837 >>> [ 63.505715] system timestamp: tv_sec: 1444656794, >>> tv_usec: 183819 >>> [ 63.506657] sys hw timestamp: tstamp: >>> 1444656871372845837, >>> tv_sec: 1444656871, tv_usec: 372845 >>> [ 63.508090] hw timestamp: tstamp: >>> 1444656871372825332, >>> tv_sec: >>> 1444656871, tv_usec: 372825 >>> >>> >>> Any help will be very much appreciated. >>> >>> Best regards, >>> >>> Mattias >>> >>> >>> >>> >>> >>> >>> >>> >>> On Thu, Oct 8, 2015 at 6:01 PM, Alexander Duyck >>> <alexander.du...@gmail.com >>> <mailto:alexander.du...@gmail.com>> wrote: >>> >>> The i350 should support timestamping all >>> packets as I >>> recall. You should probably look into getting the >>> linuxptp package and >>> seeing if you can make use of the hwstamp_ctl >>> command to >>> enable timestamping for all incoming packets >>> on the >>> adapter. >>> >>> - Alex >>> >>> On 10/08/2015 12:33 AM, Mattias Barthel wrote: >>> >>> Hi >>> thanks for answer. >>> >>> Unfortunately, the udp/ip stream just as >>> the cfm >>> stream is >>> against only one peer. >>> Furthemore on the guest side I have only >>> one vcpu >>> tied to >>> two physical siblings to minimize L2 cache >>> misses >>> on ISRs >>> due to cache coherence. >>> >>> I had an idea this morning, maybe I can >>> make use >>> of the >>> ptp hw timestaming for this? This should give >>> better accuracy. >>> Is this possible or does it only work for >>> ethertype ptp? >>> >>> On Thursday, October 8, 2015, Alexander Duyck >>> <alexander.du...@gmail.com >>> <mailto:alexander.du...@gmail.com> >>> <mailto:alexander.du...@gmail.com >>> <mailto:alexander.du...@gmail.com>>> wrote: >>> >>> The difference could be Receive Side >>> Scaling >>> (RSS). The UDP/IP >>> packets >>> can be load balanced across all queues >>> via RSS >>> based >>> on a hash of the >>> source and destination addresses. So >>> if you have >>> multiple sources >>> generating these packets, or they are >>> being >>> received >>> on multiple IP >>> addresses then it is possible the load is >>> being spread >>> out over >>> multiple >>> CPUs. Your CFM packets are on an >>> Ethertype >>> other than >>> IPv4 or IPv6 so >>> all packets will end up on queue 0 if >>> I am not >>> mistaken. >>> >>> - Alex >>> >>> On 10/07/2015 10:21 PM, Mattias >>> Barthel wrote: >>> > Hi again! >>> > >>> > I realise my message might not have >>> been so >>> clear. >>> > >>> > I will try to clarify. I am doing PM >>> with >>> software >>> timestamping. >>> > >>> > igb versions: >>> > version: 3.0.6-k2 >>> > firmware-version: 0.147-0 >>> > >>> > I do timestamping (NTP format) of the >>> packets in the >>> same place >>> for all >>> > protocols. (in igb_clean_rx_irq_adv()). >>> > >>> > For these non ip packets like IEEE >>> 801.2ag CFM ETH-DM >>> (ethertype 0x8902) I >>> > get quite a lot worse accuracy in the >>> timestamping. >>> > >>> > For UDP/IP TWAMP I get for 10k pps a >>> max >>> delay of >>> around 140us >>> back to back. >>> > For CFM ETH DM I get for the same >>> packet >>> rate and >>> packet size a >>> max delay >>> > of around 900us. >>> > >>> > So my hypothesis was: >>> > Could FW be putting these L2 packets >>> in another >>> queue with different >>> > characteristics? >>> > >>> > I tried to add a ET-type filter as >>> written in my >>> previous mail >>> but it >>> > showed no difference. >>> > >>> > >>> > Thanks for any help given! >>> > >>> > >>> > >>> > On Mon, Oct 5, 2015 at 2:19 PM, >>> Mattias Barthel >>> <mattiasbart...@gmail.com >>> <mailto:mattiasbart...@gmail.com> >>> <javascript:;>> >>> >>> > wrote: >>> > >>> >> Greetings, >>> >> >>> >> Im getting quite bad latency when >>> using the igb >>> driver on i350 >>> on linux >>> >> regarding >>> >> ETH CFM packets. This compared to >>> TWAMP >>> (IPv4 UDP >>> packets). >>> >> The environment is KVM with the >>> i350 devices in >>> PCI-passthrough. >>> >> >>> >> So i figured I add an own rx-queue >>> (filter) for >>> those types of >>> ethernet >>> >> protocol packets. >>> >> >>> >> This is what i added: >>> >> >>> >> >>> >> wr32(E1000_ETQF(4), >>> >> (1 << 31) | /* queue enable */ >>> >> (E1000_ETQF_FILTER_ENABLE) | /* enable >>> filter */ >>> >> (1 << 29) | /* enable immediate >>> interrupt */ >>> >> (0x4 << 16) | /* queue no. 4 */ >>> >> (ETH_P_8021AG)); /* 0x8902 >>> CFM eth >>> protocol type */ >>> >> >>> >> >>> >> ETH_P_8021A is 0x8902 >>> >> >>> >> >>> >> >>> >> Is this a correct/good approach? >>> >> >>> >> Thanks in advance! >>> >> >>> >> -- >>> >> Mattias Barthel >>> >> >>> > >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> _______________________________________________ >>> E1000-devel mailing list >>> E1000-devel@lists.sourceforge.net >>> <mailto:E1000-devel@lists.sourceforge.net> >>> <javascript:;> >>> https://lists.sourceforge.net/lists/listinfo/e1000-devel >>> To learn more about Intel® >>> Ethernet, visit >>> http://communities.intel.com/community/wired >>> >>> >>> Avis de confidentialité >>> >>> Les informations contenues dans le présent >>> message >>> et dans >>> toute pièce qui lui est jointe sont >>> confidentielles et >>> peuvent être protégées par le secret >>> professionnel. Ces >>> informations sont à l’usage exclusif de >>> son ou de ses >>> destinataires. Si vous recevez ce message >>> par erreur, >>> veuillez s’il vous plait communiquer >>> immédiatement >>> avec >>> l’expéditeur et en détruire tout >>> exemplaire. De >>> plus, il >>> vous est strictement interdit de le >>> divulguer, de le >>> distribuer ou de le reproduire sans >>> l’autorisation de >>> l’expéditeur. Merci. >>> >>> Confidentiality notice >>> >>> This e-mail message and any attachment >>> hereto contain >>> confidential information which may be >>> privileged >>> and which >>> is intended for the exclusive use of its >>> addressee(s). If >>> you receive this message in error, please >>> inform >>> sender >>> immediately and destroy any copy thereof. >>> Furthermore, any >>> disclosure, distribution or copying of this >>> message and/or >>> any attachment hereto without the consent >>> of the >>> sender is >>> strictly prohibited. Thank you. >>> >>> >>> >>> >>> >>> -- Mattias Barthel >>> >>> >>> >>> >>> -- Mattias Barthel >>> >>> >>> >>> Avis de confidentialité >>> >>> Les informations contenues dans le présent message et dans >>> toute pièce qui lui est jointe sont confidentielles et peuvent >>> être protégées par le secret professionnel. Ces informations >>> sont à l’usage exclusif de son ou de ses destinataires. Si >>> vous recevez ce message par erreur, veuillez s’il vous plait >>> communiquer immédiatement avec l’expéditeur et en détruire >>> tout exemplaire. De plus, il vous est strictement interdit de >>> le divulguer, de le distribuer ou de le reproduire sans >>> l’autorisation de l’expéditeur. Merci. >>> >>> Confidentiality notice >>> >>> This e-mail message and any attachment hereto contain >>> confidential information which may be privileged and which is >>> intended for the exclusive use of its addressee(s). If you >>> receive this message in error, please inform sender >>> immediately and destroy any copy thereof. Furthermore, any >>> disclosure, distribution or copying of this message and/or any >>> attachment hereto without the consent of the sender is >>> strictly prohibited. Thank you. >>> >>> >>> >>> Avis de confidentialité >>> >>> Les informations contenues dans le présent message et dans toute pièce >>> qui lui est jointe sont confidentielles et peuvent être protégées par le >>> secret professionnel. Ces informations sont à l’usage exclusif de son ou de >>> ses destinataires. Si vous recevez ce message par erreur, veuillez s’il >>> vous plait communiquer immédiatement avec l’expéditeur et en détruire tout >>> exemplaire. De plus, il vous est strictement interdit de le divulguer, de >>> le distribuer ou de le reproduire sans l’autorisation de l’expéditeur. >>> Merci. >>> >>> Confidentiality notice >>> >>> This e-mail message and any attachment hereto contain confidential >>> information which may be privileged and which is intended for the exclusive >>> use of its addressee(s). If you receive this message in error, please >>> inform sender immediately and destroy any copy thereof. Furthermore, any >>> disclosure, distribution or copying of this message and/or any attachment >>> hereto without the consent of the sender is strictly prohibited. Thank you. >>> >>> >> > > > -- > Mattias Barthel > -- Mattias Barthel
------------------------------------------------------------------------------
_______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
