I don't know if it is available in that driver or not, but there is an option in later igb drivers to support a time stamp being written with the packet data. Based on the fact that you are already seeing issues, you might want to try getting the out-of-tree driver from e1000.sf.net and seeing if you can use that with the time stamp in packet option that is enabled when you select to timestamp all incoming Rx packets.
- Alex On 10/13/2015 10:01 AM, Mattias Barthel wrote: > Yes, setting the cycles.shift to the same as on the 82580 and also > shifting the reads accordingly everything works. The sync of > shhwtstamps->syststamp too. > > My driver version is 3.0.6-k2 and linux is 3.0.8. > > Although this might not be very suited to my needs. > The incoming buffers wont be timestamped if you cant keep up with > reading the timestamping registers as I understand it. > This might be perfectly fine in the case of PTP where the packet rate > is fairly low. In my case Im looking at some 80kpps that all would > need to be timestamped. > There is a comment about this in the driver: > > *//* If this bit is set, then the RX registers contain the time stamp. > No/**/ other packet will be time stamped until we read these > registers, so/* */read the registers to make them available again. > Because only one/**/ packet can be time stamped at a time, we know > that the register/* */values must belong to this one here and > therefore we don't need to/**/ compare any of the additional > attributes stored for it./* */If nothing went wrong, then it should > have a shared tx_flags that we/**/ can turn into a > skb_shared_hwtstamps. *//* > *//* > *//* > > On Tuesday, October 13, 2015, Alexander Duyck > <alexander.du...@gmail.com <mailto:alexander.du...@gmail.com>> wrote: > > I'm assuming you are using either an older kernel or an > out-of-tree driver do I have that right? I ask because this code > doesn't resemble what is is currently in the latest Linux kernel. > > Comments on what you have found are inline below. > > - Alex > > On 10/13/2015 12:53 AM, Mattias Barthel wrote: > > Hi, I think I managed to get around this problem. > > > in function init_hw_timer(): > ---- > case e1000_i350: > case e1000_82580: > > /* > * The 82580 timesync updates the system timer every 8ns by 8ns > * and the value cannot be shifted. Instead we need to shift > * the registers to generate a 64bit timer value. As a result > * SYSTIMR/L/H, TXSTMPL/H, RXSTMPL/H all have to be shifted by > * 24 in order to generate a larger value for synchronization. > */ > adapter->cycles.shift = IGB_82580_TSYNC_SHIFT; > > ---- > > I have the I350 and I dont think this cycles shift was meant > for my adapter. > IGB_82580_TSYNC_SHIFT is 24. > > Also when doing igb_read_clock, > this shifting is applied only for the 82580 and I guess it > should not have been set in the first place for the I350: > -------- > /* > * The timestamp latches on lowest register read. For the 82580 > * the lowest register is SYSTIMR instead of SYSTIML. However > we never > * adjusted TIMINCA so SYSTIMR will just read as all 0s so > ignore it. > */ > if (hw->mac.type == e1000_82580) { > stamp = rd32(E1000_SYSTIMR) >> 8; > shift = IGB_82580_TSYNC_SHIFT; > } > > stamp |= (u64)rd32(E1000_SYSTIML) << shift; > stamp |= (u64)rd32(E1000_SYSTIMH) << (shift + 32); > ------------- > > > This is a bug in the code. The i350 should be shifted by the same > value as the 82580. Then the result would be the same as if you > had set the shift to 0. As I recall the reasoning behind the > change was to make use of as many bits of the time as possible, > and on 82580 and newer the SYSTIMH portion of the clock only > contained 8 bits of actual clock data. > > Without that change I believe there is a risk of the counter being > considered to have cycled too fast and causing errors as the cycle > counter will roll over at 40 bits instead of 64. The alternative > would be to update the cycle counter you are using so that it is > aware that it is limited to 40 bits and then use an shift of 0. > > Setting adapter->cycles.shift to 0 makes the clock run at > about the same speed as the system clock. > > Next problem I have is that the syststamp is not syncing > correctly against the system clock. > > Any help is much appreciated. > > Regards, > > Mattias > > > > > > > > > > > > > On Mon, Oct 12, 2015 at 5:04 PM, Mattias Barthel > <mattiasbart...@gmail.com <mailto:mattiasbart...@gmail.com>> > wrote: > > Hi again, > > Started to try and timestamp all packets in the I350. > > Driver version: > > driver: igb > version: 3.0.6-k2 > firmware-version: 0.147-0 > > Doing an ioctl that turns on timestamping for all packets. > > hwconfig.tx_type = HWTSTAMP_TX_OFF; > hwconfig.rx_filter = HWTSTAMP_FILTER_ALL; > hwconfig_requested = hwconfig; > ioctl(sock, SIOCSHWTSTAMP, &hwtstamp) > > The strangest thing is that although it works to read the > timestamps the clock seems to advance very slowly and/or > incorrectly. > > I just printout the HW timestamps alongside the > systemclock and > the hw stamping is advancing the nanoseconds when the > systemclock > is advancing us and seconds. > > Is there a know bug in this version of the driver that maybe > initializes the clock incorrectly or maybe reads > incorrectly from > the clock? > > > [ 63.004116] igb_rx_hwtstamp > [ 63.004684] hwtstamp: 1444656871372825302, syststamp: > 1444656871372845807 > [ 63.005721] system timestamp: tv_sec: 1444656793, > tv_usec: 683825 > [ 63.006631] sys hw timestamp: tstamp: 1444656871372845807, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.008044] hw timestamp: tstamp: 1444656871372825302, > tv_sec: > 1444656871, tv_usec: 372825 > [ 63.009395] > [ 63.104111] igb_rx_hwtstamp > [ 63.104704] hwtstamp: 1444656871372825308, syststamp: > 1444656871372845813 > [ 63.105747] system timestamp: tv_sec: 1444656793, > tv_usec: 783851 > [ 63.106669] sys hw timestamp: tstamp: 1444656871372845813, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.108075] hw timestamp: tstamp: 1444656871372825308, > tv_sec: > 1444656871, tv_usec: 372825 > [ 63.109428] > [ 63.204109] igb_rx_hwtstamp > [ 63.204683] hwtstamp: 1444656871372825314, syststamp: > 1444656871372845819 > [ 63.206018] system timestamp: tv_sec: 1444656793, > tv_usec: 884122 > [ 63.207242] sys hw timestamp: tstamp: 1444656871372845819, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.209096] hw timestamp: tstamp: 1444656871372825314, > tv_sec: > 1444656871, tv_usec: 372825 > [ 63.210897] > [ 63.304068] igb_rx_hwtstamp > [ 63.304849] hwtstamp: 1444656871372825320, syststamp: > 1444656871372845825 > [ 63.306176] system timestamp: tv_sec: 1444656793, > tv_usec: 984280 > [ 63.307391] sys hw timestamp: tstamp: 1444656871372845825, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.309252] hw timestamp: tstamp: 1444656871372825320, > tv_sec: > 1444656871, tv_usec: 372825 > [ 63.311032] > [ 63.404107] igb_rx_hwtstamp > [ 63.404659] hwtstamp: 1444656871372825326, syststamp: > 1444656871372845831 > [ 63.405695] system timestamp: tv_sec: 1444656794, > tv_usec: 83798 > [ 63.406635] sys hw timestamp: tstamp: 1444656871372845831, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.408044] hw timestamp: tstamp: 1444656871372825326, > tv_sec: > 1444656871, tv_usec: 372825 > [ 63.409405] > [ 63.504084] igb_rx_hwtstamp > [ 63.504670] hwtstamp: 1444656871372825332, syststamp: > 1444656871372845837 > [ 63.505715] system timestamp: tv_sec: 1444656794, > tv_usec: 183819 > [ 63.506657] sys hw timestamp: tstamp: 1444656871372845837, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.508090] hw timestamp: tstamp: 1444656871372825332, > tv_sec: > 1444656871, tv_usec: 372825 > > > Any help will be very much appreciated. > > Best regards, > > Mattias > > > > > > > > > On Thu, Oct 8, 2015 at 6:01 PM, Alexander Duyck > <alexander.du...@gmail.com > <mailto:alexander.du...@gmail.com>> wrote: > > The i350 should support timestamping all packets as I > recall. You should probably look into getting the > linuxptp package and > seeing if you can make use of the hwstamp_ctl command to > enable timestamping for all incoming packets on the > adapter. > > - Alex > > On 10/08/2015 12:33 AM, Mattias Barthel wrote: > > Hi > thanks for answer. > > Unfortunately, the udp/ip stream just as the cfm > stream is > against only one peer. > Furthemore on the guest side I have only one vcpu > tied to > two physical siblings to minimize L2 cache misses > on ISRs > due to cache coherence. > > I had an idea this morning, maybe I can make use > of the > ptp hw timestaming for this? This should give > better accuracy. > Is this possible or does it only work for > ethertype ptp? > > On Thursday, October 8, 2015, Alexander Duyck > <alexander.du...@gmail.com > <mailto:alexander.du...@gmail.com> > <mailto:alexander.du...@gmail.com > <mailto:alexander.du...@gmail.com>>> wrote: > > The difference could be Receive Side Scaling > (RSS). The UDP/IP > packets > can be load balanced across all queues via RSS > based > on a hash of the > source and destination addresses. So if you have > multiple sources > generating these packets, or they are being > received > on multiple IP > addresses then it is possible the load is > being spread > out over > multiple > CPUs. Your CFM packets are on an Ethertype > other than > IPv4 or IPv6 so > all packets will end up on queue 0 if I am not > mistaken. > > - Alex > > On 10/07/2015 10:21 PM, Mattias Barthel wrote: > > Hi again! > > > > I realise my message might not have been so > clear. > > > > I will try to clarify. I am doing PM with > software > timestamping. > > > > igb versions: > > version: 3.0.6-k2 > > firmware-version: 0.147-0 > > > > I do timestamping (NTP format) of the > packets in the > same place > for all > > protocols. (in igb_clean_rx_irq_adv()). > > > > For these non ip packets like IEEE 801.2ag > CFM ETH-DM > (ethertype 0x8902) I > > get quite a lot worse accuracy in the > timestamping. > > > > For UDP/IP TWAMP I get for 10k pps a max > delay of > around 140us > back to back. > > For CFM ETH DM I get for the same packet > rate and > packet size a > max delay > > of around 900us. > > > > So my hypothesis was: > > Could FW be putting these L2 packets in another > queue with different > > characteristics? > > > > I tried to add a ET-type filter as written in my > previous mail > but it > > showed no difference. > > > > > > Thanks for any help given! > > > > > > > > On Mon, Oct 5, 2015 at 2:19 PM, Mattias Barthel > <mattiasbart...@gmail.com > <mailto:mattiasbart...@gmail.com> <javascript:;>> > > > wrote: > > > >> Greetings, > >> > >> Im getting quite bad latency when using the igb > driver on i350 > on linux > >> regarding > >> ETH CFM packets. This compared to TWAMP > (IPv4 UDP > packets). > >> The environment is KVM with the i350 devices in > PCI-passthrough. > >> > >> So i figured I add an own rx-queue (filter) for > those types of > ethernet > >> protocol packets. > >> > >> This is what i added: > >> > >> > >> wr32(E1000_ETQF(4), > >> (1 << 31) | /* queue enable */ > >> (E1000_ETQF_FILTER_ENABLE) | /* enable > filter */ > >> (1 << 29) | /* enable immediate > interrupt */ > >> (0x4 << 16) | /* queue no. 4 */ > >> (ETH_P_8021AG)); /* 0x8902 CFM eth > protocol type */ > >> > >> > >> ETH_P_8021A is 0x8902 > >> > >> > >> > >> Is this a correct/good approach? > >> > >> Thanks in advance! > >> > >> -- > >> Mattias Barthel > >> > > > > > > ------------------------------------------------------------------------------ > _______________________________________________ > E1000-devel mailing list > E1000-devel@lists.sourceforge.net > <mailto:E1000-devel@lists.sourceforge.net> > <javascript:;> > https://lists.sourceforge.net/lists/listinfo/e1000-devel > To learn more about Intel® Ethernet, visit > http://communities.intel.com/community/wired > > > Avis de confidentialité > > Les informations contenues dans le présent message > et dans > toute pièce qui lui est jointe sont confidentielles et > peuvent être protégées par le secret > professionnel. Ces > informations sont à l’usage exclusif de son ou de ses > destinataires. Si vous recevez ce message par erreur, > veuillez s’il vous plait communiquer immédiatement > avec > l’expéditeur et en détruire tout exemplaire. De > plus, il > vous est strictement interdit de le divulguer, de le > distribuer ou de le reproduire sans l’autorisation de > l’expéditeur. Merci. > > Confidentiality notice > > This e-mail message and any attachment hereto contain > confidential information which may be privileged > and which > is intended for the exclusive use of its > addressee(s). If > you receive this message in error, please inform > sender > immediately and destroy any copy thereof. > Furthermore, any > disclosure, distribution or copying of this > message and/or > any attachment hereto without the consent of the > sender is > strictly prohibited. Thank you. > > > > > > -- Mattias Barthel > > > > > -- > Mattias Barthel > > > > Avis de confidentialité > > Les informations contenues dans le présent message et dans toute pièce > qui lui est jointe sont confidentielles et peuvent être protégées par > le secret professionnel. Ces informations sont à l’usage exclusif de > son ou de ses destinataires. Si vous recevez ce message par erreur, > veuillez s’il vous plait communiquer immédiatement avec l’expéditeur > et en détruire tout exemplaire. De plus, il vous est strictement > interdit de le divulguer, de le distribuer ou de le reproduire sans > l’autorisation de l’expéditeur. Merci. > > Confidentiality notice > > This e-mail message and any attachment hereto contain confidential > information which may be privileged and which is intended for the > exclusive use of its addressee(s). If you receive this message in > error, please inform sender immediately and destroy any copy thereof. > Furthermore, any disclosure, distribution or copying of this message > and/or any attachment hereto without the consent of the sender is > strictly prohibited. Thank you. > ------------------------------------------------------------------------------ _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
