Thanks again Alex,
It might be a good idea to upgrade the driver, who knows what more there is
in store for me using this one.
Anyways, regarding the timestamping in each rx buffer I think I found it.
>From the I350 datasheet:
-----
When the TSAUXC.Disable systime bit is cleared and the SRRCTL[n].Timestamp
bit is set to 1, packets received to the queue will be time stamped if they
meet one of the following conditions:
— Meet the criteria defined in the TSYNCRXCTL.Type field (See Section
8.16.1 and Section 8.16.26).
— Match the value defined in one of the ETQF registers with the 1588 time
stamp bit set (See Section 7.1.2.4) if TSYNCRXCTL.Type field defines time
stamping of L2 packets.
— Match a 2-tuple filter with the TTQF.1588 time stamp set (See Section
7.1.2.5) if TSYNCRXCTL.Type field defines time stamping of L4 packets.
When detecting a receive packet that should be time stamped, the I350 will:
• Place a 64 bit timestamp, indicating the time a packet was received by
the MAC, at the beginning of the receive buffer before the received packet.
• Set the TSIP bit in the RDESC.STATUS field of the last receive
descriptor.
• Update the RDESC.Packet Type field in the last receive descriptor. Value
in this field enables identifying that this is a PTP (Precision Time
Protocol) packet (this indication is only relevant for L2 packets).
• Update the RDESC.HDR_LEN and RDESC.PKT_LEN values to include size of
timestamp.
-----
In my driver I found this:
if (hw->mac.type == e1000_82580)
srrctl |= E1000_SRRCTL_TIMESTAMP;
So again, its only being done if its 82580. I will try to set this bit
tomorrow for the I350 before trying any driver upgrades which can be quite
painful.
Thank again!
On Tue, Oct 13, 2015 at 8:45 PM, Alexander Duyck <[email protected]>
wrote:
> You can probably just download and use the latest driver from:
> http://sourceforge.net/projects/e1000/files/igb%20stable/
>
> It should be able to build on a 3.0.8 kernel provided nothing has been
> back-ported that would change the APIs.
>
> - Alex
>
> On 10/13/2015 10:26 AM, Mattias Barthel wrote:
>
>> Thanks a lot Alex, there might still be some hope then.
>> Unfortunately Im bound to using the linux kernel 3.0.8. What out-of-tree
>> driver can I use with that, please?
>>
>> On Tuesday, October 13, 2015, Alexander Duyck <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> I don't know if it is available in that driver or not, but there
>> is an option in later igb drivers to support a time stamp being
>> written with the packet data. Based on the fact that you are
>> already seeing issues, you might want to try getting the
>> out-of-tree driver from e1000.sf.net <http://e1000.sf.net> and
>>
>> seeing if you can use that with the time stamp in packet option
>> that is enabled when you select to timestamp all incoming Rx packets.
>>
>> - Alex
>>
>> On 10/13/2015 10:01 AM, Mattias Barthel wrote:
>>
>> Yes, setting the cycles.shift to the same as on the 82580 and
>> also shifting the reads accordingly everything works. The sync
>> of shhwtstamps->syststamp too.
>>
>> My driver version is 3.0.6-k2 and linux is 3.0.8.
>>
>> Although this might not be very suited to my needs.
>> The incoming buffers wont be timestamped if you cant keep up
>> with reading the timestamping registers as I understand it.
>> This might be perfectly fine in the case of PTP where the
>> packet rate is fairly low. In my case Im looking at some
>> 80kpps that all would need to be timestamped.
>> There is a comment about this in the driver:
>>
>> *//* If this bit is set, then the RX registers contain the
>> time stamp. No/**/ other packet will be time stamped until we
>> read these registers, so/* */read the registers to make them
>> available again. Because only one/**/ packet can be time
>> stamped at a time, we know that the register/* */values must
>> belong to this one here and therefore we don't need to/**/
>> compare any of the additional attributes stored for it./* */If
>> nothing went wrong, then it should have a shared tx_flags that
>> we/**/ can turn into a skb_shared_hwtstamps. *//*
>> *//*
>> *//*
>>
>> On Tuesday, October 13, 2015, Alexander Duyck
>> <[email protected] <mailto:[email protected]>>
>> wrote:
>>
>> I'm assuming you are using either an older kernel or an
>> out-of-tree driver do I have that right? I ask because
>> this code
>> doesn't resemble what is is currently in the latest Linux
>> kernel.
>>
>> Comments on what you have found are inline below.
>>
>> - Alex
>>
>> On 10/13/2015 12:53 AM, Mattias Barthel wrote:
>>
>> Hi, I think I managed to get around this problem.
>>
>>
>> in function init_hw_timer():
>> ----
>> case e1000_i350:
>> case e1000_82580:
>>
>> /*
>> * The 82580 timesync updates the system timer every
>> 8ns by 8ns
>> * and the value cannot be shifted. Instead we need to
>> shift
>> * the registers to generate a 64bit timer value. As a
>> result
>> * SYSTIMR/L/H, TXSTMPL/H, RXSTMPL/H all have to be
>> shifted by
>> * 24 in order to generate a larger value for
>> synchronization.
>> */
>> adapter->cycles.shift = IGB_82580_TSYNC_SHIFT;
>>
>> ----
>>
>> I have the I350 and I dont think this cycles shift was
>> meant
>> for my adapter.
>> IGB_82580_TSYNC_SHIFT is 24.
>>
>> Also when doing igb_read_clock,
>> this shifting is applied only for the 82580 and I guess it
>> should not have been set in the first place for the I350:
>> --------
>> /*
>> * The timestamp latches on lowest register read. For
>> the 82580
>> * the lowest register is SYSTIMR instead of SYSTIML.
>> However
>> we never
>> * adjusted TIMINCA so SYSTIMR will just read as all 0s so
>> ignore it.
>> */
>> if (hw->mac.type == e1000_82580) {
>> stamp = rd32(E1000_SYSTIMR) >> 8;
>> shift = IGB_82580_TSYNC_SHIFT;
>> }
>>
>> stamp |= (u64)rd32(E1000_SYSTIML) << shift;
>> stamp |= (u64)rd32(E1000_SYSTIMH) << (shift + 32);
>> -------------
>>
>>
>> This is a bug in the code. The i350 should be shifted by
>> the same
>> value as the 82580. Then the result would be the same as
>> if you
>> had set the shift to 0. As I recall the reasoning behind the
>> change was to make use of as many bits of the time as
>> possible,
>> and on 82580 and newer the SYSTIMH portion of the clock only
>> contained 8 bits of actual clock data.
>>
>> Without that change I believe there is a risk of the
>> counter being
>> considered to have cycled too fast and causing errors as
>> the cycle
>> counter will roll over at 40 bits instead of 64. The
>> alternative
>> would be to update the cycle counter you are using so that
>> it is
>> aware that it is limited to 40 bits and then use an shift
>> of 0.
>>
>> Setting adapter->cycles.shift to 0 makes the clock run at
>> about the same speed as the system clock.
>>
>> Next problem I have is that the syststamp is not syncing
>> correctly against the system clock.
>>
>> Any help is much appreciated.
>>
>> Regards,
>>
>> Mattias
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Oct 12, 2015 at 5:04 PM, Mattias Barthel
>> <[email protected]
>> <mailto:[email protected]>>
>> wrote:
>>
>> Hi again,
>>
>> Started to try and timestamp all packets in the I350.
>>
>> Driver version:
>>
>> driver: igb
>> version: 3.0.6-k2
>> firmware-version: 0.147-0
>>
>> Doing an ioctl that turns on timestamping for all
>> packets.
>>
>> hwconfig.tx_type = HWTSTAMP_TX_OFF;
>> hwconfig.rx_filter = HWTSTAMP_FILTER_ALL;
>> hwconfig_requested = hwconfig;
>> ioctl(sock, SIOCSHWTSTAMP, &hwtstamp)
>>
>> The strangest thing is that although it works to
>> read the
>> timestamps the clock seems to advance very slowly
>> and/or
>> incorrectly.
>>
>> I just printout the HW timestamps alongside the
>> systemclock and
>> the hw stamping is advancing the nanoseconds when the
>> systemclock
>> is advancing us and seconds.
>>
>> Is there a know bug in this version of the driver
>> that maybe
>> initializes the clock incorrectly or maybe reads
>> incorrectly from
>> the clock?
>>
>>
>> [ 63.004116] igb_rx_hwtstamp
>> [ 63.004684] hwtstamp: 1444656871372825302, syststamp:
>> 1444656871372845807
>> [ 63.005721] system timestamp: tv_sec: 1444656793,
>> tv_usec: 683825
>> [ 63.006631] sys hw timestamp: tstamp:
>> 1444656871372845807,
>> tv_sec: 1444656871, tv_usec: 372845
>> [ 63.008044] hw timestamp: tstamp:
>> 1444656871372825302,
>> tv_sec:
>> 1444656871, tv_usec: 372825
>> [ 63.009395]
>> [ 63.104111] igb_rx_hwtstamp
>> [ 63.104704] hwtstamp: 1444656871372825308, syststamp:
>> 1444656871372845813
>> [ 63.105747] system timestamp: tv_sec: 1444656793,
>> tv_usec: 783851
>> [ 63.106669] sys hw timestamp: tstamp:
>> 1444656871372845813,
>> tv_sec: 1444656871, tv_usec: 372845
>> [ 63.108075] hw timestamp: tstamp:
>> 1444656871372825308,
>> tv_sec:
>> 1444656871, tv_usec: 372825
>> [ 63.109428]
>> [ 63.204109] igb_rx_hwtstamp
>> [ 63.204683] hwtstamp: 1444656871372825314, syststamp:
>> 1444656871372845819
>> [ 63.206018] system timestamp: tv_sec: 1444656793,
>> tv_usec: 884122
>> [ 63.207242] sys hw timestamp: tstamp:
>> 1444656871372845819,
>> tv_sec: 1444656871, tv_usec: 372845
>> [ 63.209096] hw timestamp: tstamp:
>> 1444656871372825314,
>> tv_sec:
>> 1444656871, tv_usec: 372825
>> [ 63.210897]
>> [ 63.304068] igb_rx_hwtstamp
>> [ 63.304849] hwtstamp: 1444656871372825320, syststamp:
>> 1444656871372845825
>> [ 63.306176] system timestamp: tv_sec: 1444656793,
>> tv_usec: 984280
>> [ 63.307391] sys hw timestamp: tstamp:
>> 1444656871372845825,
>> tv_sec: 1444656871, tv_usec: 372845
>> [ 63.309252] hw timestamp: tstamp:
>> 1444656871372825320,
>> tv_sec:
>> 1444656871, tv_usec: 372825
>> [ 63.311032]
>> [ 63.404107] igb_rx_hwtstamp
>> [ 63.404659] hwtstamp: 1444656871372825326, syststamp:
>> 1444656871372845831
>> [ 63.405695] system timestamp: tv_sec: 1444656794,
>> tv_usec: 83798
>> [ 63.406635] sys hw timestamp: tstamp:
>> 1444656871372845831,
>> tv_sec: 1444656871, tv_usec: 372845
>> [ 63.408044] hw timestamp: tstamp:
>> 1444656871372825326,
>> tv_sec:
>> 1444656871, tv_usec: 372825
>> [ 63.409405]
>> [ 63.504084] igb_rx_hwtstamp
>> [ 63.504670] hwtstamp: 1444656871372825332, syststamp:
>> 1444656871372845837
>> [ 63.505715] system timestamp: tv_sec: 1444656794,
>> tv_usec: 183819
>> [ 63.506657] sys hw timestamp: tstamp:
>> 1444656871372845837,
>> tv_sec: 1444656871, tv_usec: 372845
>> [ 63.508090] hw timestamp: tstamp:
>> 1444656871372825332,
>> tv_sec:
>> 1444656871, tv_usec: 372825
>>
>>
>> Any help will be very much appreciated.
>>
>> Best regards,
>>
>> Mattias
>>
>>
>>
>>
>>
>>
>>
>>
>> On Thu, Oct 8, 2015 at 6:01 PM, Alexander Duyck
>> <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> The i350 should support timestamping all
>> packets as I
>> recall. You should probably look into getting the
>> linuxptp package and
>> seeing if you can make use of the hwstamp_ctl
>> command to
>> enable timestamping for all incoming packets
>> on the
>> adapter.
>>
>> - Alex
>>
>> On 10/08/2015 12:33 AM, Mattias Barthel wrote:
>>
>> Hi
>> thanks for answer.
>>
>> Unfortunately, the udp/ip stream just as
>> the cfm
>> stream is
>> against only one peer.
>> Furthemore on the guest side I have only
>> one vcpu
>> tied to
>> two physical siblings to minimize L2 cache
>> misses
>> on ISRs
>> due to cache coherence.
>>
>> I had an idea this morning, maybe I can
>> make use
>> of the
>> ptp hw timestaming for this? This should give
>> better accuracy.
>> Is this possible or does it only work for
>> ethertype ptp?
>>
>> On Thursday, October 8, 2015, Alexander Duyck
>> <[email protected]
>> <mailto:[email protected]>
>> <mailto:[email protected]
>> <mailto:[email protected]>>> wrote:
>>
>> The difference could be Receive Side
>> Scaling
>> (RSS). The UDP/IP
>> packets
>> can be load balanced across all queues
>> via RSS
>> based
>> on a hash of the
>> source and destination addresses. So
>> if you have
>> multiple sources
>> generating these packets, or they are
>> being
>> received
>> on multiple IP
>> addresses then it is possible the load is
>> being spread
>> out over
>> multiple
>> CPUs. Your CFM packets are on an Ethertype
>> other than
>> IPv4 or IPv6 so
>> all packets will end up on queue 0 if
>> I am not
>> mistaken.
>>
>> - Alex
>>
>> On 10/07/2015 10:21 PM, Mattias
>> Barthel wrote:
>> > Hi again!
>> >
>> > I realise my message might not have
>> been so
>> clear.
>> >
>> > I will try to clarify. I am doing PM
>> with
>> software
>> timestamping.
>> >
>> > igb versions:
>> > version: 3.0.6-k2
>> > firmware-version: 0.147-0
>> >
>> > I do timestamping (NTP format) of the
>> packets in the
>> same place
>> for all
>> > protocols. (in igb_clean_rx_irq_adv()).
>> >
>> > For these non ip packets like IEEE
>> 801.2ag CFM ETH-DM
>> (ethertype 0x8902) I
>> > get quite a lot worse accuracy in the
>> timestamping.
>> >
>> > For UDP/IP TWAMP I get for 10k pps a max
>> delay of
>> around 140us
>> back to back.
>> > For CFM ETH DM I get for the same packet
>> rate and
>> packet size a
>> max delay
>> > of around 900us.
>> >
>> > So my hypothesis was:
>> > Could FW be putting these L2 packets
>> in another
>> queue with different
>> > characteristics?
>> >
>> > I tried to add a ET-type filter as
>> written in my
>> previous mail
>> but it
>> > showed no difference.
>> >
>> >
>> > Thanks for any help given!
>> >
>> >
>> >
>> > On Mon, Oct 5, 2015 at 2:19 PM,
>> Mattias Barthel
>> <[email protected]
>> <mailto:[email protected]>
>> <javascript:;>>
>>
>> > wrote:
>> >
>> >> Greetings,
>> >>
>> >> Im getting quite bad latency when
>> using the igb
>> driver on i350
>> on linux
>> >> regarding
>> >> ETH CFM packets. This compared to TWAMP
>> (IPv4 UDP
>> packets).
>> >> The environment is KVM with the
>> i350 devices in
>> PCI-passthrough.
>> >>
>> >> So i figured I add an own rx-queue
>> (filter) for
>> those types of
>> ethernet
>> >> protocol packets.
>> >>
>> >> This is what i added:
>> >>
>> >>
>> >> wr32(E1000_ETQF(4),
>> >> (1 << 31) | /* queue enable */
>> >> (E1000_ETQF_FILTER_ENABLE) | /* enable
>> filter */
>> >> (1 << 29) | /* enable immediate
>> interrupt */
>> >> (0x4 << 16) | /* queue no. 4 */
>> >> (ETH_P_8021AG)); /* 0x8902
>> CFM eth
>> protocol type */
>> >>
>> >>
>> >> ETH_P_8021A is 0x8902
>> >>
>> >>
>> >>
>> >> Is this a correct/good approach?
>> >>
>> >> Thanks in advance!
>> >>
>> >> --
>> >> Mattias Barthel
>> >>
>> >
>>
>>
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> E1000-devel mailing list
>> [email protected]
>> <mailto:[email protected]>
>> <javascript:;>
>> https://lists.sourceforge.net/lists/listinfo/e1000-devel
>> To learn more about Intel®
>> Ethernet, visit
>> http://communities.intel.com/community/wired
>>
>>
>> Avis de confidentialité
>>
>> Les informations contenues dans le présent
>> message
>> et dans
>> toute pièce qui lui est jointe sont
>> confidentielles et
>> peuvent être protégées par le secret
>> professionnel. Ces
>> informations sont à l’usage exclusif de
>> son ou de ses
>> destinataires. Si vous recevez ce message
>> par erreur,
>> veuillez s’il vous plait communiquer
>> immédiatement
>> avec
>> l’expéditeur et en détruire tout
>> exemplaire. De
>> plus, il
>> vous est strictement interdit de le
>> divulguer, de le
>> distribuer ou de le reproduire sans
>> l’autorisation de
>> l’expéditeur. Merci.
>>
>> Confidentiality notice
>>
>> This e-mail message and any attachment
>> hereto contain
>> confidential information which may be
>> privileged
>> and which
>> is intended for the exclusive use of its
>> addressee(s). If
>> you receive this message in error, please
>> inform
>> sender
>> immediately and destroy any copy thereof.
>> Furthermore, any
>> disclosure, distribution or copying of this
>> message and/or
>> any attachment hereto without the consent
>> of the
>> sender is
>> strictly prohibited. Thank you.
>>
>>
>>
>>
>>
>> -- Mattias Barthel
>>
>>
>>
>>
>> -- Mattias Barthel
>>
>>
>>
>> Avis de confidentialité
>>
>> Les informations contenues dans le présent message et dans
>> toute pièce qui lui est jointe sont confidentielles et peuvent
>> être protégées par le secret professionnel. Ces informations
>> sont à l’usage exclusif de son ou de ses destinataires. Si
>> vous recevez ce message par erreur, veuillez s’il vous plait
>> communiquer immédiatement avec l’expéditeur et en détruire
>> tout exemplaire. De plus, il vous est strictement interdit de
>> le divulguer, de le distribuer ou de le reproduire sans
>> l’autorisation de l’expéditeur. Merci.
>>
>> Confidentiality notice
>>
>> This e-mail message and any attachment hereto contain
>> confidential information which may be privileged and which is
>> intended for the exclusive use of its addressee(s). If you
>> receive this message in error, please inform sender
>> immediately and destroy any copy thereof. Furthermore, any
>> disclosure, distribution or copying of this message and/or any
>> attachment hereto without the consent of the sender is
>> strictly prohibited. Thank you.
>>
>>
>>
>> Avis de confidentialité
>>
>> Les informations contenues dans le présent message et dans toute pièce
>> qui lui est jointe sont confidentielles et peuvent être protégées par le
>> secret professionnel. Ces informations sont à l’usage exclusif de son ou de
>> ses destinataires. Si vous recevez ce message par erreur, veuillez s’il
>> vous plait communiquer immédiatement avec l’expéditeur et en détruire tout
>> exemplaire. De plus, il vous est strictement interdit de le divulguer, de
>> le distribuer ou de le reproduire sans l’autorisation de l’expéditeur.
>> Merci.
>>
>> Confidentiality notice
>>
>> This e-mail message and any attachment hereto contain confidential
>> information which may be privileged and which is intended for the exclusive
>> use of its addressee(s). If you receive this message in error, please
>> inform sender immediately and destroy any copy thereof. Furthermore, any
>> disclosure, distribution or copying of this message and/or any attachment
>> hereto without the consent of the sender is strictly prohibited. Thank you.
>>
>>
>
--
Mattias Barthel
------------------------------------------------------------------------------
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
