I'm assuming you are using either an older kernel or an out-of-tree
driver do I have that right? I ask because this code doesn't resemble
what is is currently in the latest Linux kernel.
Comments on what you have found are inline below.
- Alex
On 10/13/2015 12:53 AM, Mattias Barthel wrote:
> Hi, I think I managed to get around this problem.
>
>
> in function init_hw_timer():
> ----
> case e1000_i350:
> case e1000_82580:
>
> /*
> * The 82580 timesync updates the system timer every 8ns by 8ns
> * and the value cannot be shifted. Instead we need to shift
> * the registers to generate a 64bit timer value. As a result
> * SYSTIMR/L/H, TXSTMPL/H, RXSTMPL/H all have to be shifted by
> * 24 in order to generate a larger value for synchronization.
> */
> adapter->cycles.shift = IGB_82580_TSYNC_SHIFT;
>
> ----
>
> I have the I350 and I dont think this cycles shift was meant for my
> adapter.
> IGB_82580_TSYNC_SHIFT is 24.
>
> Also when doing igb_read_clock,
> this shifting is applied only for the 82580 and I guess it should not
> have been set in the first place for the I350:
> --------
> /*
> * The timestamp latches on lowest register read. For the 82580
> * the lowest register is SYSTIMR instead of SYSTIML. However we never
> * adjusted TIMINCA so SYSTIMR will just read as all 0s so ignore it.
> */
> if (hw->mac.type == e1000_82580) {
> stamp = rd32(E1000_SYSTIMR) >> 8;
> shift = IGB_82580_TSYNC_SHIFT;
> }
>
> stamp |= (u64)rd32(E1000_SYSTIML) << shift;
> stamp |= (u64)rd32(E1000_SYSTIMH) << (shift + 32);
> -------------
>
This is a bug in the code. The i350 should be shifted by the same value
as the 82580. Then the result would be the same as if you had set the
shift to 0. As I recall the reasoning behind the change was to make use
of as many bits of the time as possible, and on 82580 and newer the
SYSTIMH portion of the clock only contained 8 bits of actual clock data.
Without that change I believe there is a risk of the counter being
considered to have cycled too fast and causing errors as the cycle
counter will roll over at 40 bits instead of 64. The alternative would
be to update the cycle counter you are using so that it is aware that it
is limited to 40 bits and then use an shift of 0.
> Setting adapter->cycles.shift to 0 makes the clock run at about the
> same speed as the system clock.
>
> Next problem I have is that the syststamp is not syncing correctly
> against the system clock.
>
> Any help is much appreciated.
>
> Regards,
>
> Mattias
>
>
>
>
>
>
>
>
>
>
>
>
> On Mon, Oct 12, 2015 at 5:04 PM, Mattias Barthel
> <mattiasbart...@gmail.com <mailto:mattiasbart...@gmail.com>> wrote:
>
> Hi again,
>
> Started to try and timestamp all packets in the I350.
>
> Driver version:
>
> driver: igb
> version: 3.0.6-k2
> firmware-version: 0.147-0
>
> Doing an ioctl that turns on timestamping for all packets.
>
> hwconfig.tx_type = HWTSTAMP_TX_OFF;
> hwconfig.rx_filter = HWTSTAMP_FILTER_ALL;
> hwconfig_requested = hwconfig;
> ioctl(sock, SIOCSHWTSTAMP, &hwtstamp)
>
> The strangest thing is that although it works to read the
> timestamps the clock seems to advance very slowly and/or incorrectly.
>
> I just printout the HW timestamps alongside the systemclock and
> the hw stamping is advancing the nanoseconds when the systemclock
> is advancing us and seconds.
>
> Is there a know bug in this version of the driver that maybe
> initializes the clock incorrectly or maybe reads incorrectly from
> the clock?
>
>
> [ 63.004116] igb_rx_hwtstamp
> [ 63.004684] hwtstamp: 1444656871372825302, syststamp:
> 1444656871372845807
> [ 63.005721] system timestamp: tv_sec: 1444656793, tv_usec: 683825
> [ 63.006631] sys hw timestamp: tstamp: 1444656871372845807,
> tv_sec: 1444656871, tv_usec: 372845
> [ 63.008044] hw timestamp: tstamp: 1444656871372825302, tv_sec:
> 1444656871, tv_usec: 372825
> [ 63.009395]
> [ 63.104111] igb_rx_hwtstamp
> [ 63.104704] hwtstamp: 1444656871372825308, syststamp:
> 1444656871372845813
> [ 63.105747] system timestamp: tv_sec: 1444656793, tv_usec: 783851
> [ 63.106669] sys hw timestamp: tstamp: 1444656871372845813,
> tv_sec: 1444656871, tv_usec: 372845
> [ 63.108075] hw timestamp: tstamp: 1444656871372825308, tv_sec:
> 1444656871, tv_usec: 372825
> [ 63.109428]
> [ 63.204109] igb_rx_hwtstamp
> [ 63.204683] hwtstamp: 1444656871372825314, syststamp:
> 1444656871372845819
> [ 63.206018] system timestamp: tv_sec: 1444656793, tv_usec: 884122
> [ 63.207242] sys hw timestamp: tstamp: 1444656871372845819,
> tv_sec: 1444656871, tv_usec: 372845
> [ 63.209096] hw timestamp: tstamp: 1444656871372825314, tv_sec:
> 1444656871, tv_usec: 372825
> [ 63.210897]
> [ 63.304068] igb_rx_hwtstamp
> [ 63.304849] hwtstamp: 1444656871372825320, syststamp:
> 1444656871372845825
> [ 63.306176] system timestamp: tv_sec: 1444656793, tv_usec: 984280
> [ 63.307391] sys hw timestamp: tstamp: 1444656871372845825,
> tv_sec: 1444656871, tv_usec: 372845
> [ 63.309252] hw timestamp: tstamp: 1444656871372825320, tv_sec:
> 1444656871, tv_usec: 372825
> [ 63.311032]
> [ 63.404107] igb_rx_hwtstamp
> [ 63.404659] hwtstamp: 1444656871372825326, syststamp:
> 1444656871372845831
> [ 63.405695] system timestamp: tv_sec: 1444656794, tv_usec: 83798
> [ 63.406635] sys hw timestamp: tstamp: 1444656871372845831,
> tv_sec: 1444656871, tv_usec: 372845
> [ 63.408044] hw timestamp: tstamp: 1444656871372825326, tv_sec:
> 1444656871, tv_usec: 372825
> [ 63.409405]
> [ 63.504084] igb_rx_hwtstamp
> [ 63.504670] hwtstamp: 1444656871372825332, syststamp:
> 1444656871372845837
> [ 63.505715] system timestamp: tv_sec: 1444656794, tv_usec: 183819
> [ 63.506657] sys hw timestamp: tstamp: 1444656871372845837,
> tv_sec: 1444656871, tv_usec: 372845
> [ 63.508090] hw timestamp: tstamp: 1444656871372825332, tv_sec:
> 1444656871, tv_usec: 372825
>
>
> Any help will be very much appreciated.
>
> Best regards,
>
> Mattias
>
>
>
>
>
>
>
>
> On Thu, Oct 8, 2015 at 6:01 PM, Alexander Duyck
> <alexander.du...@gmail.com <mailto:alexander.du...@gmail.com>> wrote:
>
> The i350 should support timestamping all packets as I recall.
> You should probably look into getting the linuxptp package and
> seeing if you can make use of the hwstamp_ctl command to
> enable timestamping for all incoming packets on the adapter.
>
> - Alex
>
> On 10/08/2015 12:33 AM, Mattias Barthel wrote:
>
> Hi
> thanks for answer.
>
> Unfortunately, the udp/ip stream just as the cfm stream is
> against only one peer.
> Furthemore on the guest side I have only one vcpu tied to
> two physical siblings to minimize L2 cache misses on ISRs
> due to cache coherence.
>
> I had an idea this morning, maybe I can make use of the
> ptp hw timestaming for this? This should give better accuracy.
> Is this possible or does it only work for ethertype ptp?
>
> On Thursday, October 8, 2015, Alexander Duyck
> <alexander.du...@gmail.com
> <mailto:alexander.du...@gmail.com>
> <mailto:alexander.du...@gmail.com
> <mailto:alexander.du...@gmail.com>>> wrote:
>
> The difference could be Receive Side Scaling (RSS).
> The UDP/IP
> packets
> can be load balanced across all queues via RSS based
> on a hash of the
> source and destination addresses. So if you have
> multiple sources
> generating these packets, or they are being received
> on multiple IP
> addresses then it is possible the load is being spread
> out over
> multiple
> CPUs. Your CFM packets are on an Ethertype other than
> IPv4 or IPv6 so
> all packets will end up on queue 0 if I am not mistaken.
>
> - Alex
>
> On 10/07/2015 10:21 PM, Mattias Barthel wrote:
> > Hi again!
> >
> > I realise my message might not have been so clear.
> >
> > I will try to clarify. I am doing PM with software
> timestamping.
> >
> > igb versions:
> > version: 3.0.6-k2
> > firmware-version: 0.147-0
> >
> > I do timestamping (NTP format) of the packets in the
> same place
> for all
> > protocols. (in igb_clean_rx_irq_adv()).
> >
> > For these non ip packets like IEEE 801.2ag CFM ETH-DM
> (ethertype 0x8902) I
> > get quite a lot worse accuracy in the timestamping.
> >
> > For UDP/IP TWAMP I get for 10k pps a max delay of
> around 140us
> back to back.
> > For CFM ETH DM I get for the same packet rate and
> packet size a
> max delay
> > of around 900us.
> >
> > So my hypothesis was:
> > Could FW be putting these L2 packets in another
> queue with different
> > characteristics?
> >
> > I tried to add a ET-type filter as written in my
> previous mail
> but it
> > showed no difference.
> >
> >
> > Thanks for any help given!
> >
> >
> >
> > On Mon, Oct 5, 2015 at 2:19 PM, Mattias Barthel
> <mattiasbart...@gmail.com
> <mailto:mattiasbart...@gmail.com> <javascript:;>>
>
> > wrote:
> >
> >> Greetings,
> >>
> >> Im getting quite bad latency when using the igb
> driver on i350
> on linux
> >> regarding
> >> ETH CFM packets. This compared to TWAMP (IPv4 UDP
> packets).
> >> The environment is KVM with the i350 devices in
> PCI-passthrough.
> >>
> >> So i figured I add an own rx-queue (filter) for
> those types of
> ethernet
> >> protocol packets.
> >>
> >> This is what i added:
> >>
> >>
> >> wr32(E1000_ETQF(4),
> >> (1 << 31) | /* queue enable */
> >> (E1000_ETQF_FILTER_ENABLE) | /* enable filter */
> >> (1 << 29) | /* enable immediate interrupt */
> >> (0x4 << 16) | /* queue no. 4 */
> >> (ETH_P_8021AG)); /* 0x8902 CFM eth
> protocol type */
> >>
> >>
> >> ETH_P_8021A is 0x8902
> >>
> >>
> >>
> >> Is this a correct/good approach?
> >>
> >> Thanks in advance!
> >>
> >> --
> >> Mattias Barthel
> >>
> >
>
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> E1000-devel mailing list
> E1000-devel@lists.sourceforge.net
> <mailto:E1000-devel@lists.sourceforge.net> <javascript:;>
> https://lists.sourceforge.net/lists/listinfo/e1000-devel
> To learn more about Intel® Ethernet, visit
> http://communities.intel.com/community/wired
>
>
> Avis de confidentialité
>
> Les informations contenues dans le présent message et dans
> toute pièce qui lui est jointe sont confidentielles et
> peuvent être protégées par le secret professionnel. Ces
> informations sont à l’usage exclusif de son ou de ses
> destinataires. Si vous recevez ce message par erreur,
> veuillez s’il vous plait communiquer immédiatement avec
> l’expéditeur et en détruire tout exemplaire. De plus, il
> vous est strictement interdit de le divulguer, de le
> distribuer ou de le reproduire sans l’autorisation de
> l’expéditeur. Merci.
>
> Confidentiality notice
>
> This e-mail message and any attachment hereto contain
> confidential information which may be privileged and which
> is intended for the exclusive use of its addressee(s). If
> you receive this message in error, please inform sender
> immediately and destroy any copy thereof. Furthermore, any
> disclosure, distribution or copying of this message and/or
> any attachment hereto without the consent of the sender is
> strictly prohibited. Thank you.
>
>
>
>
>
> --
> Mattias Barthel
>
>
>
>
> --
> Mattias Barthel
------------------------------------------------------------------------------
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
