Re: [Linuxptp-devel] SyncE support
> -Original Message- > From: Miroslav Lichvar > Sent: Tuesday, March 16, 2021 8:59 AM > To: linuxptp-devel@lists.sourceforge.net > Subject: Re: [Linuxptp-devel] SyncE support > > On Tue, Mar 16, 2021 at 03:52:36PM +0100, Frantisek Rysanek wrote: > > On 16 Mar 2021 at 11:25, Miroslav Lichvar wrote: > > > > > > In the Intel's igc driver I saw few SYNCE registers defined, > > > but no code using them. > > > > > Whoa. igc ? oh there's an i225... didn't know about this one, thanks > > for the pointer, this definitely looks promising :-) > > Looks like a successor to the i210 generation. > > I'm sorry for misleading you. I meant the ice driver (speeds up to > 100Gb). I didn't see an out-of-tree version of the igc driver. Yep, the ice hardware is supposed to have SyncE support, though no software here yet :( Not sure about igc. > > The I225 might be an interesting NIC for experiments for a different > reason. It seems it supports the PCIe Precision Time Measurement (PTM) > feature. It is a hardware implementation of an NTP-like protocol over > PCIe, which should allow a highly accurate synchronization of the > system clock, avoiding the asymmetry on PCIe. It is not supported in > the igc driver yet, but there were some patches submitted for it. > > I measured the PCIe asymmetry with the I210 on few different > boards+CPUs and it changed a lot (between about -100ns and 100ns), so > I think PTM could make a significant improvement. > igc should support PTM, but I'm not sure why the patches haven't landed. I'm not very well connected to the igc team. > -- > Miroslav Lichvar > > > > ___ > Linuxptp-devel mailing list > Linuxptp-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/linuxptp-devel ___ Linuxptp-devel mailing list Linuxptp-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-devel
Re: [Linuxptp-devel] SyncE support
> -Original Message- > From: Frantisek Rysanek > Sent: Monday, March 15, 2021 2:40 PM > To: linuxptp-devel@lists.sourceforge.net > Subject: Re: [Linuxptp-devel] SyncE support > > Dear gentlemen, > thanks for opening this interesting topic... > > Is there any stock NIC (silicon, board), with an open-source Linux > driver, that would support SyncE off the shelf in the NIC hardware? > I believe that SyncE is supported by the E800 series 100GB hardware from Intel, but there's no existing driver/software support. > Other than that... SyncE sounds like a pretty self-contained L1 > feature, at least considering just a single NIC port. Obviously the > routing of clock signals among possibly several ports in a system and > a local master oscillator - that would be a more complicated, > proprietary story. > But at the level of a single port... what do you need? > SyncE enable / disable, and select master vs. slave role? > It would take a fairly simple extension to some existing API - makes > me wonder which one: MII, Ethtool, Netlink, or maybe the timestamping > API? I'd guess Ethtool or Netlink would be the right level of > abstraction. Or some nodes in procfs or sysfs :-) > MII is too HW-specific and the timestamping stuff operating on top of > setsockopt() feels a little too detached from hardware. > > And yes there is some additional messaging, I understand in-band in > Ethernet, which would probably have to be handled by ptp4l software, > unless offloaded to the kernel-space driver or hardware... > > Frank Rysanek > > > ___ > Linuxptp-devel mailing list > Linuxptp-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/linuxptp-devel ___ Linuxptp-devel mailing list Linuxptp-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-devel
Re: [Linuxptp-devel] SyncE support
On 18/03/2021 11:57, Frantisek Rysanek wrote: > On 17 Mar 2021 at 18:01, Miroslav Lichvar wrote: > > [on PCI-e PTM] >> >> The switches are supposed to work like NTP server and client at the >> same time (boundary clock in the PTP terminology), so all PCIe links >> have hardware timestamping on both ends. >> >> BTW, at least in theory, a network using boundary clocks should >> perform better than a similar network using transparent clocks, >> assuming the servos are well configured and the sync interval is short >> enough to minimize the time errors in the chain. Divide and conquer >> :). I think transparent clocks are meant to be the simpler and cheaper >> variant. >> > > I recall some history around the debate on BC vs TC. > From my subjective perspective anyway. > > PTP v1 used to be all about BC's. > Then in 2008 came PTP v2, and suddenly TC was all the rage, the only > right way for PTP-capable switches to function from now on. > Except that, as years were passing by, PTP-aware switches did not > exactly spawn in flocks everywhere. Rather, they remained few and far > between. And, especially the telco industry would've loved to deploy > PTP, but there was no way for their active elements in the networks > to support PTP (to turn them all into TC's). So after some years > since 2008, suddenly the Telecom Profiles came out, and were all > about BC's and partial on-path support (or rather: no on-path > support). The TC idea has a stronghold in the electric power > industry, at least on paper - practical compatibility of > implementations remains a fun topic, to use a euphemism, especially > where multi-vendor heterogeneous setups are attempted... this is > where I have a limited by ongoing practical experience :-) > > On the topic of BC's and their apparent renaissance, I've had a brief > debate about this with someone from Meinberg, and the message was: > BC's or TC's, either of them works, if done properly. Reportedly > there have been tests of up to 16 BC's in a cascade, and if the local > oscillators were stable, and the feedback loops were tuned > appropriately, the cascade just worked, there were no ill effects > (excessive residual wander at the tail end, or whatever). So it's all > down to oscillator quality and feedback loop parameters = control > theory and the respective math. Laplace, Nyquist, whatever... > > As for TC's, from my practical observations (and maybe a bit of > gossip), switch chipset vendors have a hard time implementing the > on-the-fly corrections in the hardware. Something that would work > "out of the box" for the switch box makers. And because the silicon > vendors struggle, some switch box vendors implement PTP on their own, > mangling MII on the fly using custom FPGA designs etc. > You probably know the ugly details an order of magnitude better than > I do, i.e. what it takes to make a TC switch, for P2P mode vs. E2E > mode... The "TC support in a switch" is likely never going to be pure > hardware, there's always an element of switch firmware that e.g. does > the talking with P2P peers... While E2E might look easier to support > in a switch (than P2P), I've heard a credible opinion that in E2E > mode, a switch port needs to keep track of delay transactions passing > through (stateful style - ORLY?) and making E2E work for a single > slave is not the same as making it work for many slaves :-) etc. > > > I've wandered even further off topic here... > > In PTM, having a servoed local oscillator in every PCI-e switch (BC > style) seems complicated, to the point of being impractical / > difficult to implement, if the oscillator should be half decent / > stable... > > In my wildest dreams, it would be beautiful to have a 1PPS output out > of the CPU TSC. Plug that into some SDP input on an i210, and you'd > have an idea, where the CPU's PPS edge is located, relative to the > i210 PHC. Which can in turn be disciplined by PTP, or by another > external 1PPS reference. Just a single output wire, coming from the > TSC - it wouldn't even need to be adjustable. Having a timestamp for > that CPU 1PPS event from the i210, you'd be able to calculate an > offset in TSC granularity, and subtract that offset from every > timestamp obtained via rdtsc (at the cost of a single integer sub() > instruction). A single signal, for 1PPS, coming out of the CPU > socket. Is that too much to ask? Compared to the rather complex and > imperfect PTM implementation? > > Actually this is not my wildest dream. I can extrapolate this > further. Such as: > > After the all, the TSC stands for a "Time Stamp Counter" - isn't that > ironic? As it is, the TSC is just a register counting steadily > forward. What reference does it actually have to the wall time in the > outside world, apart from the CPU core clock? (Yes I know - now with > TurboBoost, even the notion of a CPU clock is hazy.) I mean wouldn't > it be nice if the CPU would have a discrete event *input* into the > TSC? Typically useable
Re: [Linuxptp-devel] SyncE support
On 17 Mar 2021 at 18:01, Miroslav Lichvar wrote: [on PCI-e PTM] > > The switches are supposed to work like NTP server and client at the > same time (boundary clock in the PTP terminology), so all PCIe links > have hardware timestamping on both ends. > > BTW, at least in theory, a network using boundary clocks should > perform better than a similar network using transparent clocks, > assuming the servos are well configured and the sync interval is short > enough to minimize the time errors in the chain. Divide and conquer > :). I think transparent clocks are meant to be the simpler and cheaper > variant. > I recall some history around the debate on BC vs TC. >From my subjective perspective anyway. PTP v1 used to be all about BC's. Then in 2008 came PTP v2, and suddenly TC was all the rage, the only right way for PTP-capable switches to function from now on. Except that, as years were passing by, PTP-aware switches did not exactly spawn in flocks everywhere. Rather, they remained few and far between. And, especially the telco industry would've loved to deploy PTP, but there was no way for their active elements in the networks to support PTP (to turn them all into TC's). So after some years since 2008, suddenly the Telecom Profiles came out, and were all about BC's and partial on-path support (or rather: no on-path support). The TC idea has a stronghold in the electric power industry, at least on paper - practical compatibility of implementations remains a fun topic, to use a euphemism, especially where multi-vendor heterogeneous setups are attempted... this is where I have a limited by ongoing practical experience :-) On the topic of BC's and their apparent renaissance, I've had a brief debate about this with someone from Meinberg, and the message was: BC's or TC's, either of them works, if done properly. Reportedly there have been tests of up to 16 BC's in a cascade, and if the local oscillators were stable, and the feedback loops were tuned appropriately, the cascade just worked, there were no ill effects (excessive residual wander at the tail end, or whatever). So it's all down to oscillator quality and feedback loop parameters = control theory and the respective math. Laplace, Nyquist, whatever... As for TC's, from my practical observations (and maybe a bit of gossip), switch chipset vendors have a hard time implementing the on-the-fly corrections in the hardware. Something that would work "out of the box" for the switch box makers. And because the silicon vendors struggle, some switch box vendors implement PTP on their own, mangling MII on the fly using custom FPGA designs etc. You probably know the ugly details an order of magnitude better than I do, i.e. what it takes to make a TC switch, for P2P mode vs. E2E mode... The "TC support in a switch" is likely never going to be pure hardware, there's always an element of switch firmware that e.g. does the talking with P2P peers... While E2E might look easier to support in a switch (than P2P), I've heard a credible opinion that in E2E mode, a switch port needs to keep track of delay transactions passing through (stateful style - ORLY?) and making E2E work for a single slave is not the same as making it work for many slaves :-) etc. I've wandered even further off topic here... In PTM, having a servoed local oscillator in every PCI-e switch (BC style) seems complicated, to the point of being impractical / difficult to implement, if the oscillator should be half decent / stable... In my wildest dreams, it would be beautiful to have a 1PPS output out of the CPU TSC. Plug that into some SDP input on an i210, and you'd have an idea, where the CPU's PPS edge is located, relative to the i210 PHC. Which can in turn be disciplined by PTP, or by another external 1PPS reference. Just a single output wire, coming from the TSC - it wouldn't even need to be adjustable. Having a timestamp for that CPU 1PPS event from the i210, you'd be able to calculate an offset in TSC granularity, and subtract that offset from every timestamp obtained via rdtsc (at the cost of a single integer sub() instruction). A single signal, for 1PPS, coming out of the CPU socket. Is that too much to ask? Compared to the rather complex and imperfect PTM implementation? Actually this is not my wildest dream. I can extrapolate this further. Such as: After the all, the TSC stands for a "Time Stamp Counter" - isn't that ironic? As it is, the TSC is just a register counting steadily forward. What reference does it actually have to the wall time in the outside world, apart from the CPU core clock? (Yes I know - now with TurboBoost, even the notion of a CPU clock is hazy.) I mean wouldn't it be nice if the CPU would have a discrete event *input* into the TSC? Typically useable as 1PPS *input*. Having a timestamping register (MSR?) on the inside, a neighbor to the TSC, which would always contain a timestamp of the last exterior 1PPS edge
