Re: [ntp:questions] reference clock, shm and ntpd
All, please disregard my previous email about ntpd setting tick=freq=0
in the struct timex passed to adjtimex system call. A more
representative call to adjtimex contains these values:
adjtimex({modes=ADJ_OFFSET|ADJ_MAXERROR|ADJ_ESTERROR|ADJ_STATUS|ADJ_TIMECONST,
offset=32390, freq=33554432, maxerror=35039, esterror=7301,
status=STA_PLL, constant=0, precision=1, tolerance=33554432,
time={1469486352, 424327}, tick=1, ppsfreq=0, jitter=200, shift=2,
stabil=33554432, jitcnt=0, calcnt=0, errcnt=0, stbcnt=0}) = 0 (TIME_OK)
The freq value of 33million+ corresponds to a ppm value of 512, which is
above the 500 limit that the adjtimex system call expects. I should
point out again that I am on a legacy 2.6.10 Linux kernel. Perhaps my
kernel is simply not compatible with a recent ntpd codebase.
Stuart
___
questions mailing list
questions@lists.ntp.org
http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] reference clock, shm and ntpd
Hi Pearly, all,
I have re-read the suggestions on this thread, to figure out why the use
of a shm segment and ntpd does NOT give me the same performance as a
daemon doing our own calls to adjtimex.
I am running ntpd under strace, recording just the adjtimex system call.
Here is a typical system call made by ntpd:
adjtimex({modes=ADJ_OFFSET|0x8000, offset=5, freq=0, maxerror=16384000,
esterror=16384000, status=STA_PLL|STA_UNSYNC, constant=3, precision=1,
tolerance=33554432, time={1469483836, 57343}, tick=1, ppsfreq=0,
jitter=200, shift=2, stabil=33554432, jitcnt=0, calcnt=0, errcnt=0,
stbcnt=0}) = 5 (TIME_ERROR)
I note with some confusion how both the 'tick' and 'freq' members of the
struct timex are both 0. I note also the system call return of 5 where
I guess you would hope/expect a 0.
In our home-grown time adjuster, we found that a tick value of 10008 and
a varying freq value, always non-zero, did a good job of keeping Linux
system time in line with our reference clock.
Obviously ntpd is trying a completely different approach to getting
system time aligned with some reference.
Stuart
___
questions mailing list
questions@lists.ntp.org
http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] reference clock, shm and ntpd
On Thu, May 26, 2016 at 08:15:26PM -0700, Stuart Maclean wrote: > Thanks for the info Dan, I have to concede I don't even know what the > 'frequency' is or what it means, much less know why the value is 500. How > might I reduce that value? I am fairly sure my reference clock is good, and > that my Linux system clock ticks slow compared to the ref clock. You could try increasing (or decreasing?) ADJ_TICK by 5 and see if that helps. It should move the ntpd's frequency range by 500 ppm. I doubt it will help, however. I think stable ofset and frequency pushed to the limit indicates something is very wrong and it's not just local clock being too fast or slow. To me it looks like either the kernel is broken or the reference time is actually based on the local system time. Or maybe something else is messing with the clock, overwriting the ntpd's adjustments. -- Miroslav Lichvar ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] reference clock, shm and ntpd
On 05/27/2016 05:15 AM, Stuart Maclean wrote: > Thanks for the info Dan, I have to concede I don't even know what the > 'frequency' is or what it means, much less know why the value is 500. The correction, in PPM, to apply to the clock frequency. (More below...) > How might I reduce that value? I am fairly sure my reference clock is > good, and that my Linux system clock ticks slow compared to the ref clock. The other way round, I assume ;) > > I am beginning to wonder if I am actually populating the shm segment > correctly. I shall write a reader app in place on NTP, just to see if > the data transfer is working as I hoped it would. I may have usecs and > nsecs fields mixed up. > > Or perhaps my cross compile to Arm produced different code layouts for > ntp and my daemon app, in which case my data xfer is way off. I note > that the struct shmTime is not written in terms of stdint.h types like > uint32_t, but as general ints. I suppose since the producer and > consumer are by definition on the same machine, they should at least > agree on the struct layout no matter how it is defined. Hmm, maybe > padding could be in play here, or maybe I need to look at memory > barriers for my producer's writes?? > >From far away, I don't think that's the problem here. Both NTPD and your service daemon agree on the offset, so the data in the SHM segment is interpreted in a consistent way on both sides. So as long as both apps (NTPD and the acquisition daemon) are compiled with the same infrastructure, alignment issues in the SHM wouldn't be my prime suspects. This could be an issue for the call to adjtimex, in theory. After 'man 2 adjtimex' and reading /usr/include/linux/timex.h I don't think that's an issue for a 32-bit system. I have a kernel 4.4, and my sources need not necessarily to match yours, but this structure seems to be pretty stable over the last 20 years. > Regards > > Stuart > > > On 05/26/2016 02:50 PM, d...@drown.org wrote: >> > mintc=3, offset=25.970500, frequency=500.000, sys_jitter=0.00 >> , I missed that frequency=500 statement initially, but IMHO that's an indication that points into the direction of the clock/adjtimex handling in the kernel. NTPD tries to speed up the clock by 500ppm already, that is, change the clock time for 0.0005s over the next second. If the kernel does not apply any adjustment in the next second, the adjustment is applied again, cancelling the old request. Again to no effect... IMHO the most useful thing to do would be to assert that adjtimex (it's disguised as 'ntp_adjtime' in the source) works properly -- This involves checking the call sites in NTPD (what flags/parameters are used etc) and see if they are all implemented and handled on the kernel side. This could become a real headache. Cheers, Pearly ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] reference clock, shm and ntpd
Thanks for the info Dan, I have to concede I don't even know what the 'frequency' is or what it means, much less know why the value is 500. How might I reduce that value? I am fairly sure my reference clock is good, and that my Linux system clock ticks slow compared to the ref clock. I am beginning to wonder if I am actually populating the shm segment correctly. I shall write a reader app in place on NTP, just to see if the data transfer is working as I hoped it would. I may have usecs and nsecs fields mixed up. Or perhaps my cross compile to Arm produced different code layouts for ntp and my daemon app, in which case my data xfer is way off. I note that the struct shmTime is not written in terms of stdint.h types like uint32_t, but as general ints. I suppose since the producer and consumer are by definition on the same machine, they should at least agree on the struct layout no matter how it is defined. Hmm, maybe padding could be in play here, or maybe I need to look at memory barriers for my producer's writes?? Regards Stuart On 05/26/2016 02:50 PM, d...@drown.org wrote: > mintc=3, offset=25.970500, frequency=500.000, sys_jitter=0.00 , There's your problem, ntpdlimits itself to +/- 500 ppm. There's probably syslogmessages to this effect on your system. I know there's a way to raise this limit, but the limit exists because a correction that large usually signals there's either a local clock problem (hardware or the expected clock frequency is configured wrong in the kernel) or that your upstream clock is wrong (running too fast or slow) ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] reference clock, shm and ntpd
Pearly, thanks for your suggestion I investigate adjtimex. We started down this road of 'reference clock syncing to Linux system time' a while back by basically looking at ref time and system time every few secs (I think it was 10secs) and calculating the tick and freq fields of the adjtimex structure ourselves and then calling adjtimex. This seemed to work OK, I think we were able to adjust both tick and freq by very small amounts and see system time apparently adjust. I then convinced myself that doing it in a home grown fashion was not a good idea and that since the world uses ntp, we should do so also. So, my hunch is that adjtimex can be adjusted with very small adjustments, but whether it can work as used by ntpd I shall have to investigate some more. BTW, I just updated by daemon to read EVERY PPS, and to write clock time and Linux time into the SHM once per second, thinking that ntpd might work differently if it saw 'more samples' in the shm segment. This resulted in NO apparent change in the final result, still ~26ms. Stuart ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] reference clock, shm and ntpd
On 05/26/2016 07:23 PM, Stuart Maclean wrote: > Hi Pearly, > > based on your suggestion on the use of ntpq, I indeed ran those commands > and the offset does indeed say about 26ms, see here: > > associd=0 status=0418 leap_none, sync_uhf_radio, 1 event, no_sys_peer, > > version="ntpd 4.2.8p4@1.3265 Mon May 23 20:45:07 UTC 2016 (1)", > > , leap=00, stratum=1, > precision=-15, rootdelay=0.000, rootdisp=26.313, refid=csac, > > reftime=daf1adb7.a5581391 Thu, May 26 2016 17:15:35.645, > > clock=daf1adbd.54fb354f Thu, May 26 2016 17:15:41.331, peer=6499, tc=4, > > mintc=3, offset=25.970500, frequency=500.000, sys_jitter=0.00, > > clk_jitter=1.029, clk_wander=12.000 > (*grunt*) Instructive, and rules out a few possibilities. > > > I shall have to see if my ntp build process was flawed. Our deployment > target is a legacy Linux 2.6.10 system on Arm, and we have a fairly > dated cross compiler toolchain. So getting ntp to cross-compile at all > took a bit of effort, my exact sequence was this: > > $ mkdir arm-linux > $ cd arm-linux > $ ../configure --host=arm-linux --disable-GPSD --with-yielding-select > $ make > > Perhaps those extra configure options, without which the build failed, > introduce some offset of 26ms ?? That sounds unlikely to me... which proofs nothing, of course. But omitting the GPSD/JSON protocol support won't do any harm, and the method for multiplexed IO shouldn't do anything bad, either. But a few thoughts start to connect... And one thing would be *really* bad: Are you sure the implementation of 'adjtimex' is fully operational? You said you have a 32kHz crystal (probably a 32768Hz wrist watch oscillator), and properly implementing the adjtimex syscall is probably tricky with such a low frequency. But even if this call is properly implemented, there's still the interaction with the close loop controller in NTPD. AFAIR ntpd does not accumulate the residual correction that is returned after setting a slew with this call. So if your implementation of adjtimex needs a certain threshold to start working actively, there's a good chance that the FLL/PLL closed loop controller will not issue a correction that exceeds this limit once the deviation is small enough -- e.g., 26msec! (The same could happen if there's no real threshold: If at least part of the correction does not happen in the next second, nothing will happen at all: NTPD issues a correction once per second, cancelling any old correction that is not yet complete.) I *think* the non-accumulating behaviour is by intention, because accumulating the the residual creates a very different feedback characteristic. That is, it would become a different closed loop controller, and not necessarily with same stability properties. Considering the time Dr. Mills spent on those issues I don't think fiddling around there is a good idea. Is the 32768Hz oscillator the only possible time source? I would hope that there are other, much faster oscillators available to implement the system clock... These would probably result in *much* smaller internal thresholds, and consequently much smaller residual errors. Provided the assumption about the internal minimum threshold for 'adjtimex' to kick in can be held upright. > > I'll re-run ntpd on our target arm board under strace, and see it is > using e.g. some drift file I was unaware of. > Especially the drift file shouldn't be a problem, as after a while your offset becomes stable -- just not zero, as it should. > Thanks again for your help and suggestions de nada! ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] reference clock, shm and ntpd
Hi Pearly, based on your suggestion on the use of ntpq, I indeed ran those commands and the offset does indeed say about 26ms, see here: associd=0 status=0418 leap_none, sync_uhf_radio, 1 event, no_sys_peer, version="ntpd 4.2.8p4@1.3265 Mon May 23 20:45:07 UTC 2016 (1)", , leap=00, stratum=1, precision=-15, rootdelay=0.000, rootdisp=26.313, refid=csac, reftime=daf1adb7.a5581391 Thu, May 26 2016 17:15:35.645, clock=daf1adbd.54fb354f Thu, May 26 2016 17:15:41.331, peer=6499, tc=4, mintc=3, offset=25.970500, frequency=500.000, sys_jitter=0.00, clk_jitter=1.029, clk_wander=12.000 I shall have to see if my ntp build process was flawed. Our deployment target is a legacy Linux 2.6.10 system on Arm, and we have a fairly dated cross compiler toolchain. So getting ntp to cross-compile at all took a bit of effort, my exact sequence was this: $ mkdir arm-linux $ cd arm-linux $ ../configure --host=arm-linux --disable-GPSD --with-yielding-select $ make Perhaps those extra configure options, without which the build failed, introduce some offset of 26ms ?? I'll re-run ntpd on our target arm board under strace, and see it is using e.g. some drift file I was unaware of. Thanks again for your help and suggestions Stuart ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] reference clock, shm and ntpd
On 05/25/2016 02:44 AM, Stuart Maclean wrote: > Hi Pearly, thanks so much for your reply, the first one both seems to > understand what I am trying to do AND that I can understand your points. > > I am guessing that 'clockTimestamp*' are the fields from the reference > clock. Since my ref clock outputs whole seconds only, yes I set > clockTimestampUsec and clockTimestampNsec to both zero. I have also > just asserted that ntpd's shm code never writes to those values, so I am > safe writing to them just once. > > When my daemon gets system time at time of ref clock PPS, my device > driver supplies secs and usecs back to it(essentially the data that the > device driver obtains with a do_gettimeofday right in the ISR attached > to PPS event). I am not a fan of how ntp uses the term 'driver', since > I am using device driver here also, to mean 'code in the Linux kernel'. (Just nomenclatura. Cups has printer drivers, but the 'real' driver is obviously an USB or parallel port or even serial line driver. It's a matter of perspective... but when one is dealing with both levels, it gets confusing very fast. I have no good solution for that...) > > So anyway, my daemon gets secs + usecs back from the PPS event. and I > simply set them into the shmTime.receiveTimestampSec and > shmTime.receiveTimestanpUSec respectively. I clamp my > shmTime.receiveTimestampNsec to 0 throughout. > > So in fact I am only setting 3 of the 6 timestamp fields as non-zero: > the clockTimestampSec, the receiveTimestampSec and the receiveTimestampUSec. > Makes perfect sense to me. > I have precision = -20, leap = 0, samples = 3, though am not sure what > each is supposedly doing. > > Performance is odd. I can see ntpd working 'slowly' to align system > time to the reference, on the order of minutes, and that's fine. It > just never gets any closer than ~26ms. Here's a snippet of my daemon's > log, each time it grabs refclock time and PPS timestamps. You can see > the ~26000 usec offset over time. By the way, my Linux system tick is > 10ms, and my Linux clock crystal runs at 32Khz, so my linux system time > resolution is about 30us as I understand it. (So basically your precision is -16, not that it matters when we're away from that by a factor of 10**3...) > > In my log, the 'Data' value is the reference clock. The PPS values > (secs + usecs) are Linux system time at time of ref clock PPS, For this > run, I had reduced my daemon main loop from 10 secs to 5 secs: > > 20160525004117982 Data: 1464136878, PPS 1464136877 973992 > 20160525004122987 Data: 1464136883, PPS 1464136882 973906 > 20160525004127982 Data: 1464136888, PPS 1464136887 973962 > 20160525004132986 Data: 1464136893, PPS 1464136892 974045 > 20160525004137980 Data: 1464136898, PPS 1464136897 973929 > 20160525004142985 Data: 1464136903, PPS 1464136902 973940 > 20160525004147979 Data: 1464136908, PPS 1464136907 974029 > > Of course what i WANT ntpd to do is nudge that 974000 up to towards > 00, so that when the ref clock's PPS goes off, my system clock reads > same time as next ref clock data output. > That does look pretty stable to me, and yes, the offset should go away. Since you have your daemon feeding the SHM segment: What's the output of NTPQ? (the general 'ntpq -pn' as well one where you can see the clock and peer variables. That's 'ntpq -c "cv "' and 'ntpq -c "rv "', where you can get the '' from a 'ntpq -cas' and matching the lines with the output ofg 'ntpq -crv') The jitter and offset for your SHM segment provided by 'ntpq -pn' should match the one you observe from your daemon's debug output. If there's a discrepancy... I cannot start to suspect where this dead fish is rotting away, but then it's basically between your acquisition daemon (and/or it's output / debug output) and ntpq. If the offset in 'ntpq -crv' is also 26ms, then I assume a different kind of fish is decaying somewhere inside ntpd. While both aspects are not exactly wonderful, you may pardon me for hoping it's not NTPD ;) I'll be AFK and out-of-reach for some time to come, but I think that trying to correlate the output of NTPQ with the output of your acquisition daemon will provide further insight. Just don't feel shunted when I'm unresponsive for some time ;) Cheers, Pearly ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] reference clock, shm and ntpd
Hi Pearly, thanks so much for your reply, the first one both seems to understand what I am trying to do AND that I can understand your points. I am guessing that 'clockTimestamp*' are the fields from the reference clock. Since my ref clock outputs whole seconds only, yes I set clockTimestampUsec and clockTimestampNsec to both zero. I have also just asserted that ntpd's shm code never writes to those values, so I am safe writing to them just once. When my daemon gets system time at time of ref clock PPS, my device driver supplies secs and usecs back to it(essentially the data that the device driver obtains with a do_gettimeofday right in the ISR attached to PPS event). I am not a fan of how ntp uses the term 'driver', since I am using device driver here also, to mean 'code in the Linux kernel'. So anyway, my daemon gets secs + usecs back from the PPS event. and I simply set them into the shmTime.receiveTimestampSec and shmTime.receiveTimestanpUSec respectively. I clamp my shmTime.receiveTimestampNsec to 0 throughout. So in fact I am only setting 3 of the 6 timestamp fields as non-zero: the clockTimestampSec, the receiveTimestampSec and the receiveTimestampUSec. I have precision = -20, leap = 0, samples = 3, though am not sure what each is supposedly doing. Performance is odd. I can see ntpd working 'slowly' to align system time to the reference, on the order of minutes, and that's fine. It just never gets any closer than ~26ms. Here's a snippet of my daemon's log, each time it grabs refclock time and PPS timestamps. You can see the ~26000 usec offset over time. By the way, my Linux system tick is 10ms, and my Linux clock crystal runs at 32Khz, so my linux system time resolution is about 30us as I understand it. In my log, the 'Data' value is the reference clock. The PPS values (secs + usecs) are Linux system time at time of ref clock PPS, For this run, I had reduced my daemon main loop from 10 secs to 5 secs: 20160525004117982 Data: 1464136878, PPS 1464136877 973992 20160525004122987 Data: 1464136883, PPS 1464136882 973906 20160525004127982 Data: 1464136888, PPS 1464136887 973962 20160525004132986 Data: 1464136893, PPS 1464136892 974045 20160525004137980 Data: 1464136898, PPS 1464136897 973929 20160525004142985 Data: 1464136903, PPS 1464136902 973940 20160525004147979 Data: 1464136908, PPS 1464136907 974029 Of course what i WANT ntpd to do is nudge that 974000 up to towards 00, so that when the ref clock's PPS goes off, my system clock reads same time as next ref clock data output. Stuart ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] reference clock, shm and ntpd
On 05/24/2016 11:27 PM, Stuart Maclean wrote: > Back in January of this year, I posted my ntp scenario to this list: > > http://lists.ntp.org/pipermail/questions/2016-January/040609.html > > I never got a reply I could understand. > > I am now revisiting the issue of using ntp to read from a shm segment, > using driver 28. I populate that shm segment using a daemon of my own, > call it D, thus: > > every 10 secs, my daemon D reads 'whole secs since the epoch' from a > special hardware clock, call it HC. My daemon also reads PPS events > from that same clock (imagine it being like a GPS device, it can output > a PPS and a NMEA-like string containing a timestamp). I wrote a kernel > device driver which sees the PPS as an interrupt on a GPIO pin and grabs > system time and returns that as the read result of the PPS 'device' to > daemon D. > > So my daemon gets a time value from the hardware clock AND a local Linux > system time. I use these two to populate the fields of a single SHM > struct which I write and NTP then reads. > > My ntp.conf is thus: > > server 127.127.28.2 minpoll 4 > fudge 127.127.28.2 refid HC > > Things are kind-of working. I know without use of ntp, my system clock > would drift from the HC time value. It would tick slow, and the drift > is significant, seconds per hour. > > ntpd is getting my Linux system clock to stay within 25 ms of the HC > reference time, but never any nearer. ntpq -p reports an offset of > 25-26. My daemon D also reports the time grabs it gets from HC and > system time at time of PPS, and it too reports a discrepancy of 25ms. > > I do not understand this 25ms 'barrier'? Anyone enlighten me? > > I suspect I am not using ntp correctly, with respect to how it expects > to use a PPS signal. As I alluded to in my earlier email, our system is > a legacy Linux 2.6.* kernel pre-dating linuxpps support. > Other clock sources (in my case, the Meinberg service daemon) do *not* have this issue, so I guess your suspicion is right... Among the things that can go wrong I would check: Is the clock time properly set? Assuming you get your PPS pulse at the start of a new second, both 'clockTimeStampUSec' and 'clockTimeStampNSec' should be zero. Is 'receiveTimeStampUSec' the truncated result of dividing 'receiveTimeStampNSec' by 1000? Is the precision field set to something that mirrors the precision of the PPS signal? If you have 1usec accuracy, 'precision' should be set to -20. 'leap' should be zero, but I guess you have that, or you would experience other strange things. (Note: I have a tendency to mess up the meaning of clock time and receive time... If in doubt, switch the roles and see if my writing makes more sense then.) Oh, another thing that can go wrong: Do you sample the right edge of the PPS pulse? I guess you went over your code several times already, so I guess the time stamps themselves are OK. Which leaves the other things to check. Thinking about it, the PPS edge that gets evaluated becomes my prime suspect, but since I don't have details about the hardware I cannot validate or dismiss that suspicion. (While the trailing edge is not as stable in timing as the leading edge for most receivers, the PPS pulse width is stable enough to give you a pretty constant offset...) And my apologies if you have already checked these issues. Now, why do some clock sources insist in feeding two clock units, one for PPS and one for the serial data stream? Some people consider them being two clocks that are actually unrelated. I don't want to start a flame war here, so let's just say that's one way to see it. Other GNS based time sources just use one channel, and IMHO cause less trouble for NTPD and the programmer. So you should be perfectly fine with one SHM channel... provided you do the proper error handling when the sat signal deteriorates, like stopping to feed the SHM slot with dubious / untrustworthy time stamps. Cheers, Pearly ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
[ntp:questions] reference clock, shm and ntpd
Back in January of this year, I posted my ntp scenario to this list: http://lists.ntp.org/pipermail/questions/2016-January/040609.html I never got a reply I could understand. I am now revisiting the issue of using ntp to read from a shm segment, using driver 28. I populate that shm segment using a daemon of my own, call it D, thus: every 10 secs, my daemon D reads 'whole secs since the epoch' from a special hardware clock, call it HC. My daemon also reads PPS events from that same clock (imagine it being like a GPS device, it can output a PPS and a NMEA-like string containing a timestamp). I wrote a kernel device driver which sees the PPS as an interrupt on a GPIO pin and grabs system time and returns that as the read result of the PPS 'device' to daemon D. So my daemon gets a time value from the hardware clock AND a local Linux system time. I use these two to populate the fields of a single SHM struct which I write and NTP then reads. My ntp.conf is thus: server 127.127.28.2 minpoll 4 fudge 127.127.28.2 refid HC Things are kind-of working. I know without use of ntp, my system clock would drift from the HC time value. It would tick slow, and the drift is significant, seconds per hour. ntpd is getting my Linux system clock to stay within 25 ms of the HC reference time, but never any nearer. ntpq -p reports an offset of 25-26. My daemon D also reports the time grabs it gets from HC and system time at time of PPS, and it too reports a discrepancy of 25ms. I do not understand this 25ms 'barrier'? Anyone enlighten me? I suspect I am not using ntp correctly, with respect to how it expects to use a PPS signal. As I alluded to in my earlier email, our system is a legacy Linux 2.6.* kernel pre-dating linuxpps support. Any help appreciated. Stu ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
