Re: [chrony-dev] [GIT] chrony/chrony.git branch, master, updated. 1.26-26-g9a01ccc

2011-11-28 Thread Miroslav Lichvar 
On Fri, Nov 25, 2011 at 10:39:33AM +, Ed W wrote:
> On 25/11/2011 10:12, Ed W wrote:
> > As you have clearly started to push the boundaries of curve fitting
> > here, have you considered switching to some kind of Kalman filtering
> > technique?  Implemented correctly this can potentially simultaneously
> > incorporate the updating of the estimate and tracking the estimated
> > error.  I haven't thought about it enough, but it might also make it
> > simpler to incorporate exotic ideas such as more active changes in
> > measurement intervals in response to changing estimated error..? 
> 
> Hmm, some googling shows that this is a fairly well discussed idea in
> academic circles.  I found this article enlightening
> 
> http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.163.7440

Interesting. It seems this could be used to improve our temperature
compensation code to determine the coefficients automatically, but
does it apply to the more common case without temperature data?

I know two places in the chrony code which might benefit from some new
ideas. The sample weight calculation and the pruning of old samples.

Currently, on top of my todo list is NTP4 support and clock combining.
If you have any patches to improve the performance I'll be happy to
test it in the simulator (or help you set it up).

-- 
Miroslav Lichvar

---
To unsubscribe email chrony-dev-requ...@chrony.tuxfamily.org with "unsubscribe" 
in the subject.
For help email chrony-dev-requ...@chrony.tuxfamily.org with "help" in the 
subject.
Trouble?  Email listmas...@chrony.tuxfamily.org.

Re: [chrony-dev] [GIT] chrony/chrony.git branch, master, updated. 1.26-26-g9a01ccc

2011-11-25 Thread Piotr Grudzinski 
Hello,

Fuzzy logic here. I can do it, too.
---
Piotr

On Fri, Nov 25, 2011 at 5:39 AM, Ed W  wrote:

> On 25/11/2011 10:12, Ed W wrote:
> > As you have clearly started to push the boundaries of curve fitting
> > here, have you considered switching to some kind of Kalman filtering
> > technique?  Implemented correctly this can potentially simultaneously
> > incorporate the updating of the estimate and tracking the estimated
> > error.  I haven't thought about it enough, but it might also make it
> > simpler to incorporate exotic ideas such as more active changes in
> > measurement intervals in response to changing estimated error..?
>
> Hmm, some googling shows that this is a fairly well discussed idea in
> academic circles.  I found this article enlightening
>
> http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.163.7440
>
> Sorry if this is an old and well discussed area...
>
> Ed W
>
> ---
> To unsubscribe email chrony-dev-requ...@chrony.tuxfamily.org with
> "unsubscribe" in the subject.
> For help email chrony-dev-requ...@chrony.tuxfamily.org with "help" in the
> subject.
> Trouble?  Email listmas...@chrony.tuxfamily.org.
>
>

Re: [chrony-dev] [GIT] chrony/chrony.git branch, master, updated. 1.26-26-g9a01ccc

2011-11-21 Thread Miroslav Lichvar 
On Wed, Nov 16, 2011 at 11:06:27AM -0800, Bill Unruh wrote:
> On Wed, 16 Nov 2011, Miroslav Lichvar wrote:
> >Polling interval was fixed to 64 seconds, the number of samples is
> >around 30-40. With higher jitter or more stable clock (longer Allan
> 
> So that is about 1800 sec. which looks to be something like that period
> 
> I suppose not surprizingly, since the linear least squares fitting would 
> eliminate
> any higher frequencies.

I think there is not much chrony can do here to eliminate higher
frequencies besides using a shorter polling interval and increasing
the maximum number of samples.

> What is interesting is that I do not see it in the other runs, which might
> suggest that there is something a bit strange in the filtering which is taking
> place. I guess you could run the simulations with the jitter zero, and then
> the frequency wander zero to see if if one of those were driving it.

With this wander/jitter ratio the jitter is the one driving it.

Here are another simulations, jitter with wander, only jitter and only
wander.

http://mlichvar.fedorapeople.org/tmp/chrony_corr4.png
http://mlichvar.fedorapeople.org/tmp/chrony_corr4_nowander.png
http://mlichvar.fedorapeople.org/tmp/chrony_corr4_nojitter.png

With no wander the buffer is mostly full - 64 samples (except the
spike in the middle where the number of samples is 8), with no jitter
the buffer has mostly fewer than 10 samples.

> What do you do, every second add a small random amount to the frequency and a
> random amount to the offset with a roughly gaussian distribution?

Yes, but only to the frequency. The offset is updated by the
difference of the clock and kernel frequency.

-- 
Miroslav Lichvar

---
To unsubscribe email chrony-dev-requ...@chrony.tuxfamily.org with "unsubscribe" 
in the subject.
For help email chrony-dev-requ...@chrony.tuxfamily.org with "help" in the 
subject.
Trouble?  Email listmas...@chrony.tuxfamily.org.

Re: [chrony-dev] [GIT] chrony/chrony.git branch, master, updated. 1.26-26-g9a01ccc

2011-11-15 Thread Miroslav Lichvar 
On Tue, Nov 15, 2011 at 10:32:28AM -0800, Bill Unruh wrote:
> >The graphs of the clock offset now look much nicer too:
> >http://mlichvar.fedorapeople.org/tmp/chrony_corr.png
> 
> Why the large jump in time offset at the beginning with the new procedure.
> That looks suspicious. Also the oscillations in the time, while damped do not
> look very strongly damped. (Q of maybe 10)

It's just a random spike. The starting offset is 1 ms and the jitter is
1 ms too. They were two separate simulations and were not using the
same random sequence for jitter and clock wander.

That graph was with default corrtimeratio 1.0, the following one is
with 10.0.

http://mlichvar.fedorapeople.org/tmp/chrony_corr2.png

-- 
Miroslav Lichvar

---
To unsubscribe email chrony-dev-requ...@chrony.tuxfamily.org with "unsubscribe" 
in the subject.
For help email chrony-dev-requ...@chrony.tuxfamily.org with "help" in the 
subject.
Trouble?  Email listmas...@chrony.tuxfamily.org.

Re: [chrony-dev] [GIT] chrony/chrony.git branch, master, updated. 1.26-26-g9a01ccc

2011-11-15 Thread Miroslav Lichvar 
On Tue, Nov 15, 2011 at 10:28:31AM -0800, Bill Unruh wrote:
> 
> I am a bit confused both about the reason for and the details of this
> alteration. chrony determines both the offset error and the frequency error.

The main reason is that adjtime() slews at a fixed rate of 500 ppm.
On the graphs of frequency error there are huge spikes. If an
application is measuring a short interval just when the adjustment is
running, it will get 500ppm error.

> It corrects the frequency error immediately, and then adjusts the frequency to
> eliminate the offset error.

With the old code only larger (>0.2s) errors were corrected by changing
kernel frequency. Normally adjtime() was called or the PLL phase
adjustment was made (<10us) as adjtime has only microsecond
resolution.

The new code uses the frequency change for errors above 0.5 second and
the PLL for everything else (if it's available).

It certainly is possible to use the frequency change for all offset
corrections, but I think the need for scheduling a timeout to cancel
the frequency change is a big disadvantage for it to be the primary
offset adjustment method. It's not accurate as it's not known when
exactly it was started and ended, so a dispersion needs to be added to
all past samples on every adjustment, it can run out if the machine or
the process is suspended and it introduces extra process wakeups.

> I am not sure what the reason is behind the
> "eliminate small errors over a long time, and large errors quickly". This
> looks like it is heading for the step type adjustment of ntpd for larger
> offset errors.

The maximum rate is limited by the allowed range for the PLL time
constant, with the minimum time constant (4s) and the maximum offset
(0.5s) it is 12.5% in the first second.

> Is your thinking that small errors are probably not true
> offsets but noise and thus should not really be corrected?

Yes. If the offset stddev is 1 ms and we want to correct 500 us, there
is no point in slewing that in one second. OTOH, 10ms offset is
significant and should be removed quickly.

> What are the
> consequences of this change to the stability of the system?

I think it should be ok, simulations and the previous use of the PLL
for nanosecond corrections seemed fine.

-- 
Miroslav Lichvar

---
To unsubscribe email chrony-dev-requ...@chrony.tuxfamily.org with "unsubscribe" 
in the subject.
For help email chrony-dev-requ...@chrony.tuxfamily.org with "help" in the 
subject.
Trouble?  Email listmas...@chrony.tuxfamily.org.

Re: [chrony-dev] [GIT] chrony/chrony.git branch, master, updated. 1.26-26-g9a01ccc

2011-11-15 Thread Bill Unruh 

One of the issues with ntpd that started me being interested in this whole
timing stuff, and chrony was the presence of what seemed like oscillations in
the data. In this latest  one, I again see oscillations, with about a 3000s
period. Do you have any idea what those could be?

How long is the typical time scale over which chrony keeps the data to fit the
linear regression to? What is the polling period?


On Tue, 15 Nov 2011, Miroslav Lichvar wrote:


On Tue, Nov 15, 2011 at 10:32:28AM -0800, Bill Unruh wrote:

The graphs of the clock offset now look much nicer too:
http://mlichvar.fedorapeople.org/tmp/chrony_corr.png


Why the large jump in time offset at the beginning with the new procedure.
That looks suspicious. Also the oscillations in the time, while damped do not
look very strongly damped. (Q of maybe 10)


It's just a random spike. The starting offset is 1 ms and the jitter is
1 ms too. They were two separate simulations and were not using the
same random sequence for jitter and clock wander.

That graph was with default corrtimeratio 1.0, the following one is
with 10.0.

http://mlichvar.fedorapeople.org/tmp/chrony_corr2.png




--
William G. Unruh   |  Canadian Institute for| Tel: +1(604)822-3273
Physics  | Advanced Research  | Fax: +1(604)822-5324
UBC, Vancouver,BC  |   Program in Cosmology | un...@physics.ubc.ca
Canada V6T 1Z1 |  and Gravity   |  www.theory.physics.ubc.ca/

---
To unsubscribe email chrony-dev-requ...@chrony.tuxfamily.org with "unsubscribe" 
in the subject.
For help email chrony-dev-requ...@chrony.tuxfamily.org with "help" in the 
subject.
Trouble?  Email listmas...@chrony.tuxfamily.org.

Re: [chrony-dev] [GIT] chrony/chrony.git branch, master, updated. 1.26-26-g9a01ccc

2011-11-15 Thread Bill Unruh 

On Tue, 15 Nov 2011, Miroslav Lichvar wrote:


On Tue, Nov 15, 2011 at 06:55:18PM +0100, g...@tuxfamily.net wrote:

Add corrtimeratio directive

The corrtimeratio directive controls the ratio between the
duration in which the clock is slewed for an average correction
according to the source history and the interval in which the
corrections are done (usually the NTP polling interval).  Corrections
larger than the average take less time and smaller corrections take
more time, the amount of the correction and the correction time are
inversely proportional.

Increasing corrtimeratio makes the overall frequency error of
the system clock smaller, but increases the overall time error as
the corrections will take longer.

By default, the ratio is 1, which means the duration of an average
correction will be close to the update interval.


I'm not sure about the terminology and I'm open for suggestions for a
better name for this option.

In my testing the RMS frequency error can improve by factor 10 or even
more and with larger corrtimeratio it can be even better than with ntpd.
It seems there is always a compromise between time and frequency
error. The default is set so the time error is not noticeably worse
than it was before.

The graphs of the clock offset now look much nicer too:
http://mlichvar.fedorapeople.org/tmp/chrony_corr.png


Why the large jump in time offset at the beginning with the new procedure.
That looks suspicious. Also the oscillations in the time, while damped do not
look very strongly damped. (Q of maybe 10)






--
William G. Unruh   |  Canadian Institute for| Tel: +1(604)822-3273
Physics  | Advanced Research  | Fax: +1(604)822-5324
UBC, Vancouver,BC  |   Program in Cosmology | un...@physics.ubc.ca
Canada V6T 1Z1 |  and Gravity   |  www.theory.physics.ubc.ca/

---
To unsubscribe email chrony-dev-requ...@chrony.tuxfamily.org with "unsubscribe" 
in the subject.
For help email chrony-dev-requ...@chrony.tuxfamily.org with "help" in the 
subject.
Trouble?  Email listmas...@chrony.tuxfamily.org.

Re: [chrony-dev] [GIT] chrony/chrony.git branch, master, updated. 1.26-26-g9a01ccc

2011-11-15 Thread Bill Unruh 


I am a bit confused both about the reason for and the details of this
alteration. chrony determines both the offset error and the frequency error.
It corrects the frequency error immediately, and then adjusts the frequency to
eliminate the offset error. I am not sure what the reason is behind the
"eliminate small errors over a long time, and large errors quickly". This
looks like it is heading for the step type adjustment of ntpd for larger
offset errors. Is your thinking that small errors are probably not true
offsets but noise and thus should not really be corrected? What are the
consequences of this change to the stability of the system?



On Tue, 15 Nov 2011, g...@tuxfamily.net wrote:


This is an automated email from git. It was enerated because a ref
change was pushed to the repository "chrony/chrony.git".

The branch, master has been updated
  via  9a01ccc07fd5399be88218c02a0430d30cefd82b (commit)
  via  1b8deaf3547c86efd3ac567aba8fc0a5a28d79be (commit)
  via  c7d0232bb113c6ca86cd24f9e694c70bfc946ab2 (commit)
  via  79e5f2be1331fb639362cd159087343c06594351 (commit)
 from  9ab181eb9c75e42695f60c1f89bc2a6235bcfc96 (commit)

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.

- Log -
commit 9a01ccc07fd5399be88218c02a0430d30cefd82b
Author: Miroslav Lichvar 
Date:   Tue Nov 15 12:24:50 2011 +0100

   Add corrtimeratio directive

   The corrtimeratio directive controls the ratio between the
   duration in which the clock is slewed for an average correction
   according to the source history and the interval in which the
   corrections are done (usually the NTP polling interval).  Corrections
   larger than the average take less time and smaller corrections take
   more time, the amount of the correction and the correction time are
   inversely proportional.

   Increasing corrtimeratio makes the overall frequency error of
   the system clock smaller, but increases the overall time error as
   the corrections will take longer.

   By default, the ratio is 1, which means the duration of an average
   correction will be close to the update interval.

commit 1b8deaf3547c86efd3ac567aba8fc0a5a28d79be
Author: Miroslav Lichvar 
Date:   Tue Sep 13 14:53:46 2011 +0200

   Control offset correction rate in Linux driver

   The kernel currently doesn't support a linear adjustment with
   programmable rate, extend the use of the kernel PLL with locked
   frequency instead.

   Set the PLL time constant according to the correction time corresponding
   to the correction rate and corrected offset.

   On kernels with nano PLL adjtime() is no longer used.

commit c7d0232bb113c6ca86cd24f9e694c70bfc946ab2
Author: Miroslav Lichvar 
Date:   Mon Sep 5 15:45:32 2011 +0200

   Introduce offset correction rate

   We want to correct the offset quickly, but we also want to keep the
   frequency error caused by the correction itself low.

   Define correction rate as the area of the region bounded by the graph of
   offset corrected in time. Set the rate so that the time needed to correct
   an offset equal to the current sourcestats stddev will be equal to the
   update interval (assuming linear adjustment). The offset and the
   time needed to make the correction are inversely proportional.

   This is only a suggestion and it's up to the system driver how the
   adjustment will be executed.

commit 79e5f2be1331fb639362cd159087343c06594351
Author: Miroslav Lichvar 
Date:   Fri Nov 11 18:40:01 2011 +0100

   Include clock steps in calculated reference update interval

---

Summary of changes:
acquire.c   |2 +-
chrony.texi |   30 +
cmdmon.c|2 +-
conf.c  |   21 +++
conf.h  |1 +
local.c |   10 +++---
local.h |7 +++--
localp.h|5 ++-
reference.c |   64 +++---
reference.h |1 +
rtc_linux.c |2 +-
sources.c   |1 +
sys_linux.c |   76 +++
sys_netbsd.c|2 +-
sys_solaris.c   |2 +-
sys_sunos.c |2 +-
wrap_adjtimex.c |4 +-
wrap_adjtimex.h |2 +-
18 files changed, 194 insertions(+), 40 deletions(-)


hooks/post-receive
--
chrony/chrony.git

---
To unsubscribe email chrony-dev-requ...@chrony.tuxfamily.org with "unsubscribe" 
in the subject.
For help email chrony-dev-requ...@chrony.tuxfamily.org with "help" in the 
subject.
Trouble?  Email listmas...@chrony.tuxfamily.org.



--
William G. Unruh   |  Canadian Institute for| Tel: +1(604)822-3273
Physics  | Advanced Research  | Fax: +1(604)822-5324
UBC, Vancouver,BC  |   Program in Cosmology |