Re: [time-nuts] Digital temperature compensation

2014-11-02 Thread Daniel Drown

On 11/1/2014 6:03 PM, Hal Murray wrote:

dan-timen...@drown.org said:

I'm experimenting with using a temperature sensor to estimate local
oscillator frequency changes.  My goal is to have a decent holdover clock
for a NTP server with not so great GPS antenna placement.


This is for ntpd rather than chrony, but it's a good read:
   NTP temperature compensation
   http://www.ijs.si/time/temp-compensation/


Thank you, I hadn't seen this one.  I chose chrony because it has 
built-in support for temperature compensation.  I couldn't find anyone 
else talking about using it, so I posted my trial run information:

http://blog.dan.drown.org/beaglebone-black-ntpgps-server-temperature-compensation/

Chrony's support looks very similar in implementation to that page, so I 
assume that's what it was inspired by.  My trial run was exciting 
because it suggested around 25ppb-60ppb was possible.



With the colder temperatures, there's a sudden drop off in frequency  that
I'm having a hard time finding a equation that fits as nicely.


That section of the graph doesn't make sense to me.  I suspect a glitch in
your setup.


I suspect what I'm running into is some interaction of other components 
(perhaps the resistors and capacitors) with the crystal oscillator.



When does your GPS unit run out of satellites?


I've only included data when the GPS is locked and the PPS offset is 
within +/- 15us.



Where is your temperature probe?  Is it in contact with the crystal inside
the PC?  Is the PC doing anything?  In particular at the times when the data
doesn't make sense?


The probe is nearby, on a breadboard.  The system in question is a 
beaglebone black and dedicated to just NTP.  I'm also collecting the 
(less precise) CPU temperature sensor data, and there's no relationship 
between it and the frequency spikes.  I've gone ahead and attached that 
graph.



PCs can do a lot of self heating.  Try running a big workload when the
temperature is in the middle of the good range.


Yup, you can see an example of this on the blog page.

I've posted a part 2 to my blog to go over my estimation of what should 
be possible using this hardware, and came to the conclusion of around 
700ppb: 
http://blog.dan.drown.org/beaglebone-black-ntpgps-server-temperature-compensation-part-2/
___
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.

[time-nuts] Digital temperature compensation

2014-11-01 Thread Dan Drown
I'm experimenting with using a temperature sensor to estimate local  
oscillator frequency changes.  My goal is to have a decent holdover  
clock for a NTP server with not so great GPS antenna placement.


I've been sampling temperature every minute, measured by a DS18B20.   
I've been measuring local clock frequency differences, using chronyd's  
logs from the GPS's PPS.  At 28C through 21C, I get a pretty good  
result that fits a quadratic polynomial decently (0.117 ppm stddev).   
I've attached the graph of that as temp-clock-warmer.png.


With the colder temperatures, there's a sudden drop off in frequency  
that I'm having a hard time finding a equation that fits as nicely.   
All the examples I can find on the web look like third degree  
polynomials, while my data doesn't seem to fit that exactly.  The best  
result I've had so far (0.198 ppm stddev) is attached as  
temp-clock.png and uses the function:


f(x) = -abs(a * (x - 20.88)) + b * ((x - 20.88)**2) + c * ((x - 20.88)**3)
a = 0.888582
b = 0.113806
c = -0.00445763

Does anyone have any advice on how to better model this?  Has anyone  
seen this behavior?


I can provide the raw data if that would help any.
___
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.

Re: [time-nuts] Digital temperature compensation

2014-11-01 Thread Bob Camp
Hi

This sort of thing is normally done with a precisely controlled temperature 
chamber and multi day temperature ramp runs. Even then there is a bit of 
“wonder what that was, let’s try it again”. 

If you are looking at a crystal oscillator, what you have is a perturbation in 
the frequency / temperature curve. The response you get will be temperature 
rate of change dependent. 

Bob

 On Nov 1, 2014, at 4:40 PM, Dan Drown dan-timen...@drown.org wrote:
 
 I'm experimenting with using a temperature sensor to estimate local 
 oscillator frequency changes.  My goal is to have a decent holdover clock for 
 a NTP server with not so great GPS antenna placement.
 
 I've been sampling temperature every minute, measured by a DS18B20.  I've 
 been measuring local clock frequency differences, using chronyd's logs from 
 the GPS's PPS.  At 28C through 21C, I get a pretty good result that fits a 
 quadratic polynomial decently (0.117 ppm stddev).  I've attached the graph of 
 that as temp-clock-warmer.png.
 
 With the colder temperatures, there's a sudden drop off in frequency that I'm 
 having a hard time finding a equation that fits as nicely.  All the examples 
 I can find on the web look like third degree polynomials, while my data 
 doesn't seem to fit that exactly.  The best result I've had so far (0.198 ppm 
 stddev) is attached as temp-clock.png and uses the function:
 
 f(x) = -abs(a * (x - 20.88)) + b * ((x - 20.88)**2) + c * ((x - 20.88)**3)
 a = 0.888582
 b = 0.113806
 c = -0.00445763
 
 Does anyone have any advice on how to better model this?  Has anyone seen 
 this behavior?
 
 I can provide the raw data if that would help any.
 temp-clock-warmer.pngtemp-clock.png___
 time-nuts mailing list -- time-nuts@febo.com
 To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
 and follow the instructions there.

___
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.


[time-nuts] Digital temperature compensation

2014-11-01 Thread Mark Sims
You could try submitting your data to zunzun.com   It will fit it to around 
40,000 different curves and find the best ones.   

Beware that with all curve fitting formulas,  once your live data starts to 
wander out of the range of your original curve fit data,  things can go rather 
badly...   
___
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.


Re: [time-nuts] Digital temperature compensation

2014-11-01 Thread Dan Drown
Ok, I hadn't considered rate of change.  This data is currently 3  
day's worth and seems to repeat itself on both days at the same  
temperature point.  Attached is a time based graph to show that.  The  
ppm axis (on the right) is inverted to make it easier to see the  
relationship between the two.



Quoting Bob Camp kb...@n1k.org:

Hi

This sort of thing is normally done with a precisely controlled  
temperature chamber and multi day temperature ramp runs. Even then  
there is a bit of “wonder what that was, let’s try it again”.


If you are looking at a crystal oscillator, what you have is a  
perturbation in the frequency / temperature curve. The response you  
get will be temperature rate of change dependent.


Bob


On Nov 1, 2014, at 4:40 PM, Dan Drown dan-timen...@drown.org wrote:

I'm experimenting with using a temperature sensor to estimate local  
oscillator frequency changes.  My goal is to have a decent holdover  
clock for a NTP server with not so great GPS antenna placement.


I've been sampling temperature every minute, measured by a DS18B20.  
 I've been measuring local clock frequency differences, using  
chronyd's logs from the GPS's PPS.  At 28C through 21C, I get a  
pretty good result that fits a quadratic polynomial decently (0.117  
ppm stddev).  I've attached the graph of that as  
temp-clock-warmer.png.


With the colder temperatures, there's a sudden drop off in  
frequency that I'm having a hard time finding a equation that fits  
as nicely.  All the examples I can find on the web look like third  
degree polynomials, while my data doesn't seem to fit that exactly.  
 The best result I've had so far (0.198 ppm stddev) is attached as  
temp-clock.png and uses the function:


f(x) = -abs(a * (x - 20.88)) + b * ((x - 20.88)**2) + c * ((x - 20.88)**3)
a = 0.888582
b = 0.113806
c = -0.00445763

Does anyone have any advice on how to better model this?  Has  
anyone seen this behavior?


I can provide the raw data if that would help any.
temp-clock-warmer.pngtemp-clock.png___
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to  
https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts

and follow the instructions there.


___
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.



___
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.

Re: [time-nuts] Digital temperature compensation

2014-11-01 Thread Dan Drown

I gave zunzun a try and the one with the lowest root mean squared error was:
f(x) = a( x**0.5) + b( x ) + c( sin(x) ) + d( cos(x) )

It got 0.202 RMSE, so I guess I'll stick with my original function as  
it seems to be closer to what I expect will happen at colder/hotter  
temps.


You have a good point about temperatures outside my data samples.   
Once it gets hot again in the summertime, I'm sure I'll have to  
re-evaluate this.


Quoting Mark Sims hol...@hotmail.com:
You could try submitting your data to zunzun.com   It will fit it to  
around 40,000 different curves and find the best ones.


Beware that with all curve fitting formulas,  once your live data  
starts to wander out of the range of your original curve fit data,   
things can go rather badly...


___
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.


Re: [time-nuts] Digital temperature compensation

2014-11-01 Thread Bob Camp
Hi

 On Nov 1, 2014, at 5:09 PM, Dan Drown dan-timen...@drown.org wrote:
 
 Ok, I hadn't considered rate of change.  

It’s one of the limits on this sort of thing in general. It’s even more of an 
issue with a coupled mode like the one you show. Since there are an enormous 
number of possible variables, it’s always tough to know exactly how much of an 
issue you will have.

One common answer - run your profile runs at the temperature change rate you 
are most likely to see in practice. 

Another very common answer - don’t use that crystal, get one that does not have 
the problem. You can get parts with slopes  0.1 ppm / C  over 10 to 50 C. You 
might have to spend some quality time sorting them out ….

Bob

 This data is currently 3 day's worth and seems to repeat itself on both days 
 at the same temperature point.  Attached is a time based graph to show that.  
 The ppm axis (on the right) is inverted to make it easier to see the 
 relationship between the two.
 
 
 Quoting Bob Camp kb...@n1k.org:
 Hi
 
 This sort of thing is normally done with a precisely controlled temperature 
 chamber and multi day temperature ramp runs. Even then there is a bit of 
 “wonder what that was, let’s try it again”.
 
 If you are looking at a crystal oscillator, what you have is a perturbation 
 in the frequency / temperature curve. The response you get will be 
 temperature rate of change dependent.
 
 Bob
 
 On Nov 1, 2014, at 4:40 PM, Dan Drown dan-timen...@drown.org wrote:
 
 I'm experimenting with using a temperature sensor to estimate local 
 oscillator frequency changes.  My goal is to have a decent holdover clock 
 for a NTP server with not so great GPS antenna placement.
 
 I've been sampling temperature every minute, measured by a DS18B20.  I've 
 been measuring local clock frequency differences, using chronyd's logs from 
 the GPS's PPS.  At 28C through 21C, I get a pretty good result that fits a 
 quadratic polynomial decently (0.117 ppm stddev).  I've attached the graph 
 of that as temp-clock-warmer.png.
 
 With the colder temperatures, there's a sudden drop off in frequency that 
 I'm having a hard time finding a equation that fits as nicely.  All the 
 examples I can find on the web look like third degree polynomials, while my 
 data doesn't seem to fit that exactly.  The best result I've had so far 
 (0.198 ppm stddev) is attached as temp-clock.png and uses the function:
 
 f(x) = -abs(a * (x - 20.88)) + b * ((x - 20.88)**2) + c * ((x - 20.88)**3)
 a = 0.888582
 b = 0.113806
 c = -0.00445763
 
 Does anyone have any advice on how to better model this?  Has anyone seen 
 this behavior?
 
 I can provide the raw data if that would help any.
 temp-clock-warmer.pngtemp-clock.png___
 time-nuts mailing list -- time-nuts@febo.com
 To unsubscribe, go to 
 https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
 and follow the instructions there.
 
 ___
 time-nuts mailing list -- time-nuts@febo.com
 To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
 and follow the instructions there.
 
 
 temp-vs-clock.png___
 time-nuts mailing list -- time-nuts@febo.com
 To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
 and follow the instructions there.

___
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.


Re: [time-nuts] Digital temperature compensation

2014-11-01 Thread Hal Murray

dan-timen...@drown.org said:
 I'm experimenting with using a temperature sensor to estimate local
 oscillator frequency changes.  My goal is to have a decent holdover clock
 for a NTP server with not so great GPS antenna placement. 

This is for ntpd rather than chrony, but it's a good read:
  NTP temperature compensation
  http://www.ijs.si/time/temp-compensation/


 With the colder temperatures, there's a sudden drop off in frequency  that
 I'm having a hard time finding a equation that fits as nicely.

That section of the graph doesn't make sense to me.  I suspect a glitch in 
your setup.

When does your GPS unit run out of satellites?

Where is your temperature probe?  Is it in contact with the crystal inside 
the PC?  Is the PC doing anything?  In particular at the times when the data 
doesn't make sense?

PCs can do a lot of self heating.  Try running a big workload when the 
temperature is in the middle of the good range.




-- 
These are my opinions.  I hate spam.



___
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.


Re: [time-nuts] Digital temperature compensation

2014-11-01 Thread Chris Albertson
If this is just for holdover then I don't think you even want a general
solution.  Have the controller always keep the last few days of data for
temperature vs. EFC value.  Then if GPS fails use the most recent data for
the current temperature.  This makes for a self adapting system accounting
for aging too.  Not just crystal aging but the aging of the resisters, the
temperature sensor itself and other components.

I don't think holdover should fall back onto a model that was created from
data that might be months or years old.  Use hours or days old data.

This will give the desired response which is this:  If the temperature is
constant when the GPS fails the EFC value as determined by your holdover
temperature modeling should give you the EXACT same EFC value as just
before the GPS failure.  I don't think you can do that with a model. You'd
nee to use logged data.


On Sat, Nov 1, 2014 at 4:01 PM, Bob Camp kb...@n1k.org wrote:

 Hi

  On Nov 1, 2014, at 5:09 PM, Dan Drown dan-timen...@drown.org wrote:
 
  Ok, I hadn't considered rate of change.

 It’s one of the limits on this sort of thing in general. It’s even more of
 an issue with a coupled mode like the one you show. Since there are an
 enormous number of possible variables, it’s always tough to know exactly
 how much of an issue you will have.

 One common answer - run your profile runs at the temperature change rate
 you are most likely to see in practice.

 Another very common answer - don’t use that crystal, get one that does not
 have the problem. You can get parts with slopes  0.1 ppm / C  over 10 to
 50 C. You might have to spend some quality time sorting them out ….

 Bob

  This data is currently 3 day's worth and seems to repeat itself on both
 days at the same temperature point.  Attached is a time based graph to show
 that.  The ppm axis (on the right) is inverted to make it easier to see the
 relationship between the two.
 
 
  Quoting Bob Camp kb...@n1k.org:
  Hi
 
  This sort of thing is normally done with a precisely controlled
 temperature chamber and multi day temperature ramp runs. Even then there is
 a bit of “wonder what that was, let’s try it again”.
 
  If you are looking at a crystal oscillator, what you have is a
 perturbation in the frequency / temperature curve. The response you get
 will be temperature rate of change dependent.
 
  Bob
 
  On Nov 1, 2014, at 4:40 PM, Dan Drown dan-timen...@drown.org wrote:
 
  I'm experimenting with using a temperature sensor to estimate local
 oscillator frequency changes.  My goal is to have a decent holdover clock
 for a NTP server with not so great GPS antenna placement.
 
  I've been sampling temperature every minute, measured by a DS18B20.
 I've been measuring local clock frequency differences, using chronyd's logs
 from the GPS's PPS.  At 28C through 21C, I get a pretty good result that
 fits a quadratic polynomial decently (0.117 ppm stddev).  I've attached the
 graph of that as temp-clock-warmer.png.
 
  With the colder temperatures, there's a sudden drop off in frequency
 that I'm having a hard time finding a equation that fits as nicely.  All
 the examples I can find on the web look like third degree polynomials,
 while my data doesn't seem to fit that exactly.  The best result I've had
 so far (0.198 ppm stddev) is attached as temp-clock.png and uses the
 function:
 
  f(x) = -abs(a * (x - 20.88)) + b * ((x - 20.88)**2) + c * ((x -
 20.88)**3)
  a = 0.888582
  b = 0.113806
  c = -0.00445763
 
  Does anyone have any advice on how to better model this?  Has anyone
 seen this behavior?
 
  I can provide the raw data if that would help any.
 
 temp-clock-warmer.pngtemp-clock.png___
  time-nuts mailing list -- time-nuts@febo.com
  To unsubscribe, go to
 https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
  and follow the instructions there.
 
  ___
  time-nuts mailing list -- time-nuts@febo.com
  To unsubscribe, go to
 https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
  and follow the instructions there.
 
 
  temp-vs-clock.png___
  time-nuts mailing list -- time-nuts@febo.com
  To unsubscribe, go to
 https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
  and follow the instructions there.

 ___
 time-nuts mailing list -- time-nuts@febo.com
 To unsubscribe, go to
 https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
 and follow the instructions there.




-- 

Chris Albertson
Redondo Beach, California
___
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.


Re: [time-nuts] Digital temperature compensation

2014-11-01 Thread Alan Melia
Mmmm yes you can see the equation evaluation starting to rise in your 
Warmer plot, as Mark says, which will make a nonsense of the formula if 
your summer temps get above 28.


Why not a table and then interpolate between the table data points?. You 
might have more points where the changes are greater.
The colder plot looks cubic maybe for a crystal made for 20 deg C ?? But 
depending on the oscillator electronics you may have component tempcos 
affecting the frequency as well?
I suspect the turnover at 21deg C should be a smooth curve not as your 
formula predicts. Which suggests that you have too too high an order of 
polynomial I think, but you may not get a good fit with a cubic if other 
effects are present.


Alan
G3NYK


- Original Message - 
From: Dan Drown dan-timen...@drown.org

To: time-nuts@febo.com
Sent: Saturday, November 01, 2014 10:07 PM
Subject: Re: [time-nuts] Digital temperature compensation


I gave zunzun a try and the one with the lowest root mean squared error 
was:

f(x) = a( x**0.5) + b( x ) + c( sin(x) ) + d( cos(x) )

It got 0.202 RMSE, so I guess I'll stick with my original function as  it 
seems to be closer to what I expect will happen at colder/hotter  temps.


You have a good point about temperatures outside my data samples.   Once 
it gets hot again in the summertime, I'm sure I'll have to  re-evaluate 
this.


Quoting Mark Sims hol...@hotmail.com:
You could try submitting your data to zunzun.com   It will fit it to 
around 40,000 different curves and find the best ones.


Beware that with all curve fitting formulas,  once your live data  starts 
to wander out of the range of your original curve fit data,   things can 
go rather badly...


___
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to 
https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there. 


___
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.


Re: [time-nuts] Digital temperature compensation

2014-11-01 Thread Dan Drown
I think you have a good point - any model is going to have a larger  
error than the data itself.  I'll be looking into this.


Quoting Chris Albertson albertson.ch...@gmail.com:


If this is just for holdover then I don't think you even want a general
solution.  Have the controller always keep the last few days of data for
temperature vs. EFC value.  Then if GPS fails use the most recent data for
the current temperature.  This makes for a self adapting system accounting
for aging too.  Not just crystal aging but the aging of the resisters, the
temperature sensor itself and other components.

I don't think holdover should fall back onto a model that was created from
data that might be months or years old.  Use hours or days old data.

This will give the desired response which is this:  If the temperature is
constant when the GPS fails the EFC value as determined by your holdover
temperature modeling should give you the EXACT same EFC value as just
before the GPS failure.  I don't think you can do that with a model. You'd
nee to use logged data.


On Sat, Nov 1, 2014 at 4:01 PM, Bob Camp kb...@n1k.org wrote:


Hi

 On Nov 1, 2014, at 5:09 PM, Dan Drown dan-timen...@drown.org wrote:

 Ok, I hadn't considered rate of change.

It’s one of the limits on this sort of thing in general. It’s even more of
an issue with a coupled mode like the one you show. Since there are an
enormous number of possible variables, it’s always tough to know exactly
how much of an issue you will have.

One common answer - run your profile runs at the temperature change rate
you are most likely to see in practice.

Another very common answer - don’t use that crystal, get one that does not
have the problem. You can get parts with slopes  0.1 ppm / C  over 10 to
50 C. You might have to spend some quality time sorting them out ….

Bob

 This data is currently 3 day's worth and seems to repeat itself on both
days at the same temperature point.  Attached is a time based graph to show
that.  The ppm axis (on the right) is inverted to make it easier to see the
relationship between the two.


 Quoting Bob Camp kb...@n1k.org:
 Hi

 This sort of thing is normally done with a precisely controlled
temperature chamber and multi day temperature ramp runs. Even then there is
a bit of “wonder what that was, let’s try it again”.

 If you are looking at a crystal oscillator, what you have is a
perturbation in the frequency / temperature curve. The response you get
will be temperature rate of change dependent.

 Bob

 On Nov 1, 2014, at 4:40 PM, Dan Drown dan-timen...@drown.org wrote:

 I'm experimenting with using a temperature sensor to estimate local
oscillator frequency changes.  My goal is to have a decent holdover clock
for a NTP server with not so great GPS antenna placement.

 I've been sampling temperature every minute, measured by a DS18B20.
I've been measuring local clock frequency differences, using chronyd's logs
from the GPS's PPS.  At 28C through 21C, I get a pretty good result that
fits a quadratic polynomial decently (0.117 ppm stddev).  I've attached the
graph of that as temp-clock-warmer.png.

 With the colder temperatures, there's a sudden drop off in frequency
that I'm having a hard time finding a equation that fits as nicely.  All
the examples I can find on the web look like third degree polynomials,
while my data doesn't seem to fit that exactly.  The best result I've had
so far (0.198 ppm stddev) is attached as temp-clock.png and uses the
function:

 f(x) = -abs(a * (x - 20.88)) + b * ((x - 20.88)**2) + c * ((x -
20.88)**3)
 a = 0.888582
 b = 0.113806
 c = -0.00445763

 Does anyone have any advice on how to better model this?  Has anyone
seen this behavior?

 I can provide the raw data if that would help any.

temp-clock-warmer.pngtemp-clock.png___
 time-nuts mailing list -- time-nuts@febo.com
 To unsubscribe, go to
https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
 and follow the instructions there.

 ___
 time-nuts mailing list -- time-nuts@febo.com
 To unsubscribe, go to
https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
 and follow the instructions there.


 temp-vs-clock.png___
 time-nuts mailing list -- time-nuts@febo.com
 To unsubscribe, go to
https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
 and follow the instructions there.

___
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to
https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.





--

Chris Albertson
Redondo Beach, California
___
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.



___
time-nuts mailing list -- time-nuts@febo.com