Re: Time Synchronizing Between Two Servers

2007-05-07 Thread Chuck Swiger

On May 4, 2007, at 9:10 AM, RW wrote:

On Thu, 03 May 2007 11:07:34 -0400
Chuck Swiger [EMAIL PROTECTED] wrote:

Sun SPARC machines have good HW clocks, and also some of the newer
Macs also seem to have consistently low values in ntp.drift and
handle timekeeping well.


Does that matter?


A good question-- the answer seems to be that it depends.


The RTC time is almost immediately overridden by ntpdate. The
drift is a systematic error that ntpd allows for. I would
have thought that the only significant issue, is whether the system
loses timer interrupts under load.


There are limits to how rapidly ntpd will slew the clock via adjtime 
(); the smaller the intrinsic drift of the HW clock, the sooner any  
adjustment (beyond the initial stepping at system boot via ntpdate)  
will complete.  This only matters to stratum-2 and higher systems--  
anything with a primary reference clock (GPS/WWV/ACTS/etc) is going  
to sync to that and ignore the local HW clock entirely.


--
-Chuck

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Time Synchronizing Between Two Servers

2007-05-07 Thread [EMAIL PROTECTED]

On 07/05/07, Chuck Swiger [EMAIL PROTECTED] wrote:

On May 4, 2007, at 9:10 AM, RW wrote:
 On Thu, 03 May 2007 11:07:34 -0400



 Chuck Swiger [EMAIL PROTECTED] wrote:
 Sun SPARC machines have good HW clocks, and also some of the newer
 Macs also seem to have consistently low values in ntp.drift and
 handle timekeeping well.

 Does that matter?

A good question-- the answer seems to be that it depends.


A low value in ntp.drift is inconsequential compared
to a constant or near constant value, which many
motherboards do not support.



 The RTC time is almost immediately overridden by ntpdate. The
 drift is a systematic error that ntpd allows for. I would
 have thought that the only significant issue, is whether the system
 loses timer interrupts under load.

There are limits to how rapidly ntpd will slew the clock via adjtime
(); the smaller the intrinsic drift of the HW clock, the sooner any
adjustment (beyond the initial stepping at system boot via ntpdate)
will complete.  This only matters to stratum-2 and higher systems--
anything with a primary reference clock (GPS/WWV/ACTS/etc) is going
to sync to that and ignore the local HW clock entirely.


If you really need that ultimate precision, by all means
ntpd - ntpd on the LAN is probably the Right Thing,
in conjunction with close temperature control.  For most
uses (keeping two or more given machines within 10ms
or so on the same LAN) timed with one machine synced
to the outside world via ntpd is simpler at the very least.

--
--
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Time Synchronizing Between Two Servers

2007-05-07 Thread RW
On Mon, 7 May 2007 12:30:29 -0700
Chuck Swiger [EMAIL PROTECTED] wrote:

 On May 4, 2007, at 9:10 AM, RW wrote:
  On Thu, 03 May 2007 11:07:34 -0400
  Chuck Swiger [EMAIL PROTECTED] wrote:
  Sun SPARC machines have good HW clocks, and also some of the newer
  Macs also seem to have consistently low values in ntp.drift and
  handle timekeeping well.
 
  Does that matter?
 
 A good question-- the answer seems to be that it depends.
 
  The RTC time is almost immediately overridden by ntpdate. The
  drift is a systematic error that ntpd allows for. I would
  have thought that the only significant issue, is whether the system
  loses timer interrupts under load.
 
 There are limits to how rapidly ntpd will slew the clock via adjtime 
 (); the smaller the intrinsic drift of the HW clock, the sooner any  
 adjustment (beyond the initial stepping at system boot via ntpdate)  
 will complete. 

As I understand it, ntpd uses it's own kernel interface, ntp_adjtime(),
which lets it share some of its internal state with the kernel. The
kernel knows about the time and frequency errors, and makes the
corrections itself every second. 

If the time error is zeroed by ntpdate, and there's a drift-file, I
don't see that the actual drift value makes much difference. I suspect
that any quartz clock is overkill.

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Time Synchronizing Between Two Servers

2007-05-07 Thread Jeffrey Goldberg

On May 7, 2007, at 5:02 PM, RW wrote:


If the time error is zeroed by ntpdate, and there's a drift-file, I
don't see that the actual drift value makes much difference. I suspect
that any quartz clock is overkill.


As someone already mentioned, drift data doesn't really solve the  
problem if the amount of drift varies (often with temperature, and  
sometimes dramatically with sleep).  The clock on my wife's G5 iMac  
seems to be erratic, but I haven't (and won't) bother to investigate  
further.  If her system is up to 2 seconds off for a bit after waking  
from sleep, so be it.  (If I ever start using kerberos around the  
house, I will have to address that.)


If a machine is up for months, ntpdate may have been run in the  
distant past, so you can still a fair amount of error.


ntpd is really a very light weight thing.  When things are ticking  
over nicely, it may make just one query every few hours and still  
keep very good time.


Also, if you have a server facing the Internet, you may wish to run a  
public NTP service on it and contribute it to pool.ntp.org, see


 http://www.pool.ntp.org/join.html

for info.

-j


--
Jeffrey Goldberghttp://www.goldmark.org/jeff/

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Time Synchronizing Between Two Servers

2007-05-07 Thread jdow

From: Chuck Swiger [EMAIL PROTECTED]


On May 4, 2007, at 9:10 AM, RW wrote:

On Thu, 03 May 2007 11:07:34 -0400
Chuck Swiger [EMAIL PROTECTED] wrote:

Sun SPARC machines have good HW clocks, and also some of the newer
Macs also seem to have consistently low values in ntp.drift and
handle timekeeping well.


Does that matter?


A good question-- the answer seems to be that it depends.


The RTC time is almost immediately overridden by ntpdate. The
drift is a systematic error that ntpd allows for. I would
have thought that the only significant issue, is whether the system
loses timer interrupts under load.


There are limits to how rapidly ntpd will slew the clock via adjtime (); 
the smaller the intrinsic drift of the HW clock, the sooner any 
adjustment (beyond the initial stepping at system boot via ntpdate)  will 
complete.  This only matters to stratum-2 and higher systems--  anything 
with a primary reference clock (GPS/WWV/ACTS/etc) is going  to sync to 
that and ignore the local HW clock entirely.


On good operating systems, that is to say ones in which the NTP code can
get in and slightly alter the HZ clock timing, it implements a phase
locked loop with a lot of filtering on the data returned in the NTP polls.
So the concept of good oscillator is a little different from what you
might presume. It basically means one that is stable with time and with
temperature variations seen as installed. Wide temperature excursions
(in this context that could be as little as 10 degrees C) can cause
errors. The loop tries to adapt. But time setting might fall back on
frequent correction to be satisfactory.

Most PCs have floor sweepings for their crystal oscillators. Those with
good AT cut crystals are, more or less coincidentally, working at a
temperature that is fairly benign for temperature variations. The frequency
error versus temperature curve for the AT cut crystal is an S curve. It
starts negative at low frequencies (maybe -25 to -50 ppm). It runs up to
a peak, still at relatively low temperature, of about +25 to +50ppm. Then
it runs back down to the -25 to -50ppm range at around 40 to 50 degrees C
depending on the precise angles at which the crystal blank was cut. Then
it swings back upwards again.  The basic error of (only) -25 to -50 ppm
can be tweaked out with software pretty easily. (In the bad old days
trimming the capacitance across the crystal served the same purpose. And
when a precise quartz standard is needed this technique still prevails
in various forms.)

So basically if you either get lucky or the motherboard was built with a
quality (but uncompensated) quartz oscillator you may get relatively good
performance over machine room temperatures. It might be as good as a small
number of parts per million. About 11 ppm is a second a day for reference.
As long as NTP can get in and modify the division ratio, ideally on a tick
per tick basis, it can compensate out time variances to the point that you
remain remarkably accurate even after a day of being disconnected from the
network.

The second major problem is aging. Oscillators change frequency with age.
Precisely how they change depends on the care with which they are made,
the evacuation or leakage of the crystal can, whether the crystal is well
baked out, and how long it has been turned on. Once the oscillator has been
on for a few days NTP can get a decent estimate of the long term aging
characteristics of the oscillator and compensate for it as well. This is
probably why server class motherboards have their good reputation. The
oscillators are still probably floor sweepings or perhaps slightly better
quality premium floor sweepings. But if they are on 24/7 their aging
characteristics pretty much settle down and become predictable. As long
as it is predictable NTP can correct for it.

(FWIW the oscillators on the old block II GPS satellites were intended to
include one Cs standard (as an experiment) and three Rb standards. Rb
standards are not primary standards. But their phase noise qualities are
superior to Cs. Cs is a primary standard. But they do show some variance.

(The military decided they did not want enemies to be able to easily plonk
a cruise missile down Jimmy Carter's toity in the White House. So they
implemented a means to deny accuracy to the enemy. They corrupted the
clock frequencies. I built the frequency synthesizer which did this and
made comments back up the chain that this is also a prime way to ensure
maximum GPS constellation accuracy. When you have a synthesizer capable
of correcting an oscillator to parts per ten to the thirteenth accuracy
you can move the correction up or down to get it to about 2-4 parts per
10^13th. Then you can move up and down one count to cause the average over
time to be something into the ten to the fifteenth range if the oscillator's
accuracy over that interval is good enough. The comments I heard come back
down the chain amounted to yum yum.

(This is essentially what NTP does at tens to 

Re: Time Synchronizing Between Two Servers

2007-05-07 Thread jdow

From: [EMAIL PROTECTED]


On 07/05/07, Chuck Swiger [EMAIL PROTECTED] wrote:

On May 4, 2007, at 9:10 AM, RW wrote:
 On Thu, 03 May 2007 11:07:34 -0400



 Chuck Swiger [EMAIL PROTECTED] wrote:
 Sun SPARC machines have good HW clocks, and also some of the newer
 Macs also seem to have consistently low values in ntp.drift and
 handle timekeeping well.

 Does that matter?

A good question-- the answer seems to be that it depends.


A low value in ntp.drift is inconsequential compared
to a constant or near constant value, which many
motherboards do not support.



 The RTC time is almost immediately overridden by ntpdate. The
 drift is a systematic error that ntpd allows for. I would
 have thought that the only significant issue, is whether the system
 loses timer interrupts under load.

There are limits to how rapidly ntpd will slew the clock via adjtime
(); the smaller the intrinsic drift of the HW clock, the sooner any
adjustment (beyond the initial stepping at system boot via ntpdate)
will complete.  This only matters to stratum-2 and higher systems--
anything with a primary reference clock (GPS/WWV/ACTS/etc) is going
to sync to that and ignore the local HW clock entirely.


If you really need that ultimate precision, by all means
ntpd - ntpd on the LAN is probably the Right Thing,
in conjunction with close temperature control.  For most
uses (keeping two or more given machines within 10ms
or so on the same LAN) timed with one machine synced
to the outside world via ntpd is simpler at the very least.


If you have a 10ms tolerance you fall out of range rather quickly with
rather small errors. 1.1ppm over 2.4 hours is about 10ms. And that is
the range of variance that you can expect with temperature changes. That
is why NTP has a locked loop. It can sense the temperature changes over
the polling interval and compensate.

Note that typical motherboard oscillators are specified as plus or minus
100 ppm. AT cut crystals pretty much used to be 50 ppm devices until the
mass market PC crystals appeared. A reasonably good AT cut crystal should
show a plus to minus 25 ppm variance over -20 C to +70 C or there abouts.
And if the manufacturer felt good the day yours was made it will show a
turnover temperature about that of the inside of your case. But with the
really cheap crystals floating around - don't bet on it.

Machine rooms have more constant temperatures as a rule. That translates
to better stability. And the machines are on 24/7. That translates to
MUCH better stability. (Crystals age VERY rapidly for the first few
hours after turn on. This has to do with particulates and even air
molecules settling on the quartz surface when they are not oscillating.)

{^_^}Joanne
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Time Synchronizing Between Two Servers

2007-05-07 Thread RW
On Mon, 7 May 2007 18:35:13 -0500
Jeffrey Goldberg [EMAIL PROTECTED] wrote:

 On May 7, 2007, at 5:02 PM, RW wrote:
 
  If the time error is zeroed by ntpdate, and there's a drift-file, I
  don't see that the actual drift value makes much difference. I
  suspect that any quartz clock is overkill.
 
 As someone already mentioned, drift data doesn't really solve the  
 problem if the amount of drift varies (often with temperature, and  
 sometimes dramatically with sleep).  The clock on my wife's G5 iMac  
 seems to be erratic, but I haven't (and won't) bother to investigate  
 further.  If her system is up to 2 seconds off for a bit after
 waking from sleep, so be it.  (If I ever start using kerberos around
 the house, I will have to address that.)
 
 If a machine is up for months, ntpdate may have been run in the  
 distant past, so you can still a fair amount of error.
 
 ntpd is really a very light weight thing.  When things are ticking  
 over nicely, it may make just one query every few hours and still  
 keep very good time.

I was questioning the need for a low-drift system clock on a machine
that *is* running ntpd, not the need for ntpd.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Time Synchronizing Between Two Servers

2007-05-07 Thread jdow

From: Jeffrey Goldberg [EMAIL PROTECTED]


On May 7, 2007, at 5:02 PM, RW wrote:


If the time error is zeroed by ntpdate, and there's a drift-file, I
don't see that the actual drift value makes much difference. I suspect
that any quartz clock is overkill.


As someone already mentioned, drift data doesn't really solve the  
problem if the amount of drift varies (often with temperature, and  
sometimes dramatically with sleep).  The clock on my wife's G5 iMac  
seems to be erratic, but I haven't (and won't) bother to investigate  
further.  If her system is up to 2 seconds off for a bit after waking  
from sleep, so be it.  (If I ever start using kerberos around the  
house, I will have to address that.)


If a machine is up for months, ntpdate may have been run in the  
distant past, so you can still a fair amount of error.


ntpd is really a very light weight thing.  When things are ticking  
over nicely, it may make just one query every few hours and still  
keep very good time.


Also, if you have a server facing the Internet, you may wish to run a  
public NTP service on it and contribute it to pool.ntp.org, see


 http://www.pool.ntp.org/join.html

for info.


Real NTP goes up to 1024 seconds between polls in my experience. (And it
NEVER jam sets when it is working correctly. Of course, with MS crap this
is not necessarily true.)

{^_^}   Joanne
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Time Synchronizing Between Two Servers

2007-05-07 Thread Jeffrey Goldberg

On May 7, 2007, at 6:53 PM, RW wrote:


I was questioning the need for a low-drift system clock on a machine
that *is* running ntpd, not the need for ntpd.


Ah, sorry.

However, I was adding a somewhat pedantic point of distinguishing  
between low drift and inconsistent drift.  High but consistent  
drift is better than low but inconsistent drift.


Anyway, jdow obviously knows much more about this than I do.  So I  
will defer to her.


Cheers,

-j


--
Jeffrey Goldberghttp://www.goldmark.org/jeff/

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Time Synchronizing Between Two Servers

2007-05-04 Thread RW
On Thu, 03 May 2007 11:07:34 -0400
Chuck Swiger [EMAIL PROTECTED] wrote:

 Sun SPARC machines have good HW clocks, and also some of the newer
 Macs also seem to have consistently low values in ntp.drift and
 handle timekeeping well.
 

Does that matter?

The RTC time is almost immediately overridden by ntpdate. The
drift is a systematic error that ntpd allows for. I would
have thought that the only significant issue, is whether the system
loses timer interrupts under load.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Time Synchronizing Between Two Servers

2007-05-03 Thread Jeffrey Goldberg

On May 2, 2007, at 8:45 PM, Duane Hill wrote:



I have two servers that have to have their time synchronized  
between the two to within one second. What is recommended?


Currently, I have ntpd running on one and have the other  
synchronizing it's time off the first.


ntp is the right way to do things.  You could set each server as an  
ntp peer of the other.  That way.  And, as others pointed out, you  
could have one of them use an external lower stratum (closer to  
reference servers) to sync properly with the rest of the world which  
is useful if you want your logs to match up properly with the rest of  
us.


-j

--
Jeffrey Goldberghttp://www.goldmark.org/jeff/

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Time Synchronizing Between Two Servers

2007-05-03 Thread Duane Hill

On Thu, 3 May 2007, Jeffrey Goldberg wrote:


On May 2, 2007, at 8:45 PM, Duane Hill wrote:



I have two servers that have to have their time synchronized between the 
two to within one second. What is recommended?


Currently, I have ntpd running on one and have the other synchronizing it's 
time off the first.


ntp is the right way to do things.  You could set each server as an ntp peer 
of the other.  That way.  And, as others pointed out, you could have one of 
them use an external lower stratum (closer to reference servers) to sync 
properly with the rest of the world which is useful if you want your logs to 
match up properly with the rest of us.


Yes. I have the one server set to sync with the world and the other 
server syncs its time off the first. The two servers insert and update 
information in a MySQL table and one such piece if information is based on 
time. Everything we do here is all based on UTC.

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Time Synchronizing Between Two Servers

2007-05-03 Thread Chuck Swiger

[EMAIL PROTECTED] wrote:
[ ... ]

It may very well be noisier than just serving out ntp
to the local network, what with talk about elections
and such every 4 minutes, but generally everything
is kept within 0.050 seconds (and running ntpd on
all of the local machines feels like serious overkill).


Simply setting the date upon system boot and maybe once a day using cron to 
call ntpdate or whatever is probably good enough for any client machine, and 
OK for non-important servers where the exact timekeeping doesn't matter much.


ntpd is tiny by modern standards-- it's much smaller than a single Perl or 
Apache httpd+mod_Perl/PHP/whatever child process.  :-)  Normally, you choose 
your three (or more) most important servers, and run NTPd on them in a 
peer-aware ring with some external servers from the NTP pool, and then call 
ntpdate or run ntpd against only your local NTP resources for the rest of your 
machines.


Sun SPARC machines have good HW clocks, and also some of the newer Macs also 
seem to have consistently low values in ntp.drift and handle timekeeping well.


--
-Chuck

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Time Synchronizing Between Two Servers

2007-05-03 Thread Dag-Erling Smørgrav
Chuck Swiger [EMAIL PROTECTED] writes:
 Simply setting the date upon system boot and maybe once a day using
 cron to call ntpdate or whatever is probably good enough for any
 client machine, and OK for non-important servers where the exact
 timekeeping doesn't matter much.

Why, when setting up ntpd is so easy?

On your router:

# hostname
router.example.com
# cat /etc/ntp.conf
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
^D
# cat /etc.rc.conf
ntpdate_enable=YES
ntpd_enable=YES
^D
# /etc/rc.d/ntpdate start
# /etc/rc.d/ntpd start

On every other machine in your network:

# cat /etc/ntp.conf
server router.example.com
^D
# cat /etc.rc.conf
ntpdate_enable=YES
ntpd_enable=YES
^D
# /etc/rc.d/ntpdate start
# /etc/rc.d/ntpd start

Everything else is already taken care of.

DES
-- 
Dag-Erling Smørgrav - [EMAIL PROTECTED]
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Time Synchronizing Between Two Servers

2007-05-02 Thread Duane Hill


I have two servers that have to have their time synchronized between the 
two to within one second. What is recommended?


Currently, I have ntpd running on one and have the other synchronizing it's 
time off the first.


Thanks
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Time Synchronizing Between Two Servers

2007-05-02 Thread Jeff Mohler

Is that working?

If it is..seems you nailed it.


On 5/2/07, Duane Hill [EMAIL PROTECTED] wrote:



I have two servers that have to have their time synchronized between the
two to within one second. What is recommended?

Currently, I have ntpd running on one and have the other synchronizing
it's
time off the first.

Thanks
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to 
[EMAIL PROTECTED]


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Time Synchronizing Between Two Servers

2007-05-02 Thread Duane Hill

On Wed, 2 May 2007, Jeff Mohler wrote:


Is that working?

If it is..seems you nailed it.


It is working. I just didn't know if there was another way. I will 
continue on with the way it is. Thanks.

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Time Synchronizing Between Two Servers

2007-05-02 Thread Dan Nelson
In the last episode (May 03), Duane Hill said:
  On Wed, 2 May 2007, Jeff Mohler wrote:
 
  Is that working?
 
  If it is..seems you nailed it.
 
  It is working. I just didn't know if there was another way. I will
  continue on with the way it is. Thanks.

Yes, ntp is the best way to synchronise time.  If you also point one of
the machines to some pool.ntp.org servers, you will also be in synch
with the rest of the world :)

http://www.pool.ntp.org/

-- 
Dan Nelson
[EMAIL PROTECTED]
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Time Synchronizing Between Two Servers

2007-05-02 Thread [EMAIL PROTECTED]

On 02/05/07, Duane Hill [EMAIL PROTECTED] wrote:

On Wed, 2 May 2007, Jeff Mohler wrote:

 Is that working?

 If it is..seems you nailed it.

It is working. I just didn't know if there was another way. I will
continue on with the way it is. Thanks.


I prefer to have one machine (generally something
with a server class motherboard since those seem
to have better clocks) running ntpd(8) and it also
serving as the local timed(8) master (-F localhost -M).

It may very well be noisier than just serving out ntp
to the local network, what with talk about elections
and such every 4 minutes, but generally everything
is kept within 0.050 seconds (and running ntpd on
all of the local machines feels like serious overkill).

--
--
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]