On a related note, I've read the following:
http://en.wikipedia.org/wiki/Crystal_oscillator
Care must be taken to use only one crystal oscillator source when
designing circuits to avoid subtle failure modes of metastability in
electronics. If this is not possible, the number of distinct
Upgrading and Repairing PCs, 17th Edition states:
A crystal oscillator controls clock speeds using a sliver of quartz
sometimes contained in what looks like a small tin container. Newer
systems include the oscillator circuitry in the motherboard chipset, so
it might not be a visible separate
Suppose I build something like that. Is there anything I should
tinker with to tell ntpd that it has a (very) stable local clock?
fudge 127.127.1.0 stratum 1. (-:
In fact, isn't this how stratum 0s are born?
That's not the case I was interested in.
I was thinking about a PC that had a clock
According to https://buy.garmin.com/shop/shop.do?pID=223pvID=796, it
says that the receiver has an integrated magnetic base.
I had never noticed the magnet. Being curious, I tried
it with a handy knife. Yes, there is a magnet there.
Even if it doesn't, Richard's solution of some glue and a
In that case, I may need to use D cells or a power adapter for long-term
use.
The usual hack is to power it off one of the modem control signals.
It doesn't take much power. I'd add a regulator if I couldn't find
the specs on the chip.
--
These are my opinions, not necessarily my employer's.
I believe NMEA is pretty much useless without a PPS signal.
NMEA works without a PPS. It just doesn't work as well as
people would like. I'd expect offsets in the tens of milliseconds
range rather than microseconds. I don't have any data to back that
up but I can get it if it will help.
The
Why not assigning a new driver number for PCI-SyncClock32 and add a
refclock_psc.c?
It's a minor pain to maintain a driver. If it's reasonable, it's
much simpler ovarall to make a single driver support several
versions of hardware.
My guess is that the hardware out on the board is pretty much
Thanks for the clarification, and the hint with a CDMA-based solution.
Of course, ideally, I would try to have a stratum 1 clock available on
the LAN, avoiding the network to mess up my highly accurate time from
the reference clock. Actually this might be the only solution, as cell
phones might
Does anyone know what the comparative GPS timing accuracies are in
precision and degraded modes?
http://www.leapsecond.com/pages/saoff/
http://www.niceties.com/atomic.html
My feeling is that even degraded would still be within the
microsecond.
The graphs in the above URLs used gear
I see that nslookup resolves 1.us.pool.ntp.org into a list of IP
addresses. How does ntp deal with several IP addresses for a single
nerver name in ntp.conf?
It picks one and remembers it until you restart ntpd.
Most DNS servers deliver the set of addresses in random
order to distribute the
Here's what I want to do. I want to change the clock on my client
machine will automatically adjust when I change the clock on my server
machine. And I also want to record the statistics.
You are in the wrong place, then. ntpd has various defences against
machines that suddenly change their
Basically, what i need is how to add to the ntp.conf to record
statistics. Thanks
There is good documentation.
If you have the sources, it's (probably) in .../html/monopt.html
If not, google for clockstats peerstats and you should be
able to find a copy easily. (finding one that matches your
to synchronize time stamping in pharmaceutical enviromental it is possible
to connect all production machines via NTP to one Time Server.
Question is: Is there a procedure developped to validate that all
connected production machines do have the same time adjusted than the time
server? This
I'm totally confused by this series of snapshots below. Running 4.2.4p3
under NetBSD-3.1/patch, I started seeing some strange steps this morning.
I added some logging instructions in ntp.conf and restarted the daemon,
and the steps keep happening.
remote refid st t when poll
Therefore, whether I use the PIT or the LAPIC timer, ntpd will compute a
similar frequency offset (that of the crystal driving these two timers).
Right?
That's what I would expect.
If you are interested in this area, I suggest collecting and graphing
some data. Can you see any difference?
Suppose a ntpd client can pick up 5 random pool servers, and
periodically (say once a day) replace the 'worst' server
by randomly picking a new one.
I like that suggestion. But, ignoring implementation details,
how do you decide which of two servers is better?
Is a server with 20 ms of jitter
The use of broadcast mode is important because it will allow the fixed
systems on the LAN to notice the laptop within ~ 64 seconds after it is
ready to be be a server.
Broadcast mode sounds like asking for troubles when you are plugged
into the net at the site you are visiting. It would be too
I like that suggestion. But, ignoring implementation details,
how do you decide which of two servers is better?
If one would get a '*' or '+' in ntpq peers output, and the other
would not, then it is better?
That tells you which is better right now. I was looking for
something that averaged
Unless I missed something, the RTC raises the IRQ when the fraction of a
second is 0.5 NOT when it is 0.
Or is something perhaps delaying the IRQ by 500 ms every time?
In other words, when I receive the IRQ, and the RTC tells me the date,
it is, in fact, 500 ms later than this date?
It sounds
The SHM driver is only useful if you have a helper program running
outside ntpd which writes to the SHared Memory segment.
Does anything other than gpsd write info to the SHM segment?
The (atomized) NMEA driver is the correct one to use with the GPS-18LVC.
I've lost track of PPS support in
In article [EMAIL PROTECTED],
Evandro Menezes [EMAIL PROTECTED] writes:
Many former participants of the NTP pool have stated that they
continue with lingering NTP clients after they left the pool.
The reason for this is that ntpd never tries to resolve an IP address
(assuming that *.pool.ntp.org
What does ntp_gettime returns if the connection has failed?
You can do the experiment yourself. Just unplug the ethernet cable.
--
These are my opinions, not necessarily my employer's. I hate spam.
___
questions mailing list
It is de-rigueur to mention Linux and missing interrupts in the same
sentence. Even in the absence of evidence.
RedHat 7.2 shipped with IDE disks running with DMA disabled.
(I assumed that was a workaround for buggy hardware or a buggy driver.)
That was ok with light loads which covered most
It seems as if ntpd can't keep its clock syncronized. It's
drifting about 6-10 minutes per day, well over the 500 ppm limit.
After reading some of the posts on this newsgroup, I have come
to realize that debugging ntp problems can be quite complex.
Before I launch into a detailed search, I
A bad clock frequency is a hardware issue!
Or a software bug. There have been troubles with roundoff
on the value of the tick size.
--
These are my opinions, not necessarily my employer's. I hate spam.
___
questions mailing list
In article [EMAIL PROTECTED],
Patrick Nolan [EMAIL PROTECTED] writes:
On 2007-10-20, Steve Kostecke [EMAIL PROTECTED] wrote:
The hardware clock in a PC is made of exceedingly cheap components. A
common quartz wristwatch is a better clock.
I have noticed this. Until WWV-controlled clocks came
In article [EMAIL PROTECTED],
Patrick Nolan [EMAIL PROTECTED] writes:
I'm located at a university. We have a master Stratum 1 server with a
GPS clock. We also have at least four Stratum 2 servers which are
synchronized with the master. These are all very close, with delay
and offset values
I'm very confused on how ntp do the timestamp. I'm running ntpd on
vxworks. It seems to me that the receive timestamp and the Transmit
timestamp are using two different clocks, because when I use ethereal/
wireshark to look up the information of the ntp packet,
the Recevie Time Stamp is: Jan 1,
server 192.168.1.1
# by default ignore all ntp packets
restrict default ignore
# allow localhost
restrict 127.0.0.1 mask 255.255.255.255
# accept packets from...
restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.4.0
The second thing, is that ntp through NAT would get a variable latency
point (since NAT speed of most routers vary with router traffic load).
I think you are talking about CPU latency rather than wire latency.
(Wire latency doesn't depend upon NAT.)
My NAT box is a DSL router. The CPU is
I known now, in the recent kernel the internal frequency will be to 250
Mhz...
With my gentoo it's not a problem for me because i make myself my kernel ...
but if i take Mandriva or a another distribution how find this values ?
I assume you mean the scheduling clock which would be 250 Hz rather
On Linux, a simpler way can be to look at /proc/interrupts - e.g.
(probably Linux-version- and possibly config-specific):
$ (cat /proc/interrupts; sleep 10; cat /proc/interrupts) | \
awk '/timer/{prev=now; now=$2} END{printf %dHz\n, int((now-prev)/10)}'
This yields 41Hz on my Via C7
The current SHM driver looks for a sample each polling interval.
The documentation for gpsd suggests using minpoll and maxpoll of 4.
I've fixed the SHM driver to look every second. The refclock
infrastructure already had a FIFO and code to process things
like this. This reduces the jitter
Although from all the other replies (thanks everyone), it sounds like
maybe there really just aren't tons of other cheap OEM options.
Your original request wanted a PPS signal. If you don't really
need that, you have a lot more options. There are many low cost
GPS units with USB connections.
Would such a document really be all that useful? It seems to me that the
codebase is pretty small, and it should not be all that hard to figure out.
I use
grep whatever *.c
or
grep whatever -r .
If that doesn't work in the directory I'm working in,
then I cd .. and try again. After doing
I had my computer count the lines of code in one of the older versions,
about two years ago. 70,000 or so lines of code may be a small number
to you but it seems like a lot to me. I once looked at the code for the
Motorola Oncore driver to try to understand what was going on and failed.
It
Routers generally do not do NTP in any way, shape, or form! They don't
need to know the time!
That's misleading.
Routers often include anti-spam/abuse mechanisims which get logged.
It helps if the time stamps on the log files are correct. It's easier
to get that if the router itself uses NTP
I don't think it's a hardware product defect. Because we have two of
them, and we had run the same test on both of the board, we still got
the same error. Thanks,
There are two kinds of hardware (or software) errors. One is a design
error. The problem happens all the time on all units. The
Why do you feel that your router needs to know the time?
Log files for security incidents.
--
These are my opinions, not necessarily my employer's. I hate spam.
___
questions mailing list
questions@lists.ntp.org
In article [EMAIL PROTECTED],
[EMAIL PROTECTED] writes:
Does anybody know of any *practical* samples on how to
implement NTP/SNTP client?. The goal is to provide accurate
time for a program/client running on Windows Vista.
Just curious. Why do you want to roll your own as compared
to run one of
Yes. And one missing step:
After stopping ntpd, ntpdate -b [server].
The single most common cause of a large negative drift value
is a system whose clock has not been initialized before starting
ntpd (or has been stopped during a large time adjustment).
What vintage of ntpd are you using?
The NMEA driver uses 3 mode bits. I think more are available.
The docs say that flag1 is unused for the NMEA driver.
It would be really nice if there was a clean way to specify the
baud rate.
Several of the NMEA units I'm familiar with allow you to set
the baud rate they use. I expect there
In article [EMAIL PROTECTED],
[EMAIL PROTECTED] (linux newbie) writes:
HI,
Need following clarification.
Our Application needs to have two individual hardware (with DSP processor)
to have same crystal clock freqency. Though individual boards are alike, due
to environmental factors there might be
It would be really nice if there was a clean way to specify the
baud rate.
This is a general issue -- there are other drivers (e.g. the HP 58503)
that also assume a fixed baud rate and parity/stop bits. They cause a
lot of problems when trying to use not-quite-identical hardware (as in
1. Accord GPS Clock spits out NMEA at 9600 baudrate
The NMEA specs call for 4800. Is there some way to set
the Accord box to run at 4800? That seems only sensible
if they are in the NMEA market.
2. It has custom NMEA format for GPS (and the driver is
intended to use this )
I
I don't know; it seems to me better for the refclock driver to specify a
default set of comm parameters that will work for the standard device
that refclock supports, but have a universal mechanism that would allow
overriding those params from the config file and would work for any driver.
Over the years and with some effort several drivers now support more
than one device. All TrueTime models are supported by one driver; all
Spectracom models by another and all telephone modem services by a
third. With few exceptions, a NMEA radio is a NMEA radio and I strongly
suspect many of
Perhaps the best way is to add a completely new command refclock and
load it up with all kinds of options, including those now done by the
fudge command. The command would not need the dotted quad, which could
be replaced by the driver name already in a table. The syntax could in
principle be
Too many packaging of ntpd configure a local clock driver
in their sample configuration.
Does anybody know why that got started?
What urban legend do we have to stomp out before that practice will go away?
--
These are my opinions, not necessarily my employer's. I hate spam.
It would be very handy if one could monitor NTP with the same tools used
for monitoring other aspects of performance, rather than having to use
special programs like ntpq and parse its output. I'm thinking of this an
addition to the present tools, though, not a replacement.
Is there anything
Don't. Just have the Linux servers point to external servers like the
pool and then you don't have to change anything later. You are likely to
get better quality time service that way. Tardis is a bit short on
technical information so it's hard to tell what it does, how it does it
and how
assID‡73 status14 reach, 1 event, event_reach,
srcadrmaster1, srcport3, dstadr0.0.0.0,
dstport3, leap
You see, it's garbage again. It's hard to keep up under this circumstances.
My copy wasn't garbled.
It looked like this:
assID=8773 status=1014 reach, 1 event, event_reach,
The nodes' drifts are high:
# cat /etc/ntp.drift
node-A: 499.206
node-B: 497.070
500 ppm is the limit.
The next day, after restarting ntpd on the nodes and resetting
the time on all nodes with ntpdate, everything worked as
expected with the time syncing properly, no false tickers, and the
One possibility may be that the GBS18 has switched to Garmin mode
instead of NMEA mode.
That seems unlikely. You have to send it a sensible command.
Garbage on the line is not likely to do that.
On the other hand, maybe he ran some software that knows
about Garmin GPS units...
My GPS18
Some stratum 2 servers I get time from are actually polling stratum 1's
that have a pretty significant round-trip (more than 70ms), so is this any
better than having my ntp getting its time from a stratum 3 or 4 that's
closer to its own upstream sources?
A long round-trip time isn't evil all
Why is the '.SHM.'-clock not even considered? Because of the negative offset?
Negative offsets are not a problem. They happen all the time.
My guess is that something your new driver is setting up
makes your clock look not-very-good. It's probably one of the
obscure constants that aren't well
I am seeing strange behaviour on my _x86_64 Fedora 7 desktop
workstation with regard to the system-cmos time that `adjtimex'
reports.
That leaves the RTC doing the jumping. But having an RTC that is
runing nearly 1 ppm slower than my system clock and which jumps
ahead every 10 seconds seems
When I wrote that I was unaware of the html page. I'm a Unix guy and I
generally don't even consider looking for html docs - I am used to (and
expect) man pages.
Me too.
Would it help to ship dummy man pages that just pointed to
the html documentation?
--
These are my opinions, not
The radio receiver will be late by of the
order of millisec, while the gps will be on time to microseconds.
Of course you may not care, but someone who has two onboard time sources
would care, I would think.
The radio propagation delay can be fudged out.
How stable is
Currently I'm syncing against NMEA/PPS and seeing quiet a big offset:
remote refid st t when poll reach delay offset jitter
==
xGPS_NMEA(0) .GPS.0 l7 16 3770.000
To my pc I have connected a garmin 18lvc and a HBG radio clock receiver.
Now what I would like to do is:
- let the hbg receiver set the current time and when the HBG signal is
not available (bad reception or the 59 seconds while it is receiving
the broadcastmessage) use the PPS signal of the
Ok but the odd thing is: a friend of mine has the exact same garmin 18
lvc but not this big offset?
Your PPS isn't working. That could be either hardware or software.
What OS are you using? What version of ntpd?
What is your friend using?
How about taking your unit over to his place and
In article [EMAIL PROTECTED],
Bill Unruh [EMAIL PROTECTED] writes:
I have been keeping track of the clocks on my various systems. One is
synced using ntp from a GPS 18LVM pps source. The others are synced from
What is a GPS 18LVM?
I've seen it mentioned several times recently. From the
On recent Linux kernels, I think the drift file is always bad after reboot.
HZ=100, no dynamic ticks aka tickless system (CONFIG_NO_HZ not set). I think
I even tried with a kernel command line option lpj= but it didn't help.
If the system is rebooted, ntpd stabilizes to a new different drift
I am afraid I simply do not believe this. NMEA is lucky to get a ms not a
usec. The offset on the NMEA should be a lot bigger than .001
The NMEA driver includes built-in PPS support.
--
These are my opinions, not necessarily my employer's. I hate spam.
However, you give me an idea. Why not shut down the burst when the clock
filter delivers the first sample? Gotta think about that.
That would make sense if the goal is to get an answer on a lossy
system.
I think the background in this discussion was not loss, but delays
in setting up
First, somebody gets to decide if this is really a bug in the NTP code or if
it is a bug in GCC.
It could also be a glitch in the included header files.
--
These are my opinions, not necessarily my employer's. I hate spam.
___
questions mailing
What does all over the map mean. they will be withing a few tens of
microseconds of that server.
I'd expect a few to tens of ms rather than microseconds.
If you manually set the time on the server, I'd expect it to
take a while for the clients to catch up.
This question comes up often enough
So, no, I am comparing apples to apples ( the offsets as determined from
the ntp packet exchange mechanism which both use and both report).
Another approach is to setup a 3rd machine to watch both.
--
These are my opinions, not necessarily my employer's. I hate spam.
I was talking about what people could expect from software that behaved
well; I think you are describing what ntpd actually does here. My point
was that ntpd's ability to tolerate really rotten links is irrelevant
for most users, who are only about 20ms away from their ISP's time
server, and
Iam trying to install ISC NTP-4.2.4 on RHEL 4. But Im not able to start
the services.
I firstly removed the default rpm and then compiled the latest version,
configured the ntp.conf file, but Iam not able to start the service. I
give the command /etc/init.d/ntpd start. # prompt comes back.
I have problem with my stratum1 box. My box include: freebsd
5.2-RELEASE without PPS , ntp-4.2.0_1, Intel Pentium III, 40 G IDE,
Germin GPSmap60 , connect with serial com. In GPSmap60 set as NMEA.
Freebsd box can get NMEA input string. My configuration of ntp.conf
like this:
server
So its the DISK I/O thats causing loss of ticks ?
My first Red Hat system defaulted to no-DMA for IDE disks. (Yes,
that was a long time ago.) With that setup, it was simple to generate
lots of lost timer interrupts: just keep the disk busy doing reads.
(Seeks don't count. Read consecutive
I've never used the GPS18LVC (I have a Motorola M12+T) but if you know
your exact location you only need one satellite to get the time. There
are four equations in four unknowns (lattitude, longitude, elevation,
and time). Not knowing any of these variables, you need four
satellites. Once
Does the GPS18LVC provide an oscillator to serve time even when there
are not enough satellites in sight?
No.
--
These are my opinions, not necessarily my employer's. I hate spam.
___
questions mailing list
questions@lists.ntp.org
I didn't find a reasonable place to insert it in the wiki so
I parked it here for now.
This is a hack that just reads a disk as fast as it can.
It was very good at tickling the lost-interrupt problem
on an old RH system which was not using DMA on the disks.
In article [EMAIL PROTECTED],
[EMAIL PROTECTED] (Peter Laws) writes:
I've resuscitated a left-for-dead TS 2100 and now am in need of a Trimble
Bullet III, or equivalent, active GPS antenna.
It's a current item for Symmetricom and they want more dinero than I want
to put into this 12 year old
In article [EMAIL PROTECTED],
Harlan Stenn [EMAIL PROTECTED] writes:
In article [EMAIL PROTECTED], [EMAIL PROTECTED] (Hal Murray) writes:
Hal I didn't find a reasonable place to insert it in the wiki so I parked
Hal it here for now.
Hal http://www.megapathdsl.net/~hmurray/hacks/read.c
How
I am willing to believe, barring contrary evidence, that your faithfully
implimented the protocol. (and that the Linux people faithfully implimented
the protocol in the kernel discipline routines).
Don't be so quick to believe that things work as expected.
I seem to remember a report of some
The basic idea is to do the time stamping in hardware deep in
the network adaper. That avoids lots and lots of jitter.
Yes, PTP can yield an accuracy better than 100 ns if both the NICs at the
clients and the server support hardware timestamping of sent/received PTP
packets.
I am still
Yes, PTP can yield an accuracy better than 100 ns if both the NICs at the
clients and the server support hardware timestamping of sent/received PTP
packets.
On the other hand, also *every* network node between the PTP endpoints has
to be PTP-aware and compensate the packet delay it introduces,
The ntp log file shows when NTP steps the time. But then the potential harm
is already done. Especially if the time moves backward, our server might
have serious trouble. Is there a log event which indicates that the time is
going to be reset in order to enable us to take appropriate action
My current problem is that drift settles at 82ppm (what I called 90 in
previous email) in one run and then 32ppm in another run (with a reboot
between). This is similar to the problem I had with stepping disabled
where drift would go from +500ppm in one run and then swing all the way
to -500ppm
Fine! Don't touch anything, happy man, or it might well tomber en
marche. Real men don't want the eleven-minutes mode. It is not only
extremely inaccurate by itself, but it also steps on the toes of those
tools that are able to manage the RTC properly.
Would somebody please collect this info on
As I've stated before, I don't believe the oscillator is really this
unstable, but I could be wrong. After all, my CPU measurements varied
much more than yours, especially from one run to the next. However, I'm
still open to the possibility that linux's approach to speed measurement
is less
Can someone point me to a good document that
shows how to setup a Time Server? I have an
isolated network that cannot get to the Internet
to sync time. I have Solaris 8,9,10 and Red Hat Linux
Advanced Server 4 servers as potential time server
candidates.
I don't know of any document that covers
I assume that you mean to use both the refclock_atom and the refclock_nmea
drivers. (127.127.22.0 and 127.127.20.0)
The NMEA driver automagically uses the PPS stuff, whether you want
it to or not.
It would be nice to have a flag to disable that feature so you
could collect data on the NMEA text
Well, no. 1 or 2 sec is 10-20PPM which is on the good side. 43 sec per day
is like 500PPM which is definitely on the high side. 5-10sec per day is
more typical. Note that chrony(on linux) will fix 43s/day. (It will use the
fast
slew-- ie changing the tick size-- as well as the slow slew.) ntp as
Do you know any code that cares if that is wrong by 10% (which would be
10PPM) Ie, is 10% error insane?
Is 1% (1PPM)?
Ie, .05% seems a bit extreme for that.
I used to do a lot of performance measurements.
For the stuff I was doing, 10% is easy to spot. 1% is borderline.
--
These are
The problem here is that the distribution does not contain a decent
assortment of example configuration files for common configurations. So
the OS distributors/aggregators/vendors each cobble together their own
one size fits all configuration file.
But does a local refclock make sense in a
$!/bin/sh
if ! ntpq|grep '* name.of.router'/dev/null; then
mail -s NTP sync has been lost [EMAIL PROTECTED]
fi
And put the shell script into crontab to run every minute.
(your milage may vary)
That could lead to a lot of clutter in your mailbox.
--
These are my opinions, not necessarily my
I actually don't really understand why a GPS would improve my
synchronisation.
If your master server has GPS, its time will be more stable.
That means your clients can sync to a stable clock rather than
trying to track a clock that is wobbling around. (Things
are simpler and work better when
Sorry it's actually difficult for me to precise this topic for
confidentiality reason of course and also because I'm an intern and thus
there for a short period. Hence I don't know exactly why myself. This is
just one of the requirement of my project.
Keeping a handful of machines synchronized to
I am using NTPd and I have guessed that it uses the High Resolution Timer,
brought to the linux kernel since the 2.6.16 release.
Does anyone know if it said somewhere in the documentation or any other
official source, a trusty one.
ntpd just calls the kernel.
Recent Linux kernels have several
I haven't seen that problem since I added
clocksource=acpi_pm
on the boot time parameters.
What version of the kernel do you have? 2.6.17 does not have clocksource
parameter or an acpi_pm option. Is there anywhere where these different
clock parameters are defined?
I'm using 2.6.23.
Does the HPET support enable microsecond resolution for hardware clock
reads in Linux (the HPET hardware is supposedly capable of this)?
I don't know. A quick test with my kernel seemed to ignore
that chunk of the command line. I may have it turned off in
the config file.
--
These are my
I would like to connect to any server to receive a string where it is
written the istant time (possibly hh.mm.ss.xxx ). I found several sites
where I may read hh.mm.ss then downloading the page and reading it I could
get the string hh.mm.ss but
I don't know of any servers that do that.
In the
In article [EMAIL PROTECTED],
Noob [EMAIL PROTECTED] writes:
Hello,
One of my systems had been running for ~6 weeks. During this time,
the frequency offset computed by ntpd remained stable between
-6.6 ppm and -6.2 ppm.
On April 30, the time offset started climbing and reached 14 ms in
On April 30, the time offset started climbing and reached 14 ms in
approximately 90 minutes. ntpd started bumping the frequency offset,
down to -9.4 ppm.
Is this the Linux TSC bug? Are you running a recent Linux kernel?
Did you reboot the system at that time?
--
These are my opinions, not
I didn't check the amount of the jump. I would not expect a jump of more
than a few ppm (due to the crystal itself), and not more than about once a
year.
It was 3 or 4 ppm.
That's big enough that I think it very unlikely, but not impossible.
Even if it happened only once per year, somebody
1 - 100 of 324 matches
Mail list logo
