Re: Introduction of long term scheduling

[Zefram]

Daniel R. Tobias wrote:
>Formulas for UTC, as actually defined at the time, go back to 1961
>here:

But that involves rubber seconds, which is quite a big complication to add
to your TAI<->UTC conversion.  If you're going to handle pre-1972 times,
I think you really need to decide what you'll do prior to 1961 (when
UTC doesn't exist at all) and prior to 1955 (when there is no atomic
timescale).  POSIX punts on this: prior to 1972 time_t is implicitly
tied to an unspecified variety of UT.  (I wrote a bit about this aspect
of time_t in the Wikipedia article [[Unix time]].)

To be rigorously correct, applications should have distinct
representations for TAI dates, UTC dates, and UT dates.  TAI dates can't
legitimately exist that precede the first atomic timescale, and UTC dates
can't legitimately exist that precede either 1961 or 1972 (depending on
which concept of UTC is in use).  UT dates can exist for any time back to
four or five gigayears ago.  TAI and UT dates can exist arbitrarily far
in the future; future UTC dates can be considered, but some have only a
tentative existence.  Conversions can't be done very far into the future.

How things would work out in a system that pervasively used this
rigorously correct model I'm not sure.  We've already discussed
the aspects relating to present and future times.  For distant past
times it'd be necessary to explicitly process timestamps in UT form.
Possibly TT could also be used in some form, for interval calculations
in the pre-caesium age.

Whatever timescales are used for pre-1955 dates are, of course, also
available for present and future dates.  Perhaps, in this system,
many applications dealing with distant past dates would just use TT and
vague-UT for present and future dates as well.  This sounds sensiblish
to me: sub-second contemporary timestamps and also pre-caesium timestamps
in the same application seems like a specialised requirement.

-zefram

Re: Introduction of long term scheduling

[Tony Finch]

On Sun, 7 Jan 2007, Daniel R. Tobias wrote:
>
> Formulas for UTC, as actually defined at the time, go back to 1961
> here:

You helpfully snipped the part where I said that it probably isn't
worth implementing rubber seconds.

> ftp://maia.usno.navy.mil/ser7/tai-utc.dat
> It appears the offset was 1.4228180 seconds at the start of this.

If you extend the initial period backwards to the start of 1958 then the
offset drops close to zero.

Tony.
--
f.a.n.finch  <[EMAIL PROTECTED]>  http://dotat.at/
FITZROY: WEST BACKING SOUTHWEST 5 TO 7, OCCASIONALLY GALE 8. ROUGH OR VERY
ROUGH, OCCASIONALLY HIGH. SHOWERS, THEN RAIN. MODERATE OR GOOD.

Re: Introduction of long term scheduling

[Daniel R. Tobias]

On 8 Jan 2007 at 0:15, Tony Finch wrote:

> How did you extend the UTC translation back past 1972 if the undelying
> clock followed TAI? I assume that beyond some point in the past you say
> that the clock times are a representation of UT. However TAI matched UT in
> 1958 and between then and 1972 you somehow have to deal with a 10s offset.

Formulas for UTC, as actually defined at the time, go back to 1961
here:

ftp://maia.usno.navy.mil/ser7/tai-utc.dat

It appears the offset was 1.4228180 seconds at the start of this.

--
== Dan ==
Dan's Mail Format Site: http://mailformat.dan.info/
Dan's Web Tips: http://webtips.dan.info/
Dan's Domain Site: http://domains.dan.info/

Re: Introduction of long term scheduling

[Tony Finch]

On Sun, 7 Jan 2007, M. Warner Losh wrote:
>
> Having tried it, there are a lot of little 33 second anomolies in many
> applications :-(.

How did you extend the UTC translation back past 1972 if the undelying
clock followed TAI? I assume that beyond some point in the past you say
that the clock times are a representation of UT. However TAI matched UT in
1958 and between then and 1972 you somehow have to deal with a 10s offset.
It would be over-engineering to implement rubber seconds for the whole
system when only very few applications need them. I suppose you could
invent a leap second schedule for the 1960s, but perhaps it's more
sensible to define the underlying timescale to be TAI+10 so that your
system makes the universal->atomic time split at the point when UTC was
established.

> I've toyed with the idea of running the kernel in TAI and having 'smart'
> processes tell libc they want no UTC translation and having all the
> TAI<->UTC translation happen in libc (also hacking those FS that want
> UTC time to be able to get it).

It would seem sensible to me to fix time_t's lack of range at the same
time as fixing its model of time. Perhaps the transition model to follow
is the one used on non-BSD systems for large file support, which allows
the developer to either set a compile time flag to get the new behaviour,
or use a wide form of the API, e.g. stat64() instead of stat().

Tony.
--
f.a.n.finch  <[EMAIL PROTECTED]>  http://dotat.at/
ROCKALL: SOUTHWEST 6 BECOMING CYCLONIC 6 TO GALE 8, PERHAPS SEVERE GALE 9
LATER. VERY ROUGH BECOMING HIGH. SHOWERS, RAIN LATER. MODERATE OR GOOD.

Re: Introduction of long term scheduling

[Tony Finch]

On Sun, 7 Jan 2007, M. Warner Losh wrote:
>
> [POSIX time] is designed to be UTC, but fails to properly implement
> UTC's leap seconds and intervals around leapseconds.

>From the historical point of view I'd say that UNIX time was originally
designed to be some vague form of UT, and the POSIX committee retro-fitted
a weak form of UTC synchronization.

Tony.
--
f.a.n.finch  <[EMAIL PROTECTED]>  http://dotat.at/
DOGGER FISHER GERMAN BIGHT HUMBER: SOUTHWEST, VEERING NORTHWEST FOR A TIME, 6
TO GALE 8, OCCASIONALLY SEVERE GALE 9 IN DOGGER. ROUGH OR VERY ROUGH. RAIN OR
SHOWERS. MODERATE OR GOOD.

Re: Introduction of long term scheduling

[M. Warner Losh]

In message: <[EMAIL PROTECTED]>
Rob Seaman <[EMAIL PROTECTED]> writes:
: What is correct is to have a 61 second minute occasionally, neither
: to redo the first second of the next day, nor to repeat the last
: second of the current day.

Unfortunately, that's not POSIX time_t.  And when you are implementing
unix kernel APIs, that is the only game in town, unless you invent
your own.  ntp_gettime comes close to implementing things right (on
FreeBSD the time state is synchronous to top of second), but even that
is a mess.

You can't do both 61 second minutes and also have the 'naive math'
that time_t requires.

I agree with others that have said that ntp should be TAI or GPS based
(eg a linear count of seconds in a timescale that doesn't have leap
seconds), and that leap second corrections should happen in userland
like is done for timezones.  That's really the only sane way to do
things.  Having tried it, there are a lot of little 33 second
anomolies in many applications :-(.  I've toyed with the idea of
running the kernel in TAI and having 'smart' processes tell libc they
want no UTC translation and having all the TAI<->UTC translation
happen in libc (also hacking those FS that want UTC time to be able to
get it).  Maybe I'll try it and see what breaks since the line between
applications and libraries can make a per-process flag hard to cope
with if your application links with a lot of libraries.

Warner

Re: Introduction of long term scheduling

[Rob Seaman]

Tony Finch wrote:


As http://www.eecis.udel.edu/~mills/leap.html shows, NTP (with kernel
support) is designed to stop the clock over the leap second, which I
don't call "correct". Without kernel support it behaves like a
"pinball
machine"  (according to Mills).


Warner Losh wrote:


It implements exactly what ntpd wants.  I asked Judah Levine when
determining what was pedantically correct during the leap second.


Well, I didn't say that that NTP, and Mills and Levine themselves,
currently form a timekeeping model self-consistent with UTC, rather I
was trying to suggest that POSIX wasn't the only game in town.  I've
made more successful rhetorical choices...


I also consulted with the many different resources avaialable to
deterimine what the right thing is.


What is correct is to have a 61 second minute occasionally, neither
to redo the first second of the next day, nor to repeat the last
second of the current day.

Presumably no one would object if the ITU made it easier to obtain a
copy of 460.4.

Rob

Re: Introduction of long term scheduling

[Zefram]

M. Warner Losh wrote:
> But ntp_gettime returns a timespec
>for the time, as well as a time_state for the current time status,
>which includes TIME_INS and TIME_DEL for psotive and negative leap
>second 'warning' for end of the day so you know there will be a leap
>today, and TIME_WAIT for the actual positive leap second itself
>(there's nothing for a negative leapsecond, obviously).

Actually the interface is more complicated than that.  TIME_WAIT indicates
that a leap second recently occurred, and continues to be returned
until the command bit that initiated the leap second has been cleared.
The status during a leap second is meant to be TIME_OOP.  TIME_OK is the
normal state.  If the clock is not properly synchronised then TIME_ERROR
(a.k.a. "TIME_ERR" or "TIME_BAD") is returned instead of the leap state:
the leap second engine still operates in this situation (at least it
does on Linux), but you don't get to see the state variable.

Mills's paper "A Kernel Model for Precision Timekeeping"
 says that the
time_t should repeat the last second of the day.  That would give you
these behaviours:

398 TIME_DEL 398 TIME_OK  398 TIME_INS
400 TIME_WAIT399 TIME_OK  399 TIME_INS
401 TIME_WAIT400 TIME_OK  399 TIME_OOP
402 TIME_WAIT401 TIME_OK  400 TIME_WAIT

Actually, though, the paper doesn't require the state change to be
atomic with the change of time_t.  It's allowed to be slightly delayed.
(It is in fact delayed a few milliseconds on Linux.)  So what is actually
seen is this:

398.5 TIME_DEL 398.5 TIME_OK  398.5 TIME_INS
399.0 TIME_DEL 399.0 TIME_OK  399.0 TIME_INS
400.5 TIME_WAIT399.5 TIME_OK  399.5 TIME_INS
401.0 TIME_WAIT400.0 TIME_OK  400.0 TIME_INS
401.5 TIME_WAIT400.5 TIME_OK  399.5 TIME_OOP
402.0 TIME_WAIT401.0 TIME_OK  400.0 TIME_OOP
402.5 TIME_WAIT401.5 TIME_OK  400.5 TIME_WAIT

There is enough information in there to fully decode it, but it means
looking at more states than would be required in the nominal version.
For example, if you see TIME_INS with a time that appears to be just
after midnight, then you're actually inside a positive leap second.
The second that time_t repeats is neither the second before midnight
[399, 400] nor the second after midnight [400, 401], but something
between those, encompassing midnight, approximately [399.005, 400.005].

Interestingly, once you've got all the decode logic required to handle
this, it's also possible to handle a system where time_t repeats the
first second of the next day.  This system wouldn't use the TIME_OOP
state at all, using TIME_INS to indicate the leap second.  It's just
the extreme end of where the repeated second could be placed.

I think this is a horrible mess and all the leap second adjustments should
happen in user space.  (Filesystems that use UTC-based timestamps would
require the kernel to be able to translate too, but this shouldn't affect
the clock or the APIs.)  NTP has problems because the leap second handling
was grafted on and it has no memory of leap seconds.  Synchronisation
and kernel APIs should use a plain linear count of TAI seconds, because
they're not concerned with time-of-day per se.  Anything that does care
about time-of-day can do the conversion itself, just like anything that
cares about day-of-week or day-of-year.  UTC is a calendar.

-zefram

Re: Introduction of long term scheduling

[M. Warner Losh]

In message: <[EMAIL PROTECTED]>
Rob Seaman <[EMAIL PROTECTED]> writes:
: > If by "some limp attempt" you mean "returns the correct time" then you
: > are correct.
:
: It's not the correct time under the current standard if the
: timekeeping model doesn't implement leap seconds correctly.  I don't
: think this is an impossible expectation, see http://
: www.eecis.udel.edu/~mills/exec.html, starting with the Levine and
: Mills PTTI paper.

It implements exactly what ntpd wants.  I asked Judah Levine when
determining what was pedantically correct during the leap second.  I
also consulted with the many different resources avaialable to
deterimine what the right thing is.  Of course, there are different
explaintions about what the leap second should look like depending on
if you listen to Dr. Levine or Dr Mills.  Dr. Mills web site says
'redo the first second of the next day' while Dr. Levine's
leapsecond.dat file says 'repeat the last second of the day.'
Actually, both of them hedge and say 'most systems implement...'  or
some variation on that theme.

It is possible to determine when you are in a leap second using ntp
extensions with their model.  Just not with POSIX interfaces.  The
POSIX interfaces are kludged, while the ntpd ones give you enough info
to know to print :59 or :60, but POSIX time_t is insufficiently
expressive, by itself, to know.  But ntp_gettime returns a timespec
for the time, as well as a time_state for the current time status,
which includes TIME_INS and TIME_DEL for psotive and negative leap
second 'warning' for end of the day so you know there will be a leap
today, and TIME_WAIT for the actual positive leap second itself
(there's nothing for a negative leapsecond, obviously).

So I stand by my "returns the correct time" statement.

Warner

Re: Introduction of long term scheduling

[M. Warner Losh]

In message: <[EMAIL PROTECTED]>
Tony Finch <[EMAIL PROTECTED]> writes:
: On Sat, 6 Jan 2007, M. Warner Losh wrote:
: >
: > Most filesystems store time as UTC anyway...
:
: POSIX time is not UTC :-)

True.  It is designed to be UTC, but fails to properly implement UTC's
leap seconds and intervals around leapseconds.

Warner

Re: Introduction of long term scheduling

[Poul-Henning Kamp]

In message <[EMAIL PROTECTED]>, David Malone writes:

>FWIW, I believe most hospitals are more than capable of looking
>after equipment with complex maintenance schedules.

It is not just a questoin of ability, to a very high degree
cost is much more important.

--
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
[EMAIL PROTECTED] | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

Re: Introduction of long term scheduling

[Tony Finch]

On Sun, 7 Jan 2007, Rob Seaman wrote:
>
> It's not the correct time under the current standard if the
> timekeeping model doesn't implement leap seconds correctly.  I don't
> think this is an impossible expectation, see http://
> www.eecis.udel.edu/~mills/exec.html, starting with the Levine and
> Mills PTTI paper.

As http://www.eecis.udel.edu/~mills/leap.html shows, NTP (with kernel
support) is designed to stop the clock over the leap second, which I
don't call "correct". Without kernel support it behaves like a "pinball
machine" (according to Mills).

Tony.
--
f.a.n.finch  <[EMAIL PROTECTED]>  http://dotat.at/
SOUTHEAST ICELAND: CYCLONIC 6 TO GALE 8, BECOMING VARIABLE 4 FOR A TIME. ROUGH
OR VERY ROUGH. OCCASIONAL RAIN OR WINTRY SHOWERS. MODERATE OR GOOD.

Re: Introduction of long term scheduling

[Tony Finch]

On Sat, 6 Jan 2007, M. Warner Losh wrote:
>
> Most filesystems store time as UTC anyway...

POSIX time is not UTC :-)

Tony.
--
f.a.n.finch  <[EMAIL PROTECTED]>  http://dotat.at/
SOUTHEAST ICELAND: CYCLONIC 6 TO GALE 8, DECREASING 5 OR 6 LATER. ROUGH OR
VERY ROUGH. OCCASIONAL RAIN OR WINTRY SHOWERS. MODERATE OR GOOD.

Re: Introduction of long term scheduling

[David Malone]

> So you think it is appropriate to demand that ever computer with a
> clock should suffer biannual software upgrades if it is not connected
> to a network where it can get NTP or similar service ?

> I know people who will disagree with you:

> Air traffic control
> Train control
> Hospitals

> and the list goes on.

FWIW, I believe most hospitals are more than capable of looking
after equipment with complex maintenance schedules. They have
endoscopes, blood gas analysers, gamma cameras, MRI machines,
dialysis machines and a rake of other stuff that has a schedule
requiring attention more regurally than once every 6 months.

I am not sure how much un-networked equipment that requires UTC to
1 second and doesn't already have a suitable maintenance schedule
exists in hospitals.

David.

Re: Introduction of long term scheduling

[Rob Seaman]

Warner Losh wrote:


Actually, every IP does not have a 1's complement checksum.  Sure,
there is a trivial one that covers the 20 bytes of header, but that's
it.  Most hardware these days off loads checksumming to the hardware
anyway to increase the throughput.  Maybe you are thinking of TCP or
UDP :-).  Often, the packets are copied and therefore in the cache, so
the addition operations are very cheap.


Ok.  I simplified.  There are several layers of checksums.  I
designed an ASCII encoded checksum for the astronomical FITS format
and should not have been so sloppy.  "They do it in hardware" could
be taken as an argument for how time should be handled, as well.


Adding or subtracting two of them is relatively easy.


Duly stipulated, your honor.


Converting to a broken down format or doing math
with the complicated forms is much more code intensive.


And should the kernel be expected to handle "complicated forms" of
any data structure?


Dealing with broken down forms, and all the special cases usually
involves
multiplcation and division, when tend to be more computationally
expensive than the checksum.


Indeed.  May well be.  I would suggest that the natural scope of this
discussion is the intrinsic requirements placed on the kernel, just
as it should be the intrinsic requirements of the properly traceable
distribution and appropriate usage of time-of-day and interval
times.  Current kernels (and other compute layers, services and
facilities) don't appear to implement a coherent model of
timekeeping.  Deprecating leap seconds is not a strategy for make the
model more coherent, rather, just the timekeeping equivalent of
Lysenkoism.


Having actually participated in the benchmarks that showed the effects
of inefficient timekeeping, I can say that they have a measurable
effect.  I'll try to find references that the benchmarks generated.


With zero irony intended, that would be thoroughly refreshing.


If by "some limp attempt" you mean "returns the correct time" then you
are correct.


It's not the correct time under the current standard if the
timekeeping model doesn't implement leap seconds correctly.  I don't
think this is an impossible expectation, see http://
www.eecis.udel.edu/~mills/exec.html, starting with the Levine and
Mills PTTI paper.


You'd think that, but you have to test to see if something was
pending.  And the code actually does that.


Does such testing involve the complex arithmetic you describe above?
(Not a rhetorical question.)  The kernel does a heck of a lot of
conditional comparisons every second.


Did I say anything about eviscerating mean solar time?


Well, these side discussions get a little messy.  The leap second
assassins haven't made any particular fuss about kernel computing
issues, either, just previous and next generation global positioning
and "certain spread spectrum applications" and the inchoate fear of
airplanes falling from the sky.

The probability of the latter occurring seems likely to increase a
few years after leap seconds are finally eradicated - after all,
airplanes follow great circles and might actually care to know the
orientation of the planet.  Hopefully, should such a change occur
courtesy of WP7A, all pilots, all airlines and all air traffic
control centers will get the memo and not make any sign errors in
implementing contingent patches.  It's the height of hubris to simply
assume all the problems vanish with those dastardly leap seconds.  (I
don't suppose the kernel currently has to perform spherical trig?)

Note that the noisy astronomer types on this list are all also
software types, we won't reject computing issues out of hand.


I'm just suggesting that some of the suggested ideas have real
performance issues that means they wouldn't even be considered as
viable options.


Real performance issues will be compelling evidence to all parties.
Real performance issues can be described with real data.


True, but timekeeping is one of those areas of the kernel that extra
overhead is called so many times that making it more complex hurts a
lot more than you'd naively think.


Either the overhead in question is intrinsic to the reality of
timekeeping - or it is not.  In the latter case, one might expect
that we could all agree that the kernel(s) in question are at fault,
or that POSIX is at fault.  I have little sympathy for the suggestion
that having established that POSIX or vendors are at fault that we
let them get away with it anyway.  Rather, workaround any limitations
in the mean time and redesign properly for the future.

If, however, the overhead is simply the cost of doing timekeeping
right, then I submit that it is better to do timekeeping right than
to do it wrong.  Doing it right certainly may involve appropriate
approximations.  Destroying mean solar time based civil time-of-day
is not appropriate.

Of course, we have yet to establish the extent of any problem with
such overhead.  It sounds like you have expertise in this area.
Asse

Re: Introduction of long term scheduling

[Poul-Henning Kamp]

In message <[EMAIL PROTECTED]>, "M. Warner Losh" writes
:
>In message: <[EMAIL PROTECTED]>
>Tony Finch <[EMAIL PROTECTED]> writes:
>: On Sat, 6 Jan 2007, Ashley Yakeley wrote:
>: >
>: > Presumably it only needs to know the next leap-second to do this, not
>: > the whole known table?
>:
>: Kernels sometimes need to deal with historical timestamps (principally
>: from the filesystem) so it'll need a full table to be able to convert
>: between POSIX time and atomic time for compatibility purposes.
>
>Most filesystems store time as UTC anyway...

Actually, I tend to think these are in the minority, but most of
the non-UTC ones are of minor significance.

--
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
[EMAIL PROTECTED] | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.