Dear Temporal CF enthusiasts,
I agree "proleptic_gregorian_utc" is a good label for something. I express no
view as to what! ;-)
There is a danger in using phrases like 'UTC Timestamps'. UTC is a timescale
usually used in conjunction with the Gregorian calendar (and proleptic
Gregorian) and with leap seconds included by definition if not in practice.
'Timestamps' may be in a variety of formats, and one of the ISO8601 options is
common. However, a number of groups, the IETF in particular, have taken great
pains to ensure that such timestamps make an well ordered sequence, *without
inferring any duration or calculations*. So 'timestamps' should be taken as
that - a fully ordered sequence of labels. However, different computers systems
may have different clocks, so in total, they become a set of partially ordered
labels, depending on the any epochs or datums that can be established, such as
by NNTP protocols etc.
Any calculations on timestamps, whether fully or partially ordered, must make
calendrical assumptions to perform calculations, unless the full details are
known.
HTH, but it may not help achieve consensus, Chris
-----Original Message-----
From: CF-metadata [mailto:[email protected]] On Behalf Of
[email protected]
Sent: Wednesday, April 29, 2015 2:56 PM
To: [email protected]
Subject: CF-metadata Digest, Vol 144, Issue 20
Send CF-metadata mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific than "Re:
Contents of CF-metadata digest..."
Today's Topics:
1. Re: How to define time coordinate in GPS? (Jonathan Gregory)
2. Re: How to define time coordinate in GPS? (Jim Biard)
----------------------------------------------------------------------
Message: 1
Date: Wed, 29 Apr 2015 14:39:50 +0100
From: Jonathan Gregory <[email protected]>
To: [email protected]
Subject: Re: [CF-metadata] How to define time coordinate in GPS?
Message-ID: <[email protected]>
Content-Type: text/plain; charset=us-ascii
Dear Jim, Chris et al.
I'm using the word "calendar" in a CF-consistent way, I believe. Maybe it's not the best word for
the concept, but nonetheless we have an attribute called "calendar", and its sole function is
indicate which algorithm is used to translate between components of time (YMDhms) and elapsed time (in units
of time since a reference time). So perhaps that is consistent with "calendar"
being a collection of algorithms, in Chris's text, but it's more specific than that. It
has a particular function in the interpretation of CF time values (usually coordinates).
CF sect 4.4.1 says "In order to calculate a new date and time given a base date,
base time and a time increment one must know what calendar to use."
and I think that is the sense in which I am using "calendar".
In the "real" world, we often start with UTC timestamps that have leap
seconds accounted for, yet convert them to elapsed times using
calculators that don't account for leap seconds. This can actually
lead to elapsed time values that encode a time discontinuity and
cannot be counted on to produce accurate differences between every
pair of values.
This is a problem, I agree. We should avoid that problem for future data by
making the conventions more precise about which calculator should be used
(which calendar, in the CF sense). We can't decide for sure what was done when
encoding past data, but the conventions string records the version of CF used.
I'm suggesting that we need to do two things. One is to more precisely
define what sorts of times can be used in the time reference part of
the units attribute. I just reread section 4.4, and it actually says
that the time is UTC or a time zone offset from it. I think it should
stay that way and the wording strengthened to make it clearer.
Yes, it does say that. It's a quote from the udunits man page. However I don't
think the issue of leap seconds has been carefully considered before, so we
don't have to assume that's what it meant exactly, especially as udunits does
not support lead seconds. As previously said, and I think you may agree, it is
likely that nearly all existing time values have been encoded *without* leap
seconds, and therefore *not* UTC strictly. Therefore my alternative suggestion
is that we should add some text here that says we don't necessarily imply leap
seconds are included by mentioning UTC. This must be the case, because the same
format of time unit is used for calendars that definitely do not ever include
leap seconds i.e. all the non-real-world ones. UTC is mentioned simply as a way
to refer to the time-zone which contains the Greenwich meridian, without summer
time.
The other thing I think we need to do is provide a way to indicate
that the elapsed times in a time variable are true elapsed times that
are certain to be free of leap second discontinuities, or are possibly
contaminated with leap second discontinuities. In connection with this
we would need to add clarifying language in the CF conventions to
educate people on the importance of using time calculators that are
aware of leap seconds when moving between UTC timestamps and elapsed
time values. This could be handled by adding a modifier to a calendar
name in the calendar attribute, or it could be handled by adding a new
attribute to hold this information. I think that coming up with one or
more new calendar names is a more confusing and less useful way to
accomplish this.
I don't think we should define a new attribute, because this distinction is one which applies only to the real-world
calendar. It's therefore more robust and simpler to indicate it as a modifier to the name when applicable in the the
calendar att, so making a new calendar name, in effect. But given this discussion I agree that calling it just
"UTC" may not be clear enough. The real-world calendar is called "standard" or
"gregorian". I would propose a new possibility "proleptic_gregorian_utc", meaning the proleptic
Gregorian calendar with leap seconds inserted as applicable since 1958.
Best wishes
Jonathan
------------------------------
Message: 2
Date: Wed, 29 Apr 2015 09:56:07 -0400
From: Jim Biard <[email protected]>
To: "Arctur, David K" <[email protected]>,
"[email protected]" <[email protected]>
Subject: Re: [CF-metadata] How to define time coordinate in GPS?
Message-ID: <[email protected]>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Randall, are you lurking out there? :-)
On 4/29/15 9:37 AM, Arctur, David K wrote:
Please pardon this digression, but is xkcd following this thread?
http://xkcd.com/1514/ - just came out last week? ;-)
Cheers,
David Arctur
?
On Apr 28, 2015, at 4:41 PM, Jim Biard <[email protected]
<mailto:[email protected]>> wrote:
Jonathan,
As I've mentioned in other emails, I don't think TAI and GPS are
calendars, but I understand where you are coming from.
When you are working in the model domain you are generating your own
timestamps, so you are self-consistent and have no problems. As far as
that goes, there are models that use no_leap or all_leap calendars,
and they are again self-consistent and have no problems. In the "real"
world, we often start with UTC timestamps that have leap seconds
accounted for, yet convert them to elapsed times using calculators
that don't account for leap seconds. This can actually lead to elapsed
time values that encode a time discontinuity and cannot be counted on
to produce accurate differences between every pair of values. I'm not
sure what we should call that, but I don't like the idea of naming it
with a calendar.
It is true that a large majority of files "in the wild" probably
either use time at a resolution that is coarse enough that none of
this matters or managed to avoid having any leap seconds creep in to
their elapsed time values.
I'm not suggesting that we add anything to the units attribute. I am
suggesting that we need to do two things. One is to more precisely
define what sorts of times can be used in the time reference part of
the units attribute. I just reread section 4.4, and it actually says
that the time is UTC or a time zone offset from it. I think it should
stay that way and the wording strengthened to make it clearer. Since
there are no leap seconds defined before the TAI epoch date, I think
there would be no problem with saying that times in years before Jan
1, 1958 are assumed to have no leap seconds. (Call it proleptic UTC if
you like.)
The other thing I think we need to do is provide a way to indicate
that the elapsed times in a time variable are true elapsed times that
are certain to be free of leap second discontinuities, or are possibly
contaminated with leap second discontinuities. In connection with this
we would need to add clarifying language in the CF conventions to
educate people on the importance of using time calculators that are
aware of leap seconds when moving between UTC timestamps and elapsed
time values. This could be handled by adding a modifier to a calendar
name in the calendar attribute, or it could be handled by adding a new
attribute to hold this information. I think that coming up with one or
more new calendar names is a more confusing and less useful way to
accomplish this.
It seems to me that the historical default for existing datasets is
that the leap second handling has to be "unknown". For most (maybe
all) of the model datasets, the actual situation is "no leap second
discontinuities". (Do you actually have any models where time in the
output has a resolution fine enough and span wide enough for it to
matter anyway?) For many "real world" datasets, the actual situation
is "possibly contaminated with leap second discontinuities", since the
times started as UTC timestamps and were converted to elapsed times
using calculators that didn't consider leap seconds. (But as I
mentioned, most of them probably aren't affected by the error.) We
have two communities that have different defaults, so we shouldn't
pick one over the other.
Grace and peace,
Jim
On 4/28/15 12:52 PM, Jonathan Gregory wrote:
Dear Jim
I agree that TAI and GPS aren't distinct CF calendars if they differ
only because of the epoch. In CF and udunits, the reference time is
part of the the units, as you know; it's not a property of the
calendar. I referred to GPS calendar just to contrast it with UTC.
Unlike you, I don't live only in the real world. :-) Many models use
the default CF calendar (called standard or gregorian) as an
approximation to the real world. In these models there are no leap
seconds, and it would not be correct to call this UTC, or to include
the leap seconds in the encoding and decoding of time cooordinates.
In any case, as we understand, most software doesn't allow for leap
seconds. For those two reasons, I propose that we more precisely
define the default (standard, gregorian) calendar to say explicitly
that it does not include leap seconds i.e. every day is 86400 s long
exactly. This is probably correct for most of the data which has been written
with this calendar.
I call UTC a calendar because it affects the encoding, since it
implies leap seconds. You suggest, I think, that the treatment of
leap seconds should be indicated in the reference time in the units
string. This doesn't seem quite right to me because the units doesn't
say anything about the encoding. We use the same format of units
string for all calendars. udunits does mention UTC in respect of the
format of this string because of wanting to talk about time zones,
not because of leap seconds. Time zones could be used in models too
(though I don't know of a case). Hence I still propose that we should
regard UTC as a calendar, if it's correct that the leap seconds in
use in the real world are part of the definition of UTC. This means
we should define a a new calendar, which is not the default, in which
time units have just the same format as usual i.e. udunits, but the
encoding and decoding of values is done including leap seconds
according to UTC. The same value, with the same units string, may
translate into a different date-time (by a number of
seconds) according to whether the calendar is default (standard,
gregorian) or utc. It wouldn't make sense to use this calendar for
dates before the introduction of UTC, so it should be illegal to do
so. We could do this by prohibiting reference times earlier than the
start of UTC and negative values of time in this calendar.
Maybe I haven't grasped your point properly?
Best wishes
Jonathan
----- Forwarded message from Jim Biard <[email protected]
Date: Tue, 28 Apr 2015 11:35:16 -0400
From: Jim Biard<[email protected]>
To:[email protected]
Subject: Re: [CF-metadata] How to define time coordinate in GPS?
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0)
Gecko/20100101 Thunderbird/31.6.0
Jonathan,
I see where you are coming from, and there's validity in that line of
thought. Leap seconds represent a finer-grained adjustment to the
overall date/time system being used. I still think it makes good
sense to add a new attribute to declare whether or not leap second
handling was used or strengthen our standards for time variables so
that problems are averted.
From a human understandability perspective, a calendar attribute of
"GPS" or "UTC" will be somewhat confusing. In my experience, people
don't speak of the UTC calendar, it's UTC time. Further, TAI and GPS
time don't really concern themselves with anything but counts of
seconds since an epoch date & time. People convert the counts to time
stamps for convenience, but they are actually more equivalent to the
Julian Day Number (JDN) than they are to the Gregorian or Julian
calendar. TAI and GPS time have different epochs, and TAI is more
accurate, but they both are running counts of seconds that aren't
tied to the motions of the Earth. As a result, I think that it's
improper to talk about a GPS calendar or TAI calendar.
What is being exposed by this discussion is the reality that any of
us (myself included) have often ignored or been unaware of the fact
that the time calculators (time handling software) we used when
filling our time variables with elapsed times weren't giving us true
counts of seconds since the epochs we wrote into our units
attributes. If you are working at a resolution of days or hours, this
will probably never cause a problem. If you are working at a
resolution of minutes or less and working over a time span of greater
than two years, it may well have caused at least occasional small
problems. If you are working with full-resolution polar-orbiting
satellite data, one second represents ~7 km of satellite motion, so
such errors can produce significant geolocation errors.
A set of elapsed seconds since a Gregorian/UTC epoch that were
calculated from Gregorian/UTC time stamps without regard for leap
seconds and which crossed a leap second boundary are not "GPS"
seconds. Nor are they "UTC" seconds. They are, strictly speaking,
elapsed times into which one or more step errors have been
introduced. As I mentioned in a previous email, as long as you use
the same time calculator to extract time stamps as you did to get
elapsed times from input time stamps, you won't notice anything. You
may notice a problem if you are taking differences between elapsed
times and a leap second boundary gets involved.
As I've considered all of this more, I'm tending to favor the second
option I suggested.
We could also be more strict, and say the epoch time stamp in the units
attribute must always be in UTC. The question would then be reduced to
whether or not leap seconds were counted into the elapsed times stored
in the time variable. In this case, we could add a "leap_seconds"
attribute which would have a value of "UTC" if UTC leap seconds were
counted into the elapsed times, and "none" if not. This would also allow
for some other system of leap seconds to be used. (I don't know if there
are others.) For backward compatibility, considering history, the
default value for this attribute would probably be "UTC".
Clearly, having epoch time stamps with time zone offsets from UTC, as
described in the CF conventions, would be OK as well. I'm also open
to other namings for the new attribute and for its possible values.
The leap seconds only become an issue in certain rather specific
instances, so I think that such an attribute, along with a bit of
discussion in the document, would likely be sufficient to warn those
people that may find themselves negatively affected by improper leap
second handling.
Grace and peace,
Jim
On 4/27/15 1:07 PM, Jonathan Gregory wrote:
Dear Jim
I don't think calendars are the right place to encode this. We
could add a new "time_system" attribute where you would declare
whether your time stamps and elapsed times were based on UTC, GPS, TAI, etc.
If we take this route, we should require the elapsed times to
encode leap seconds if the time system is UTC, and state that the
default time system is UTC.
I think this is a calendar issue because the calendar is the set of
rules which translate between components of time (YMDhms) and
elapsed time (in fixed time units) since the reference time. Your
later email seems to me to be consistent with that. In the real
world, the elapsed interval (expressed e.g. as the number of
seconds) between the ref YMDhms and the actual YMDhs depends on whether your
calendar includes leap seconds (UTC) or not (GPS).
It seems that GPS is the calendar likely to have been assumed in
existing CF datasets, so it would be logical to say that the default
is the real- world calendar without leap seconds. Have I
misunderstood something? If we regard this as a property of the calendar, we
don't need a new attribute.
Best wishes
Jonathan
_______________________________________________
CF-metadata mailing list
[email protected]
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
--
<CicsLogoTiny.png> <http://www.cicsnc.org/>Visit us on
Facebook <http://www.facebook.com/cicsnc> *Jim Biard*
*Research Scholar*
Cooperative Institute for Climate and Satellites NC
<http://cicsnc.org/> North Carolina State University
<http://ncsu.edu/> NOAA National Centers for Environmental Information
<http://ncdc.noaa.gov/> /formerly NOAA?s National Climatic Data
Center/
151 Patton Ave, Asheville, NC 28801
e: [email protected] <mailto:[email protected]>
o: +1 828 271 4900
/We will be updating our social media soon. Follow our current
Facebook (NOAA National Climatic Data Center
<https://www.facebook.com/NOAANationalClimaticDataCenter> and NOAA
National Oceanographic Data Center
<https://www.facebook.com/noaa.nodc>) and Twitter (@NOAANCDC
<https://twitter.com/NOAANCDC> and @NOAAOceanData
<https://twitter.com/NOAAOceanData>) accounts for the latest
information./
_______________________________________________
CF-metadata mailing list
[email protected] <mailto:[email protected]>
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
