Jonathan,
We are definitely getting much closer to full agreement. I continue to
think that a separate time_system (or some such) attribute would be a
much better way to handle this than modifying the calendar attribute,
and that space-separated calendar modifiers would be next best after
that, but I will bow to the apparent majority and agree to your proposal
for modified definitions of the general reference time and Gregorian
calendar, the addition of a new gregorian_utc calendar, etc, as you just
outlined.
There are two remaining things that I would like to see.
1. A section that explains the importance for data producers and
consumers of using the right time handling routines for the input
time data and the calendar chosen if your time resolution makes you
sensitive to errors on the order of 1-30 seconds, pointing out (for
example) that using the standard *nix time functions with sets of
correct UTC timestamps will produce incorrect elapsed time values
under the right conditions. Such a section would also point out to
data consumers that if errors on this level are significant to them,
they shouldn't assume that existing observational datasets are free
of these errors.
2. A standard mechanism for data producers to indicate that they have,
in fact, taken the extra care with their time calculations,
whichever calendar they may have chosen - that is, the elapsed time
values are guaranteed to be free of any leap second related offsets
or discontinuities. This would give data consumers greater
confidence in cases where such errors matter.
Grace and peace,
Jim
On 5/14/15 11:49 AM, Jonathan Gregory wrote:
Dear Jim
I think I completely agree with what you have written here! Thanks. Therefore
I wonder what we are discussing now, and like Chris I'm unsure of the status
of the proposal. So I will take the liberty of repeating the essential
points of what I have suggested before, and if you disagree with it again, in
the light of this new understanding maybe we will both understand why :-)
* Don't mention "UTC" in the description of time units. Instead say the time
zone of the Greenwich meridian, without summmer/daylight-saving time.
* Clarify that in the CF convention the choice of "calendar" implies the
particular set of rules that is used to convert between date-times (YYYY-MM-DD
hh:mm:ss i.e. sets of six numbers) and time coordinates in units of elapsed
time since a reference time. The calendar is identified by the calendar att of
of the time coordinate variable. It's a property of the time coord variable.
* Require the calendar to be specified i.e. no default, and abolish the
"standard" calendar (currently a synonym for the default). These are backward-
incompatible changes for data-writers, but of course they do not invalidate any
data written with old CF versions.
* Redefine the "gregorian" calendar to mean that leap seconds are not included
in the time conversions. All days have 86400 seconds in this calendar. This
might not be correct for some existing data written with calendar="gregorian"
but we cannot know. For new data, writers should take care they choose the
right calendar, and that's the reason for not allowing a default, so they have
to consider the issue.
* Introduce a new calendar "gregorian_utc" which does include leap seconds
in the time conversions according to UTC.
I've omitted the Julian/Gregorian issues. If we can agree on the UTC issue
first, we can come back to that.
What do you and all think?
Best wishes
Jonathan
----- Forwarded message from Jim Biard <[email protected]> -----
Date: Wed, 13 May 2015 13:35:17 -0400
From: Jim Biard <[email protected]>
To: [email protected]
Subject: Re: [CF-metadata] How to define time coordinate in GPS?
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0)
Gecko/20100101 Thunderbird/31.6.0
Jonathan,
I'd say this is a lot of where the misunderstanding lies.
I agree that if the calendar and time system are specified as
Gregorian/NLS, then you should not involve leap seconds in your
calculations going between times represented as HH:MM:SS and elapsed
times since your epoch. But if the calendar and time system are
specified as Gregorian/UTC, then you should involve leap seconds in
those same calculations. But in either case you *should* end up with
a set of elapsed times in your time variable that don't have any
leap second offsets or discontinuities.
Let's say I have a machine that counts the number of total seconds
and 86,400 second periods (standard days) that have gone by since I
started it, which was on Jan 1, 1970. (A UNIX system with a
super-accurate CPU clock.) When I have a leap day, I don't add or
subtract anything from the counts the machine has. What I do is use
the algorithms specified by the Gregorian calendar to represent
February as having one more day than usual so that my YYYY-MM-DD
calendar date stays more closely aligned with Earth's position in
its orbit. Leap seconds are just like that. I don't add or subtract
anything from the counts my machine has. I use the algorithms
specified by the UTC time system to represent June 30 (in a year
when I have a leap second) as having one more second in the last
minute of the last hour so that my HH:MM:SS clock time stays more
closely aligned with Earth's position in its rotation.
In the particular case of *nix systems using the Network Time
Protocol, getting the system to display the correct UTC time
actually is accomplished by subtracting a second from the elapsed
time being counted by the system, thus ensuring that any long term
recording of elapsed times since the computer was started does, in
fact, include leap second discontinuities. This can add a whole
other facet to the general problem of how to get all of this right
as a data producer if I'm getting my time from my *nix machine. I
don't know for sure how Windows boxes handle it, but I don't think
it's actually any better.
Grace and peace,
Jim
On 5/13/15 12:54 PM, Jonathan Gregory wrote:
Dear Jim
I think your last email (which didn't go to the email list) explains why we
have not been understanding each other about your points 2 and 3:
There shouldn't be (or there is no) good reason to use a different
calendar to encode and decode, but this is exactly what has been
done with regard to times. As a netCDF file producer, if you have a
set of observations that have attached correct Gregorian/UTC date &
time strings and you use a POSIX calculator to encode your times
since your correct Gregorian/UTC reference date & time string, then
you are at risk of producing incorrect elapsed time values. (Whether
you actually do or not depends on factors mentioned in previous
emails.) The contents of time variables *should* always be true
counts of time that has elapsed since the reference epoch and
*should* never, ever encode leap second induced discontinuities or
offsets.
That's not how I understand what we've been discussing. I think that if
calendar="gregorian[_nls]" then the encoding/decoding should be done without
leap seconds (86400 s per day), but if calendar="gregorian_utc" the encoding/
decoding should be done with leap seconds according to UTC. This why I see
the issue as being one that relates to the calendar, and not to the units
or the reference time, which are calendar-neutral. Is this what we disagree on?
I can see there's a difficulty with the UTC calendar that if a second is taken
out there will be two consecutive seconds with the same timestamp. This is
a problem for encoding - the data-writer would have to be careful that he
chose the right one. It's not a problem for using the encoded time coords,
which will be monotonic and run forward at 1 second per second, nor for
decoding, which will correctly produce the repeated timestamp. Inserting a
leap second is the same kind of discontinuity as the change from Julian to
Gregorian calendar. It means that there is a range of date-times which cannot
be encoded because they are not valid, but decoding is no problem. Similarly
there are dates which are valid in some calendars but not others, such as
30 Feb being valid in the 360_day calendar.
I think my view is consistent with the general treatment of calendars by CF.
The encoding/decoding algorithm differs, but the same units string could
always be used.
Best wishes
Jonathan
_______________________________________________
CF-metadata mailing list
[email protected]
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
--
CICS-NC <http://www.cicsnc.org/> Visit us on
Facebook <http://www.facebook.com/cicsnc> *Jim Biard*
*Research Scholar*
Cooperative Institute for Climate and Satellites NC <http://cicsnc.org/>
North Carolina State University <http://ncsu.edu/>
NOAA National Centers for Environmental Information <http://ncdc.noaa.gov/>
/formerly NOAA?s National Climatic Data Center/
151 Patton Ave, Asheville, NC 28801
e: [email protected] <mailto:[email protected]>
o: +1 828 271 4900
/We will be updating our social media soon. Follow our current
Facebook (NOAA National Climatic Data Center
<https://www.facebook.com/NOAANationalClimaticDataCenter> and NOAA
National Oceanographic Data Center
<https://www.facebook.com/noaa.nodc>) and Twitter (@NOAANCDC
<https://twitter.com/NOAANCDC> and @NOAAOceanData
<https://twitter.com/NOAAOceanData>) accounts for the latest
information./
_______________________________________________
CF-metadata mailing list
[email protected]
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
----- End forwarded message -----
_______________________________________________
CF-metadata mailing list
[email protected]
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
--
CICS-NC <http://www.cicsnc.org/> Visit us on
Facebook <http://www.facebook.com/cicsnc> *Jim Biard*
*Research Scholar*
Cooperative Institute for Climate and Satellites NC <http://cicsnc.org/>
North Carolina State University <http://ncsu.edu/>
NOAA National Centers for Environmental Information <http://ncdc.noaa.gov/>
/formerly NOAA’s National Climatic Data Center/
151 Patton Ave, Asheville, NC 28801
e: [email protected] <mailto:[email protected]>
o: +1 828 271 4900
/We will be updating our social media soon. Follow our current Facebook
(NOAA National Climatic Data Center
<https://www.facebook.com/NOAANationalClimaticDataCenter> and NOAA
National Oceanographic Data Center <https://www.facebook.com/noaa.nodc>)
and Twitter (@NOAANCDC <https://twitter.com/NOAANCDC> and @NOAAOceanData
<https://twitter.com/NOAAOceanData>) accounts for the latest information./
_______________________________________________
CF-metadata mailing list
[email protected]
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
