subject:"Re\: \[cf\-convention\/cf\-conventions\] Add calendars gregorian_tai and gregorian

@martinjuckes I started out with a similar idea a few years ago. ;-)

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436760085

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

@ChrisBarker-NOAA If the UTC time stamps they start with are valid, then the 
round-trip recovered time stamps will be valid UTC. (With all the standard 
caveats about any instances where time measurements fell with leap second 
spans.) The epoch time stamp does not have to be acquired for it to be valid 
UTC.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436767916

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

@JimBiardCics wrote:

> The epoch time stamp in the units attribute is correct UTC. The time stamps 
> that went into creating the time variable contents are all correct UTC, and 
> (except for time stamps falling with the span of a leap second unless you 
> detect and correct) the time stamps extracted from the time variable will be 
> correct UTC.

I"m still not sure. imagine this case:

someone starts with a bunch of UTC timestamps from, say 2018, right about now.

They decide to store the data in their CF file as: "seconds since 
2000-01-01T00:00:00".

They compute the number of seconds with a no-leap-second-aware library.

Is that timestamp Correct UTC??? Does it matter??

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436763397

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

@JonathanGregory wrote:

> I don't think we should do (c). As I've said, we've discussed it many times 
> before.

yes, and I strongly advocated against it at the time. But I don't recall anyone 
bringing up the use-case at hand, so it seemed quite unnecessary.

> Obviously it does have advantages, but there's a big disadvantage of 
> complexity of software and convention in introducing another way,

exactly why I didn't like the idea back in the day.

> however marvellous it may be, to do something we can already do adequately.

This thread shows quite clearly, that we can not do it adequately -- exactly 
why I've re-considered.

> But I think it should be the subject of another issue.

Indeed -- if I can find the time, I'll post one, if only to document that it's 
been re-considered and rejected.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436764935

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

@ChrisBarker-NOAA I think you got your case 2b correct the first time. The 
epoch time stamp in the **`units`** attribute is correct UTC. The time stamps 
that went into creating the time variable contents  are all correct UTC, and 
(except for time stamps falling with the span of a leap second unless you 
detect and correct) the time stamps extracted from the time variable will be 
correct UTC.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436759438

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-11-07 Thread Martin

@JimBiardCics : I'm sorry if I misrepresented your views, but I think I have 
understood, I'm just using different words. In the case 2b in your comment [a 
few hours 
ago](https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436722293)
 you refer to data encoded as if there are no leap seconds. For data encoded in 
this way, `x seconds since time-stamp_0` is equivalent to a `time-stamp_1` 
calculated on the basis of fixed 60 second minutes, fixed 86400 second days. In 
addition, you are saying that `time-stamp_0` in this calendar system **is** the 
UTC time-stamp. I'm calling this relationship between the time stamp in the new 
calendar system and the UTC time stamp a mapping. 

I think we could find a CF solution to these different perspectives, and also 
perhaps address concerns about implications for past data, by adding a new 
attribute to express the relation between the calendar time used in the file 
and UTC time. There are at least 4 different options for converting from UTC to 
a calendar with fixed lengths days which have been discussed in this issue:

- TAI: the time stamp is offset: convert to UTC by subtracting leap second 
count;
- GPS: as for TAI, but with an additional fixed offset;
- Jim case 2b: use the same time stamp;
- model: not really defined  might as well use the same time stamp;

If we add a new attribute (e.g. `utc_conversion`) which can take one of the 
values `tai`, `gps`, `model` and a 4th for Jim's use case (e.g. `direct`), this 
would cover all use cases, I think. If undefined, as it would be for historical 
data, the interpretation would be fuzzy, as it is now.

With this approach, the case 2b would be covered by `calendar = 
"gregorian_fixed"` and `utc_conversion = "direct"`.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436758942

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

@JonathanGregory wrote:

> What I propose is that data written with the next version of CF (in which we 
> have made the calendar changes) will have no ambiguity.

That would be nice -- but I simply don't think is possible / practical. Many, 
many, users do not understand (or even know about) leap-seconds, TAI vs UTC, 
many libraries do not support leap seconds, and if they do, users probably 
don't know if they do or what the implications are.

So we need to have a way to encode time that says: these data are ambiguous 
with regard to leap seconds, and I don't care. And that's what the current 
gregorian calendar effectively is -- the problem is that folks that DO care 
have no way to express their data.

I do think that it would be good to to say that a correct "gregorian" time 
variable be "metrical", though as there are existing data sets where that is 
not the case, I'm not sure if CF can say that a file is compliant with CF x.y, 
but not CF x.y+1

> UTC time which is written without leap seconds (and therefore non-metrical 
> for some intervals) is gregorian. So you will always know how to decode the 
> time coordinates correctly, won't you?

no -- because the datetimestamp may or may not include (some) leap seconds, and 
you don't know whether the producer intended it to be decoded with or without 
leap seconds.

 I fear that there must be something in what you write which I haven't 
interpreted properly.
> 
> Yes, the model gregorian calendar has days which are always 86400 seconds. 
> The days are always the same length, but the length of the second does not 
> vary. This is not the real world!
> 
> 

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436748497

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-11-07 Thread JonathanGregory

@ChrisBarker-NOAA, you didn't ask me, but I applaud your summary! Thanks.

I don't think we should do (c). As I've said, we've discussed it many times 
before. Obviously it does have advantages, but there's a big disadvantage of 
complexity of software and convention in introducing another way, however 
marvellous it may be, to do something we can already do adequately. But I think 
it should be the subject of another issue.

I agree that (b) is not perfect, but it's in use, and that's not surprising 
because it's easy. You may not even need a library to compute timedeltas in 
this system, because the formula is fairly simple.

It appears that I differ from Jim in thinking that (1) and (2b) should both be 
called `gregorian`. I think that's good because it's no change from the present 
convention for models and not much change for obs, and because they are 
interpreted and encoded in the same way.


-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436729719

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

@JimBiardCics: I jsut realized that I may hve made a mistake:

> b) Have a calendar that:
> 
> The datetimestamp is correct UTC
> The timedeltas have been computed from datetimestamps without leap-seconds
> When decoding, leap-seconds should NOT be used in order to recover the 
> original, correct timestamps. 

I now think that the datetimestamp is NOT correct UTC, but rather, computed 
from the original UTC data without leap-seconds. So it may or may not be UTC 
depending on what epoch is used.

Which is correct??



-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436724237

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

@JimBiardCics wrote:
>  I'm not sure how to reframe my argument. I am specifically excluding 
> modelers from the topic. It is only relevant for real-world time values.

I think the way to re-frame it is to start from use-cases -- and, indeed, the 
docs should reflect this as well when we've settled on something. So I'll try:

1) modelers writing output from their models which are generally done (in the 
model) as time-delta since a_timestamp can use the current "gregorian" 
calendar, which has always been ambiguous with regard to leap-seconds -- but it 
works perfectly well for this use case.

so modelers need nothing new.

2) Folks collecting data with an instrument (Or some such) that start with a 
bunch of "correct" UTC datetimestamps (with leap seconds applied) need a way to 
encode this data so that it can be accurately described an recovered. Possible 
solutions:

 a) Have a proper UTC gregorian calendar -- to be used correctly:
   - the datetimestamp is correct UTC, leap seconds and all
   - the timedeltas have been computed with leap-seconds included
   - when decoding leap-seconds need to be taken into account in order to get 
accurate datetimestamps back.

b) Have a calendar that:
   - The datetimestamp is correct UTC
   - The timedeltas have been computed from datetimestamps without  leap-seconds
   - When decoding, leap-seconds should NOT be used in order to recover the 
original, correct timestamps.

c) Have a way to directly store datetimestamps in CF (probably ISO 8601 strings)
   - this would add a new way to encode time to CF, which we are all resistant 
to.

a) Is the "right" way to do it - so I think we should add this to CF. However, 
the reality is that most folks (producers or consumers) don't have easy access 
to a leap-seconds aware tools. So, at least for the moment, hardly anyone can 
actually use a proper UTC calendar. 

c) is another "right" way to do it -- so I'm advocating for that. However, 
other folks may not agree, so it may never happen. And (c) is easy to do with 
commonly available tools. However there are a lot of existing datasets and 
workflows that do (b) already, so it would be nice to provide a way for these 
folks to clearly define their data so it can be used correctly without needing 
to change their workflow.

So we need (b).

Downsides to (b):
 - the pendant in me thinks it's "just wrong", and we really shouldn't codify 
such things in a standard.
 - the pragmatist in me thinks that converting to-from datetimestamps that have 
leap seconds to-from timedeltas with a non-leapseconds aware library works MOST 
of the time -- with a very high value of MOST -- but it does not work correctly 
EVERY time. -- so codifying this workflow is inviting hidden (exceedingly rare) 
bugs and confusion.
 - the timedeltas actually stored are not "metrical" -- that is, they don't 
accurately reflect the amount of passed time (being off by some number of leap 
seconds)
 - folks that really care about sub-second-level accuracy should be using TAI 
or GPS time anyway

But "practicality beats purity" -- folks are doing this now, folks will 
continue to do it in the future, and it does work almost all the time.

Final plea -- maybe because I'm a pedant, but I still don't "like" this 
approach -- I think we should add a way to directly encode datetimestamps, and 
then recommend that it be used for this use-case.

 @JimBiardCics: does this correctly capture your proposal?

@martinjuckes: does this clarify it for you?  

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436722293

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

@martinjuckes I disagree with your characterization of what I am advocating 
for, but I'm not sure how to reframe my argument. I am specifically excluding 
modelers from the topic. It is only relevant for real-world time values.

Regarding the epoch time stamp, I agree that we need to be careful to 
differentiate between ISO 8601 and UTC. ISO 8601 makes no claim for the 
calendar informing the time stamp contents. It just identifies the generic 
year, month, day, hour, minute, second, subsecond delineations and specifies 
how to order them.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436711211

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

@martinjuckes Where do you find that BIPM does this? I couldn't find it.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436709271

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

@JonathanGregory I'm thinking of existing files (built pre - CF-1.8) that have 
time variables marked by the **`gregorian`** calendar. They may well not match 
the new definition, but people won't likely look back to CF-1.7 or earlier to 
see the difference between the old and new definitions. It's the existing mixed 
population that concerns me.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436699524

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

@JonathanGregory I think we can likely leave LORAN-C off for now. There was the 
original request for GPS time, I have, myself wished there was a TAI time when 
I was building a dataset that depends on it.

I still think we should provide a path for data producers to clearly signal to 
their users that they can depend on the data in their time variables to produce 
(almost always) correct UTC times stamps if they follow the right prescription. 
The **`gregorian`** calendar, if modified as @JonathanGregory described above, 
could lead people to think that all time variables carrying this calendar 
follow that rule. There are, very possibly, time variables that have metrical 
time in them which are marked as **`gregorian`**, and it would be incorrect to 
treat them as if they didn't. As a specific example of a potential problem, if 
I know for a fact that my time variable is of the 'converted UTC time stamps' 
variety, I can use a leap second table to correct the values to make them 
metrical. If I do this to a file that came from a model or where the values 
were already metrical, I am going to get incorrect results.

I think the **`gregorian`** calendar, with appropriate warnings added, is 
perfectly fine for modelers. As near as I can tell from the interwebs, this 
calendar doesn't speak at all to the question of leap seconds at all. The time 
naturally implied by this calendar is days, and the length of the second varies 
to keep exactly 86,400 of them in each day.



-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436681688

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

Regarding the "broken" data. It's only broken from a purist compliance 
perspective. It works quite well from a practical perspective.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436664787

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

I can't find a way to remove 'cf-metadata' as a contributor on this issue, so I 
don't have a way to stop emails from this issue going to the CF-Metadata list, 
if they are getting in there that way. @davidhassell do you know of a way? (I'm 
not getting any emails on this issue from cf-metadata, but someone seems to be 
doing so.)

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436664501

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-11-07 Thread JonathanGregory

Thanks for explaining "uncorrected GPS", Jim.

The CF `calendar` attribute specifies (a) the interpretation of the reference 
timestamp in the `units`, (b) the rules for converting between timestamps and 
time coordinates. Because of (a), I agree that TAI, GPS and Loran-C are all 
different calendars, although they use the same rules (Gregorian calendar, no 
leap seconds). They differ by a constant number of seconds, as Jim has 
explained. I don't think it's redundant to distinguish these calendars.

If the use-case originally was for GPS, we should certainly support that one. 
TAI has often been mentioned in this discussion, and seems reasonable to 
include. Loran-C hasn't been mentioned before, as far as I remember. Do we know 
whether anyone wants to use it with CF-netCDF? If not, I don't think we should 
add it. We should wait until someone asks.

Those who are writing new datasets with the new CF standard using metrical UTC 
time with leap seconds should use the new gregorian UTC calendar (whatever we 
call it). I believe we agree on that.

My opinion is that those who write new datasets with the new CF standard where 
they convert UTC timestamps to time coordinates *without* allowing for leap 
seconds (the naive algorithm) should continue to call it `gregorian`. The 
interpretation of the timestamps as being in the Gregorian calendar, and the 
rules for conversion, are the same as for the model metrical Gregorian 
calendar, so it makes sense to call them the same. If `calendar="gregorian"`, 
the reference timestamp in the `units` must not have 60 for the seconds, even 
when that would be legal as a UTC timestamp.

This does change the interpretation of `gregorian` but I don't think it's 
backwards incompatible. The change is simply to remove an ambiguity. It would 
be reasonable to apply the same interpretation to existing `gregorian` data 
i.e. decode it without leap seconds. In most cases, that would be correct, but 
we can't be sure about it. In future, however, we will know for sure, because 
`gregorian` will not be used for the case where leap seconds have been allowed 
for.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436596583

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-11-06 Thread JonathanGregory

I agree, I think! We should keep gregorian for converting UTC timestamps to 
time coordinates and back without leap seconds (naively). That's what I said 
too. In what respect am I disagreeing with your final paragraph? Jonathan

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436267468

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-11-06 Thread Martin

Hello,  @JimBiardCics , @ChrisBarker-NOAA ,

Jonathan points out that the timestamp is usually interpreted in the calendar 
stated  [in a comment added 3 days 
ago](https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435587942).
 I think it is the case that all existing calendars supported by CF were 
created outside CF. This is clearly true for the Julian and Gregorian calendars 
created by Julius Ceasar and Pope Gregory respectively. Other calendars, such 
as the 360 day calendar, came from the climate modelling community. Adding 
support for TAI and UTC calendars would follow this pattern of providing a 
mechanism for encoding information in calendar systems which have been defined 
by others. 

All the existing calendars use a common framework for mapping instants in time 
into a YY-MM-DD hh:mm:ss time stamp and back, with a fully reversible mapping. 
Your proposed calendar introduces a mapping which is not reversible 
(`2016-12-31T23:59:60Z` and `2017-01-01T00:00:00Z` both map onto the same value 
of the time variable in the example @JimBiardCics gives above). I'm puzzled by 
the suggestion that we shouldn't worry about this because it only happens at a 
few leap seconds. Isn't the point of this discussion to handle leap seconds 
accurately in the convention? I haven't been able to find a clear definition of 
"non-metric" in the discussion above. Would this be a good definition, that the 
mapping between time stamp and time since epoch is not reversible? I agree with 
Jonathan that this encoding issue is not really a change of calendar. There 
will also be people getting their time values one hour wrong because of failing 
to encode the spring and autumn clock changes properly -- but that is a problem 
the users need to sort out, the convention is clear.

As Jonathan points out, models use a fixed 86400 second day, which resemble the 
TAI calendar, and we need to support this. Chris [writing in from the CF 
list](https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-434723577)
 [I'm not sure who this is] points out that the true Gregorian calendar has 
been defined by BIPM as including leap seconds, whereas CF refers to fixed 
days. 

There appear to be divergent views on whether `gregorian`, as a CF calendar 
identifier can be made into a precise term or not. One view is that we want to 
upgrade the definition to take into account emerging requirements for higher 
precision, and a 2nd view is that the term has been ambiguous in the past, so 
it has to stay ambiguous. As far as I can see, giving the term a precise 
meaning which is within the range of its possible interpretations does nothing 
to undermine past data. Data written when this term had an imprecise meaning 
will always lack that precision, whatever we do. 






-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436215565

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-11-05 Thread JimBiardCics

@ChrisBarker-NOAA @JonathanGregory Again, please back up a step or two and stop 
thinking in terms of UTC and TAI and don't focus on the time stamps. What we 
have in a time variable containing fully metric elapsed times is, in essence, 
TAI with an offset subtracted. It is elapsed time since an epoch, where the 
values may be smaller than the equivalent TAI elapsed time by the value of the 
TAI elapsed time at the moment of the **`units`** epoch time stamp. There are 
numerous ways to get this state of existence, and many of them never need to 
touch TAI directly.

As a data user, the most important question I have is, "How do I use these 
values?" Assuming that the data is observational data and that I am a user that 
cares about sub-minute accuracy, I need to know two things to use the time 
variable:
1. Are the time values metrical? (What I had been calling fully metric.) (How 
about that name, @JonathanGregory?)
1. What time system was used for the epoch time stamp in the **`units`** 
attribute?

If the answer to 1) above is no, then I have a third question:
- Were the elapsed time values obtained in a well-understood fashion so that I 
can use them even though they aren't metrical?

We are only having this discussion because we spent years not caring about the 
distinction, and because software packages didn't care about the distinction 
either. My goal here is to provide a means for data producers to allow data 
users to answer the 2-3 questions above with the least amount of confusion.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436038332

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-11-05 Thread JonathanGregory

Dear Jim and Chris

Thank for your latest exchanges. What I said before them is similar to what 
Chris says, I think.

(1) I used the name **gregorian_utc** for what Jim called **gregorian_metric**. 
In this calendar, timestamps (including the reference time) are UTC, and the 
conversion uses leap seconds. I think to call it "metric" could be confusing 
(although I understand why you choose that word) because "metric" might be 
understood to mean "not Imperial" i.e. not feet and pounds! :-)

(2) What Jim calls **gregorian_nonmetric** is what is usually done for 
real-world data. The times are UTC but have been encoded without leap seconds. 
That's fine and workable. It's just an encoding, with the drawback that time is 
not exactly metric.

(3) However, this same scheme is what is used for some models. They follow the 
real-world Gregorian calendar, but the model world has no leap seconds. Hence 
the encoding without leap seconds is accurate, and the times are exactly metric.

I therefore still think that we should call both (2) and (3) **gregorian**, 
because that's no change from present. Their timestamps are UTC (or UTC-like if 
model) but their time conversions have no leap seconds and date-times for leap 
seconds cannot be encoded. We should call the new calendar (1) something else. 
We could call it **gregorian_leaps** for example, if **gregorian_utc** is 
misleading because (2) can also encode UTC. If we introduce this new calendar 
(1), we should prohibit the use of **gregorian** to mean this in future, so 
there will be no ambiguity for data written in future.

As we have all said, we could also have a pure **gregorian_tai** calendar, in 
which timesteps are TAI, conversions have no leap seconds and times are metric. 
I think we *should* add this if there is a use-case for it.

CF has often discussed introducing timestamps as an alternative to time 
coordinates, and equally often we've decided against it. I would appeal not to 
start that again - at least, not in this issue!

Best wishes

Jonathan


-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436029042

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-11-05 Thread JimBiardCics

@ChrisBarker-NOAA UTC is, in practical terms, a specification for how to turn 
TAI (a count of SI seconds since the TAI epoch) into time stamps that are 
synchronized with the motions of the earth. It uses a combination of the 
Gregorian calendar and leap seconds to achieve this goal. It does not, strictly 
speaking, provide for a way to deal with times before the TAI epoch. You have 
no choice but to assume there are no leap seconds, with an unknown impact (but 
there were no satellites then, and probably no time sequences long enough with 
fine-enough resolution to matter).

It's an inversion from where all this started, but we could, as you suggested, 
call the fully metric calendar **`UTC`** or **`gregorian_utc`**. Such a 
calendar would have all the characteristics of **`gregorian_metric`** in my 
previous comment.

Here's my objection to that name. I'm concerned that this will lead people to 
think that they can't use that calendar for anything other than 'UTC times'. 
This is why I used **`gregorian_utc`** for the other case in earlier versions 
of this proposal. It was to signal that the time variable contains naively 
converted UTC time stamps, and shouldn't be used for anything else.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-436005691

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-11-05 Thread Chris Barker

"""
I think your proposal to allow binary time stamps is a good one. Or even string 
time stamps. It could have a standard name of time_stamp. For example, if you 
organize a binary time stamp in 4-bit fields (which can be marked up using 
flag_masks and flag_meanings attributes) in high-to-low bit order 
-MM-DD-HH-mm- where  represents centiseconds, you have a monotonic 
result.

yup -- though thinking more, I think we should probably just go with strings -- 
I don't think there is any existing standard for a binary form, so users would 
need to be writing custom byte-manipulation code both to read and write these. 
Until/unless it gets built into the netcdf libs, it would be a pretty big 
barrier to entry.

"""
I will say, however, that we have this large user base in the wild that isn't 
working this way. Even if we added this as an option, I feel like we need to 
give recognition to the existing use case.
"""
Recognition is not the same as endorsement -- codifying it into CF is 
endorsement. And this new calendar will apply only to new files anyway, users 
will need to make at least a small change it their workflow to use it -- if we 
gave tham an almost-as-easy alternative, then we're OK, yes?





-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435993460

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-11-05 Thread JimBiardCics

@ChrisBarker-NOAA I love your name proposals!

I think your proposal to allow binary time stamps is a good one. Or even string 
time stamps. It could have a standard name of **`time_stamp`**. For example, if 
you organize a binary time stamp in 4-bit fields (which can be marked up using 
**`flag_masks`** and **`flag_meanings`** attributes) in high-to-low bit order 
-MM-DD-HH-mm- where  represents centiseconds, you have a monotonic 
result.

I will say, however, that we have this large user base in the wild that isn't 
working this way. Even if we added this as an option, I feel like we need to 
give recognition to the existing use case.

Regarding TAI time, there are two different potential situations, and we need 
to make sure we aren't confusing them. I think that confusion is my fault. 
There is fully metric elapsed time, which can be constructed any number of 
ways, among them being:
- Get TAI time counts and subtract them from an epoch TAI time count. The epoch 
TAI time count must be converted into a time stamp for use in the **`units`** 
attribute. That time stamp could be TAI or UTC.
- Get TAI time stamps and convert them to elapsed times since a TAI epoch. In 
this case, the conversion is done without consideration of leap seconds and 
produces correct results. The TAI time stamp could be used directly in the 
**`units`** attribute or first converted to UTC.
- Get UTC time stamps and convert them to elapsed times since a UTC epoch. In 
this case the conversion is done __with__ consideration of leap seconds and 
produces correct results. The UTC time stamp could be used directly in the 
**`units`** attribute or first converted to TAI.
- Get GPS week number and elapsed time counts and convert them to elapsed times 
since an epoch week number and elapsed time count. The epoch week number and 
elapsed time count must be converted into a time stamp for use in the 
**`units`** attribute. That time stamp could be TAI or UTC.

Then there is TAI time itself. Which is, at core, a count of seconds since 
1958-01-01 00:0:00. It is represented for human benefit as time stamps using 
the Gregorian calendar as years, months, days, hours, minutes, and seconds, 
etc, but no attempt is made to keep this synchronized with the rotation of the 
earth - no leap seconds modifying the time stamps.

A TAI calendar is an option. In that case the epoch time stamp would be TAI. 
But as I showed above, there are actually many different ways to get to fully 
metric elapsed time. I believe this is what we need to tell users about through 
a new calendar. We want them to know that the data producer is guaranteeing 
that the elapsed times in the time variable are fully metric. The particular 
time system used for the epoch date stamp is important, because you need to 
know what it is, but I don't see a strong motivation for TAI over UTC. If you, 
as a data producer, are messing with TAI, then I think it is reasonable to 
assume that you will have software that will convert to UTC.

So if we back up a moment and stop considering which of many sources the data 
producer might have used to get her fully metric elapsed times, and if we take 
the simplifying approach of mandating UTC epoch time stamps, we see that all 
the different source cases fit in one new calendar. We don't need 4 (or 5 or 6) 
new calendars to account for the different possible sources for the fully 
metric elapsed times.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435980225

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-11-03 Thread larsbarring

Dear all,

[Sorry for joining in late, I am currently and for the coming two weeks 
travelling with only infrequent internet access]. 

I have just read Jim's invitation to the email list thread on 360_day months to 
come over here. In that thread I just waited for the email conversation a 
couple of years ago on calendars and TAI to come into play. But it happened 
here instead. Many thanks Jim for putting together the background text, it did 
a good job in structuring my thoughts on this. And I can only agree with Chris 
and Jonathan, this is a complicated thread and I am not sure that I have 
grasped the full depth of all aspects. Hence, the following questions of 
understanding:

1. If I understand previous threads and Jim's background text, the need for 
high-resolution linear (metric) time comes from the satellite community. Why is 
that (I know that satellites are moving at high speed)? If someone wants to 
compare a scene/pixel/vertical profile/etc. from time1 with the corresponding 
scene/pixel/vertical profile/etc. at time2, is really a 'few seconds' that 
important given the sampling granularity (i.e. how often a certain point is 
revisited and/or the spatial and temporal 'averaging' involved in acquiring the 
data)? Is it because i) the precise timing (as such) IS in fact important, or 
is it because ii) the precise timing determines the precise geolocation of the 
data (which is reasonable considering the speed of the platform)? If it is 
because of i) then it is clear to me that a highly precise linear (metric) 
calendar is needed. If, however, ii) is the underlying reason for this 
discussion I am not so sure. Is it the right way to go for CF to conceptually 
merge the need for spatial accuracy with the already complex machinery for CF 
calendar and time representation?  Should not this be handled in the data 
preparation step.  
 
2. Jim's background text states that "the synchronization of time stamps with 
the cycles of the planet, is where calendars come into play, and this is where 
things get ugly." I fully agree, and when we now are discussing in some depth 
the CF representation of time and calendars, I would like not to forget another 
important use case for CF, namely the climate modelling (or broader 
geophysical) communities. Here seconds is useful as a SI unit, but data 
requiring this precision is rare if at all existing (LES/CFD modellers using 
CF, anyone?). The two time units "day" and "year" both have specific 
geophysical meaning, where the current definition 1 day = 84600 seconds is to 
my knowledge perfectly all right for all models. But the (sort of) time unit 
"year" is differently but well defined in the model representation of the 
planet. And I would argue that this is such an important geophysical model and 
real world quantity that it should be better suppported by CF. In various 
threads over the years I have seen the argument that the Udunits time units 
month and year should not be allowed by CF. I understand the argument, I think, 
but just because it is difficult (so far) within the present CF and Udunits 
versions it should not be ruled out that easily. Could we penetrate this in 
some more depth now when dealing with these matters, or should it be another 
thread?

3. Mostly a comment: a while back there was a thread on "gregorian" calendar 
vs. "mixed_gregorian_julian" calendar. To improve clarity and precision in 
defined concepts and calendars, would it not be better to deprecate "gregorian" 
and "standard", and instead use "mixed_julian_gregorian" (julian first then 
gregorian). Otherwise it might become confusing if we end up with "gregorian", 
"gregorian_utc" and "gregorian_tai".


Kind regards,
Lars



-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435609575

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-11-02 Thread Chris Barker

@j

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435549755

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

@martinjuckes I appreciate your concerns. I think that the two new calendars I 
have proposed provide a clear path for anyone who needs to do metric math with 
time.

**`new calendar a`** signals to a user just how they should go about getting to 
metric times from the time variable contents if they need to do that. They will 
need a leap seconds file and appropriate software, but they can do it with no 
problem. **`new calendar b`** signals to a user that the time variable contents 
are already metric. It also signals to a user that if he is interested in 
getting UTC time stamps, he will need a leap seconds file and appropriate 
software.

**`new calendar a`** also signals to a user that doesn't need to do metric math 
with time that she can simply convert the elapsed times to time stamps without 
need for leap seconds files or extra software.

Let's consider the situation we have now, where the only game in town for 
current data is, effectively, **`gregorian`**. Users have no idea how to handle 
the times. If accuracy to the 1 - 37 seconds level is required, users have had 
to obtain information from the data provider, figure out what is going on 
through experimentation, or are unaware of the reason their results are 
sometimes off.

Educating people about time systems and encouraging them to use "smarter" 
software is a good thing. Writing CF conventions in such a fashion that we make 
them too difficult to use in the name of consistency is not a good thing. I 
know it doesn't sound like it is "too difficult", but the scientists I work 
with have a hard time dealing with a number of aspects of CF. I feel confident 
they would throw their hands up if told they had to start using software that 
knew how to handle leap seconds.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435511718

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-11-02 Thread Chris Barker

I'm worried about the fact that applications that care about leap seconds
will probably also require more precision (at least microsecond, maybe
nanosecond?

Python’s datetime is microsecond precision. And numpy’s is (or can be)
nanosecond:-)

But I don’t think subclassing would be all that easy.

It might be pretty easy to simply add and remove the leap seconds before
and after a computation.

CHB



—

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
,
or mute the thread

.


-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435502296

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-11-02 Thread Martin

Hello All,

I accept @ChrisBarker-NOAA that my objection to the solution @JimBiardCics has
called `gregorian_utc` is pedantic, but I still consider it important. My
objection has nothing to do with the name, but is to do with the fact that an
increment of 1 second in the time variable may be interpreted as 2 seconds.
This is fine in a an index, but not in a variable with standard name `time`.
More on this below.

The alternative suggested by @ChrisBarker-NOAA above, of providing support in
CF for time stamps to be stored without forcing the user to go through
error-prone translations looks preferable to me, and far more robust. Ideally,
they should be able to store not only the time stamp information but also some
information about where it came from.

Coming back to whether a one second adjustment to time coordinates matters. As
I said, I'm being pedantic without apology. We all know that information put in
files contains many uncertainties, and errors in time information greater that
a second are certainly common. The issue here is about the precision and
robustness of the standard. In the example described by @JimBiardCics
[above](https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435181134)
, a variable of variable value of `39` should, according to the proposed
convention, be interpreted as `40` on the assumption that a user who is unable
to process the time stamp accurately has reliably provided the full information
about the method he is using to process the time stamp. I think this well known
and perhaps overused comic from [xkcd](https://xkcd.com/927/) sums it up: there
are two many standards for dealing with leap seconds ... adding another one
looks like adding more confusion to me. I'm not convinced that "UTC converted
to elapsed time without taking leap seconds into account" can reliably handled
in this way. If the consequences of the neglect of leap seconds are
predictable, everything is fine, but software is seldom that reliable. There
may also be issues around the data being stored. The user who is providing the
data may integrating or differentiating the data, making use of his/her time
increment which, according to the proposed standard would be different from the
time increment seen by applications which apply the leap-second adjustment.

As pointed out in a comment by Chris
[above](https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435152076)
it is not that hard to find a library to deal with leap seconds. It may be
worth trying to get such support incorporated into cf-python and making the
community more aware of the issue by updating the convention to allow users to
choose between fixed length days (i.e. TAI) or (UTC).

--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435464856

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

In python, it seems like something could be built around 
[astropy](http://docs.astropy.org/en/stable/time/).

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435431651

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

No - I was just making a comment about why a new tool will be needed to deal 
with those applications

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435447428

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-11-02 Thread Chris Barker

It seems this issue has become a bit entangled with multiple problems in search 
of a solution— they are all related to calendars, but also could each be 
independently implemented. Maybe separate issue for each? 

In the meantime, I think these are what’s on the table:

- calendar(s) used for climate modeling — 360 days per year. 
  - proposed solution: we come up with a name and definition and we’re done.
   - question: do climate modelers use “time stamps” that look like the 
Gregorian calendar? That is: March 15, 2025? (And of course, potentially 
February 30, 2100)? If so then this is all pretty straightforward. If, on the 
other hand, they use another system, like year 2025, day 154. Then we should 
have a way to directly encode that. 
- is there any ambiguity in what “2016-06-15” means? Does anyone care about 
that ambiguity? That is, saying a timestamp is UTC clearly identifies it, but 
what does a timestamp in 360_day_year calendar mean? Can it be converted to UTC?

OK, that’s enough for a phone...

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435435361

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

@jswhit I expect they well might. We aren't limiting the resolution by anything 
we are doing here, are we?

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435444196

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

After reading that link I can appreciate that XKCD comic even more :-)

The [cftime](http://github.com/Unidata/cftime) utility that I've worked on is 
quite primitive and will probably never be able to serve applications that 
require accounting for leap seconds.  I agree that it is important to have some 
sort of calendar designation that signifies that this precision is needed, so 
that tools such as mine can return an error message, instead of an incorrect 
answer.   **gregrorian_tai** or **proleptic_gregorian_tai** seem like perfectly 
acceptable names to me.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435423548

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

Here's a good compendium of information about time systems.

https://www.ucolick.org/~sla/leapsecs/timescales.html

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435417475

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

@jswhit There are no leap seconds prior to - depending on your viewpoint - 
somewhere between 1958 to 1970, so **`gregorian_utc`** (**`new calendar a`**) 
collapses to **`gregorian`** for dates prior to that range. I have no 
preference over where that should be considered proleptic or not. I tend to 
think that a time series using the proleptic Gregorian calendar would not care 
about UTC as such.

What do you think?

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435417291

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

Here's a bit of comic relief on this topic, courtesy of XKCD.
https://xkcd.com/2050/

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435411435

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

Is **gregorian_utc** a proleptic gregorian calendar, or is the same as 
**gregorian** (mixed Julian/gregorian) with a known relationship with UTC?

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435406498

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

@JonathanGregory 
I think the new calendars, as you have described them, are not what I am 
proposing, and I don't think it is the direction we should go. I think the 
names were causing confusion in the discussion, and I get the impression that 
it has influenced your thinking too. This is why I started referring to the 
calendars by generic names - **`new calendar a`** and **`new calendar b`**.

Are you assuming that time values in the time variables for both of the new 
calendars will be fully metric (no non-linearities or unexpected offsets)?

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435381045

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-11-01 Thread JonathanGregory

Dear Jim, Martin, Chris et al.

Thanks for this discussion and specially many thanks to Jim for spending the 
time to make a proposal. It was indeed an epic discussion on the email list, 
and it's great to see an outcome.

I agree with the proposal as Jim has made it, if I understand it correctly, 
except that I agree with Martin that the reference time in the time units 
should be given in the calendar specified by the calendar attribute. If we have 
calendar="julian", zero "days since 1917-10-25" means 7 Nov 1917 in the 
gregorian calendar (which is "proleptic" UTC). It does not mean 25 Oct 1917 in 
the gregorian calendar.

In CF, as Chris says, a "calendar" is a way to translate from 
time-since-a-timestamp to another timestamp (timestamp being year, day, month, 
minute, second). The calendar attribute identifies the rules for converting 
between a time coordinate (with time units) and a timestamp (components of 
time). A crude way to picture these "rules" is that a calendar is defined by an 
ordered list of all the valid times. For the current set of CF calendars, it 
would be sufficient to list all the valid dates (-MM-DD) in the calendar, 
which have a spacing of 1 day (86400 s), but with the complication of leap 
seconds we have to regard the calendar as being defined by an ordered list of 
all the valid timestamps (-MM-DD HH:MM:SS) in the calendar, which have a 
spacing of 1 second. The reference time in the time-units is one of these 
timestamps. To work out the time coordinate for a given timestamp, you count 
how many valid timestamps there are between the reference and the one you want.

Martin's interpretation of the reference time would modify Jim's proposal 
slightly, according to my understanding, which is as follows:

gregorian_tai. The reference date in the time-units is TAI, not UTC. The 
calendar has all the Gregorian dates but no leap seconds.

gregorian_utc. The reference date in the time-units is UTC. The calendar has 
all the Gregorian dates and includes leap seconds.

gregorian. The reference date in the time-units is UTC, if it's real-world 
data. The calendar has all the Gregorian dates but no leap seconds, so the 
rules for converting between time coordinates and timestamps are the same as 
for gregorian_tai. The timestamps can be encoded and decoded successfully by 
these rules but the difference between two time coordinates which span a leap 
second will be incorrect. That is, the time coordinates are not "metric", in 
Jim's sense. As Jim has explained, most real-world data is probably like this, 
because the timestamp comes from a clock which is synchronised to UTC but it 
was converted to a time coordinate according to the no-leap-second rules. 
Moreover, there is a lot of climate model data which uses this calendar, and 
it's *correct* because there are no leap seconds in the model world. It's not 
really "UTC" but it's what the model uses. In the model world, the time 
coordinates are metric.

I support introducing gregorian_tai and gregorian_utc for real-world 
applications which require time coordinates that take account of leap seconds. 
Should it be prohibited to use these two new calendars for dates before 
1958-1-1, or are they gregorian before that date? I assume it should be 
prohibited to use gregorian_utc for dates in the future, since the leap seconds 
are unknown.

Best wishes

Jonathan

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435227916

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

@martinjuckes 
Providing concrete examples like you did is quite useful. I should probably 
have done so myself well before this. My apologies.

Let's say I am acquiring data once every 10 seconds using a GPS receiver to get 
UTC time stamps. Around the last leap second I will have:
- **`2017-12-31T23:59:40Z`**
- **`2017-12-31T23:59:50Z`**
- **`2017-12-31T23:59:60Z`**
- **`2017-01-01T00:00:09Z`**
- **`2017-01-01T00:00:19Z`**
- **`2017-01-01T00:00:29Z`**

This represents the worst case scenario.

Here is what my time variable will contain using the two new calendars:
- Time variable using **`new calendar a`** (UTC converted to elapsed time 
without taking leap seconds into account)
   - units = "seconds since 2017-12-31 23:59:40"
   - data = [ 0, 10, 20, 29, 39, 49 ]
- Time variable using **`new calendar b`** (UTC time stamps converted to 
elapsed time taking leap seconds into account)
   - units = "seconds since 2017-12-31 23:59:40"
   - data = [ 0, 10, 20, 30, 40, 50 ]

In the first case the time variable encodes an error. This is what makes the 
contents non-metric. Comparison of the intervals before and after the leap 
second yields a value of 10 seconds, but the center interval is 9 seconds. It's 
not that the seconds aren't SI seconds. It's that we have encoded an error. In 
the second case, the time variable contents are fully metric.

What happens if we extract UTC time stamps  from these two different time 
variables? We get:
1. Time variable using **`new calendar a`** (Converted to UTC time stamps 
without taking leap seconds into account)
   - **`2017-12-31T23:59:40Z`**
   - **`2017-12-31T23:59:50Z`**
   - **`2017-01-01T00:00:00Z`** - error here
   - **`2017-01-01T00:00:09Z`**
   - **`2017-01-01T00:00:19Z`**
   - **`2017-01-01T00:00:29Z`**
1. Time variable using **`new calendar a`** (Converted to UTC time stamps 
taking leap seconds into account - requires custom software)
   - **`2017-12-31T23:59:40Z`**
- **`2017-12-31T23:59:50Z`**
   - **`2017-12-31T23:59:60Z`**
   - **`2017-01-01T00:00:09Z`**
   - **`2017-01-01T00:00:19Z`**
   - **`2017-01-01T00:00:29Z`**
1. Time variable using **`new calendar b`** (Converted to UTC time stamps 
without taking leap seconds into account)
   - **`2017-12-31T23:59:40Z`**
   - **`2017-12-31T23:59:50Z`**
   - **`2017-01-01T00:00:00Z`** - error here
   - **`2017-01-01T00:00:10Z`** - error here
   - **`2017-01-01T00:00:20Z`** - error here
   - **`2017-01-01T00:00:30Z`** - error here
1. Time variable using **`new calendar b`** (Converted to UTC time stamps 
taking leap seconds into account)
   - **`2017-12-31T23:59:40Z`**
- **`2017-12-31T23:59:50Z`**
   - **`2017-12-31T23:59:60Z`**
   - **`2017-01-01T00:00:09Z`**
   - **`2017-01-01T00:00:19Z`**
   - **`2017-01-01T00:00:29Z`**

In the first case, we get back (except on the leap second) the time stamps that 
were the inputs to the variable. There is an error in the results, but notice 
that if the times were each one second later, you wouldn't see any error in the 
results.

In the second case, the savvy user understands from the calendar used that 
there will be an error in the time stamp on the leap second. That user captures 
that special case with custom software and fixes the one time stamp.

In the third case, the unfortunate user doesn't pay attention to what it means 
to have a variable that uses **`new calendar b`** and converts the metric 
elapsed times to UTC without considering leap seconds. They get a one second 
error that propagates. But they ignored the calendar, so we can't really help 
them.

In the fourth case, the savvy user understands from the calendar used that the 
elapsed times are metric, and that if she wants to get an accurate UTC time 
stamp, she must use a process that is aware of leap seconds. She does so, and 
gets fully correct results back.

The reason for the two different new calendars is that the first, erroneous 
time variable produces correct results (with one exception) if the users are 
only interested in getting time stamps. If they convert the values back to UTC 
time stamps without taking leap seconds into account, they will get back the 
original inputs the vast majority of the time. And that is well sufficient for 
a large number of users.

The second, "correct" time variable produces correct results if the user is 
savvy. Notice that the savvy user can also get correct results from the first 
time variable if they want to. The savvy user can also get exact elapsed times 
from the first variable by adding leap seconds as needed if they need exact 
elapsed time values.
The only truly bad scenario here is the one in which an un-savvy user tries to 
get UTC time stamps from the "correct" time variable without taking leap 
seconds into account.

This is why I have proposed the two new calendars. They allow data producers to 
signal to savvy users how to use the contents. **`new calendar a`** provides 
maximum accommodation of the needs of less savvy users without posing

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-11-01 Thread Chris Barker

> As I look at it, the case I am describing above is not the same as the "I 
> don't know about all this and it doesn't matter to me" 

I agree -- THAT case is currently handled (defacto) by "gregorian", and I think 
we all agree that we should clarify that in the docs.

My concern is the "It matters to me,  but I don't really understand exactly 
what the systems and libraries I'm working with do and what their limitations 
are" use case.

And I think those folks would be poorly served by a CF standard that appeared 
to support their use case, but might give incorrect results.

For the "I really know exactly what my requirements are, and I know that the 
tools I use satisfy them" crowd -- they can use proper TAI or a datetime 
library that handles leap seconds.

As for the distinction between producers and consumers -- that is a big issue 
-- consumers are likely to be less knowledgeable, so you do want a way to tell 
them what to do with the tools they have access to. So the idea of a way to 
tell them: "decode this with a non-leap-second-aware library and you will get 
what you expect makes sense.

@JimBiardCics wrote:
>  they would be perfectly happy with storing the time stamps as strings 
> instead of converting to elapsed time since an epoch. But CF mandates elapsed 
> time since an epoch, so that is what they do. Their user base likely feels 
> the same way. Precision is desired, but metricity is not an issue.

Funny you should say this -- on the bus this morning, I was thinking about 
this, and realized exactly this -- the use case is:

- I have a bunch of UTC timestamps (e.g.  2018-11-01T12:15:32)
- I want to store them in a time variable in a CF compliant way
- I want users of that file to get exactly the same timestamps back when they 
use the data.

And the fact is that without the existence of leap seconds in commonly 
available libraries, it's actually hard to do that properly with the CF 
time-elapsed-since-a-timestamp. 

So the solution to this is to provide, in CF, a way to encode timestamps 
directly.  ISO8601 strings are probably the easiest way to do it, but you could 
store them in a more compact binary format if you like (like the python 
datetime object does internally).

I was part of the conversation a while ago about doing this, and did not 
support it because it seemed like simply another way to encode the same 
information. But I've come to the conclusion now that it is NOT the same 
information with a different encoding.

Storing timestamps directly (as a string or other format) is a way to store not 
a time axis (with nice metric properties, etc), but to store, well, timestamps 
-- like "this measurement was taken at this date and time".

So my personal opinion:

1) CF should accommodate the use-case Jim is advocating for.

2) The best way to do that is to have a format for storing timestamps -- maybe 
we could even call it "timestamp" or something, rather than "time".

3) If (2) is not acceptable to the CF community, then JIm's proposal is 
probably the only practical solution.

NOTE: I have come around because I've come to a new understanding of the 
problem we are trying to solve. I thought it was:

"We need a way for people to store data  that are working with inappropriate 
tools for their use-case, that will create invalid data in the CF file, but 
that will probably be able to be recovered with standard tools"

>From that perspective, you can probably see why I thought it was not a good 
>idea.

Now I think the problem we are trying to solve is:

We need a way for people to store precise-to-the-leap-second UTC timestamps 
such that they and their users will get the exact same timestamps back when 
processing the file"

I imagine most of you will agree that that is a reasonable use-case for CF.

Jim's proposal is a way to accommodate this use-case using the current CF 
approach to encoding datetime and commonly available tools. However, it is also 
a way that is prone to confusion, errors and hidden bugs.

Thus I propose a way to directly store timestamps, which will address the 
problem at hand, in a clear and much less error prone way.

I want to thank Jim for his persistence in continuing to push on this in the 
face of my intransigence :-)

Final thought -- we have also brought up issues in this thread about whether to 
properly support TAI, etc -- maybe a new issue is in order for that?

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435179676

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-11-01 Thread Chris Barker

@martinjuckes wrote:

> Rather than create a kludgy fix in the standard, couldn't we provide people 
> with clearer guidance? The leap seconds are not that complicated  though 
> it took be a while to get the information and it be useful to make that bit 
> easier for our users, e.g. the authorise IETF list of leap seconds is here .. 
> and this list will be updated at least 6 months before a new leap seconds is 
> introduced. 

I have to agree with Jim here -- guidance is not the problem -- UTC is well 
defined, and as you say the leap seconds are published, and the math is not 
that hard. The problem is libraries: no datetime library that I know of handles 
leap seconds properly (OK -- I just googled, how hard is that :-) -- the 
BoostDatetime lib does handle leap seconds). So I'll rephrase, "no datetime 
library commonly used by the CF community" handles leap seconds. 

As a community, maybe we could do something about that, but as a data standard, 
we're stuck with the state of practice as it is.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435152076

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-11-01 Thread Martin

hi @JimBiardCics .. my previous comment (2 above) was a response to your post 4 
above, before I saw your comment addressed to me (3 above) ... 

Firstly, in response to your most recent question:
If you add 120 seconds to  `2016-12-31T23:59:00Z (UTC)` you get 
`2017-01-01T00:00:59Z (UTC)` because of the leap second. You appear to be 
suggesting that `2017-01-01T00:00:59Z` could be encoded with `119 
unit_formerly_known_as_seconds since 2016-12-31T23:59:00Z` in the case of the 
calendar formerly known as `gregorian_utc`. The 
`unit_formerly_known_as_seconds` is not the same as SI seconds. `120 seconds` 
has a unique meaning for all other calendars which is not modified in any way 
by the choice of calendar. A system in which two physical seconds map onto one 
unit of measure is introducing substantial new complexity. 

I didn't realise that you has previously considered the introducing a 
TAI-timestamped calendar. I don't understand your conclusion that this would 
require 3 new calendars. To be, it appears that we would want two versions of 
`gregorian`, one with leap seconds (i.e. UTC) and the other without (fixed 
minutes, as used for TAI). Can't we get away with one new calendar, and a 
clarification of the definition of `gregorian`?

If I've understood correctly (sorry if I've been a bit slow) you are concerned 
about users who have a sequence of UTC time stamps such as:
- 2016-12-31T23:59:40Z
- 2016-12-31T23:59:50Z
- 2016-12-31T23:59:60Z
- 2017-01-01T00:00:10Z
and would like to encode them as times in `seconds since 2016-12-31T23:59:40Z`. 
 The correct answer would be [0,10,20,31]. If their software is not leap-second 
aware they might get [0,10,20,30], but they could also get [0,10,0,30] (60 
seconds identified as 0) or [10,10,NaN,30] (software fails silently on seeing 
60 seconds).  

Rather than create a kludgy fix in the standard, couldn't we provide people 
with clearer guidance? The leap seconds are not that complicated  though it 
took be a while to get the information and it be useful to make that bit easier 
for our users, e.g. the authorise IETF list of leap seconds is 
[here](https://www.ietf.org/timezones/data/leap-seconds.list) .. and this list 
will be updated at least 6 months before a new leap seconds is introduced. 

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435109348

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

@martinjuckes How is the calendar formerly known as **`gregorian_utc`** not 
consistent with use of SI time units?

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435087156

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-11-01 Thread Martin

However, calendar  formerly known as `gregorian_utc`, as described at the start 
of this discussion, is not consistent with use of SI time units, and so could 
not be used with standard name `time`. What is described here has no relation 
to the concept of calendar as it currently exists in the CF convention. I can't 
see how such a major step in increased complexity can be justified. 

Why are you not prepared to consider working with the structures we have?




-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435086726

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

@martinjuckes My perspective is that we will continue to honor SI seconds and 
days, weeks, and fortnights (2 weeks) as integer multiples of SI seconds. The 
thing that can be quite confusing in all this is that the number of seconds 
that has elapsed since any given epoch is the same in both the UTC and TAI time 
systems. The time stamps for the epoch may be different, but the elapsed time 
is always exactly the same. The same is true of 360-day, lunar, 365-day, 
Julian, and Gregorian calendars, when used in the real world. The date stamp 
for a given epoch will be different in each, but the number of days that have 
elapsed since that epoch will be the same in all the calendars.

Leap days and leap seconds are a mechanism to keep the date and time stamps 
synchronized with the orbital and rotational motion of the earth relative to 
the sun. Adding leap days to date stamps keeps the Vernal Equinox to within one 
day of the same month and day. Adding leap seconds to time stamps keeps 
midnight at 0 degrees longitude to within one second of the same hour, minute, 
and second. Leap days and seconds are only about time stamps, not about how 
much time has passed. The Gregorian calendar says, "Let's keep the months and 
days lined up with the seasons." The UTC time system says, "Let's also keep the 
hours and minutes lined up with the solar days." The TAI time system says, 
"Let's let the hours and minutes do their own thing and slip relative to solar 
days."

When we had this discussion on the CF metadata listserv a few years ago, I 
started out advocating for a new calendar that would be entirely TAI, so epoch 
time stamps would be in TAI. This meant we would need three new calendars 
instead of two. I became convinced that the side effects of this would cause 
more problems than it would solve.

Users who didn't realize that the TAI epoch time stamps were not UTC (or didn't 
go to the trouble of getting leap-second-aware software to get time stamps from 
the time variable) would have maximal discrepancy between the times they 
thought they had and the actual times (different amounts in different years up 
to 37 seconds for data from 2017 and 2018). Data producers who want to store 
monotonic, metric elapsed times in their time variables won't find it hard to 
generate accurate UTC time stamps for their time variable epochs, so there 
didn't seem to be a compelling case for giving people the option of going "full 
TAI".

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435086285

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

@ChrisBarker-NOAA 
>> People who are converting precise and accurate UTC time stamps into values 
>> in time variables using the tools most available to software developers for 
>> handling time without sensitivity to leap seconds are creating time 
>> variables that have the potential to be non-monotonic because of leap 
>> seconds. They have done it in the past, they are doing it now, and they will 
>> very likely continue to do so.

> Agreed -- and we need to accommodate that -- which we already do implicitly 
> with "gregorian", and we should make it explicit in the sense that the whole 
> leap-second thing is ambiguous.

As I look at it, the case I am describing above is not the same as the "I don't 
know about all this and it doesn't matter to me" case. These are people for 
whom precise and accurate time matters to the level they are taking it, but 
they would be perfectly happy with storing the time stamps as strings instead 
of converting to elapsed time since an epoch. But CF mandates elapsed time 
since an epoch, so that is what they do. Their user base likely feels the same 
way. Precision is desired, but metricity is not an issue.

When I think about all of this, I find myself looking at two slightly different 
category lists. One is for producers, the other is for consumers.

For producers:
1. The distinctions are unimportant. (old **`gregorian`** is good enough)
1. Precision is important but metricity is not. The inputs to the netCDF 
creation process are accurate UTC time stamps. Using leap seconds to create 
fully metric and monotonic time variables imposes a burden on users to find 
software that handles leap seconds properly. (**`new calendar a`**)
1. Precision is important and metricity is important. The kind of input to the 
netCDF creation process doesn't matter, because they are going to do whatever 
it takes to generate accurate time variables. Their target users are going to 
do whatever it takes to use the time properly. (**`new calendar b`**)

For consumers:
1. The distinctions are unimportant. (old **`gregorian`** is good enough)
1. Precision is important but metricity is not. As long as they can get 
accurate UTC time stamps they are happy. They would rather not deal with leap 
seconds if they don't have to.  (**`new calendar a`**)
1. Precision is important and metricity is important. They likely want to do 
math with the elapsed time values themselves. Either new calendar works for 
them, because they are going to do whatever it takes to get what they need as 
long as they know what is in the time variable. (**`new calendar a`** is best, 
but **`new calendar b`** can be made to work)

I agree wholeheartedly that the ideal scenario would be to insist that the 
contents of every time variable be fully metric and monotonic under all 
circumstances. I think the practical reality is that the vast majority of both 
producers and consumers fall in their respective categories 1 and 2. As kludgy 
as it is, I think it is a good idea to allow **`new calendar a`** (the calendar 
formerly known as **`gregorian_utc`**) as well as **`new calendar b`** (the 
calendar formerly known as **`gregorian_tai`**).

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-435071298

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-10-31 Thread Chris Barker

> People who are converting precise and accurate UTC time stamps into values in 
> time variables using the tools most available to software developers for 
> handling time without sensitivity to leap seconds are creating time variables 
> that have the potential to be non-monotonic because of leap seconds. They 
> have done it in the past, they are doing it now, and they will very likely 
> continue to do so.

Agreed -- and we need to accommodate that -- which we already do implicitly 
with "gregorian", and we should make it explicit in the sense that the whole 
leap-second thing is ambiguous.

>Telling a significant group of data producers that they must change their 
>software in a way that will cause widespread problems for their users is a 
>good way to get them to ignore you.

Indeed. The question at hand is do we codify as correct what is, at best, a 
kludgy improper process?

As we've all said, the ambiguity is a non-issue for the vast number of CF use 
cases.

So the question really is -- are there folks for whom:

- Leap-seconds matter
- They know and understand the issues of UTC vs TAI, etc.
- They have no way to produce a "correct" time variable

I honestly don't know -- but it's been suggested that we can use UTC always as 
the timestamp, 'cause anyone that understands the distinction (and cares about 
it) will have access to a leap-second-aware-library. In which case, no -- there 
isn't a large user base.

But given the dirth of leap-second aware libs, maybe we have no choice to to 
codify this kludgy encoding.







-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-434816875

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-10-31 Thread JimBiardCics

On another front, I'm perfectly happy to entertain different names for the new 
calendars. I'm not a particular fan of **`gregorian_tai`** or `gregorian_utc`**.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-434805593

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-10-31 Thread Martin

Hello @JimBiardCics , thanks again for a detailed response. 

I support the aim of allowing users to place either a UTC or TAI time stamp in 
the units statement (so that they can do whatever fits with their existing 
processes) and making it possible for them to declare which they are using. The 
suggestion of using the calendar attribute for this makes sense.

I think we are agreed now that there is a unique and well defined mapping 
between these time stamps, and there is a unique and well defined way of 
calculating an elapsed time (in the SI sense) between any two such time stamps. 
 I don't see how the layers of complexity need to come into this. The TAI time 
stamp counts up with 86400 seconds per TAI day, while the UTC has a known 
selection of days with an extra second in the final minute. 

All we can do is define these things clearly, we can't force users to adopt 
best practice. As you say, some people don't have accurate clocks, just as some 
record temperature without accurate thermometers.  

I would disagree with you on one point: a non-monotonic data array is no 
problem, but a non-monotonic coordinate array is a problem. As Chris commented, 
people who need to know about this sort of problem are likely to have sorted it 
out before they get around to writing NetCDF files. 

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-434771654

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-10-31 Thread JimBiardCics

@ChrisBarker-NOAA @martinjuckes 
For the moment, let's set aside the question of names for the calendars.

There is nothing at all wrong with specifying that the epoch time stamp in the 
**`units`** attribute always be a correct UTC time stamp. In fact, allowing the 
epoch time stamp to be from a TAI or UTC clock will increase the chances that 
the data will be handled incorrectly. If you are sophisticated enough to care 
about TAI, you will have no problem dealing with a UTC time stamp.

I am explicitly assuming that all UTC time stamps are correct and accurate at 
the times they were acquired or constructed. If you are getting your time stamp 
from a PC that isn't actively synced by a time server, you shouldn't bother to 
use either of these new calendars.

When you read time out of a GPS unit, you can get a count of seconds since the 
GPS epoch, and I believe you can get a GPS time stamp that doesn't contain leap 
seconds (like TAI time stamps, but with a fixed offset from TAI), but most 
people get a UTC time stamp. The GPS messages carry the current leap second 
count and receivers apply it by default when generating time stamps. That's 
something I learned about from Aaron Sweeney a year or two ago - well after the 
big discussion we had about all of this a few years back.

There are quite a lot of high-precision data acquisition platforms out there 
that start with accurate UTC time stamps obtained from GPS receivers. Many of 
them don't care about metricity. They just want their time stamps, but CF tells 
them that they must store time as elapsed time since an epoch.

There's not really such thing as an elapsed time that is **UTC** vs **TAI**. At 
core, true elapsed time - a count of SI seconds since an event - is the same 
for both. The UTC *time stamp* for that event may not be identical to the TAI 
*time stamp* for that same event, but they both reference the same event, and 
the elapsed time since that event is the same no matter which time system you 
are using.

The UTC time system provides a prescription in terms of leap seconds for how to 
keep UTC *time stamps* synchronized with the rotation of the earth. Just like 
the Gregorian calendar system provides a prescription in terms of leap days for 
how to keep date stamps synchronized with the orbit of the earth. The only 
difference - and it is an admittedly important one - is that the UTC leap 
second prescription does not follow a fixed formula like the Gregorian leap day 
prescription does. The UTC time system declares that the time stamp 
**`1958-01-01 00:00:00`** references the same time as the matching TAI time 
stamp. The TAI time system provides no prescription for keeping TAI *time 
stamps* synchronized with the rotation of the earth.

Time stamps - whether UTC, TAI, GPS, or something else - are, in essence, a 
convenience for humans. No matter what time system or calendar system you use, 
the number of seconds or days that has elapsed since any given date and time is 
the same.

In a perfect world, all data producers would use a leap-second-aware function 
to turn their lists of time stamps into elapsed time values and all time 
variables would be "perfect". That would also force all users of those time 
variables to use a leap-second-aware function to turn the elapsed time values 
into time stamps. But that's not the world we live in. Does naive conversion of 
UTC time stamps into elapsed times have the potential to produce non-monotonic 
time coordinate variables that violate the CF conventions? Yes. Does it cause 
any real problems (for the vast majority of cases and instances of time) if 
people use this "broken" method for encoding and decoding their time stamps? No.

At the end of the day, it doesn't matter much what process you used to create 
your elapsed time values. *For the small number cases where differences of 1 - 
37 seconds matter*, we are trying to make a way for data producers to signal to 
data users how they should handle the values in their time variables while 
staying within the existing CF time framework and acknowledging CF and world 
history regarding the way we deal with time (which isn't very consistent and is 
composed of successive corrective overlays over centuries).

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-434732069

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-10-30 Thread JimBiardCics

@martinjuckes The point here is that we have an existing way to represent time 
that has been used for quite a few years now. This was never a problem for 
climate model data or data acquired on hourly or longer time intervals. We may 
at some future point (CF 2.0?, CF 3.0?) want to consider some significantly 
different way of handling time. For CF 1.* we want to find a way to accommodate 
satellite and other high frequency data acquisition systems without imposing 
unneeded burdens on anyone.

CF says that the purpose of the calendar attribute is to tell you what you need 
to know to convert the values in a time variable into time stamps. We aren't 
telling them (at least not directly) how we obtained the values in the time 
variable. We are telling them how to use them. @JonathanGregory, @marqh, and I 
came to the conclusion that, while there may be cases we didn't consider, 
pretty much every time variable anyone might create (within reason) would fall 
into one of three categories:
* The elapsed time values are fully metric and their relationship to the UTC 
epoch date and time is accurate. (The **`gregorian_tai`** case.) You must take 
leap seconds into account when converting these time values into UTC time 
stamps if you want fully accuracy.
* The elapsed time values are almost certainly not fully metric and their 
relationship to the UTC epoch date and time is probably not accurate, but if 
you convert them to UTC time stamps without adding any offsets, using a method 
that does not take leap seconds into account, you will get UTC time stamps with 
full accuracy. (The **`gregorian_utc`** case.)
* We don't have a clue about the metricity or accuracy of the elapsed time 
values or the epoch date and time. At least not to within 37 seconds. And we 
don't care. (The updated **`gregorian`** case.)

Time representation is a monster lurking just under the surface. Everything was 
fine until we looked down there. The only pure time is counts of SI seconds (or 
fractions thereof) since some agreed starting point. Everything else is 
burdened with thousands of years of history and compromise.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-434449056

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

@JimBiardCics wrote:

>  I'm calling the UTC time 'non-metric' because a time variable for data 
> sampled at a fixed rate based on UTC time stamps converted to elapsed time 
> since an epoch without dealing with leap seconds may contain deviations from 
> what you would expect. If you attempt to 'do math' with the contents, you may 
> find that adding an interval to a time does not produce the time you expected 
> and subtracting two times does not produce the interval expected.

Thanks, got it.

My take on this -- if you do that, you have created invalid, incorrect data. CF 
should not encode this as a valid thing to do. As far as I'm concerned, it's 
the same as if you did datetime math with a broken library that didn't do 
leapyears correctly.

And frankly, I'm not sure HOW we could accommodate it anyway -- I'm still a bit 
confused about exactly when leap seconds are applied to what (that is, I think 
*nix systems, for instance, will set the current time with leap seconds -- so 
the "UTC" timestamp is really "TIA as of the last time it was reset". Which I 
think is the concern here -- if you are collecting data, and are getting a 
timestamp from a machine, you don't really know which leap-seconds have been 
applied.

But again, that's broken data

If we were to try to accommodate this kind of broken data, I have no idea how 
one would do it? One of the reasons that leap seconds are not used in most time 
libraries is that they are not predictable. So a lib released last year may 
compute a different result than one released today -- how could we even encode 
that in CF?!?!

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-434395345

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

About @martinjuckes' CDL:

I am really confused as to why anyone would ever want to do that.

I remember a thread a while back about encoding time as ISO 8601 strings, 
rather than "time_unit_ since timestamp" -- at the time, I opposed the idea, 
but now we have an even better reason why.

If we stick to the current CF convention, then all we need to do is specify the 
TIA calendar as a valid calendar (and clarify UTC vs TIA) -- that's it -- 
nothing else needs to change, and there is no ambiguity.

Is the goal here to be able to specify TIA in a way that users can use it 
without a TIA-aware library? I think that's simply a bad idea -- if you don't 
have a TIA aware library, you have no business working with TIA times (at least 
if you care about second-level precision).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-434381101

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

2018-10-30 Thread JimBiardCics

@ChrisBarker-NOAA As @martinjuckes mentioned in his comment, I'm calling the 
UTC time 'non-metric' because a time variable for data sampled at a fixed rate 
based on UTC time stamps converted to elapsed time since an epoch without 
dealing with leap seconds may contain deviations from what you would expect. If 
you attempt to 'do math' with the contents, you may find that adding an 
interval to a time does not produce the time you expected and subtracting two 
times does not produce the interval expected.

Let's say I have acquired data at regular intervals, **dt**, and I have 
captured the time by grabbing accurate UTC time stamps for each acquisition. If 
I construct my time variable by naively converting my acquired time stamps to 
elapsed time since my epoch UTC time stamp, I might have one or more problems 
lurking in my time variable.
1. If a leap second occurred over the span of my time variable, there will be a 
point in the times where one of three problems appear.
   1. If **dt** is greater than 1 second, there will be an interval between 
successive time values that is less than **dt**. The formula **t[i] = t[0] + 
i*dt** won't hold for all the times in the variable.
   1. If **dt** is equal to 1 second, the variable also won't be monotonic. 
There will be a pair of time values that are identical.
   1. If **dt** is less than 1 second, the variable won't be monotonic, but 
rather than a pair of time values that are identical, there may be a section 
where one or more time values are less than a preceding value.
1. If one or more leap seconds occurred in the time range between my epoch time 
stamp and my first acquired time stamp, my time values will be internally 
consistent, but the whole set of values will be smaller than expected.

The non-monotonicity problem is one that I don't even want to get into. And, 
again, for someone measuring things once an hour (for example) this is all 
pretty ignorable.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-434381038

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)

> The only difficulty is that the translation from Calendar A to Calendar B is 
> undefined for dates after 2019-06-30 -- for a standard such as CF this is 
> problematic.

Is it? It's terribly problematic for libraries -- which is why most of them 
don't deal with it at all. But for CF, I'm not sure it matters:

datetimes are encoded as:

a_time_unit since a_datetime_stamp.

TIA and UTC define seconds the same way. And no one should use a time unit that 
isn't  clearly defined multiple of seconds.

The only difference between TIA and UTC here is when you want to convert that 
encoding to a datetime stamp -- exactly what datetime stamp you get depends on 
whether you are using TIA or UTC, but the seconds are still fine.

If someone encodes a datetime in the future, and specifies TIA time, then 
clients will not be able to properly convert it to a datetimestamp -- but that 
isn't a CF problem.



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/148#issuecomment-434372733

Re: [cf-convention/cf-conventions] Add calendars gregorian_tai and gregorian_utc (#148)