Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
On Tue, Jan 15, 2019 at 10:09 PM Gary E. Miller wrote: > Yo Warner! > > On Tue, 15 Jan 2019 17:45:32 -0700 > Warner Losh wrote: > > > > > Except that's not a leap second. struct tm doesn't do leap seconds > > > > really. It kinda sorta does, but it kinda sorts doesn't. > > > > > > Well, it does in a modern Linux kernel. Otherwise NTPD would fail > > > on the leap second, and that only happens every year so. :-) > > > > > > > You are wrong. The kernel doesn't use struct tm. > > I never said the kernel uses struct tm. I said you can query the kernel > for > the time in a struct tm. > What system call is that? A quick grep didn't turn anything up for me. All the calls I'm aware of are library calls that get the system time and then break it up. > > The kernel does this > > by ticking either in TAI or by incrementing a TAI offset. ntpd uses a > > different interface that exposes these things to it so it can > > broadcast the leap second indicators correctly. > > Yup. > > > > > First, when you convert back and forth to time_t, you lose leap > > > > second information. It's impossible to convert back and forth > > > > from at least the UTC time zone. There's some 'right' time zones, > > > > but they aren't strictly speaking POSIX compliant because it uses > > > > the wrong math. Non-conforming, but damned useful and not wrong. > > > > > > Which is why you can't use time_t in UTC, but time_t in TAI > > > can work, as well as struct tm, usually. Lot's of latent bugs... > > > > > > > No. time_t is not TAI or GPS. It is UTC. Adjusted UTC. > > No, time_t is just a way to count seconds. See man difftime() for a > use of time_t that is unrelated to any particular epoch. > This is the biggest mistake people make with POSIX time_t. It's not just a way to count seconds. It is the funky seconds since 1970 thing. By definition. difftime takes two time_t's and returns a double that's the difference between them. This is only a time_t when time_t is defined to be a double (which is allowed, but apart from IBM, nobody has done that), otherwise it is not. Typically, time_t is some int type, so difftime isn't a good example here. Also, the open group standard is silent about what it's supposed to do across a leap second, but since it operates on time_t, it's unclear if a time_t of 1230767 and 1230768002 should be 3 or 4 since those two times straddle 2009 December 31, 23h 59m 60s. I've seen spirited debate on both sides of that issue. > It is the > > number of SI seconds since 1970, minus all the positive leap seconds, > > plus all the negative ones. > > Ciration please? I already cited the Linux doc that says otherwise. > The Linux doc does not define the standard. The POSIX doc does (from the Open Group 2017 edition): 3.150 Epoch The time zero hours, zero minutes, zero seconds, on January 1, 1970 Coordinated Universal Time (UTC). 4.16 Seconds Since the Epoch A value that approximates the number of seconds that have elapsed since the Epoch. A Coordinated Universal Time name (specified in terms of seconds (tm_sec), minutes (tm_min), hours (tm_hour), days since January 1 of the year (tm_yday), and calendar year minus 1900 (tm_year)) is related to a time represented as seconds since the Epoch, according to the expression below. If the year is <1970 or the value is negative, the relationship is undefined. If the year is >=1970 and the value is non-negative, the value is related to a Coordinated Universal Time name according to the C-language expression, where tm_sec, tm_min, tm_hour, tm_yday, and tm_year are all integer types: tm_sec + tm_min*60 + tm_hour*3600 + tm_yday*86400 + (tm_year-70)*31536000 + ((tm_year-69)/4)*86400 - ((tm_year-1)/100)*86400 + ((tm_year+299)/400)*86400 which is equivalent to what I said based on how leap seconds work. I've done a lot with POSIX time_t and what is and isn't stated in the standard. Linux may implement something that's not strictly POSIX compliant, but the root of the problem with time_t is POSIX defines it very particularly and when people deviate from the standard, or make their own up when the standard is silent, they do so is somewhat divergent ways (which is part of the problem as well as leap seconds not being defined in time_t as there's no way to represent them uniquely in a time_t, despite them existing in the non-uniform radix UTC notation). > > > If you squint at it, GPS time (gps weeks, time of week) is just > > > another notation very similar to time. And we know that works. > > > > > > > GPS time has no leap seconds. It's a monotonic count since 1980. > > Correct. > > > > Since future leap seconds are unknown, and unknowable, there is no > > > possible valid way to do this on the time scale of years. > > > > > > > Correct. This is the fundamental flaw of UTC. > > Or a flaw of the universe. For some reason the earth refuses to rotate > uniformly. > It's a fundamental flaw of UTC. There are other ways to implement something akin to UTC
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
On 1/15/19 9:14 PM, jimlux wrote: > Ground processing only - for user convenience I see somebody else mentioned SPICE, which I assumed you knew all about. >> >> Or more simply, how accurate are the satellite's GPS week and >> millisecond values? > > Time is set by the messages from the OEM-6xx series GPS receiver and > its 1pps - It's GPS time, not UTC. > > if there's a GPS outage, time propagates forward on the onboard > oscillator which isn't great. When GPS resumes, time jumps. Copacetic. Just like holdover specs on rackmount units. >>> We want to synchronize an event on the ground with an event on the >>> spacecraft, so, given that someone on the ground is going to do >>> something on 21 January 2019 at 12:34:56Z, we need to be able to >>> command the spacecraft to do that at the corresponding Week:Second. >>> >>> Right now, they use a website >>> >>> For example: https://www.labsat.co.uk/index.php/en/gps-time-calculator >>> >>> which, by the way, doesn't use leap seconds, so it's off by some 18 >>> seconds >>> >> Do they ignore the 18 seconds? What precision is required for this >> application? > > I think the folks at the labsat website just coded a naive > implementation. > > > For this application, <1 second uncertainty is needed (technically, > one can set events to occur to within 1 microsecond, but that's > counted down from the 1pps using the internal clock, with the > "correct" 1pps done by knowing it's week:second) Ok, so you set a trigger onboard and want to synchronize a state change at the ground station to some tolerance significantly under a second? If those are PC clocks their native imprecision will presumably leave a significant likelihood of ending up in the wrong second whatever the software does, but you should be able to minimize this relative to current practice. Do you have an example use case for what the operators might be doing that requires synchronization? How bad is it to be off by one or more seconds at one end or the other? Is this something like resetting both ends of a telemetry channel to different frequency or bit rate or some such? > And that's exactly how we can deal with it - it's perfectly reasonable > to say "folks, there's a leap second a'coming, run for the hills til > it gets here" Or even, "there might be a leap second, take a coffee break" since these can only happen twice a year. Or lockout such commands for the first and last UTC hour of each month if you want to be really sure without having to keep track of the file. This assumes your NTP installation will handle leap seconds correctly. We rely on Meinberg to get it right for us via both NTP and IRIG. (Same would apply to PTP.) > > For spacecraft flying via JPL ops, we work in TAI (or also, MET - > Mission Elapsed Time or SCLK - spacecraft clock), since all the nav is > done that way.. we also have to deal with relativistic effects and > light time - the whole "what do you do with time" on that scale has > seen 40 years of development. Right. The presumption is that JPL will get it right :-) Mostly we're interested in hearing about your cool systems. >> This conversion is so simple it seems unnecessary to rely on somebody >> else's library. What you might need is to arrange for a local, reliable, >> vetted copy of the leap seconds table. So perhaps folks here could >> comment on their best practices for retrieving same in an operational >> environment? > > or just set up an operational procedure that defines some file or > something under configuration control that has the "GPS time" and the > "Week:seconds" for the same time that is "after the last leap second". I think this is operationally equivalent. There is an operational advantage in leveraging the standard file. ROb ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
Yo Warner! On Tue, 15 Jan 2019 17:45:32 -0700 Warner Losh wrote: > > > Except that's not a leap second. struct tm doesn't do leap seconds > > > really. It kinda sorta does, but it kinda sorts doesn't. > > > > Well, it does in a modern Linux kernel. Otherwise NTPD would fail > > on the leap second, and that only happens every year so. :-) > > > > You are wrong. The kernel doesn't use struct tm. I never said the kernel uses struct tm. I said you can query the kernel for the time in a struct tm. > The kernel does this > by ticking either in TAI or by incrementing a TAI offset. ntpd uses a > different interface that exposes these things to it so it can > broadcast the leap second indicators correctly. Yup. > > > First, when you convert back and forth to time_t, you lose leap > > > second information. It's impossible to convert back and forth > > > from at least the UTC time zone. There's some 'right' time zones, > > > but they aren't strictly speaking POSIX compliant because it uses > > > the wrong math. Non-conforming, but damned useful and not wrong. > > > > Which is why you can't use time_t in UTC, but time_t in TAI > > can work, as well as struct tm, usually. Lot's of latent bugs... > > > > No. time_t is not TAI or GPS. It is UTC. Adjusted UTC. No, time_t is just a way to count seconds. See man difftime() for a use of time_t that is unrelated to any particular epoch. > It is the > number of SI seconds since 1970, minus all the positive leap seconds, > plus all the negative ones. Ciration please? I already cited the Linux doc that says otherwise. > > If you squint at it, GPS time (gps weeks, time of week) is just > > another notation very similar to time. And we know that works. > > > > GPS time has no leap seconds. It's a monotonic count since 1980. Correct. > > Since future leap seconds are unknown, and unknowable, there is no > > possible valid way to do this on the time scale of years. > > > > Correct. This is the fundamental flaw of UTC. Or a flaw of the universe. For some reason the earth refuses to rotate uniformly. RGDS GARY --- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 g...@rellim.com Tel:+1 541 382 8588 Veritas liberabit vos. -- Quid est veritas? "If you can’t measure it, you can’t improve it." - Lord Kelvin pgpO5SYiDWPC3.pgp Description: OpenPGP digital signature ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
JPL has a nice library for computing spacecraft location and pointing, SPICE , with interfaces in Fortran, C, IDL, Matlab, while others have created interfaces in other languages. Relevant here is the extensive well-tested time conversions . It's important to keep the leap second kernel (LSK) updated (as well as SPK, PCK kernel files). Although not mentioned explicitly in SPICE, GPS time lags TAI by 19s so conversion is straight forward. For IDL and IDL users (and some other languages), the Common Data Format (CDF) time conversion routines for CDF_TIME_TT2000 can be used for converting between TT and UTC, and GPS is an easy conversion. Again requires keeping leap second table updated. A Python library is at . ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
On Tue 2019-01-15T15:18:00-0800 jimlux hath writ: > It would be much easier for the operators if they could just *enter* the UTC > time, and have the software calculate the GPS Week:millisecond (or vice > versa) The 1970 CCIR decision to begin having leap seconds was made without any technical docuements about how to implement them. At a public meeting after the CCIR decision and prior to 1972 one of the delegates to the ongoing CCIR working group meetings was asked if he could talk about the details "unless that is sub judice". In the context that was a jab from someone who was not happy with the decision for leap seconds. But everyone was going to have to be using the new broadcast time scheme, so it was not funny then to be joking about technical details of the system being secret. That became even less funny over the decades. More disturbing were the contributions to the 1972 CCDS meeting where one of the technical delegates to the CCIR process wrote It is clear that the recommendation of the CCIR concering the UTC system is limited to "standard frequency and time-signal transmissions" in the sense of Study Group 7 of the CCIR. This recommendation does not concern other programs and other methods of broadcasting time, such as radio or television announcments of the hour, hourly services for maritime or air navigation, public clocks, etc. and Again, UTC is a way of providing AT and UT in the same broadcast. The CCIR feels responsible for transmitting AT and UT to users in a reliable fashion and with appropriate technical means. It does not matter to the CCIR that users employ AT, UT, or UTC for their particular problems. The responsibility of the CCIR ends with the transmission of UTC by the specified stations. Another delegate both to the CCIR and that CCDS meeting wrote We should limit ourselves to discussing this issue with people who are well aware of the problems involved. Those are basically the folks who made the CCIR decision saying Many folks don't have to use this, especially us. and Blame the victim if they don't use the right thing. and Let's not talk about this. So the result was that the technical folks did not talk about leap seconds. Nobody was ever tasked with setting up a scheme for transmitting information about leap seconds that was more automated and reliable than the BIH giving letters to the post office to send to various national time service agencies. But before the technical folks washed their hands they arranged for various international assemblies to produce statements approving of the leap seconds. Subsequently the bureaucratic folks took those statements to other national and international bodies and achieved approval for UTC with leap seconds in arenas where the original technical folks had admitted they might not be the best strategy. At this point in history it is unclear who of these technical folks believed that the CCIR decision had limited scope and who were being shrewdly tacit about their ongoing goals for international approval. The only internationally official clue that UTC might not be the best tool was CCIR recommendation 485 https://www.itu.int/rec/R-REC-TF.485-2-199006-W/en which allowed that TAI might be preferable to UTC for some purposes. This rec was withdrawn in 1997. That was just before the "leap seconds must die" process began in earnest in 1999. Ironically at the 14th CCTF meeting in 1999 the head of the BIPM opined that anyone who did not like leap seconds should use TAI as given in rec 485. This betrays that not even the head of BIPM, who define TAI, was aware that the ITU had withdrawn that rec two years earlier. As Captain said in Cool Hand Luke Whut we've got heyah is failya tuh c'municate. https://www.youtube.com/watch?v=V2f-MZ2HRHQ -- Steve Allen WGS-84 (GPS) UCO/Lick Observatory--ISB 260 Natural Sciences II, Room 165 Lat +36.99855 1156 High Street Voice: +1 831 459 3046 Lng -122.06015 Santa Cruz, CA 95064 https://www.ucolick.org/~sla/ Hgt +250 m ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
On 1/15/19 6:34 PM, Rob Seaman wrote: Hi Jim, The use case is this - I have a satellite in orbit which does everything internally in terms of GPS week and millisecond of week. Folks on the ground work in UTC, since that's what the plethora of PCs are using. Just to verify. The satellite does use GPS as its reference clock, not some manual upload (in a similar way as you describe) from ground stations? The routines you're talking about aren't (directly) affecting the state of the clock on the spacecraft? Ground processing only - for user convenience Or more simply, how accurate are the satellite's GPS week and millisecond values? Time is set by the messages from the OEM-6xx series GPS receiver and its 1pps - It's GPS time, not UTC. if there's a GPS outage, time propagates forward on the onboard oscillator which isn't great. When GPS resumes, time jumps. We want to synchronize an event on the ground with an event on the spacecraft, so, given that someone on the ground is going to do something on 21 January 2019 at 12:34:56Z, we need to be able to command the spacecraft to do that at the corresponding Week:Second. Right now, they use a website For example: https://www.labsat.co.uk/index.php/en/gps-time-calculator which, by the way, doesn't use leap seconds, so it's off by some 18 seconds Do they ignore the 18 seconds? What precision is required for this application? I think the folks at the labsat website just coded a naive implementation. For this application, <1 second uncertainty is needed (technically, one can set events to occur to within 1 microsecond, but that's counted down from the 1pps using the internal clock, with the "correct" 1pps done by knowing it's week:second) Sometimes cross checked for consistency by, say, the page at http://leapsecond.com/java/gpsclock.htm It would be much easier for the operators if they could just *enter* the UTC time, and have the software calculate the GPS Week:millisecond (or vice versa) Leap Seconds, should they occur, would be handled by "don't do anything around or across the leap second". Or could you simply forbid activities during a few hours of June 30 / December 31? If the operators are at JPL this would be late afternoon on whatever day of the week. That is, just assume there's always a leap second at each opportunity. As it happens the ops aren't at JPL for this spacecraft. And that's exactly how we can deal with it - it's perfectly reasonable to say "folks, there's a leap second a'coming, run for the hills til it gets here" For spacecraft flying via JPL ops, we work in TAI (or also, MET - Mission Elapsed Time or SCLK - spacecraft clock), since all the nav is done that way.. we also have to deal with relativistic effects and light time - the whole "what do you do with time" on that scale has seen 40 years of development. And, we could rebaseline the conversion i.e. the calculation would be SecondsAfterBaseDate = Seconds(UserEnteredDate) - Seconds(BaseDate) DesiredTime = Seconds(6 Jan 1980) + SecondsAfterBaseDate GPSWeek = Floor(DesiredTime/(86400*7)) GPSMilliseconds = (DesiredTime - GPSWeek*86400*7) * 1000 Which works quite nicely as long as the interval between BaseDate and UserEnteredDate does not include a leap second. This conversion is so simple it seems unnecessary to rely on somebody else's library. What you might need is to arrange for a local, reliable, vetted copy of the leap seconds table. So perhaps folks here could comment on their best practices for retrieving same in an operational environment? or just set up an operational procedure that defines some file or something under configuration control that has the "GPS time" and the "Week:seconds" for the same time that is "after the last leap second". Note that if your use cases include retroactive requirements then even if leap seconds cease you will need to maintain the leap seconds table (perhaps compiled into the source through some date). (For leap second warriors, redefining UTC makes it more complicated, not less, when all use cases are considered.) Not really - for those kinds of use cases there's already issues and solutions with reconciling local observations of clocks against outside time scales. the "don't operate during the leap" and "keep track" work just fine. ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] Running on TAI
Hi Hal, Again, it doesn't appear that the current discussion has anything to do with astronomical use cases. >> The trick is to find a source that will set a POSIX system to TAI, and then >> to avoid the gotchas that happen when such a system interacts with other >> POSIX systems. > What do the systems that point telescopes use for maintaining their system > clock? Do they use NTP or a special program that listens to GPS or a local > Hydrogen maser or ...? Radio telescopes (or LIGO, for instance) may have extremely high precision requirements (and thus fancy clocks), at least as much for frequency as time. Most other telescopes need accurate time more than high precision time. Telescopes tend to be in remote locations (mountains, Antarctica, in orbit, etc) and generally rely on GNSS clocks. These vary in quality according to the observatory budget. NTP is used across whatever mountaintop network is deployed, but may be poorly behaved across microwave links to remote campus pools, for instance. > Some GPSDOs have the option to return GPS time rather than UTC. Our Meinberg ref clocks can deliver GPS, UTC, or TAI (and thus other derived timescales). > GPS is a fixed offset from TAI. So it's easy to run your system on TAI. All > it takes is a little sysadmin hacking, no programming. Yes, but there may be significant operational details, e.g., we have red clocks for UTC in our control rooms and white clocks for local time, but might want green for TAI or so forth. There are a lot of potential timekeeping requirements depending on the science and logistics. > You won't easily get redundancy from checking with other NTP servers. If you > have several friends running similar systems, you could set up a private NTP > network. We operate four telescopes currently on two mountaintops. Three GNSS clocks (one campus spare) plus a couple of IRIG-driven linux kernel implemented stratum-0 ref clocks per telescope provide plenty of NTP stratum-1 references. Don't know about a private NTP network, the campus stratum-2 pool clocks generally don't get given the time of day, so to speak, but we want something remote for fail-over. (Would have to be a pretty intrusive local network event to get there.) > With some work it would be reasonable to setup a ntp server that took in UTC > but gave out TAI. That would be similar to the way leap smearing works. We > could use a separate port number to minimize confuzion. Also, NIST has a UT1 reference, but I for one haven't had luck connecting. > As far as I know, the kernel doesn't know anything about what type of time it > is maintaining. It just sets the clock as directed and goes tick, tick, > tick. > The trouble is with the time conversion routines. They all assume a time_t > is UTC. So if you ran your system on TAI, all your log files would be off by > 37 seconds. "Off" from what? If the system is set to GPS, TAI, UT1 or whatever, the log files arguably should be as well. > How hard would it be to insert a wrapper in front of the time conversion > routines? The idea is to rename all the current time conversion routines > from > foo to foo_utc and implement foo_tai that concerts between UTC and TAI. It > would need a list of leap seconds. The default implementation of foo would > call foo_utc but an environment variable or such would switch to using > foo_tai. Or use Steve's TZ-based solution. Rob ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
Hi Jim, > The use case is this - I have a satellite in orbit which does > everything internally in terms of GPS week and millisecond of week. > Folks on the ground work in UTC, since that's what the plethora of PCs > are using. > Just to verify. The satellite does use GPS as its reference clock, not some manual upload (in a similar way as you describe) from ground stations? The routines you're talking about aren't (directly) affecting the state of the clock on the spacecraft? Or more simply, how accurate are the satellite's GPS week and millisecond values? > We want to synchronize an event on the ground with an event on the > spacecraft, so, given that someone on the ground is going to do > something on 21 January 2019 at 12:34:56Z, we need to be able to > command the spacecraft to do that at the corresponding Week:Second. > > Right now, they use a website > > For example: https://www.labsat.co.uk/index.php/en/gps-time-calculator > > which, by the way, doesn't use leap seconds, so it's off by some 18 > seconds > Do they ignore the 18 seconds? What precision is required for this application? > Sometimes cross checked for consistency by, say, the page at > > http://leapsecond.com/java/gpsclock.htm > > It would be much easier for the operators if they could just *enter* > the UTC time, and have the software calculate the GPS Week:millisecond > (or vice versa) > > Leap Seconds, should they occur, would be handled by "don't do > anything around or across the leap second". > Or could you simply forbid activities during a few hours of June 30 / December 31? If the operators are at JPL this would be late afternoon on whatever day of the week. That is, just assume there's always a leap second at each opportunity. > And, we could rebaseline the conversion > > i.e. the calculation would be > > SecondsAfterBaseDate = Seconds(UserEnteredDate) - Seconds(BaseDate) > > DesiredTime = Seconds(6 Jan 1980) + SecondsAfterBaseDate > > GPSWeek = Floor(DesiredTime/(86400*7)) > > GPSMilliseconds = (DesiredTime - GPSWeek*86400*7) * 1000 > > Which works quite nicely as long as the interval between BaseDate and > UserEnteredDate does not include a leap second. > This conversion is so simple it seems unnecessary to rely on somebody else's library. What you might need is to arrange for a local, reliable, vetted copy of the leap seconds table. So perhaps folks here could comment on their best practices for retrieving same in an operational environment? Note that if your use cases include retroactive requirements then even if leap seconds cease you will need to maintain the leap seconds table (perhaps compiled into the source through some date). (For leap second warriors, redefining UTC makes it more complicated, not less, when all use cases are considered.) On the other hand, if your "plethora of PCs" keep good time (via NTP or SNTP), and you don't care to better than a second (which your current 18s offset suggests might be the case), and there are no after-the-fact use cases, do you need anything other than a procedural rule about avoiding such activities at the end of June and December? That is, put your effort into making your NTP installation as reliable as possible. Rob Seaman, University of Arizona ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
On Tue, Jan 15, 2019 at 1:50 PM Gary E. Miller wrote: > Yo Warner! > > On Tue, 15 Jan 2019 12:43:46 -0700 > Warner Losh wrote: > > > Except that's not a leap second. struct tm doesn't do leap seconds > > really. It kinda sorta does, but it kinda sorts doesn't. > > Well, it does in a modern Linux kernel. Otherwise NTPD would fail > on the leap second, and that only happens every year so. :-) > You are wrong. The kernel doesn't use struct tm. The kernel does this by ticking either in TAI or by incrementing a TAI offset. ntpd uses a different interface that exposes these things to it so it can broadcast the leap second indicators correctly. > > First, when you convert back and forth to time_t, you lose leap second > > information. It's impossible to convert back and forth from at least > > the UTC time zone. There's some 'right' time zones, but they aren't > > strictly speaking POSIX compliant because it uses the wrong math. > > Non-conforming, but damned useful and not wrong. > > Which is why you can't use time_t in UTC, but time_t in TAI > can work, as well as struct tm, usually. Lot's of latent bugs... > No. time_t is not TAI or GPS. It is UTC. Adjusted UTC. It is the number of SI seconds since 1970, minus all the positive leap seconds, plus all the negative ones. Now, you can use that type bogusly to hold any count from any epoch with any rules you like, but such use is not POSIX conforming. Many people abuse time_t to store TAI, it's true, but that's not its definition, and you'll wind up confusing things because of the 40 second offset between the two. > If you squint at it, GPS time (gps weeks, time of week) is just another > notation very similar to time. And we know that works. > GPS time has no leap seconds. It's a monotonic count since 1980. > > Second, many of the tm_ fields can create a time that's N 's > > in the future by adding N to that structure and converting. So tm_sec > > == 60 could be a leap second, or it could be the second after tm_sec > > == 59. > > Since future leap seconds are unknown, and unknowable, there is no > possible valid way to do this on the time scale of years. > Correct. This is the fundamental flaw of UTC. > > > But gpsd and ntpd try real hard to be correct. Sadly, with leap > > > smear, there is no consensus on what is 'correct'. > > > > > > > Both leap seconds and leap second smear must die. Others have > > different opinions, some strongly held. Not all the contradictory > > opinions are necessarily wrong. > > Good luck with that. > Yea. Having done timing systems for a decade now a decade ago, I always felt that was the best solution. The BIH guys broke time by trying to fix it. :( Warner ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] Running on TAI
On 1/15/19 4:19 PM, Hal Murray wrote: The trick is to find a source that will set a POSIX system to TAI, and then to avoid the gotchas that happen when such a system interacts with other POSIX systems. What do the systems that point telescopes use for maintaining their system clock? Do they use NTP or a special program that listens to GPS or a local Hydrogen maser or ...? Some GPSDOs have the option to return GPS time rather than UTC. GPS is a fixed offset from TAI. So it's easy to run your system on TAI. All it takes is a little sysadmin hacking, no programming. You won't easily get redundancy from checking with other NTP servers. If you have several friends running similar systems, you could set up a private NTP network. With some work it would be reasonable to setup a ntp server that took in UTC but gave out TAI. That would be similar to the way leap smearing works. We could use a separate port number to minimize confuzion. As far as I know, the kernel doesn't know anything about what type of time it is maintaining. It just sets the clock as directed and goes tick, tick, tick. The trouble is with the time conversion routines. They all assume a time_t is UTC. So if you ran your system on TAI, all your log files would be off by 37 seconds. How hard would it be to insert a wrapper in front of the time conversion routines? The idea is to rename all the current time conversion routines from foo to foo_utc and implement foo_tai that concerts between UTC and TAI. It would need a list of leap seconds. The default implementation of foo would call foo_utc but an environment variable or such would switch to using foo_tai. Some modules (Python skyfield package from Rhodes Mill) explicitly deal with this. The time object has a method for each time scale.. These return utc in various formats time.utc_jpl() time.utc_iso() time.utc_datetime() These return other timescales time.tai time.tt time.J time.ut1 I confess I have used them blindly without checking, but I can do that quickly. ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
[LEAPSECS] Running on TAI
> The trick is to find a source that will set a POSIX system to TAI, and then > to avoid the gotchas that happen when such a system interacts with other > POSIX systems. What do the systems that point telescopes use for maintaining their system clock? Do they use NTP or a special program that listens to GPS or a local Hydrogen maser or ...? Some GPSDOs have the option to return GPS time rather than UTC. GPS is a fixed offset from TAI. So it's easy to run your system on TAI. All it takes is a little sysadmin hacking, no programming. You won't easily get redundancy from checking with other NTP servers. If you have several friends running similar systems, you could set up a private NTP network. With some work it would be reasonable to setup a ntp server that took in UTC but gave out TAI. That would be similar to the way leap smearing works. We could use a separate port number to minimize confuzion. As far as I know, the kernel doesn't know anything about what type of time it is maintaining. It just sets the clock as directed and goes tick, tick, tick. The trouble is with the time conversion routines. They all assume a time_t is UTC. So if you ran your system on TAI, all your log files would be off by 37 seconds. How hard would it be to insert a wrapper in front of the time conversion routines? The idea is to rename all the current time conversion routines from foo to foo_utc and implement foo_tai that concerts between UTC and TAI. It would need a list of leap seconds. The default implementation of foo would call foo_utc but an environment variable or such would switch to using foo_tai. -- These are my opinions. I hate spam. ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
Yo jimlux! On Tue, 15 Jan 2019 15:18:00 -0800 jimlux wrote: > It would be much easier for the operators if they could just *enter* > the UTC time, and have the software calculate the GPS > Week:millisecond (or vice versa) I've been wanting that CLI tool for a while. I'll add it to the gpsd TODO list. gpsd already has gpsd_gpstime_resolve() to go from GPS weeks and tow to UTC, the reverse would be easy. Then a command line tool that used that would allow for regression testing. RGDS GARY --- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 g...@rellim.com Tel:+1 541 382 8588 Veritas liberabit vos. -- Quid est veritas? "If you can’t measure it, you can’t improve it." - Lord Kelvin pgpaJzNJL5dc2.pgp Description: OpenPGP digital signature ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
On 1/15/19 11:41 AM, Rob Seaman wrote: Isn't this covered (for instance) in section 20.3.3.5.2.4(b) of IS-GPS-200J (https://www.gps.gov/technical/icwg/IS-GPS-200J.pdf)? The various libraries and operating systems listed deal with none of the reference time scales (GPS, UTC, or TAI, for that matter) completely and correctly. There's no obvious connection here with astronomical use cases. We rely on our reference clocks' vendor to have implemented IS-GPS-200J and relevant standards correctly. These devices operate off single board linux cards, but I presume they wrote the critical code themselves to handle leap seconds, but also other precision timekeeping issues not entertained by POSIX and other mass market software. Our clocks handle time scales other than UTC, but since our customer (NASA PDCO) specifies UTC, that's what we deliver. The bread-and-butter leap second discussions on this list seem irrelevant to simply meeting an engineering requirement to adhere to whatever standard. If POSIX, python, excel, whatever don't provide canned tools to convert seamlessly between GPS and UTC, or to handle format conversions from week:ms to sexagesimal, such tools must be implemented outside those languages/libraries. There are a vast number of other science-related data engineering details that POSIX (for example) also does not implement. Indeed - but before leaping[sic] into coding it up, I thought I'd ask around You also asked.. > /I'll probably test it for the cases I'm interested in (Ruby, Python, > Excel, Matlab, Octave), but if someone else has already done it, then > I've got something to cross check against./ I would add MySQL/MariaDB and PostgreSQL, as well as TCL and C/C++ to that list. If JPL were to support libraries implementing some useful subset of timekeeping utilities under a smorgasbord of such languages, members of this list would undoubtedly be delighted to cross-check the results and put them to good use (via GitHub or what have you). --- I don't know that JPL is interested in a generalized library collection. It would be useful.. a sort of "cross platform" collection.. At JPL we tend to just solve the problem at hand, and release just that software. --- That said, it isn't clear what the original use case involves. Something about spacecraft using GPS week numbers and ground systems using UTC? But what do you need to do exactly? And is this onboard or on the ground? --- The use case is this - I have a satellite in orbit which does everything internally in terms of GPS week and millisecond of week. Folks on the ground work in UTC, since that's what the plethora of PCs are using. We want to synchronize an event on the ground with an event on the spacecraft, so, given that someone on the ground is going to do something on 21 January 2019 at 12:34:56Z, we need to be able to command the spacecraft to do that at the corresponding Week:Second. Right now, they use a website For example: https://www.labsat.co.uk/index.php/en/gps-time-calculator which, by the way, doesn't use leap seconds, so it's off by some 18 seconds Sometimes cross checked for consistency by, say, the page at http://leapsecond.com/java/gpsclock.htm It would be much easier for the operators if they could just *enter* the UTC time, and have the software calculate the GPS Week:millisecond (or vice versa) Leap Seconds, should they occur, would be handled by "don't do anything around or across the leap second". And, we could rebaseline the conversion i.e. the calculation would be SecondsAfterBaseDate = Seconds(UserEnteredDate) - Seconds(BaseDate) DesiredTime = Seconds(6 Jan 1980) + SecondsAfterBaseDate GPSWeek = Floor(DesiredTime/(86400*7)) GPSMilliseconds = (DesiredTime - GPSWeek*86400*7) * 1000 Which works quite nicely as long as the interval between BaseDate and UserEnteredDate does not include a leap second. ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
On Tue 2019-01-15T10:50:10-0800 Tom Van Baak hath writ: > Nope. All minutes have 60 seconds in Excel. And in Windows. And > in Unix/POSIX. Really any computer system that uses a fixed "epoch" > and a ticking "counter" is ruined by leap seconds. The systems differ > in their epoch's, they all differ in their counters, they can all be > set to UTC, except they all fail when there's a positive or negative > leap second. Nowhere did the folks who were there at the inception of leap seconds say so explicitly, but from reading how things worked over the history of the time services and what the folks paid to provide those services wrote, I am reasonably sure that they also believed that all minutes consisted of 60 seconds, and all days consisted of 86400 seconds. I believe that the concept in their heads was that occasionally there are two calendar days which are not adjacent by one SI second. Or, if the earth rotation were to speed up enough, there might be two calendar days which overlap by one SI second. The notation 23:59:60 is merely a tag for conveniently indicating when a second was inserted between two calendar days. The clock is disconnected from the calendar. Understandably this notion should trigger objections that it is technically barren. At this point I would answer "Yes, and they knew that." They were being paid by their governments to provide the legal and operational time of their nations. Their job had always been to tell the people of their country how to set their clocks. So when a senator or cabinet official asked "Can you tell me what time it is?" they had learned over the course of their careers that the answer was not to wander off into a long technical discussion about how sausage was made, but to smile and say "Yes." Similarly for the 1970 CCIR decision about leap seconds: when facing a chamber full of folks at an international assembly that had the power to dictate how every member nation should set their clocks in the same way that was consistent with the laws of all those nations, the answer was to smile and say "Yes, everyone has agreed that this is the best way to do it." Their job was not to say that the technical folks who made that decision had already decided to disregard that in the systems that they controlled, broadcast navigational signals based on TAI, and make almanacs based on UT (not UTC). -- Steve Allen WGS-84 (GPS) UCO/Lick Observatory--ISB 260 Natural Sciences II, Room 165 Lat +36.99855 1156 High Street Voice: +1 831 459 3046 Lng -122.06015 Santa Cruz, CA 95064 https://www.ucolick.org/~sla/ Hgt +250 m ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
On Tue 2019-01-15T13:10:54-0800 Gary E. Miller hath writ: > > What is the epoch that was used for TAI? > "TAI in this form was synchronised with Universal Time at the beginning > of 1958," That is conceptually the case, but even as a concept it deserves explanation about how it was that the USNO A.1 atomic time scale which was started with that same epoch differed from the BIH atomic time scale by 0.035 seconds. Furthermore, the epoch of A.1 was 1958-01-01T00:00:00 UT2 whereas the epoch of the BIH atomic time scale was 1958-01-01T20:00:00 UT2, but this does not explain the difference between the two time scales. That 1958 epoch is not the basis for the current incarnation of TAI. Tonight I will scan the pages of Bulletin Horaire with the answer. -- Steve Allen WGS-84 (GPS) UCO/Lick Observatory--ISB 260 Natural Sciences II, Room 165 Lat +36.99855 1156 High Street Voice: +1 831 459 3046 Lng -122.06015 Santa Cruz, CA 95064 https://www.ucolick.org/~sla/ Hgt +250 m ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
Yo Steve! On Tue, 15 Jan 2019 12:52:43 -0800 Steve Allen wrote: > On Tue 2019-01-15T12:34:11-0800 Gary E. Miller hath writ: > > Yes, and no. time_t is just seconds since an epoch. Which epoch > > is not well defined. The epoch may well be anything. See "man > > difftime". > > That evokes a challenge for all time nuts that I can make based on > reading Bulletin Horaire. > > What is the epoch that was used for TAI? According to wikipedia: https://en.wikipedia.org/wiki/International_Atomic_Time "As of 31 December 2016, when another leap second was added,[3] TAI is exactly 37 seconds ahead of UTC." > I expect that many can give an answer, and that nobody can give the > correct answer. "TAI in this form was synchronised with Universal Time at the beginning of 1958," > > The best bet is to ask the kernel for the current TAI time, and work > > with that. Use the leap seconds file to convert TAI to UTC in your > > code. Or just use TAI for everything. > > The trick is to find a source that will set a POSIX system to TAI, Easy: NTPD. > and > then to avoid the gotchas that happen when such a system interacts > with other POSIX systems. So don't let it, except by NTPD. RGDS GARY --- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 g...@rellim.com Tel:+1 541 382 8588 Veritas liberabit vos. -- Quid est veritas? "If you can’t measure it, you can’t improve it." - Lord Kelvin pgpzDR6oI7G_A.pgp Description: OpenPGP digital signature ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
On Tue 2019-01-15T12:34:11-0800 Gary E. Miller hath writ: > Yes, and no. time_t is just seconds since an epoch. Which epoch > is not well defined. The epoch may well be anything. See "man difftime". That evokes a challenge for all time nuts that I can make based on reading Bulletin Horaire. What is the epoch that was used for TAI? I expect that many can give an answer, and that nobody can give the correct answer. > The best bet is to ask the kernel for the current TAI time, and work > with that. Use the leap seconds file to convert TAI to UTC in your > code. Or just use TAI for everything. The trick is to find a source that will set a POSIX system to TAI, and then to avoid the gotchas that happen when such a system interacts with other POSIX systems. -- Steve Allen WGS-84 (GPS) UCO/Lick Observatory--ISB 260 Natural Sciences II, Room 165 Lat +36.99855 1156 High Street Voice: +1 831 459 3046 Lng -122.06015 Santa Cruz, CA 95064 https://www.ucolick.org/~sla/ Hgt +250 m ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
Yo Warner! On Tue, 15 Jan 2019 12:43:46 -0700 Warner Losh wrote: > Except that's not a leap second. struct tm doesn't do leap seconds > really. It kinda sorta does, but it kinda sorts doesn't. Well, it does in a modern Linux kernel. Otherwise NTPD would fail on the leap second, and that only happens every year so. :-) > First, when you convert back and forth to time_t, you lose leap second > information. It's impossible to convert back and forth from at least > the UTC time zone. There's some 'right' time zones, but they aren't > strictly speaking POSIX compliant because it uses the wrong math. > Non-conforming, but damned useful and not wrong. Which is why you can't use time_t in UTC, but time_t in TAI can work, as well as struct tm, usually. Lot's of latent bugs... If you squint at it, GPS time (gps weeks, time of week) is just another notation very similar to time. And we know that works. > Second, many of the tm_ fields can create a time that's N 's > in the future by adding N to that structure and converting. So tm_sec > == 60 could be a leap second, or it could be the second after tm_sec > == 59. Since future leap seconds are unknown, and unknowable, there is no possible valid way to do this on the time scale of years. > > But gpsd and ntpd try real hard to be correct. Sadly, with leap > > smear, there is no consensus on what is 'correct'. > > > > Both leap seconds and leap second smear must die. Others have > different opinions, some strongly held. Not all the contradictory > opinions are necessarily wrong. Good luck with that. RGDS GARY --- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 g...@rellim.com Tel:+1 541 382 8588 Veritas liberabit vos. -- Quid est veritas? "If you can’t measure it, you can’t improve it." - Lord Kelvin pgpTX8DP2zyyK.pgp Description: OpenPGP digital signature ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
On Tue 2019-01-15T11:49:00-0800 Tom Van Baak hath writ: > Still, I bet more astronomers use Python than Excel to point > telescopes. How do you handle the lack of leap second support in > Python? I can't say that anyone uses Python to point a telescope, but it does not matter if they do. See preprint 678 from the 2011 Future of UTC meeting proceedings http://hanksville.org/futureofutc/2011/preprints/index.html In short: 19th century telescopes point by moving them manually. 20th century telescopes are built like battleships and raw pointing is only a bit more accurate than steering a battleship. 21st century telescopes are robotic, but they still have flexure and slop that requires them to update their pointing models regularly, and those slop parameters easily soak up one second. The APF telescope at Lick is robotic. Its pointing system produces a time mismatch alarm whenever there is a leap second and requires a reboot. At the most recent summer leap second we were able to see a star before and after, and there was a jump in the centering. The software chose a different offset in the pointing for that night. -- Steve Allen WGS-84 (GPS) UCO/Lick Observatory--ISB 260 Natural Sciences II, Room 165 Lat +36.99855 1156 High Street Voice: +1 831 459 3046 Lng -122.06015 Santa Cruz, CA 95064 https://www.ucolick.org/~sla/ Hgt +250 m ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
Yo Steve! On Tue, 15 Jan 2019 11:46:02 -0800 Steve Allen wrote: > I would love to find pointers to such strategies and code. In the gpsd code look in the file: gpsd/timebase.c In the NTPsec code look in the files: ntpd/ntp_leapsec.c and ntptime/ntptime.c RGDS GARY --- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 g...@rellim.com Tel:+1 541 382 8588 Veritas liberabit vos. -- Quid est veritas? "If you can’t measure it, you can’t improve it." - Lord Kelvin pgpi1hHH2P7J1.pgp Description: OpenPGP digital signature ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
Yo Tom! On Tue, 15 Jan 2019 11:40:07 -0800 "Tom Van Baak" wrote: > But the underlying time_t cannot represent leap seconds, can it? Yes, and no. time_t is just seconds since an epoch. Which epoch is not well defined. The epoch may well be anything. See "man difftime". time_t dates to the first edition of "Unix Programmer's Manual, November 3, 1971". No timezone or other basis is given for time_t. They just said "system time". The first leap second was in 1972! time_t never caught up. https://www.bell-labs.com/usr/dmr/www/1stEdman.html In 1973, just after the third edition, time_t was pegged to GMT. According to wikipedia, GMT is mean solar time, not UTC. The seconds are not the same length, and no leap seconds: https://en.wikipedia.org/wiki/Greenwich_Mean_Time Some long time after that the doc changed from GMT to UTC, but the code did not. time() returns a time_t which pretends to be UTC, but ignores leap seconds. From a current "man time": "This value is not the same as the actual number of seconds between the time and the Epoch, because of leap seconds and because system clocks are not required to be synchronized to a standard reference." So if you need "real" UTC, POSIX is no help. And nothing is a help for future dates since the future leapseconds are unknown. The only safe way to compute times (to the second) in the past it to do it yourself using the leapseconds file. > So how does that gmtime hackery work? NTPD, or PTP, or something, tells your kernel what the 'real' current UTC is, including the current leap seconds. gmtime() just returns that. NTPD can get the current and past leap second info from the leapseconds file, from another NTPD, or from a GPS. > Also, presumably Python is based on POSIX? Does it use time_t or > gmtime? Why did Jim's quick Python experiment fail? Out of my area of understanding. But I'll guess that like almost every other time keeping system it is just ignoring leap seconds. Double counting the leap second itself as two 59's in a row. Pretending to be UTC while actually being an approximation of GMT. And, yes, during the leap second, all sorts of implementation dependent, unknown, and unknowable, stuff is happening. The best bet is to ask the kernel for the current TAI time, and work with that. Use the leap seconds file to convert TAI to UTC in your code. Or just use TAI for everything. To add more weirdness, Google, and its NTPD servers, don't use UTC because they use leap smearing. And the upcoming GPS week rollover... RGDS GARY --- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 g...@rellim.com Tel:+1 541 382 8588 Veritas liberabit vos. -- Quid est veritas? "If you can’t measure it, you can’t improve it." - Lord Kelvin pgpGNEZTAztr4.pgp Description: OpenPGP digital signature ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
On Tue 2019-01-15T12:52:18-0700 Warner Losh hath writ: > This suggests strongly that those models of time are fatally flawed and > should be revisited. I am probably the only person who has ever read all of Bulletin Horaire. https://www.ucolick.org/~sla/leapsecs/bhissues.html I can say that with certainty for the copies in the Lick library because I had to apply xacto knife to separate some of the pages which were not cut at the time of printing and never opened before I did. Over the lifetime of the BIH the staff were prone to be chatty and include transcriptions of the meetings which many times resulted in directions that the BIH must do more work in a different fashion. Those transcriptions point to the dramatis personae at those and other meetings, and to other documents. I can say unequivocally that the people who instituted leap seconds in radio broadcast time signals knew that the idea was fatally flawed. They also knew that they did not have the legal authority *not* to include leap seconds in the radio broadcast time signals, and they were not happy about the situation. They intentionally did not use UTC in their ongoing technical work. They intentionally avoided discussing the technical problems in public, redacted the transcripts of meetings that showed dissension, and arranged forced votes to produce statements indicating that various bodies agreed with the notion of leap seconds. -- Steve Allen WGS-84 (GPS) UCO/Lick Observatory--ISB 260 Natural Sciences II, Room 165 Lat +36.99855 1156 High Street Voice: +1 831 459 3046 Lng -122.06015 Santa Cruz, CA 95064 https://www.ucolick.org/~sla/ Hgt +250 m ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
On Tue, Jan 15, 2019 at 12:41 PM Rob Seaman wrote: > Isn't this covered (for instance) in section 20.3.3.5.2.4(b) of > IS-GPS-200J (https://www.gps.gov/technical/icwg/IS-GPS-200J.pdf)? The > various libraries and operating systems listed deal with none of the > reference time scales (GPS, UTC, or TAI, for that matter) completely and > correctly. > > This suggests strongly that those models of time are fatally flawed and should be revisited. > There's no obvious connection here with astronomical use cases. We rely on > our reference clocks' vendor to have implemented IS-GPS-200J and relevant > standards correctly. These devices operate off single board linux cards, > but I presume they wrote the critical code themselves to handle leap > seconds, but also other precision timekeeping issues not entertained by > POSIX and other mass market software. Our clocks handle time scales other > than UTC, but since our customer (NASA PDCO) specifies UTC, that's what we > deliver. > Spinning rocks are hard, eh? the non-uniform nature of UTC is a big reason it's hard to implement correctly in software (especially when there's code size or speed constraints). It's impossible to implement without tables that must constantly be updated. While this is natural and acceptable in many environments, it presents operational difficulties for others. The ITU standard that tries to keep the spinning rock and atomic time in sync is flawed in that it doesn't adequately accommodate all situations. POSIX has similar sorts of flaws, but the details differ. > The bread-and-butter leap second discussions on this list seem irrelevant > to simply meeting an engineering requirement to adhere to whatever > standard. If POSIX, python, excel, whatever don't provide canned tools to > convert seamlessly between GPS and UTC, or to handle format conversions > from week:ms to sexagesimal, such tools must be implemented outside those > languages/libraries. There are a vast number of other science-related data > engineering details that POSIX (for example) also does not implement. > Yes. Despite the efforts of the POSIX committee to make it easy to implement time, those efforts have resulted in a fatally flawed standard that leads to the "1 second errors are fine, don't bother me" attitude that pervades many adjacent APIs to the POSIX APIs. It was the number one frustration that I had when I had to deal with vendors that didn't get the leap seconds pedantically correct and didn't give a @!##@ that they didn't because there was no financial penalty for not doing so. It was maddening. Warner > Rob Seaman, Lunar and Planetary Laboratory > > -- > On 1/15/19 11:50 AM, Tom Van Baak wrote: > > Jim -- I'm replying via the LEAPSECS list because we love leap second > questions here. > > List -- Jim is a long-time member of time-nuts, works at JPL, and does > wonderful stuff. > > From Jim Lux: > > I'm working with a variety of things which work in UTC or GPS > week/millisecond, so we're doing a lot of conversion back and forth. > (the spacecraft puts out time in week:millisecond, all the ground > systems processing is in UTC) > > The question comes up when converting back and forth, and whether > various libraries handle leap seconds correctly. > For now, I can use a hack of not computing back to 6 Jan 1980, but use > an epoch like 15 Dec 2018 (week 2031: 518,400.000 seconds) and hope > there's no leap second in the offing. > > Yes, that's one way. But that doesn't sound safe or correct to me. > > > For instance, in Python, if I do a datetime(2016,12,31,0,0,0) + > timedelta(hours=30) does it come out as 1/1/2017 6:00:00 or 5:59:59 (it > comes out 0600) > > Similarly, does Excel's time formatting allow for some minutes having an > extra second, or does it just assume all minutes are 60 seconds. > > Nope. All minutes have 60 seconds in Excel. And in Windows. And in > Unix/POSIX. Really any computer system that uses a fixed "epoch" and a > ticking "counter" is ruined by leap seconds. The systems differ in their > epoch's, they all differ in their counters, they can all be set to UTC, > except they all fail when there's a positive or negative leap second. > > > I'll probably test it for the cases I'm interested in (Ruby, Python, > Excel, Matlab, Octave), but if someone else has already done it, then > I've got something to cross check against. > > This is a good question for LEAPSECS. I suspect those packages are well-known > here. > > > (python does NOT know about leap seconds) > > import datetime > > d = datetime.datetime(2016,12,31) > > dt = datetime.timedelta(hours=30) > > d > Out[4]: datetime.datetime(2016, 12, 31, 0, 0) > > dt > Out[5]: datetime.timedelta(1, 21600) > > d+dt > Out[6]: datetime.datetime(2017, 1, 1, 6, 0) > > Right, this is typical for almost any software developed anywhere. Leap > seconds are such a weird exception, almost no commercial software, web > frontend or backend, or mobile phones, or mobile apps deal with them >
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
On Tue, Jan 15, 2019 at 11:58 AM Gary E. Miller wrote:
> Yo Tom!
>
> On Tue, 15 Jan 2019 10:50:10 -0800
> "Tom Van Baak" wrote:
>
> > Nope. All minutes have 60 seconds in Excel. And in Windows. And in
> > Unix/POSIX.
>
> Not quite. Check out "man gmtime" for one way that POSIX represents time.
>
Yes and no. time_t doesn't have leap seconds. At all. Ever. So from that
perspective, POSIX can't do leap seconds. time_t is so engrained in the
other interfaces of POSIX, they can't have leap seconds either.
> Specifically:
>
>struct tm {
> [...]
>int tm_sec;/* Seconds (0-60) */
>
> gmtime() is perfectly capable of reporting 61 seconds in a minute.
>
> Failure to account for this causes mnay problems.
>
> What it does not do well is accounting for leap seconds that are in
> the past or future.
>
Except that's not a leap second. struct tm doesn't do leap seconds really.
It kinda sorta does, but it kinda sorts doesn't.
First, when you convert back and forth to time_t, you lose leap second
information. It's impossible to convert back and forth from at least the
UTC time zone. There's some 'right' time zones, but they aren't strictly
speaking POSIX compliant because it uses the wrong math. Non-conforming,
but damned useful and not wrong.
Second, many of the tm_ fields can create a time that's N 's in the
future by adding N to that structure and converting. So tm_sec == 60 could
be a leap second, or it could be the second after tm_sec == 59.
> > Right, this is typical for almost any software developed anywhere.
> > Leap seconds are such a weird exception, almost no commercial
> > software, web frontend or backend, or mobile phones, or mobile apps
> > deal with them correctly.
>
> But gpsd and ntpd try real hard to be correct. Sadly, with leap smear,
> there is no consensus on what is 'correct'.
>
Both leap seconds and leap second smear must die. Others have different
opinions, some strongly held. Not all the contradictory opinions are
necessarily wrong.
Warner
___
LEAPSECS mailing list
LEAPSECS@leapsecond.com
https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
> If you set your POSIX system clock at TAI minus 10 seconds then > it keeps track of the number of seconds which have been counted > in the internationally approved radio broadcast time scales. So, wait. In order to use Python for any proper UTC processing you have to configure your PC to run on TAI? That sounds, sort of inconvenient if you're trying to do anything with precise time. It's the same as Excel. Still, I bet more astronomers use Python than Excel to point telescopes. How do you handle the lack of leap second support in Python? /tvb ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
On Tue 2019-01-15T12:41:27-0700 Rob Seaman hath writ: > The bread-and-butter leap second discussions on this list seem > irrelevant to simply meeting an engineering requirement to adhere to > whatever standard. If POSIX, python, excel, whatever don't provide > canned tools to convert seamlessly between GPS and UTC, or to handle > format conversions from week:ms to sexagesimal, such tools must be > implemented outside those languages/libraries. There are a vast number > of other science-related data engineering details that POSIX (for > example) also does not implement. IEEE 1588 version 2 (PTP) makes it clear that the intended time scale on the underlying hardware is TAI counted from some epoch. It must be the case that vendors selling such hardware have chosen solution strategies and written code to handle converting between civil time and system time. I would love to find pointers to such strategies and code. -- Steve Allen WGS-84 (GPS) UCO/Lick Observatory--ISB 260 Natural Sciences II, Room 165 Lat +36.99855 1156 High Street Voice: +1 831 459 3046 Lng -122.06015 Santa Cruz, CA 95064 https://www.ucolick.org/~sla/ Hgt +250 m ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
On Tue 2019-01-15T11:40:07-0800 Tom Van Baak hath writ: > Also, presumably Python is based on POSIX? Does it use time_t or > gmtime? Why did Jim's quick Python experiment fail? Guido said that the mainline of Python would not support leap seconds. -- Steve Allen WGS-84 (GPS) UCO/Lick Observatory--ISB 260 Natural Sciences II, Room 165 Lat +36.99855 1156 High Street Voice: +1 831 459 3046 Lng -122.06015 Santa Cruz, CA 95064 https://www.ucolick.org/~sla/ Hgt +250 m ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
Isn't this covered (for instance) in section 20.3.3.5.2.4(b) of IS-GPS-200J (https://www.gps.gov/technical/icwg/IS-GPS-200J.pdf)? The various libraries and operating systems listed deal with none of the reference time scales (GPS, UTC, or TAI, for that matter) completely and correctly. There's no obvious connection here with astronomical use cases. We rely on our reference clocks' vendor to have implemented IS-GPS-200J and relevant standards correctly. These devices operate off single board linux cards, but I presume they wrote the critical code themselves to handle leap seconds, but also other precision timekeeping issues not entertained by POSIX and other mass market software. Our clocks handle time scales other than UTC, but since our customer (NASA PDCO) specifies UTC, that's what we deliver. The bread-and-butter leap second discussions on this list seem irrelevant to simply meeting an engineering requirement to adhere to whatever standard. If POSIX, python, excel, whatever don't provide canned tools to convert seamlessly between GPS and UTC, or to handle format conversions from week:ms to sexagesimal, such tools must be implemented outside those languages/libraries. There are a vast number of other science-related data engineering details that POSIX (for example) also does not implement. Rob Seaman, Lunar and Planetary Laboratory -- On 1/15/19 11:50 AM, Tom Van Baak wrote: > Jim -- I'm replying via the LEAPSECS list because we love leap second > questions here. > > List -- Jim is a long-time member of time-nuts, works at JPL, and does > wonderful stuff. > > From Jim Lux: >> I'm working with a variety of things which work in UTC or GPS >> week/millisecond, so we're doing a lot of conversion back and forth. >> (the spacecraft puts out time in week:millisecond, all the ground >> systems processing is in UTC) >> >> The question comes up when converting back and forth, and whether >> various libraries handle leap seconds correctly. >> For now, I can use a hack of not computing back to 6 Jan 1980, but use >> an epoch like 15 Dec 2018 (week 2031: 518,400.000 seconds) and hope >> there's no leap second in the offing. > Yes, that's one way. But that doesn't sound safe or correct to me. > >> For instance, in Python, if I do a datetime(2016,12,31,0,0,0) + >> timedelta(hours=30) does it come out as 1/1/2017 6:00:00 or 5:59:59 (it >> comes out 0600) >> >> Similarly, does Excel's time formatting allow for some minutes having an >> extra second, or does it just assume all minutes are 60 seconds. > Nope. All minutes have 60 seconds in Excel. And in Windows. And in > Unix/POSIX. Really any computer system that uses a fixed "epoch" and a > ticking "counter" is ruined by leap seconds. The systems differ in their > epoch's, they all differ in their counters, they can all be set to UTC, > except they all fail when there's a positive or negative leap second. > >> I'll probably test it for the cases I'm interested in (Ruby, Python, >> Excel, Matlab, Octave), but if someone else has already done it, then >> I've got something to cross check against. > This is a good question for LEAPSECS. I suspect those packages are well-known > here. > >> (python does NOT know about leap seconds) >> >> import datetime >> >> d = datetime.datetime(2016,12,31) >> >> dt = datetime.timedelta(hours=30) >> >> d >> Out[4]: datetime.datetime(2016, 12, 31, 0, 0) >> >> dt >> Out[5]: datetime.timedelta(1, 21600) >> >> d+dt >> Out[6]: datetime.datetime(2017, 1, 1, 6, 0) > Right, this is typical for almost any software developed anywhere. Leap > seconds are such a weird exception, almost no commercial software, web > frontend or backend, or mobile phones, or mobile apps deal with them > correctly. > > I'm hoping a number of LEAPSECS members can chime in. There are astronomers > here who must be used to solving issues like this, in python or other tools, > so their input would be valuable. > > /tvb > > ___ > LEAPSECS mailing list > LEAPSECS@leapsecond.com > https://pairlist6.pair.net/mailman/listinfo/leapsecs ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
> Yo Tom!
>
> On Tue, 15 Jan 2019 10:50:10 -0800
> "Tom Van Baak" wrote:
>
> > Nope. All minutes have 60 seconds in Excel. And in Windows. And in
> > Unix/POSIX.
>
> Not quite. Check out "man gmtime" for one way that POSIX represents time.
>
> Specifically:
>
>struct tm {
> [...]
>int tm_sec;/* Seconds (0-60) */
>
> gmtime() is perfectly capable of reporting 61 seconds in a minute.
Hi Gary,
Nice to hear from you.
But the underlying time_t cannot represent leap seconds, can it? So how does
that gmtime hackery work?
Also, presumably Python is based on POSIX? Does it use time_t or gmtime? Why
did Jim's quick Python experiment fail?
/tvb
___
LEAPSECS mailing list
LEAPSECS@leapsecond.com
https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
On Tue 2019-01-15T10:50:10-0800 Tom Van Baak hath writ: > Jim -- I'm replying via the LEAPSECS list because we love leap second > questions here. > > List -- Jim is a long-time member of time-nuts, works at JPL, and does > wonderful stuff. If you set your POSIX system clock at TAI minus 10 seconds then it keeps track of the number of seconds which have been counted in the internationally approved radio broadcast time scales. If you set your clock that way then you can use the "right" zones of the IANA tz distribution and get conversions between the internal system time and the civil time that has been specified in the international agreements. https://www.ucolick.org/~sla/leapsecs/right+gps.html Note that there are caveats to this scheme. -- Steve Allen WGS-84 (GPS) UCO/Lick Observatory--ISB 260 Natural Sciences II, Room 165 Lat +36.99855 1156 High Street Voice: +1 831 459 3046 Lng -122.06015 Santa Cruz, CA 95064 https://www.ucolick.org/~sla/ Hgt +250 m ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
Yo Tom!
On Tue, 15 Jan 2019 10:50:10 -0800
"Tom Van Baak" wrote:
> Nope. All minutes have 60 seconds in Excel. And in Windows. And in
> Unix/POSIX.
Not quite. Check out "man gmtime" for one way that POSIX represents time.
Specifically:
struct tm {
[...]
int tm_sec;/* Seconds (0-60) */
gmtime() is perfectly capable of reporting 61 seconds in a minute.
Failure to account for this causes mnay problems.
What it does not do well is accounting for leap seconds that are in
the past or future.
> Right, this is typical for almost any software developed anywhere.
> Leap seconds are such a weird exception, almost no commercial
> software, web frontend or backend, or mobile phones, or mobile apps
> deal with them correctly.
But gpsd and ntpd try real hard to be correct. Sadly, with leap smear,
there is no consensus on what is 'correct'.
RGDS
GARY
---
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
g...@rellim.com Tel:+1 541 382 8588
Veritas liberabit vos. -- Quid est veritas?
"If you can’t measure it, you can’t improve it." - Lord Kelvin
pgpdXnu3Zyhrw.pgp
Description: OpenPGP digital signature
___
LEAPSECS mailing list
LEAPSECS@leapsecond.com
https://pairlist6.pair.net/mailman/listinfo/leapsecs
Re: [LEAPSECS] leapseconds, converting between GPS time (week, second) and UTC
Jim -- I'm replying via the LEAPSECS list because we love leap second questions here. List -- Jim is a long-time member of time-nuts, works at JPL, and does wonderful stuff. >From Jim Lux: > I'm working with a variety of things which work in UTC or GPS > week/millisecond, so we're doing a lot of conversion back and forth. > (the spacecraft puts out time in week:millisecond, all the ground > systems processing is in UTC) > > The question comes up when converting back and forth, and whether > various libraries handle leap seconds correctly. > For now, I can use a hack of not computing back to 6 Jan 1980, but use > an epoch like 15 Dec 2018 (week 2031: 518,400.000 seconds) and hope > there's no leap second in the offing. Yes, that's one way. But that doesn't sound safe or correct to me. > For instance, in Python, if I do a datetime(2016,12,31,0,0,0) + > timedelta(hours=30) does it come out as 1/1/2017 6:00:00 or 5:59:59 (it > comes out 0600) > > Similarly, does Excel's time formatting allow for some minutes having an > extra second, or does it just assume all minutes are 60 seconds. Nope. All minutes have 60 seconds in Excel. And in Windows. And in Unix/POSIX. Really any computer system that uses a fixed "epoch" and a ticking "counter" is ruined by leap seconds. The systems differ in their epoch's, they all differ in their counters, they can all be set to UTC, except they all fail when there's a positive or negative leap second. > I'll probably test it for the cases I'm interested in (Ruby, Python, > Excel, Matlab, Octave), but if someone else has already done it, then > I've got something to cross check against. This is a good question for LEAPSECS. I suspect those packages are well-known here. > (python does NOT know about leap seconds) > > import datetime > > d = datetime.datetime(2016,12,31) > > dt = datetime.timedelta(hours=30) > > d > Out[4]: datetime.datetime(2016, 12, 31, 0, 0) > > dt > Out[5]: datetime.timedelta(1, 21600) > > d+dt > Out[6]: datetime.datetime(2017, 1, 1, 6, 0) Right, this is typical for almost any software developed anywhere. Leap seconds are such a weird exception, almost no commercial software, web frontend or backend, or mobile phones, or mobile apps deal with them correctly. I'm hoping a number of LEAPSECS members can chime in. There are astronomers here who must be used to solving issues like this, in python or other tools, so their input would be valuable. /tvb ___ LEAPSECS mailing list LEAPSECS@leapsecond.com https://pairlist6.pair.net/mailman/listinfo/leapsecs
