“Europe's Galileo and China's BeiDou use 12- and 13-bit week numbers, so we have quite a bit of time before they reset to zero: February 2078 for Galileo and January 2163 for BeiDou.”
China always takes the long view. From: AF <af-boun...@af.afmug.com> On Behalf Of Forrest Christian (List Account) Sent: Thursday, January 2, 2020 7:58 PM To: AnimalFarm Microwave Users Group <af@af.afmug.com> Subject: Re: [AFMUG] mass GPS issues All of the affected modules should be able to be flashed. Whether or not the facility is in place to do so depends a lot on the design choices made. I purposefully have ensured that there is some way to update firmware in the field for all of these modules, with varying levels of pain. I'm going to make a educated guess and say that if you have a rackinjector in place, there is a good chance you won't have much pain in the event this firmware needs to be updated. The worst case scenario will require a cable being physically plugged into the GPS receiver - I know this is specifically the case with the SyncBox junior Aux ports since there is no other way to get a serial console to the GPS module due to the AUX port pinning. How much of this is a firmware update vs a configuration change in the modules, I can't say for sure at this point. I think we're heading into the speculation category further than I want to do until we have a better idea what is really involved here. The good news is that things are progressing. On Thu, Jan 2, 2020 at 6:31 PM Ken Hohhof <af...@kwisp.com <mailto:af...@kwisp.com> > wrote: The crucial question seems to be, do we mark our calendars for 4 years from now, midnight Moscow time, New Years Eve 2023? So we can power cycle a bunch of GPS receivers? Or is there some kind of retroactive fix that can be applied? Or do we have 4 years to physically replace the devices? If so, with what? I guess it’s not the end of the world to replace a bunch of Syncboxes on towers if we have 4 years to do it, as long as we have a replacement device available. I’d like to think we could log into a Sync/Power/RackInjector and squirt new firmware into the attached GPS receiver, but I doubt it. Same with a Cambium UGPS. Not sure about ePMP. From: AF <af-boun...@af.afmug.com <mailto:af-boun...@af.afmug.com> > On Behalf Of Forrest Christian (List Account) Sent: Thursday, January 2, 2020 7:12 PM To: AnimalFarm Microwave Users Group <af@af.afmug.com <mailto:af@af.afmug.com> > Subject: Re: [AFMUG] mass GPS issues Yes, and it happens a lot apparently. I.E. it looked weird, but it wasn't. On Thu, Jan 2, 2020 at 6:05 PM Mike Hammett <af...@ics-il.net <mailto:af...@ics-il.net> > wrote: Was the weird clock update from GPS PRN 29 sent about 20 minutes before midnight Moscow time just coincidence? ----- Mike Hammett <http://www.ics-il.com/> Intelligent Computing Solutions <https://www.facebook.com/ICSIL> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> <https://www.linkedin.com/company/intelligent-computing-solutions> <https://twitter.com/ICSIL> <http://www.midwest-ix.com/> Midwest Internet Exchange <https://www.facebook.com/mdwestix> <https://www.linkedin.com/company/midwest-internet-exchange> <https://twitter.com/mdwestix> <http://www.thebrotherswisp.com/> The Brothers WISP <https://www.facebook.com/thebrotherswisp> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> _____ From: "Forrest Christian (List Account)" <li...@packetflux.com <mailto:li...@packetflux.com> > To: "AnimalFarm Microwave Users Group" <af@af.afmug.com <mailto:af@af.afmug.com> > Sent: Thursday, January 2, 2020 6:53:14 PM Subject: Re: [AFMUG] mass GPS issues For us, we have only been shipping GPS+GLONASS since late in 2017 or 2018, so this is the first GLONASS rollover for us. Remember this is still speculation here, but I am starting to feel like this most likely will turn out to be the underlying cause. All of us with the issue seem to be running AXN5.1.1 which was built at the end of 2017. This might be specific to that firmware version/build. There are lots of GPS modules out there of varying quality, but few which are really suitable for 1PPS synchronization. Believe me, I've evaluated enough of them to know. I even evaluated a GPS module earlier this year which was specifically targeted to timing applications (including survey-in and single-satellite operation) and discovered that it was far worse timing-wise than the ones we're using. The mediatek based modules seem to be a good mix of low cost and really good 1PPS performance. Pretty much any others are multiples of the price. This is probably why we've all ended up using the same modules. But if you're looking for a cheap GPS receiver and don't care about 1PPS you might end up with something else. And probably a different firmware build. TOTAL SPECULATION: One suspicion I have is that the behavior of the device might be related to which constellation it is preferring, because if GLONASS is saying 2016 and GPS is saying 2020 (according to your broken firmware), then you might find a module which thinks it's 2016 based on GLONASS but is getting these confusing signals from GPS which says it's 2019/2020. An alternate module that is convinced it's 2019/2020 might not be happy about GLONASS trying to say it's 2016. The behavior would probably have to be different in both cases and possibly would alternate between the behaviors if one was not preferred for some RF reason. On Thu, Jan 2, 2020 at 5:14 PM Ken Hohhof <af...@kwisp.com <mailto:af...@kwisp.com> > wrote: I’m not sure it’s true that only Cambium products were affected. It appears to be more a certain 3rd party GPS+GLONASS receiver module that Packetflux uses in their latest production of Syncboxes and that Cambium apparently uses in their latest UGPS production. Are Packetflux sync products used to time anything but Cambium? Probably not. Certainly some other manufacturers must OEM that particular module, you’d think some newer auto GPS nav products or autosteer tractors or ICBMs or whatever must have gone bonkers. Even if they only use location data and not time sync or a 1pps signal, it seemed like the GPS sats just disappeared. Older Cambium and Packetflux products as well as Last Mile Gear CTM-1 and CTM-2 seemed unaffected, I assume because they used different, older modules inside. It’s interesting that Nate reports his ePMP APs tracked fewer satellites but not down to zero, and recovered without a power cycle. Different GPS+GLONASS module? Different firmware in the module? Inquiring minds want to know. From: AF <af-boun...@af.afmug.com <mailto:af-boun...@af.afmug.com> > On Behalf Of Matt Hoppes Sent: Thursday, January 2, 2020 5:42 PM To: AnimalFarm Microwave Users Group <af@af.afmug.com <mailto:af@af.afmug.com> > Subject: Re: [AFMUG] mass GPS issues So why didn’t this happen four years ago? And why were only Cambium type products affected? On Jan 2, 2020, at 6:13 PM, Forrest Christian (List Account) <li...@packetflux.com <mailto:li...@packetflux.com> > wrote: The answer to your question is complicated. The short answer is that not in the way you stated it, as there isn't good enough clock hardware in the rackinjector to do this holdover, and the current electrical architecture isn't set up to permit the generation of a clock internally. We're looking at some options though so you might see something like this in the future. We're also looking at doing a hardware revision to the control board to permit a high quality holdover oscillator to be added. This work was underway well before this event, but hasn't progressed to the point that there are really any details as to what form this might take and when it might happen. On Thu, Jan 2, 2020 at 4:00 PM Eric Muehleisen <ericm...@gmail.com <mailto:ericm...@gmail.com> > wrote: A great feature would be for the RackInjector to generate it's own hold-over sync if it were to loose GPS. Any possibility of including something like that in the next firmware update? On Tue, Dec 31, 2019 at 8:45 PM Forrest Christian (List Account) <li...@packetflux.com <mailto:li...@packetflux.com> > wrote: One note is that on later RackInjectors, rebooting the rackinjector not only won't reboot attached devices, it also won't reboot the GPS receiver. This is to permit reboots of the control interface without affecting connected device at all. This includes *most* firmware upgrades. So for the later boards, you'll need to reset it on the GPS status page. On Tue, Dec 31, 2019 at 6:29 PM Andreas Wiatowski <andr...@silo.ca <mailto:andr...@silo.ca> > wrote: Thanks Forrest, let us know what you find. I can confirm that I have done the power cycle on the Rack injectors....we have seen the same issue occur again...feels like interference. It has been suggested by some to switch to Autosync+Freerun to avoid loss of client use...I guess it really depends on your networks reuse scheme. <image861500.jpg> Internet. Phone. TV. Andreas Wiatowski CEO/Founder Silo <tel:1-866-727-4138,%20ext%20600> 1-866-727-4138, ext 600 | <mailto:andr...@silo.ca> andr...@silo.ca <http://www.silo.ca/> silo.ca <image461146.png> <image041427.png> > On Dec 31, 2019, at 8:05 PM, Forrest Christian (List Account) > <li...@packetflux.com <mailto:li...@packetflux.com> > wrote: > > [EXTERNAL] > Just an update/random informational message, a lot of which is pretty obvious > from this thread: > > 1) As is pretty obvious at this point, something happened either with the GPS > constellation or with the firmware which runs in the GPS module, or some > combination of those. What many of you might not be aware of is that many > vendors use the same GPS modules from the same vendor. Generally a certain > portion of the uGPS, PacketFlux, ePMP all will use the same module, along > with other vendors. Each of the vendors (including PacketFlux) has changed > the exact model of the module over the years, for instance the GPS+glonass > modules are in newer devices, and GPS only are in older ones. I'm working > through determining whether this is just the GPS+GLONASS modules which are > affected (it seems like it might be the case), or if it's a few different > types. Note that because this seems like it might be confined to a certain > type (or types) of module, that it might turn out to be only certain > date/model ranges of each type of gear which are affected. > > 2) Restart of the module seems to clear the issue. For PacketFlux gear, > anyone with a SiteMonitor or RackInjector should be able to do this remotely. > In the SiteMonitor, the row on the binary tab is labeled something like > 'SyncPipe power'. There might be multiple rows depending on how many > SyncInjector/PowerInjectors you have - this corresponds to the power port on > each injector, so you'll have to turn off the row which is for the unit which > the GPS receiver is attached to. In the RackInjector, it's a bit simpler > since it's just a button on the GPS page. > > 3) I'm currently working through trying to find anyone who really knows what > went on here. None of the usual notification locations seem to have any data. > Probably too early to tell. I might luck out and be able to get a GPS > constellation recording I can replay here to replicate. > > One thing which might be helpful is for those of you who log this stuff, if > you can provide a pretty close UTC time that this occurred, along with the > approximate LAT/LONG, I might be able to correlate this with a specific event. > > - Forrest > -- > AF mailing list > AF@af.afmug.com <mailto:AF@af.afmug.com> > http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com <mailto:AF@af.afmug.com> http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- - Forrest -- AF mailing list AF@af.afmug.com <mailto:AF@af.afmug.com> http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com <mailto:AF@af.afmug.com> http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- - Forrest -- AF mailing list AF@af.afmug.com <mailto:AF@af.afmug.com> http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com <mailto:AF@af.afmug.com> http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- - Forrest -- AF mailing list AF@af.afmug.com <mailto:AF@af.afmug.com> http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com <mailto:AF@af.afmug.com> http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- - Forrest -- AF mailing list AF@af.afmug.com <mailto:AF@af.afmug.com> http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- - Forrest
-- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com
