Re: [AFMUG] mass GPS issues

Thu, 02 Jan 2020 18:09:42 -0800

Maybe most likely we have 4 years to forget about it.


bp
<part15sbs{at}gmail{dot}com>
On 1/2/2020 5:30 PM, Ken Hohhof wrote:

The crucial question seems to be, do we mark our calendars for 4 years from now, midnight Moscow time, New Years Eve 2023?  So we can power cycle a bunch of GPS receivers?

 

Or is there some kind of retroactive fix that can be applied?

 

Or do we have 4 years to physically replace the devices?  If so, with what?  I guess it’s not the end of the world to replace a bunch of Syncboxes on towers if we have 4 years to do it, as long as we have a replacement device available.

 

I’d like to think we could log into a Sync/Power/RackInjector and squirt new firmware into the attached GPS receiver, but I doubt it.  Same with a Cambium UGPS.  Not sure about ePMP.

 

From: AF <af-boun...@af.afmug.com> On Behalf Of Forrest Christian (List Account)
Sent: Thursday, January 2, 2020 7:12 PM
To: AnimalFarm Microwave Users Group <af@af.afmug.com>
Subject: Re: [AFMUG] mass GPS issues

 

Yes, and it happens a lot apparently.   I.E. it looked weird, but it wasn't.

 

 

 

On Thu, Jan 2, 2020 at 6:05 PM Mike Hammett <af...@ics-il.net> wrote:

Was the weird clock update from GPS PRN 29 sent about 20 minutes before midnight Moscow time just coincidence?

From: "Forrest Christian (List Account)" <li...@packetflux.com>
To: "AnimalFarm Microwave Users Group" <af@af.afmug.com>
Sent: Thursday, January 2, 2020 6:53:14 PM
Subject: Re: [AFMUG] mass GPS issues

For us, we have only been shipping GPS+GLONASS since late in 2017 or 2018, so this is the first GLONASS rollover for us.    Remember this is still speculation here, but I am starting to feel like this most likely will turn out to be the underlying cause.

 

All of us with the issue seem to be running AXN5.1.1 which was built at the end of 2017.  This might be specific to that firmware version/build.

 

There are lots of GPS modules out there of varying quality, but few which are really suitable for 1PPS synchronization.   Believe me, I've evaluated enough of them to know.  I even evaluated a GPS module earlier this year which was specifically targeted to timing applications (including survey-in and single-satellite operation) and discovered that it was far worse timing-wise than the ones we're using.  The mediatek based modules seem to be a good mix of low cost and really good 1PPS performance.   Pretty much any others are multiples of the price.  This is probably why we've all ended up using the same modules.   

 

But if you're looking for a cheap GPS receiver and don't care about 1PPS you might end up with something else.    And probably a different firmware build.

 

TOTAL SPECULATION:  One suspicion I have is that the behavior of the device might be related to which constellation it is preferring, because if GLONASS is saying 2016 and GPS is saying 2020 (according to your broken firmware), then you might find a module which thinks it's 2016 based on GLONASS  but is getting these confusing signals from GPS which says it's 2019/2020.   An alternate module that is convinced it's 2019/2020 might not be happy about GLONASS trying to say it's 2016.   The behavior would probably have to be different in both cases and possibly would alternate between the behaviors if one was not preferred for some RF reason.

 

On Thu, Jan 2, 2020 at 5:14 PM Ken Hohhof <af...@kwisp.com> wrote:

I’m not sure it’s true that only Cambium products were affected.  It appears to be more a certain 3rd party GPS+GLONASS receiver module that Packetflux uses in their latest production of Syncboxes and that Cambium apparently uses in their latest UGPS production.  Are Packetflux sync products used to time anything but Cambium?  Probably not.  Certainly some other manufacturers must OEM that particular module, you’d think some newer auto GPS nav products or autosteer tractors or ICBMs or whatever must have gone bonkers.  Even if they only use location data and not time sync or a 1pps signal, it seemed like the GPS sats just disappeared.

 

Older Cambium and Packetflux products as well as Last Mile Gear CTM-1 and CTM-2 seemed unaffected, I assume because they used different, older modules inside.  It’s interesting that Nate reports his ePMP APs tracked fewer satellites but not down to zero, and recovered without a power cycle.  Different GPS+GLONASS module?  Different firmware in the module?  Inquiring minds want to know.

 

 

From: AF <af-boun...@af.afmug.com> On Behalf Of Matt Hoppes
Sent: Thursday, January 2, 2020 5:42 PM
To: AnimalFarm Microwave Users Group <af@af.afmug.com>
Subject: Re: [AFMUG] mass GPS issues

 

So why didn’t this happen four years ago?  And why were only Cambium type products affected?


On Jan 2, 2020, at 6:13 PM, Forrest Christian (List Account) <li...@packetflux.com> wrote:

The answer to your question is complicated.   The short answer is that not in the way you stated it, as there isn't good enough clock hardware in the rackinjector to do this holdover, and the current electrical architecture isn't set up to permit the generation of a clock internally.   We're looking at some options though so you might see something like this in the future.    We're also looking at doing a hardware revision to the control board to permit a high quality holdover oscillator to be added.    This work was underway well before this event, but hasn't progressed to the point that there are really any details as to what form this might take and when it might happen.

 

On Thu, Jan 2, 2020 at 4:00 PM Eric Muehleisen <ericm...@gmail.com> wrote:

A great feature would be for the RackInjector to generate it's own hold-over sync if it were to loose GPS. Any possibility of including something like that in the next firmware update?

 

On Tue, Dec 31, 2019 at 8:45 PM Forrest Christian (List Account) <li...@packetflux.com> wrote:

One note is that on later RackInjectors, rebooting the rackinjector not only won't reboot attached devices, it also won't reboot the GPS receiver.   This is to permit reboots of the control interface without affecting connected device at all.   This includes *most* firmware upgrades.

 

So for the later boards, you'll need to reset it on the GPS status page.

 

 

 

On Tue, Dec 31, 2019 at 6:29 PM Andreas Wiatowski <andr...@silo.ca> wrote:

Thanks Forrest, let us know what you find. I can confirm that I have done the power cycle on the Rack injectors....we have seen the same issue occur again...feels like interference.

It has been suggested by some to switch to Autosync+Freerun to avoid loss of client use...I guess it really depends on your networks reuse scheme.

<image861500.jpg>

Internet.​
​Phone.
​TV.

Andreas Wiatowski

CEO/Founder

Silo

1-866-727-4138, ext 600

 | 

andr...@silo.ca

silo.ca

  

<image461146.png>

 

 

<image041427.png>

> On Dec 31, 2019, at 8:05 PM, Forrest Christian (List Account) <li...@packetflux.com> wrote:
>
>  [EXTERNAL]
> Just an update/random informational message, a lot of which is pretty obvious from this thread:
>
> 1) As is pretty obvious at this point, something happened either with the GPS constellation or with the firmware which runs in the GPS module, or some combination of those. What many of you might not be aware of is that many vendors use the same GPS modules from the same vendor. Generally a certain portion of the uGPS, PacketFlux, ePMP all will use the same module, along with other vendors. Each of the vendors (including PacketFlux) has changed the exact model of the module over the years, for instance the GPS+glonass modules are in newer devices, and GPS only are in older ones. I'm working through determining whether this is just the GPS+GLONASS modules which are affected (it seems like it might be the case), or if it's a few different types. Note that because this seems like it might be confined to a certain type (or types) of module, that it might turn out to be only certain date/model ranges of each type of gear which are affected.
>
> 2) Restart of the module seems to clear the issue. For PacketFlux gear, anyone with a SiteMonitor or RackInjector should be able to do this remotely. In the SiteMonitor, the row on the binary tab is labeled something like 'SyncPipe power'. There might be multiple rows depending on how many SyncInjector/PowerInjectors you have - this corresponds to the power port on each injector, so you'll have to turn off the row which is for the unit which the GPS receiver is attached to. In the RackInjector, it's a bit simpler since it's just a button on the GPS page.
>
> 3) I'm currently working through trying to find anyone who really knows what went on here. None of the usual notification locations seem to have any data. Probably too early to tell. I might luck out and be able to get a GPS constellation recording I can replay here to replicate.
>
> One thing which might be helpful is for those of you who log this stuff, if you can provide a pretty close UTC time that this occurred, along with the approximate LAT/LONG, I might be able to correlate this with a specific event.
>
> - Forrest
> --
> AF mailing list
> AF@af.afmug.com
> http://af.afmug.com/mailman/listinfo/af_af.afmug.com

--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com


 

--

- Forrest

--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com

--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com


 

--

- Forrest

--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com


 

--

- Forrest


--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com

 

--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com


 

--

- Forrest




-- 
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com

Reply via email to