I wanted to reply to an email or two here, but didn't know which one - This seems as good of an email to reply to as any, since Mark makes exactly the correct points.
Let me show with visuals what the problem actually is.... Without any additional introduction, let's start with this picture: (tek00011.png). [image: Inline image 2] This is the sync pulse, as put out by a Geniuine CMM Micro, connected to 100M of cable, and with a Cambium 450 AP on the end of the cable. The yellow trace shows the voltage over time - it starts at 23.1V, then drops to nothing for around 140 microseconds, then turns back on. (each light vertical grid line is 20uS) The actual timing comes off of the rising edge of the pulse - that is, where it turns back on. In this case, it's nice and sharp. Somewhere between 0 and 24V there's a threshold which is actually the point where the radio sees the transition. In essence it translates through electronics, not software, the pulse from the analog voltages you see to a nice set of 0's and 1's, where 0 would be below the threshold, and 1 would be above the threshold. So, the radio is waiting to see this go from 0 to 1, or below the threshold to above the threshold. Where the real world comes into play is the 'bouncing' of the signal on the left side of the waveform. When someone talks about ringing, this is one of the types of waveforms you expect to see. If this was a short cable, the signal would just drop to zero, but since it's a longer cable, the cable 'resists' the voltage being turned off and creates the bouncing you see as the energy dissipates. There's a lot of interactions which can cause this including capacitace and inductance of the cable, the impedance of the radio, the speed at which the signal is turned off, and so on. But the main point to be made is that it isn't the CMM Micro putting out a bad signal - instead it's the cable reacting to a correct signal. Now here's where the problem occurs. Let's assume that threshold on the radio happens to be where the darker horizontal line is. Now what does the radio see? It sees a whole bunch of 1's while the power is on, then it sees a very brief bunch of zeros while the signal drops initially, then a brief bunch of ones while the signal goes above the threshold, then a bunch of zeros for the ~140us while the signal is below the threshold, and then back to one for a long period until the next pulse. If the radio is simply watching for a transition from zero to one to start the timing, in this case, the radio is going to start transmitting 140us early. If the ringing doesn't occur (for instance on a shorter cable run), it will instead start transmitting at the correct time. There is a very easy way to get around this issue, and that is to simply wait to see 40uS worth of zeros in a row before accepting a pulse as the 'start' pulse. This would cause the ringing to be completely ignored. It also solves issues where there's ringing on the turn-on as well. Please remember that the image above of this issue is shown with all cambium gear - it's a CMM Micro, and a 450. To give you a brief idea of the struggles we had with the 450i, let me share a few other pictures: [image: Inline image 4] This picture (tek00008) is a Revision H syncinjector into a short cable and a 450i. Note no ringing on the turn-off. We purposefully turn off the signal slowly to help avoid inductive spikes and similar. Compare the following (tek00009): [image: Inline image 5] This is the exact same setup, a revsion H syncinjector into a 450i, EXCEPT I added 100m of cable. Note the ringing. I happen to know that the threshold for the 450i is about where the arrow is on the right hand side of the screen. In this case the 450i refused to transmit at all because of the bouncy nature of the signal. Again, the 'fix' I described would solve this issue, just wait for the signal to settle down before trusting it. By comparison a CMM4 signal looks like this (tek0010): [image: Inline image 7] (CMM4 into 100m of cable, 450i). This signal transitions slowly, which is both it's benefit and bane. The benefit is no ringing. The disadvantage is that the slow turn-on edge makes the timing of the signal very dependent on the threshold. Imagine 3 different radios, with three different thresholds, either because of manufacturing differences or because they're different models. There's nearly 40uS there between fully off and fully on. Where exactly on that curve is 'turn on'. Someone at cambium probably knows, I have a pretty good idea (but it varys from radio to radio), but because of what I know, I wonder how the CMM4 provides consistent sync at all. The reality is - it probably is or was 'close enough'. One final picture (tek00020): [image: Inline image 9] This is the waveform out of a Revision I0 syncinjector. We rounded off the start of the drop off to prevent the ringing (the sharp edge of the drop was effectively caused it). We also tried to maintain the fast rise time on turn-on through all of the thresholds of interest to minimize the difference from radio designs. This seems to have solved the 450i issue while maintaining compatibility with earlier radios. Hopefully this explains a bit of what is going on here, and the challenges. Personally, my feeling is that cambium could invest a bit of software work and get this fixed, permanently. I do know that they're working on various related sync issues. I'm going to reply specifically to a couple of items in a couple of emails, but I wanted to get this out there to start. On Fri, Jan 22, 2016 at 11:16 AM, Mark Radabaugh <[email protected]> wrote: > I do see the SM’s drop session when the unit goes from sync source to a > different source - there is definitely a shift between the various > sources. There are telnet commands in the newer software to disable and > enable various sync sources without rebooting the unit - very helpful for > getting a AP back where you want it without a reboot, at the risk of > causing a reregistration of some or all of the SM’s. > > If I had to speculate on the actual issue it would be a combination of a > couple of things. > > a) Cambium has never (to my knowledge) published a detailed specification > of the sync over power requirements. As a result the 3rd party vendors > have had to reverse engineer the design. > b) Cambium was not consistent in the timing from their own equipment - > internal, CMM3, CMM4, iGPS > c) The POE cabling and voltage has changed a number of times (this had to > happen - nothing that can be done about that, standards move on) > d) The ’sync detection’ circuit in the AP’s has changed a number of > times. Again this had to happen, components and designs change. It’s > nearly impossible to make them precisely the same. > e) The 3rd party timing source vendors have tried to work around timing > issue, while Cambium has tried to tweak things to deal with 3rd party > timing - and they end up confusing each other when (a) doesn’t exist. > > I think I got off topic…. Anyway - since this is sync over power and > the signal is a chop in the DC voltage (a square wave) a couple of things > happen when you attach a long cable to it. The inductance and capacitance > of the wire distorts the square wave. The leading and trailing edge > becomes a slope, possibly with some ringing on top of that. The detection > circuit in the AP has to decide where on the slope it’s going to ‘detect’ > the timing. Even if that point is precisely the same voltage on every AP > revision, the variation in cable length, inductance, and capacitance is > going to make each installation different. More knobs could be added to > the AP, but it’s going to be mighty hard to figure out what your supposed > to set them to. > > Sync over power, while an elegant solution to timing on the original FSK > radios, has outlived itself - timing is much more critical as data rates go > up and sync over power isn’t going to cut it in it’s current form. > > So why does this one AP go wonky? My guess would be an interaction > between the SyncInjector, something unique about the cabling to this > specific AP (how and what the cable is strapped to, if the installer did > something different with drip loops, slack storage, surge suppressor > components, etc.) that causes the square wave to either ring or slew, and > the AP either misinterprets this or tries to compensate for it and fails. > That doesn’t mean the SyncInjector did anything wrong - it’s probably > doing exactly what it’s supposed to do. The AP isn’t really necessarily > wrong either - it’s doing what it can to recover a mangled signal. > > Nobody is truly wrong, it’s just a broken system. > > Mark > > > > On Jan 22, 2016, at 9:56 AM, George Skorup <[email protected]> wrote: > > > > OK, wow. I feel so not alone now. > > > > A week or two ago I had this happen... I have two sites about 8 miles > apart. One has a full 3.6 cluster (N,S,E,W) on a SyncInjector. The other is > a single NE sector on a Parasitic SyncPipe. Same frequency and none of the > SMs on either sector can hear the other APs, they all face away (i.e. the > two sites don't line up together for any SM), so standard frequency reuse > scenario. We had some speed complaints and I found all of the SMs on the > North sector at 8x/1X downlink. Rebooted the whole cluster with no > resolution. Figured something happened to that AP. Then I figured I'd check > all other APs for any issues. That's when I found this NE sector running on > the on-board GPS. Everything looked completely normal on this sector (SM > links, etc). As soon as I disabled the on-board GPS to force it back onto > the SyncPipe, everything returned to normal at the other site. I still > don't understand that. It seems like the iGPS AP's timing was slightly > different and interfering with the uplink on the other (the sectors can > hear each other) which was causing a ton of retries so it backed off the > downlink? That's all I can figure. > > > > Also, before I started disabling the on-board GPS on all APs in a > cluster, one would fail over from power port to on-board and I'd get LBT > events. So I do believe there's a difference in the timing, PacketFlux vs > on-board, aka CMM3 vs CMM4. > > > > The other thing I see is dropped sessions when AutoSync switches to or > from on-board GPS and timing port/power port (PacketFlux on both). > Sometimes it's only the SMs actively passing traffic that go idle. I > believe the pulse is different and the AP has to re-align the frame. I > could be wrong. > > > > I've also mentioned this to Cambium multiple times now. I believe the > on-board GPS, even with it disabled because they don't actually turn the > receiver off, plays a role with AutoSync going stupid. Again, I could be > wrong. > > > > I remember you mentioned that you're seeing very close 3.6 SMs getting > fairly bad downlink SNR due to them hearing the opposite AP on the same > freq, even with the high f/b OEM Laird sectors. We're definitely seeing > that too. > > > > Anyway, that's all I got for now. I'm not blaming anyone or pointing > fingers, mostly because I'm not entirely sure it's Cambium's or > PacketFlux's fault, could be both, I don't know. Shit happens. But shit has > been happening for a really long time and I'm very frustrated. Aaron knows > this, I send him stuff all the time. :) > > > > On 1/22/2016 5:38 AM, Mark Radabaugh wrote: > >> Inline > >> > >>> On Jan 22, 2016, at 12:02 AM, George Skorup <[email protected]> wrote: > >>> > >>> Sounds like you mean that sector's frame timing is drifting? I haven't > seen that. But I have seen sectors go nuts where all of the SMs show 12dB > SNR and MIMO-A downlink. For no reason. Reboot the AP and it's gone. > >>> > >> I do think the sector�s frame timing is drifting. The end result is > exactly the same thing you are seeing - all the SM end up with low SNR and > MIMO-A. Reboot it and it goes away for 8 to 12 hours before it�s back. > It truly smells like timing drift. I don�t think it keeps drifting > though - I think the AP works it�s way off to some sort of offset limit > on the timing and then sits there. Cambium was playing around with > something in the software with a �debounce timing� that (I think) > compared the incoming pulse with the internal clock for a sanity check and > then rejected pulses outside the window. I�m not sure they didn�t end > up putting some type of averaging or compensating calculation in that might > be ending up working it�s way to a limit. > >> > >> > >>> On full cluster sites, we're using SyncInjectors, and only > SyncInjectors. I.E. I disable the on-board and timing ports on all sectors. > If the sync pulse from the injector drops, oh well. FreeRun is a PITA > especially with LBT. > >> That�s my usual practice, for the same reasons. > >> > >>> And I've said this many, many times. There's a difference between the > on-board GPS which is CMM4 aligned and PacketFlux stuff which is CMM3 > aligned. The frames don't match when you have a mix of this in a cluster. > >> I don�t have any way to measure the on-board timing pulse so I > can�t say where it�s at relative to a sync injector, but I am getting > very good performance with the north AP on the same frequency as the south > with one on a SyncInjector and the other on internal GPS. SNR on both > sectors is stable and modulation is where I expect it to be. > >> > >>> It's very obvious when this problem crops up. The AP session list > shows most SMs sitting at 8X/1X. And sessions with HP show the normal VC as > 8X/1X, HP VC as 1X/1X. > >> Yep > >> > >>> I know at one site in particular, the SyncInjector doesn't show any > 1PPS active events, yet the APs show a few inSync and outSync counts. Those > could be actual losses in timing, but Forrest said it needs to see > something like 3-4 in a row before it's logged in the event counter. > However, at another site, the timing is very stable and the APs show zero > outSync and one inSync count. But weird things are still happening. > >> Same - the SyncInjector logs don�t show anything unusual, nor does > the AP. > >> > >>> Another thing I've noticed is that a loss in the sync pulse doesn't > always show up in the AP event log. Or I'll see a loss message, but no > acquired message after that. > >>> > >>> And I've been seeing this weird stuff for well over a year. I just > don't know what else to do. CMMs, CTMs and new radios do not fit in the > budget. > >> Using the internal GPS isn�t a viable permanent solution for this, > but it�s working for the moment, and I have not seen a recurrence of the > problem in a week. I�m going to try a CMM to see if it makes any > difference, and if that fails I�ll probably resort to dragging a timing > port device up there and using that. > >> > >> Mark > > > > -- *Forrest Christian* *CEO**, PacketFlux Technologies, Inc.* Tel: 406-449-3345 | Address: 3577 Countryside Road, Helena, MT 59602 [email protected] | http://www.packetflux.com <http://www.linkedin.com/in/fwchristian> <http://facebook.com/packetflux> <http://twitter.com/@packetflux>
