Hi all. We faced this problem with our GUPPI machine, and Randy McCullough etal came up with a scheme to synchronize the 4 streams from 2 ibobs. It is indeed the case that the streams do not naturally remain synchronized. If you use the OOB signals, extra transfers are inserted into the data stream. Etc.
I will get Jason Ray or Randy to send you our synchronizer model file, which is implemented in an extra FPGA on the BEE2. We fought this problem for many months. It is indeed real... We need a packet-enabled PFB and FFT, and then we could discard this synchronizer business, but until then, we are stuck using a synchronous system. We are running at 1.6 Gs/s. John > Hi Jason, > > I agree you could be fooled if, by chance, the ibob counters happen to be > synced. But, I think the likelihood of such an occurrence is low enough > that it probably wouldn't fool you very often or for very long. If the 1 > pps sync *didn't* work, the values in the snap blocks would almost > certainly > be misaligned by many more than 1 or 2 counts. > > But, I think this depends on how your sync generator is setup (whether the > counter feeding the snaps will free run upon programming, prior to setting > a > sync pulse period). In this case, the counter will only be regularly > reset > upon applying a period, and is thus a somewhat random function of the time > you enter a value into a sw register - thus making the likelihood of > synced > counters w/out a 1pps sync unlikely. > > Triggering on the 1pps is indeed a more robust and specific test. > > - Andrew > > > > > > > On 8/28/09 2:58 AM, "Jason Manley" <[email protected]> wrote: > >> Yeah, but what you're wanting to test here is if they are indeed >> aligned. Agreed that the test using internal syncs would work if they >> did trigger on a 1PPS, but then there's not much point in checking. So >> you want to check if they did properly trigger on a 1PPS and if the >> ibobs are correctly sync'd. >> >> The problem with internal sync triggering >> ========================================= >> So let's imagine that they don't properly sync on a 1PPS. Then their >> onboard counters were not reset at the same time and are now running >> in sync, but are misaligned. >> Now let's say you are triggering your snap blocks off an internal >> sync, generated from a local counter. You might trigger the snap >> blocks on all the IBOBs and have themy wait for their next sync pulses >> (which is still aligned on each ibob with their respective counter >> values) before capturing a bunch of data (presumably a counter), >> Depending on your internal sync period (let's imagine a sync period of >> 1second ~2^27), each snap block could contain the same counter values, >> making it look like they're in sync, though in fact they might be out >> by as much as a second, because the triggers could occur any time >> within a second of the software arm. I have seen this happen with low >> amplitude 1PPS inputs. >> >> Debugging IBOB sync >> =================== >> If you trigger off the external 1PPS, then you are assured of absolute >> timing, rather than the internal sync, which may or may not be aligned >> (it is precisely this that you need to test). Bearing in mind that the >> exact trigger point of the 1PPS on each IBOB may jitter by up to 1 >> clock period (hence your snap's counter values might be misaligned by >> up to 2 values). I have seen this with low amplitude or slewed (poor >> distribution network) 1PPS pulses. >> >> Visual check good >> ================= >> Also, if you output the sync pulse of each ibob to an LED, you can >> usually see pretty quickly if they are reasonably closely sync'd or >> not. While not conclusive, it provides a nice sanity check. >> >> A deployable check for sync >> =========================== >> A nice way of building-in a check for sync in the deployed system is >> to have the external 1PPS enable a register of your sync generator's >> current counter. With a reasonably slow sync (I like 2^27 ~ 1 second), >> you read the registered values from all the IBOBs at the same time. >> Then you compare them in software. They should all be within 2 counts >> of each other. >> >> Jason >> >> >> On 28 Aug 2009, at 11:35, Andrew Siemion wrote: >> >>> Hi Jason, >>> >>> If the ibobs are clocked synchronously (as in Richard's setup), how >>> could >>> the internal syncs be out of alignment (after proper triggering on a >>> 1pps?). >>> >>> As I understand it, once you trigger on the 1pps the two ibob >>> internal syncs >>> should be perfectly aligned, both in cycle number and time (subject >>> to some >>> possible glitch)? >>> >>> - Andrew >>> >>> >>> >>> >>> On 8/28/09 2:23 AM, "Jason Manley" <[email protected]> wrote: >>> >>>> Placing snap blocks on separate ibobs will not be a definitive test, >>>> but would certainly be helpful. You would have to trigger them from >>>> the external 1PPS, which is not guaranteed to be perfectly aligned >>>> across boards after the first arm-trigger-sync operation (you may >>>> jitter by one or two clocks). Bear this in mind when interpreting >>>> the >>>> two captures... that the values could be out by up to 2 counts. >>>> >>>> Triggering the snap blocks off the internally-generated sync pulses >>>> would defeat the purpose entirely, since the syncs would force the >>>> local counters to trigger the snap blocks at the same values even if >>>> they don't occur at the same time. >>>> >>>> Jason >>>> >>>> On 28 Aug 2009, at 11:10, Andrew Siemion wrote: >>>> >>>>> Hi Richard, >>>>> >>>>> Did you add snap blocks to the counters prior to the xaui links, >>>>> triggered on a sync, to ensure that the ibobs are indeed >>>>> synchronized? >>>>> >>>>> - Andrew >>>>> >>>>> >>>>> On 8/28/09 2:02 AM, "Richard Armstrong" >>>>> <[email protected]> wrote: >>>>> >>>>>> Dear CASPERites, >>>>>> >>>>>> I've been trying for some time to synchronise two XAUI streams from >>>>>> two separate IBOB boards in a BEE2 board. I've not met with much >>>>>> success. I'd be really grateful any ideas or help from anyone who >>>>>> may have experience with this, or perhaps a better general >>>>>> understanding of the issues I'm encountering. >>>>>> >>>>>> The test system I've got is as follows: I'm clocking my IBOBs >>>>>> synchronously and aligning them with a 1PPS input. Each IBOB has a >>>>>> 31-bit counter which is reset off a 'sync' pulse. The sync pulse is >>>>>> generated by a circuit that is aligned with the 1PPS signal. I'm >>>>>> sending the counter value multiplexed with the sync in a demux-by-2 >>>>>> XAUI block to the BEE2. At the BEE2, I'm testing the alignment of >>>>>> the counters. >>>>>> >>>>>> In my mind, as long as the ibob data rate is low enough, the XAUI >>>>>> interface should provide complete clock-domain isolation. That is, >>>>>> the bee2 should be able to be clocked at any rate, including a >>>>>> lower rate than the ibob (provided, again that the data transfer >>>>>> rate is low enough). >>>>>> >>>>>> Worryingly, and unexpectedly, I find the counter values way out of >>>>>> alignment, and perhaps worse than this, there seems to be jitter on >>>>>> the difference between the counter values from cycle to cycle. >>>>>> Andrew and Peter helped me out for a couple of days on their way to >>>>>> Paris. We corrected some of the design errors (namely, selecting >>>>>> demux-by-2 for 32-bit data), made sure the ibob clock was running >>>>>> slower than the bee2 clock, and changed the length of the sync >>>>>> pulse. However, we were unable to resolve the fundamental issues. >>>>>> >>>>>> This week I've been trying to clock the BEE2 off an IBOB-generated >>>>>> clock with an LVTTL-to-LVPECL clock driver. I changed the BEE2 >>>>>> design and changed a jumper to select user-clock on the BEE2 board. >>>>>> Unfortunately, this doesn't seem to help much (although, I could be >>>>>> doing it wrong) and I'm finding with this scheme that the BEE2 has >>>>>> the tendancy to kill the running process after a few seconds to >>>>>> minutes. >>>>>> >>>>>> I know many of the more experienced CASPERites advise asynchronous >>>>>> designs for this reason. However, I'm pretty sure someone else must >>>>>> have disregarded this advice (or perhaps discovered this advice) by >>>>>> pursuing XAUI synchronisation. Were these same issues seen? What's >>>>>> the best clocking scheme in practice (fully synchronous or >>>>>> downstream boards clcoked faster)? Does anyone have any ideas on >>>>>> jitter? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Richard >>>> >>> >>> >> > > >
