Thanks Billy.
I will add, perhaps not unlike Richard's setup, in the ATA scheme the
iBob's output of clock/2 is sent thru an active LVTTL-LVPECL repeater,
then over 50 ohm coax cable, to a small terminating
adapter board to provide a valid signal for the BEE2's userclock
source chip (U7). The FPGAs then multiply this input clock freq by 2
for internal use (as others have already discussed).
There are lots of schemes that can provide the BEE2's FPGAs a
valid userclock. In any case, perhaps while waiting for a compile
to run, I'd measure the userclock U7 inputs and outputs
to verify they are as expected.
Matt
On Fri, 28 Aug 2009, William Chauncey Barott wrote:
All-
FWIW, ATA had this problem as well with the iBob/BEE2 beamformer.
Problem was not only synchronizing iBob streams in a BEE2, but
synchronizing streams from multiple BEE2s. It took a while to come up
with a good solution. Besides the firmware solution, the problem was
closely coupled to iBob FPGA temperature and XAUI cable length (Read:
Make sure your iBob FPGAs are actively cooled, and use 1m or at the
longest 1.5m copper XAUI cables to get the most stable links. Use fiber
where longer runs are required).
In our current setup, all iBobs are driven synchronously by the sample
clock, and a derived lower-frequency clock is driven out of an iBob to
clock the userclock input on all BEE2s (as has already been discussed in
this thread). We use the FIFO on the yellow block itself as a buffer
for the XAUI link (it's full/empty status is monitored by software, so
we know if likely errors have occurred).
Our stream alignment is done as has been discussed, using OOB sync
pulses (we have a 1pps and a 100pps, I forget which we sync off of).
For diagnostic purposes, we also keep track of how many times the stream
latencies have to be adjusted - IE, if they slip and are realigned.
Our current setup is quite robust - links are now stable and there are
few-to-no realignments after startup. Happy to share the model if it'd
be useful.
Billy
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Andrew Siemion
Sent: Friday, August 28, 2009 6:21 AM
To: Jason Manley
Cc: [email protected]
Subject: Re: [casper] XAUI synchronisation
Hi Jason,
I agree you could be fooled if, by chance, the ibob counters happen to
be
synced. But, I think the likelihood of such an occurrence is low enough
that it probably wouldn't fool you very often or for very long. If the
1
pps sync *didn't* work, the values in the snap blocks would almost
certainly
be misaligned by many more than 1 or 2 counts.
But, I think this depends on how your sync generator is setup (whether
the
counter feeding the snaps will free run upon programming, prior to
setting a
sync pulse period). In this case, the counter will only be regularly
reset
upon applying a period, and is thus a somewhat random function of the
time
you enter a value into a sw register - thus making the likelihood of
synced
counters w/out a 1pps sync unlikely.
Triggering on the 1pps is indeed a more robust and specific test.
- Andrew
On 8/28/09 2:58 AM, "Jason Manley" <[email protected]> wrote:
Yeah, but what you're wanting to test here is if they are indeed
aligned. Agreed that the test using internal syncs would work if they
did trigger on a 1PPS, but then there's not much point in checking. So
you want to check if they did properly trigger on a 1PPS and if the
ibobs are correctly sync'd.
The problem with internal sync triggering
=========================================
So let's imagine that they don't properly sync on a 1PPS. Then their
onboard counters were not reset at the same time and are now running
in sync, but are misaligned.
Now let's say you are triggering your snap blocks off an internal
sync, generated from a local counter. You might trigger the snap
blocks on all the IBOBs and have themy wait for their next sync pulses
(which is still aligned on each ibob with their respective counter
values) before capturing a bunch of data (presumably a counter),
Depending on your internal sync period (let's imagine a sync period of
1second ~2^27), each snap block could contain the same counter values,
making it look like they're in sync, though in fact they might be out
by as much as a second, because the triggers could occur any time
within a second of the software arm. I have seen this happen with low
amplitude 1PPS inputs.
Debugging IBOB sync
===================
If you trigger off the external 1PPS, then you are assured of absolute
timing, rather than the internal sync, which may or may not be aligned
(it is precisely this that you need to test). Bearing in mind that the
exact trigger point of the 1PPS on each IBOB may jitter by up to 1
clock period (hence your snap's counter values might be misaligned by
up to 2 values). I have seen this with low amplitude or slewed (poor
distribution network) 1PPS pulses.
Visual check good
=================
Also, if you output the sync pulse of each ibob to an LED, you can
usually see pretty quickly if they are reasonably closely sync'd or
not. While not conclusive, it provides a nice sanity check.
A deployable check for sync
===========================
A nice way of building-in a check for sync in the deployed system is
to have the external 1PPS enable a register of your sync generator's
current counter. With a reasonably slow sync (I like 2^27 ~ 1 second),
you read the registered values from all the IBOBs at the same time.
Then you compare them in software. They should all be within 2 counts
of each other.
Jason
On 28 Aug 2009, at 11:35, Andrew Siemion wrote:
Hi Jason,
If the ibobs are clocked synchronously (as in Richard's setup), how
could
the internal syncs be out of alignment (after proper triggering on a
1pps?).
As I understand it, once you trigger on the 1pps the two ibob
internal syncs
should be perfectly aligned, both in cycle number and time (subject
to some
possible glitch)?
- Andrew
On 8/28/09 2:23 AM, "Jason Manley" <[email protected]> wrote:
Placing snap blocks on separate ibobs will not be a definitive test,
but would certainly be helpful. You would have to trigger them from
the external 1PPS, which is not guaranteed to be perfectly aligned
across boards after the first arm-trigger-sync operation (you may
jitter by one or two clocks). Bear this in mind when interpreting
the
two captures... that the values could be out by up to 2 counts.
Triggering the snap blocks off the internally-generated sync pulses
would defeat the purpose entirely, since the syncs would force the
local counters to trigger the snap blocks at the same values even if
they don't occur at the same time.
Jason
On 28 Aug 2009, at 11:10, Andrew Siemion wrote:
Hi Richard,
Did you add snap blocks to the counters prior to the xaui links,
triggered on a sync, to ensure that the ibobs are indeed
synchronized?
- Andrew
On 8/28/09 2:02 AM, "Richard Armstrong"
<[email protected]> wrote:
Dear CASPERites,
I've been trying for some time to synchronise two XAUI streams
from
two separate IBOB boards in a BEE2 board. I've not met with much
success. I'd be really grateful any ideas or help from anyone who
may have experience with this, or perhaps a better general
understanding of the issues I'm encountering.
The test system I've got is as follows: I'm clocking my IBOBs
synchronously and aligning them with a 1PPS input. Each IBOB has a
31-bit counter which is reset off a 'sync' pulse. The sync pulse
is
generated by a circuit that is aligned with the 1PPS signal. I'm
sending the counter value multiplexed with the sync in a
demux-by-2
XAUI block to the BEE2. At the BEE2, I'm testing the alignment of
the counters.
In my mind, as long as the ibob data rate is low enough, the XAUI
interface should provide complete clock-domain isolation. That is,
the bee2 should be able to be clocked at any rate, including a
lower rate than the ibob (provided, again that the data transfer
rate is low enough).
Worryingly, and unexpectedly, I find the counter values way out of
alignment, and perhaps worse than this, there seems to be jitter
on
the difference between the counter values from cycle to cycle.
Andrew and Peter helped me out for a couple of days on their way
to
Paris. We corrected some of the design errors (namely, selecting
demux-by-2 for 32-bit data), made sure the ibob clock was running
slower than the bee2 clock, and changed the length of the sync
pulse. However, we were unable to resolve the fundamental issues.
This week I've been trying to clock the BEE2 off an IBOB-generated
clock with an LVTTL-to-LVPECL clock driver. I changed the BEE2
design and changed a jumper to select user-clock on the BEE2
board.
Unfortunately, this doesn't seem to help much (although, I could
be
doing it wrong) and I'm finding with this scheme that the BEE2 has
the tendancy to kill the running process after a few seconds to
minutes.
I know many of the more experienced CASPERites advise asynchronous
designs for this reason. However, I'm pretty sure someone else
must
have disregarded this advice (or perhaps discovered this advice)
by
pursuing XAUI synchronisation. Were these same issues seen? What's
the best clocking scheme in practice (fully synchronous or
downstream boards clcoked faster)? Does anyone have any ideas on
jitter?
Thanks,
Richard
