To test for bad links, I do send a counter periodically, but for
alignment I use a single pulse, because I can't afford the bandwidth
waste to send a counter all the time. I also need to align across
boards, for which a pulse is much easier to work with.
I suggest reasonably slow sync periods for this reason. I like a
period of about 2^27, because then you can also output it to an LED
for a visual sanity check. This corresponds to about 1 second at
200MHz using one pulse for on, one for off. Else 1/2 second LED with
negedge delay of ~2^23 which is also ok). This is definitely more than
the latency through the links.
Expect jitter to vary constantly by up to 2 words on good links. Bad
links this could be much higher as internal buffers fill, channels
realign and buffers then flush. In this case, try a shorter cable or
tweak the PHY settings. I have a circuit for detecting bad links too,
which I can share with you at the workshop or you can play with the
packetised correlator design in the SVN.
I clock downstream boards faster, so that the downstream DSP pipelines
can keep up with the datarates from the upstream boards' pipelines.
Jason
On 28 Aug 2009, at 11:55, Richard Armstrong wrote:
Hi Jason,
This sounds reasonable. Would you recommend the OOB signalling to be
implemented with a single pulse or a timestamp/counter?
I'm still pretty worried that my relative delay is _many_ clock
cycles (more than can be buffered for long sync periods). Also, is
the jitter expected to vary on such short timescales?
I'll take a look at Billy's designs now.
Also, on clocking: how are your boards clocked? The
downstream=faster approach?
~R
2009/8/28 Jason Manley <[email protected]>
I have successfully been able to do what you want (multiple ibobs to
BEE2 sync'd) and also been able to sync multiple BEE2s using this
scheme for the packetised correlator.
The reason for the jitter is because the XAUIs are clocked off their
own separate clocks (156.25MHz crystals on the boards). These are
asynchronous no matter how you clock the FPGA. The latencies on the
different ports are also not guaranteed and could change between
reboots/FPGA reloading.
The short of it is that you need to have RX fifos on the BEE. These
need some control logic to align the data, based on received signals
from the IBOBs. I use XAUI out of band with sync signals for this
purpose, but you could interleave these signals in the main
datastream if you're only clocking-in every second value.
Billy at CASPER should have a reasonable idea of how to do this as
we block-diagrammed a suitable system on the whiteboard before I
left Berkeley for his 3GHz correlator which has the same problem as
yours. Perhaps ask him for his model files.
Today is my last day at the office before I take a month's holiday
and will be back in time for the CASPER workshop. I can help you
with this interface at the workshop if you don't come right before
then.
Jason
On 28 Aug 2009, at 11:02, Richard Armstrong wrote:
Dear CASPERites,
I've been trying for some time to synchronise two XAUI streams from
two separate IBOB boards in a BEE2 board. I've not met with much
success. I'd be really grateful any ideas or help from anyone who
may have experience with this, or perhaps a better general
understanding of the issues I'm encountering.
The test system I've got is as follows: I'm clocking my IBOBs
synchronously and aligning them with a 1PPS input. Each IBOB has a
31-bit counter which is reset off a 'sync' pulse. The sync pulse is
generated by a circuit that is aligned with the 1PPS signal. I'm
sending the counter value multiplexed with the sync in a demux-by-2
XAUI block to the BEE2. At the BEE2, I'm testing the alignment of
the counters.
In my mind, as long as the ibob data rate is low enough, the XAUI
interface should provide complete clock-domain isolation. That is,
the bee2 should be able to be clocked at any rate, including a lower
rate than the ibob (provided, again that the data transfer rate is
low enough).
Worryingly, and unexpectedly, I find the counter values way out of
alignment, and perhaps worse than this, there seems to be jitter on
the difference between the counter values from cycle to cycle.
Andrew and Peter helped me out for a couple of days on their way to
Paris. We corrected some of the design errors (namely, selecting
demux-by-2 for 32-bit data), made sure the ibob clock was running
slower than the bee2 clock, and changed the length of the sync
pulse. However, we were unable to resolve the fundamental issues.
This week I've been trying to clock the BEE2 off an IBOB-generated
clock with an LVTTL-to-LVPECL clock driver. I changed the BEE2
design and changed a jumper to select user-clock on the BEE2 board.
Unfortunately, this doesn't seem to help much (although, I could be
doing it wrong) and I'm finding with this scheme that the BEE2 has
the tendancy to kill the running process after a few seconds to
minutes.
I know many of the more experienced CASPERites advise asynchronous
designs for this reason. However, I'm pretty sure someone else must
have disregarded this advice (or perhaps discovered this advice) by
pursuing XAUI synchronisation. Were these same issues seen? What's
the best clocking scheme in practice (fully synchronous or
downstream boards clcoked faster)? Does anyone have any ideas on
jitter?
Thanks,
Richard
--
Richard Armstrong
ʞn˙ɔɐ˙xo˙oɹʇ...@ƃuoɹʇsɯɹɐ˙pɹɐɥɔıɹ
--
Richard Armstrong
ʞn˙ɔɐ˙xo˙oɹʇ...@ƃuoɹʇsɯɹɐ˙pɹɐɥɔıɹ
+44 (0) 79 0682 9979 (UK mobile)
+44 (0) 1865 273597 (Office)
