Hi,
Here is a plan for clocking the Sinara system. Please comment.
Crate clock distribution
The crate distributes a 100MHz clock on a RTM RF backplane. This clock
is typically externally supplied from a high quality source, but it is
desirable to include a 100MHz oscillator on the clock module for
turnkey/standalone operation (with limited timing performance).
In a multi-crate system, all crates should receive the same 100MHz clock.
RTIO clock on Metlino
-
In root mode, the Metlino receives the 100MHz clock and turns it into a
200MHz RTIO clock that it uses as reference clock for its DRTIO
transmitters.
In satellite mode, the Metlino recovers the RTIO clock from the fiber.
The following clock resources should be available on Metlino to support
this operation:
* Si5324 for 100->200MHz in root mode, and CDR jitter filtering in
satellite mode.
* 200MHz XO to transceiver clock pin for providing a CDR reference in
satellite mode.
The Si5324 shall have its two clock outputs connected to a transceiver
clock input (so that we can transmit back synchronously and at fixed
latency) and to a general purpose clock input on the FPGA.
Transceiver<->fabric clock routing inside the FPGA is of poor quality,
so we want to be able to mitigate that.
There are three options for connecting the 100MHz RTM clock to the
Metlino (required in root mode):
1) make the Metlino double-width and connect it to RTM (preferred option
IMO)
2) add one clock cable inside the crate - could be mechanically
difficult or impossible
3) require two 100MHz inputs in root mode - this makes using the
built-in oscillator unwieldy, and will affect single-crate systems
RTIO clock on Sayma
---
Sayma cards recover their RTIO clock from the backplane's transceiver
links. This requires the same hardware as the Metlino in root mode: XO +
Si5324, connected in the same way.
RTIO clock frequency flexibility
The FPGA's built-in transceiver PLLs are not very flexible, so if easy
changing of the RTIO clock frequency is desired, we should consider
replacing the XOs with PLL synthesizer chips.
General purpose oscillator
--
Sayma and Metlino shall include a general purpose XO of e.g. 125MHz,
connected to a general purpose FPGA clock input pin. Transceiver clock
inputs can be routed to the fabric (with degraded timing), so this is
not absolutely necessary, but this is a simple addition that make the
boards a bit friendlier to developers. This XO becomes necessary if we
use a transceiver PLL chip that needs to be configured before it outputs
a clock.
JESD204 synchronization procedure
-
The FPGA shall align SYSREF with designated RTIO clock edges. The
alignment should be better than a DAC clock cycle and reproducible
across reboots.
The FPGA first roughly aligns SYSREF within one cycle before a desired
RTIO clock edge by asserting the synchronization signal of the clock
chip, which resets its dividers. This alignment has an uncertainty of
one DAC clock cycle. The FPGA then analyzes SYSREF by repeatedly
sampling it with a known clock while scanning a calibrated I/O input
delay (IDELAYE3) element to measure its phase with a high precision. It
then derives a phase correction value that it programs into the relevant
registers of the clock chip to delay SYSREF by the duration that aligns
it with the RTIO clock.
This mechanism is limited by the resolution of the IDELAY scanning
technique and of the clock chip's fine delay block. For this to work,
those resolutions should be good enough, i.e. significantly smaller than
a DAC clock period.
The clock chip we should use to generate SYSREF and the DAC clock is
AD9516-1 or HMC7044. HMC7044 looks like a better choice.
AD9516-1 has more coarse delay range. HMC7044 has more outputs, less
noise, and a higher resolution fine delay block.
Other clock chips that were considered but not selected were:
* AD9528 - maximum output frequency too low.
* HMC7043 - no PLL, cannot multiply the input clock.
This technique can be implemented on the AD9154 FMC cards, which also
use a AD9516-1 (assuming its fine delay block has enough resolution).
Alternate synchronization procedure #1
--
In case we encounter problems with the fine delay, we may be able to get
away by using the coarse delay only (which has a resolution of one DAC
clock period). This will require storing some information about the
choice of clock edge made from one boot to the next in order to ensure
constant latency across reboots. This technique was proposed by Robert
on IRC yesterday.
Alternate synchronization procedure #2
--
If the above techniques fail, we can distribute SYSREF at the source and
use it to sync HMC7044 clock chips (which contain a SYSREF retimer that
makes
