[ARTIQ] Fwd: Sinara multi-crate / DRTIO switches

[Robert Jördens]

On Sat, Nov 5, 2016 at 2:50 PM, Sébastien Bourdeauducq  wrote:
> does anyone have serious plans to use more than one Sinara crate?
>
> A crate already contains one Metlino and up to 12 Sayma cards, which means
> 96 DAC and 96 ADC channels. A Metlino could also be connected to several
> Kaslis (if the Kasli ends up being made).
>
> Multi-crate configurations require slightly complicated gateware support for
> DRTIO switches. The bandwidth between crates will also be limited to 10Gbps.
> Crossing each switch will incur 100ns-200ns of latency which will manifest
> itself in two significant ways:
> 1) slower response times.
> 2) blocking the kernel CPU by twice the latency (round-trip) when it needs
> to enquire about the space available in a remote RTIO FIFO.
>
> There is currently a plan to support multi-crate in the hardware (this
> future-proofing simply means adding some SFPs, essentially) but no plan to
> support it in the gateware.
>
> DRTIO switch support is also beneficial to the serial protocol between the
> Sayma AMC and Sayma RTM FPGAs - the current plan is to use a dumb protocol
> that doesn't have good timing resolution and is inefficient for things like
> SPI transfers, essentially a more open version of Channel Link II.

A few things to note here:

* The decision we want to taking here is whether we will support only
the simple hierarchy of a single Metlino *directly* connected to a
bunch of terminal child devices devices (Sayma, Kasli) or whether we
allow support for hierarchies deeper than a single level of DRTIO
links to be built without having to rewrite parts of the stack.
* The slower response times would only affect channels that are deeper
in the DRTIO tree than the first level. It doesn't degrade performance
of existing channels. I.e. it affects channels that are on a Sayma
that's attached to a *satellite* Metlino. And the latency is
physically given by that hierarchy already. This latency also needs to
be viewed in comparison to the CPU latency which is µs.
* The deeper hierarchy is also required to support DRTIO over the SFP
and SATA ports on Sayma or in fact any DRTIO device connected to a
Sayma. And it could come in handy for the AMC-RTM link as Sébastien
describes.
* The number of digital channels available in the simple single-level
DRTIO hierarchy might be more limiting than the number of ADC/DAC
channels.

--
Robert Jördens.
___
ARTIQ mailing list
https://ssl.serverraum.org/lists/listinfo/artiq

Re: [ARTIQ] Sinara multi-crate / DRTIO switches

[Sébastien Bourdeauducq]

On Saturday, November 05, 2016 09:57 PM, Grzegorz Kasprowicz wrote:

We can use multiple 10Gbit links in parallel between Metlinos


That's doable, but we'd have to write the gateware for inter-lane 
synchronization while keeping deterministic latency, which is a bit tricky.

___
ARTIQ mailing list
https://ssl.serverraum.org/lists/listinfo/artiq

Re: [ARTIQ] Sinara multi-crate / DRTIO switches

[Grzegorz Kasprowicz]

We can use multiple 10Gbit links in parallel between Metlinos but the
latency would be the same.
___
ARTIQ mailing list
https://ssl.serverraum.org/lists/listinfo/artiq

[ARTIQ] Sinara multi-crate / DRTIO switches

[Sébastien Bourdeauducq]

Hi,

does anyone have serious plans to use more than one Sinara crate?

A crate already contains one Metlino and up to 12 Sayma cards, which 
means 96 DAC and 96 ADC channels. A Metlino could also be connected to 
several Kaslis (if the Kasli ends up being made).


Multi-crate configurations require slightly complicated gateware support 
for DRTIO switches. The bandwidth between crates will also be limited to 
10Gbps. Crossing each switch will incur 100ns-200ns of latency which 
will manifest itself in two significant ways:

1) slower response times.
2) blocking the kernel CPU by twice the latency (round-trip) when it 
needs to enquire about the space available in a remote RTIO FIFO.


There is currently a plan to support multi-crate in the hardware (this 
future-proofing simply means adding some SFPs, essentially) but no plan 
to support it in the gateware.


DRTIO switch support is also beneficial to the serial protocol between 
the Sayma AMC and Sayma RTM FPGAs - the current plan is to use a dumb 
protocol that doesn't have good timing resolution and is inefficient for 
things like SPI transfers, essentially a more open version of Channel 
Link II.


Sébastien
___
ARTIQ mailing list
https://ssl.serverraum.org/lists/listinfo/artiq