Re: [ARTIQ] Sinara multi-crate / DRTIO switches
Joe, I think some clarification is badly required about what DRTIO does and does not. DRTIO gives you: 1) time transfer 2) low-latency low-level control of remote RTIO channels 3) an auxiliary low-bandwidth low-priority general-purpose data channel (which can be used for moninj, flashing boards, monitoring temperature, etc.) It is *not* a general-purpose networking or distributed computing protocol. On Tuesday, November 08, 2016 11:13 PM, Joe Britton wrote: Crossing each switch will incur 100ns-200ns of latency This has implications for some experiments. 10 m (10 km) fiber propagation is 48 ns (48 us). Demonstration experiments involving heralded entanglement of a pair of nodes (2 crates) have a low probability of success (~1e-6) and are repeated continuously (~1 MHz). Why does it have to be 2 crates? Are the hundreds of channels of a single crate not enough to drive a few ion traps? You'll have slow entanglement in your system at some point anyway as you plan to go long distances. 1) slower response times. 2) blocking the kernel CPU by twice the latency (round-trip) when it needs to enquire about the space available in a remote RTIO FIFO. Any implementation that requires round-trip communication to complete DRTIO is very bad due to fiber/free-space propagation delays. To first order all DRTIO should assume receiving devices are ready to receive and handle errors by a) reporting to master crate b) logging for post-processing. To second order it's fine for low-traffic advisory signaling like "FIFO 80% full." Plan for future deployments where communication propagation delays are 100's us. I advise against running DRTIO over such high-latency links. Even if we find all sorts of clever tricks to hide the latency in the "write to a remote FIFO" case, any branching would unavoidably require a roundtrip. Even toggling an output TTL in response to an input TTL edge would take 2x 100's us. Instead the nodes should have more autonomy (e.g. contain their own CPUs) and the links should be just time transfer + general purpose networking, i.e. White Rabbit. (The reasons we don't do DRTIO over White Rabbit are latency, Ethernet overhead for small packets, and difficulties in prioritizing traffic) > A current implementation using soft-core switching seems an adequate > compromise provided the system is designed in such a way that a future > gateware implementation is straight forward. In anticipation of a future all-gateware implementation of DRTIO routing is use of a dedicated soft-core CPU helpful? Not at all. Sébastien ___ ARTIQ mailing list https://ssl.serverraum.org/lists/listinfo/artiq
Re: [ARTIQ] Sinara multi-crate / DRTIO switches
> does anyone have serious plans to use more than one Sinara crate? Absolutely. One of my primary motives for supporting DRTIO is coordination of multiple crates. Use case is ARTIQ coordinating entanglement distribution between a pair of qubit systems (each with its own crate) separated by an optical fiber. Future (>3 years) is same but for >2 nodes. See the following paper for one approach using trapped ion qubits. http://iontrap.umd.edu/wp-content/uploads/2015/06/NphysModNet2015.pdf > Multi-crate configurations require slightly complicated gateware support for > DRTIO switches. >The bandwidth between crates will also be limited to 10Gbps. 10 Gb/s doesn't strike me as a limitation for the foreseeable future. > Crossing each switch will incur 100ns-200ns of latency This has implications for some experiments. 10 m (10 km) fiber propagation is 48 ns (48 us). Demonstration experiments involving heralded entanglement of a pair of nodes (2 crates) have a low probability of success (~1e-6) and are repeated continuously (~1 MHz). With each rep the success (or failure) is typically reported to nodes so the rep rate is limited by node-node communication latency. Adding an additional 200 ns for DRTIO routing in the case of 10 m separation is a significant added cost and may preclude the use of ARTIQ for early experiments (eg defect qubits where T2~ 10's us). For larger node separation it is fractionally smaller. >There is currently a plan to support multi-crate in the hardware (this >future-proofing simply >means adding some SFPs, essentially) but no plan to support it in the gateware. A current implementation using soft-core switching seems an adequate compromise provided the system is designed in such a way that a future gateware implementation is straight forward. > 1) slower response times. > 2) blocking the kernel CPU by twice the latency (round-trip) when it needs to > enquire about the space available in a remote RTIO FIFO. Any implementation that requires round-trip communication to complete DRTIO is very bad due to fiber/free-space propagation delays. To first order all DRTIO should assume receiving devices are ready to receive and handle errors by a) reporting to master crate b) logging for post-processing. To second order it's fine for low-traffic advisory signaling like "FIFO 80% full." Plan for future deployments where communication propagation delays are 100's us. In anticipation of a future all-gateware implementation of DRTIO routing is use of a dedicated soft-core CPU helpful? > DRTIO switch support is also beneficial to the serial protocol between the > Sayma AMC and > Sayma RTM FPGAs - the current plan is to use a dumb protocol that doesn't > have good timing >resolution and is inefficient for things like SPI transfers, essentially a >more open version of Channel >Link II. In this case you're relying on in-crate timing distribution. Seems fine. ___ ARTIQ mailing list https://ssl.serverraum.org/lists/listinfo/artiq
Re: [ARTIQ] Sinara multi-crate / DRTIO switches
On Saturday, November 05, 2016 09:57 PM, Grzegorz Kasprowicz wrote: We can use multiple 10Gbit links in parallel between Metlinos That's doable, but we'd have to write the gateware for inter-lane synchronization while keeping deterministic latency, which is a bit tricky. ___ ARTIQ mailing list https://ssl.serverraum.org/lists/listinfo/artiq
Re: [ARTIQ] Sinara multi-crate / DRTIO switches
We can use multiple 10Gbit links in parallel between Metlinos but the latency would be the same. ___ ARTIQ mailing list https://ssl.serverraum.org/lists/listinfo/artiq
[ARTIQ] Sinara multi-crate / DRTIO switches
Hi, does anyone have serious plans to use more than one Sinara crate? A crate already contains one Metlino and up to 12 Sayma cards, which means 96 DAC and 96 ADC channels. A Metlino could also be connected to several Kaslis (if the Kasli ends up being made). Multi-crate configurations require slightly complicated gateware support for DRTIO switches. The bandwidth between crates will also be limited to 10Gbps. Crossing each switch will incur 100ns-200ns of latency which will manifest itself in two significant ways: 1) slower response times. 2) blocking the kernel CPU by twice the latency (round-trip) when it needs to enquire about the space available in a remote RTIO FIFO. There is currently a plan to support multi-crate in the hardware (this future-proofing simply means adding some SFPs, essentially) but no plan to support it in the gateware. DRTIO switch support is also beneficial to the serial protocol between the Sayma AMC and Sayma RTM FPGAs - the current plan is to use a dumb protocol that doesn't have good timing resolution and is inefficient for things like SPI transfers, essentially a more open version of Channel Link II. Sébastien ___ ARTIQ mailing list https://ssl.serverraum.org/lists/listinfo/artiq
