Re: [ARTIQ] Sinara multi-crate / DRTIO switches
Joe, I think some clarification is badly required about what DRTIO does and does not. DRTIO gives you: 1) time transfer 2) low-latency low-level control of remote RTIO channels 3) an auxiliary low-bandwidth low-priority general-purpose data channel (which can be used for moninj, flashing boards, monitoring temperature, etc.) It is *not* a general-purpose networking or distributed computing protocol. On Tuesday, November 08, 2016 11:13 PM, Joe Britton wrote: Crossing each switch will incur 100ns-200ns of latency This has implications for some experiments. 10 m (10 km) fiber propagation is 48 ns (48 us). Demonstration experiments involving heralded entanglement of a pair of nodes (2 crates) have a low probability of success (~1e-6) and are repeated continuously (~1 MHz). Why does it have to be 2 crates? Are the hundreds of channels of a single crate not enough to drive a few ion traps? You'll have slow entanglement in your system at some point anyway as you plan to go long distances. 1) slower response times. 2) blocking the kernel CPU by twice the latency (round-trip) when it needs to enquire about the space available in a remote RTIO FIFO. Any implementation that requires round-trip communication to complete DRTIO is very bad due to fiber/free-space propagation delays. To first order all DRTIO should assume receiving devices are ready to receive and handle errors by a) reporting to master crate b) logging for post-processing. To second order it's fine for low-traffic advisory signaling like "FIFO 80% full." Plan for future deployments where communication propagation delays are 100's us. I advise against running DRTIO over such high-latency links. Even if we find all sorts of clever tricks to hide the latency in the "write to a remote FIFO" case, any branching would unavoidably require a roundtrip. Even toggling an output TTL in response to an input TTL edge would take 2x 100's us. Instead the nodes should have more autonomy (e.g. contain their own CPUs) and the links should be just time transfer + general purpose networking, i.e. White Rabbit. (The reasons we don't do DRTIO over White Rabbit are latency, Ethernet overhead for small packets, and difficulties in prioritizing traffic) > A current implementation using soft-core switching seems an adequate > compromise provided the system is designed in such a way that a future > gateware implementation is straight forward. In anticipation of a future all-gateware implementation of DRTIO routing is use of a dedicated soft-core CPU helpful? Not at all. Sébastien ___ ARTIQ mailing list https://ssl.serverraum.org/lists/listinfo/artiq
Re: [ARTIQ] Fwd: Sinara multi-crate / DRTIO switches
> * The decision we want to taking here is whether we will support only > the simple hierarchy of a single Metlino *directly* connected to a > bunch of terminal child devices devices (Sayma, Kasli) or whether we > allow support for hierarchies deeper than a single level of DRTIO > links to be built without having to rewrite parts of the stack. If I wanted a simple hierarchy I'd just use point-to-point BNC cables. Support of an arbitrary graph (not just hierarchical) is desired. Routing based on a static lookup table is fine. > * The slower response times would only affect channels that are deeper > in the DRTIO tree than the first level. It doesn't degrade performance > of existing channels. I.e. it affects channels that are on a Sayma > that's attached to a *satellite* Metlino. And the latency is > physically given by that hierarchy already. This latency also needs to > be viewed in comparison to the CPU latency which is µs. The utility of ARTIQ for future quantum information experiments is considerably hampered by reliance on CPUs with ~us latency. ___ ARTIQ mailing list https://ssl.serverraum.org/lists/listinfo/artiq
Re: [ARTIQ] Sinara multi-crate / DRTIO switches
> does anyone have serious plans to use more than one Sinara crate? Absolutely. One of my primary motives for supporting DRTIO is coordination of multiple crates. Use case is ARTIQ coordinating entanglement distribution between a pair of qubit systems (each with its own crate) separated by an optical fiber. Future (>3 years) is same but for >2 nodes. See the following paper for one approach using trapped ion qubits. http://iontrap.umd.edu/wp-content/uploads/2015/06/NphysModNet2015.pdf > Multi-crate configurations require slightly complicated gateware support for > DRTIO switches. >The bandwidth between crates will also be limited to 10Gbps. 10 Gb/s doesn't strike me as a limitation for the foreseeable future. > Crossing each switch will incur 100ns-200ns of latency This has implications for some experiments. 10 m (10 km) fiber propagation is 48 ns (48 us). Demonstration experiments involving heralded entanglement of a pair of nodes (2 crates) have a low probability of success (~1e-6) and are repeated continuously (~1 MHz). With each rep the success (or failure) is typically reported to nodes so the rep rate is limited by node-node communication latency. Adding an additional 200 ns for DRTIO routing in the case of 10 m separation is a significant added cost and may preclude the use of ARTIQ for early experiments (eg defect qubits where T2~ 10's us). For larger node separation it is fractionally smaller. >There is currently a plan to support multi-crate in the hardware (this >future-proofing simply >means adding some SFPs, essentially) but no plan to support it in the gateware. A current implementation using soft-core switching seems an adequate compromise provided the system is designed in such a way that a future gateware implementation is straight forward. > 1) slower response times. > 2) blocking the kernel CPU by twice the latency (round-trip) when it needs to > enquire about the space available in a remote RTIO FIFO. Any implementation that requires round-trip communication to complete DRTIO is very bad due to fiber/free-space propagation delays. To first order all DRTIO should assume receiving devices are ready to receive and handle errors by a) reporting to master crate b) logging for post-processing. To second order it's fine for low-traffic advisory signaling like "FIFO 80% full." Plan for future deployments where communication propagation delays are 100's us. In anticipation of a future all-gateware implementation of DRTIO routing is use of a dedicated soft-core CPU helpful? > DRTIO switch support is also beneficial to the serial protocol between the > Sayma AMC and > Sayma RTM FPGAs - the current plan is to use a dumb protocol that doesn't > have good timing >resolution and is inefficient for things like SPI transfers, essentially a >more open version of Channel >Link II. In this case you're relying on in-crate timing distribution. Seems fine. ___ ARTIQ mailing list https://ssl.serverraum.org/lists/listinfo/artiq
