Daniel, thanks for the feedback. "A few" comments in response: tl;dr: distributing a reference clock at something like 10MHz-100MHz over the backplane, and generating the required DAC/upconverter clocks locally using VCOs seems to be by far the nicest, most scalable and flexible solution. There are, however, potential issues with phase drift/noise which could cause problems with this approach. However, I'm not convinced that there aren't equally difficult problems with any other approach. And, in any case, I'm optimistic that this won't be the case and the VCOs will be fine. We've started doing some tests on this and have more planned, and hope to have at least a partial answer soon.
1) You're right that for the most critical applications the single-loop PLL I described previously could start to become a limitation (although, we did gates with 99.9999% fidelity with a clock source that had a noise level pretty similar to the VCO I suggested). In practice, one would probably want to go for a 2-loop solution anyway to avoid having such a high divider ratio (240) in a single loop, which can lead to a significant noise contribution from the PFD (scales as 10log10(N) ). An example 2-loop PLL might be: begin by locking a 100MHz VCO to the 10MHz backplane-distributed reference using a lock bandwidth of ~1kHz. This loop would be designed to "clean up" the 10MHz reference clock by rejecting cross-talk, and needs to use a VCO with extremely good close-in phase noise. A choice of VC(S)O could be a Crystek CVSS-945 ($20 in single quantity from Digi-Key http://www.digikey.co.uk/product-detail/en/crystek-corporation/CVSS-945-100.000/744-1612-ND/3711616 although, there are fancier options if one really wants to push things). One would then lock the 2.4GHz VCO to the 100MHz with a loop bandwidth of ~500kHz. In this system, the phase noise should be within ~20dB of the Wenzel 1GHz oscillator at all frequencies, with the VCO and Wenzel oscillator being comparable by ~1MHz. This does assume that the PLL doesn't degrade the VCO noise significantly, which is something we plan to test. While it's more engineering work to design and test this system, it's work that can be done once on a prototype board and then copied into the final design. I'd generally prefer that to requiring each user to sort out their own external oscillator, power splitters, amps, cables, etc. Assuming we can get close to the VCO data sheet noise, I can't see the difference between the VCOs and Wenzel oscillator being significant, particularly given that... 2) Distributing the Wenzel oscillator's signal without degrading its phase noise is going to be pretty tough. For example, the best clock distribution IC that I know of is the HMC830. Looking at figure 22 of its data sheet suggests that even this IC will degrade the Wenzel oscillator's phase noise considerably. In general, clock switches tend to be far worse than this (which is one reason that I'd really rather not put a switch in the clock path to select between the front-panel and backplane clocks). I wouldn't trust a switch/distribution IC to work at those phase noise levels unless its data sheet explicitly gave a phase noise plot. A similar remark goes for the ERA-4SM amplifier you suggest using on the DACs output. For close-in phase noise, amplifier noise figure is meaningless. In general, for work where phase noise really matters, no amplifiers should be used unless their datasheet actually gives a phase noise plot -- and I certainly wouldn't trust a cheap MCL "general purpose" Darlington amp with my Wenzel oscillator without some careful measurements! 3) The DAC's phase noise contribution is going to be worse than the locked VCO's noise (compare the 100MHz trace on figure 33 of the AD9154 data sheet to the VCO datasheets) for all frequencies above 1kHz. Below 1kHz, the locked VCO's noise should be reduced close to the level of the 10MHz reference source, which could be a Wenzel oscillator if one wants. FWIW, the situation is the same even using something like an AD9914, which is about the lowest noise digital RF source I've seen. Similarly, before one gets _too_ worried about the possibility of a few residual cross-talk spurs near the noise floor on the DAC clock, let's see what the DAC's output spectrum actually looks like. (but then, maybe I'm just scarred by the forest of low-level spurs on our current DDS boards...) 4) In practice, I think that very low-level phase noise isn't going to be the thing that stops us all getting gate errors <1e-6. For example, I'd guess that duty-cycle-dependent thermal amplitude/phase transients in the DAC/amplifier will be far more of a problem in practice. Obviously, that's not a reason to be sloppy about the clock noise, but it does mean that it's probably not worth complicating the external wiring by introducing an external oscillator just to save a few dB... ---- As you say, one could distribute a 1GHz reference over the backplane as well as the 10MHz. While this can be a good idea in some situations, I don't think it would help in our case because: 5) In practice, it's likely one would end up dividing the 1GHz reference down to ~100MHz on the AMC anyway, so we'd loose the benefits one typically gets from using a higher frequency reference. One obvious reason for the need to divide is that the 2.4GHz DAC clock isn't an integer multiple of 1GHz. As a result, we'd need to use a frac-N PLL, which would essentially just divide the 1GHz reference down to something like 200MHz before multiplying back up to 2.4GHz. A second reason we'd need to divide the 1GHz is that (unless backplane cross-talk turns out not to be too much of a problem) we will want a narrow-bandwidth "clean-up" PLL to remove pickup from the reference clock. Given the low bandwidth of this loop, it'll need a VCO with extremely good close-in phase noise. My understanding is that the best close-in phase noise always comes from VCSOs in the ~10MHz-100MHz range. Because of this, I think that a 2-loop solution with the first loop running at around 100MHz is going to give the best performance. So, again, we'd want to divide the 1GHz reference down to ~100MHz on the AMC/FMC board. Note that, while it's tempting to assume that division will reduce the phase noise by 20*log10(n), at these low phase noise levels that's not a given due to the divider's noise floor. This can take away much of the benefit of using the higher frequency reference in the first place. It's also nice to avoid dividers as far as possible, as they introduce phase ambiguities. If sufficient care isn't taken, this will prevent deterministic synchronisation of the RF output phases across different boards. To avoid this, one has to pick a divider that can be externally reset accurately and think carefully about how/when it is reset.This isn't the end of the world, but isn't trivial. 6) The only major benefit of the 1GHz clock that I can see is that one is 20dB (40dB) less sensitive to pickup on the backplane compared with 100MHz (10MHz) reference due to the lower multiplication factor. However: a) I'd expect this improvement to be at least partially cancelled out by the fact that pickup/channel-channel cross-talk will generally be worse at higher frequencies. b) At higher frequencies the backplane losses can start to become significant. To achieve optimum phase noise performance from a clock buffer, one has to be careful to maintain a good input level (6dB can make a difference here). b) In practice, it's likely to be far more important to consider how much noise we expect at the different frequencies; the big win will come from putting the clock at a frequency which nothing else in the backplane runs at. It's not clear to me that 1GHz is better from this perspective than 10MHz (see point 7). c) In any case, I expect (hope) that backplane cross-talk won't actually be an issue at all if we design our loop filters properly and think about what signals we send down the backplane, (again see point 7) If the residual cross-talk on the "cleaned" 100MHz signal turns out to be an issue when using a 10MHz reference frequency, it may be worth switching to a 100MHz reference frequency. My preference for 10MHz isn't that strong, and just stems from the fact that 10MHz is so ubiquitous and there is so much high-quality 10MHz kit around. But, one could always distribute 10MHz between racks and put a 100MHz VCSO + PLL on the MCH to generate the 100MHz system clock. 7) Okay, the next point is even further outside my area of expertise than the rest of this email, so feel free to call BS on this one, but: it's important when choosing the backplane clock frequency to consider the spectrum of the noise we're trying to avoid. If we encode all data going down the backplane using (e.g.) 8b/10b then the signals should be contained in the range f_clk/2 to f_clk/10, where f_clk is the digital communication clock frequency. The upper frequency here is obtained for data that transitions on every clock cycle, while the lower limit is for data that consists of a a run of 5 0s or 1s (the longest allowed by the encoding). As a result, there should be effectively no noise below f_clk/10. Even if we restrict ourselves to _only_ using GTX at 10GSPS then this is only just not an issue for the 1GHz clock. While, a 10MHz clock should be okay for any communication data rate above 100MHz, which works well for standard LVDS. ---- Aside from noise-related issues, I'd generally prefer to avoid using an externally generated DAC clock because: 8) We're quite keen that our next generation hardware should support deterministic board-board phase synchronisation. In other words: suppose one programs the same RF frequency onto DACs on different FMC boards. We would like the phase relationship between the RF outputs to be unchanged by power-cycling the hardware (or just re-running the synchronisation routine). The ways we've thought about achieving this rely on the DAC clock being edge-aligned with the system reference clock (which the RTIO clock is generated from). This is hard to guarantee if the DAC clock is generated externally. Is this kind of synchronisation something you guys have thought about much? Do you have a plan for how you're going to do it? ---- 9) Your point about phase drift is a good one; this is actually one of the things that I think will be most crucial to get right in the new hardware. However: a) As you comment, it's not that scalable b) Related to (a), it doesn't seem future-proof either: you are assuming that we won't ever want to use a second RF clock? But, what if, e.g. at some point you want to upgrade to a DAC running at a different (higher) frequency, but don't want to bin all your existing hardware? If this happens, a second external oscillator is needed and the solution gets messier. But, beyond that, one becomes sensitive to the phase drifts between the two different external oscillators. In the past, I've seen significant (>10deg over something like 20 mins, after 24hrs of warmup) phase drifts between the 3GHz outputs of high-end Agilent synths that are nominally phase-locked together at 10MHz. Presumably this comes from the fact that these sources are designed with phase noise at frequencies above, say 1Hz, in mind, rather than long term drift. Now, I'd expect the Wenzel Oscillator to be better on that front (simpler than a synth so less to drift). But I'd still want to test it. c) You're assuming that passive clock distribution is always more stable than active. That's definitely true if one uses run of the mill components. But, compare e.g. the temp co of an ADCLK950 (50fs/K) to the channel-channel phase drifts of many MCL passive splitters. The splitters I've looked at have been worse than this. I think that in general, active or passive, this is a tough problem. d) The advantage of keeping all clock generation on board is that it can be tested as part of the design. If phase drifts turn out to be a problem, the design can be refined or small regions can be ovenized until the drifts are acceptable. Once that's done, everyone who uses the hardware knows what they're getting, without having to test their own external oscillators. ---- Sorry that was such a long one! But, that about covers my thoughts on clock distribution. I'd be interested to know what you think. Best, Tom P.S. Acknowledgement: many of the better ideas in the above are due to Weida Zhang and Chris Ballance... ________________________________ From: Slichter, Daniel H. (Fed) [daniel.slich...@nist.gov] Sent: 01 April 2016 22:02 To: Thomas Harty; artiq@lists.m-labs.hk Subject: RE: FW: initial specification of the project These are good thoughts, Tom, thanks for sending this detailed message. I imagine Greg will have some insights and measurements as well from Creotech’s experience in building the RF distribution backplane for the RTMs. A few comments inline below: Take the 2.4GHz DAC clock as an example. A decent (relatively small and inexpensive) choice of VCO for the PLL might be a UMX-630-D16-G (http://www.rfmd.com/store/products/all-products/oscillators/ultra-low-noise-cro-vcos/umx-630-d16-g-1.html). To get an idea of the lock bandwidth required, we can compare the phase noise of this VCO with a typical high-quality lab 10MHz source, such as a SRS FS725 Rubidium Standard (http://www.thinksrs.com/products/FS725.htm) scaled from 10MHz to 2.4GHz by adding 20*log10(240)=48dB. By 10kHz offset frequency, the typical VCO noise is -108dBc/Hz, compared with a (scaled) noise for the FS725 of ~-105dBc/Hz. Beyond that, the VCO is significantly better*. For good phase noise performance, one is better off distributing a higher frequency clock. For example, use an external 1 GHz crystal or oscillator disciplined to a 10 MHz atomic reference with low loop bandwidth. The phase noise at 1 GHz at 10 kHz offset for these sorts of things can be -160 dBc/Hz (e.g. Wenzel GMXO-PLD), which is substantially better than the scaled (multiplied) phase noise from an FS725 rubidium standard (-115 dBc/Hz @ 1 GHz, 10 kHz offset). Now you are way below the phase noise from the VCO, which would be -116 dBc/Hz scaled at 1 GHz. And the VCO you quote is a particularly fancy one because it is disciplined to an onboard CRO already, so one is not likely to find a substantially better VCO. How much will this level of phase noise affect things? If we go with -96 dBc/Hz at 10 GHz (scaled), assuming a VCO, we can compare with Fig 2 in http://arxiv.org/abs/1602.04551. This would put us about a third of the way from the “lab-grade” at -75 dbc/Hz to the “precision” at -130 dBc/Hz. This frequency offset corresponds to ~100 us operation times, so the phase noise would give an infidelity of about 10^-4 per gate there. Taking the 100 kHz offset, we have -118 dBc/Hz at 10 GHz with VCO, which is about 2/3 of the way from “lab grade” to “precision” and would limit gate fidelity to a few times 10^-6 per gate at ~10 us gate times. This is obviously a very coarse way of ballparking things, because one has to look at the entire spectrum, especially the integrated power in the white noise floor, but it gives a rough figure of merit. Given this admittedly very coarse analysis, I would prefer to at least have the option to pipe in a nice low-noise external clock directly, although it seems a backplane clock (especially if one distributes a higher frequency on the backplane) could work for many applications. Presumably, your concern is some combination of: a) We won't be able to build a PLL on the AMC/FMC board which has good enough phase noise @2.4GHz, regardless of the quality of the 10MHz reference we feed it This is one concern, but it appears the chip you suggest could get us decent phase noise for many applications. b) The PLL won't do a good enough job of removing noise on the reference clock far outside the loop bandwidth (say, outside the range 9MHz - 11MHz if one wants to be pessimistic) If you have a really good onboard oscillator that you are only disciplining with the backplane reference, this is not so much of an issue. But this then requires substantially more engineering. c) Even with a narrow-bandwidth (~1kHz) PLL with carefully chosen loop filter, and taking care over what other signals we send down the backplane, there will be too much pickup on the clock line close to 10MHz that can't be filtered off This is potentially an issue. d) Something else I'm missing? If each board in a crate has its own PLL, you will have more phase drift between the outputs of separate cards than if you distribute the DAC clock directly to all boards from a single source. Obviously the direct clock distribution doesn’t scale to many crates, but you would at least know that an entire crate’s worth of DACs are fully synchronized, as opposed to just one AMC card’s worth. If you have any measurements relevant to this, I'd be interested to see them. We're currently building hardware, one purpose of which will be to test how well we can generate a low phase noise 1GHz DDS clock from a 10MHz signal distributed down a uTCA backplane. Creotech probably have some measurements of this from when they built their RF backplane for the RTMs; I don’t have any data of my own. Glad you are doing some testing of your own. I would see if you can’t distribute the 1 GHz clock directly and use the PLL for cleanup only? * Of course, actually achieving this noise level will require some work in terms of simulation/testing the PLL design, and may need a multi-loop PLL to prevent the high divider ratio from causing problems with the PFD... This issue can be substantially ameliorated by distributing a higher frequency clock (e.g. 1 GHz) on the backplane as described above. Best, Daniel
_______________________________________________ ARTIQ mailing list https://ssl.serverraum.org/lists/listinfo/artiq
