Hi Clifford and All, Thanks for your responses. After looking into our design further, it looks like we are moving toward a somewhat narrower bandwidth, which makes it compatible with the Skarab ADC. Since this is an entirely new platform for me, I'll need some help visualizing the architecture needed for a 10 antenna (or 16, if 2^n is required) dual-pol system. I assume that the paradigm of a packet-switched correlator has not changed, so we are talking about a set of F-engines feeding a set of X-engines. With 3, 4-input ADCs, we could get 6 dual-pol antennas into one Skarab, but are the FPGA resources able to handle so many F-engines? Can someone estimate for me what the practical limit (number of inputs/Skarab) would be, for a generic F-engine design? For our ROACH-2-based EOVSA correlator, we actually have both F- and X-engine functions on each ROACH-2, but what would be the preferred architecture in the Skarab case?
To update the design concept from what I initially posted, we now want to digitize roughly 1 - 1.2 GHz (depends on how broadband we can make the horn feed) from 10 dual-pol antennas, channelizing to 4096 or more channels. We'll need ~10 ms dump times for the science case, which for baselines for 10 antennas comes to about 360 MB/s, if I count correctly, but if we have to dump baselines for all 16 inputs it becomes about 1 GB/s. This is no problem for a 40 Gbe system, I guess. Could anyone venture a ballpark guess as to the cost of a Skarab-based system to do this? Many Thanks, Dale On Thu, Nov 9, 2017 at 5:21 AM, Clifford van Dyk <cliffordvan...@gmail.com> wrote: > Hi Dale > > In case you are interested in considering the Skarab platform, I have > attached the latest Skarab product glossies to this email (there is also > a lot if info on the Casper Skarab web pages). Currently, only one > 4-channel, 3 GSPS 14-bit ADC mezzanine exists, that can (practically) be > populated on any of the Skarab's 4 mezzanine sites. Since one site is > dedicated to 4 x 40 Gb Ethernet, this has a practical limit of 12 ADC > channels per Skarab. While the ADC is not capable of delivering a 2 GHz > instantaneous bandwidth you are after (Nyquist is limited to 1.5 GHz), > it has a number of other attractive features, like multiple built-in > DDCs behind each ADC channel. This may offload some of the high speed > trickery/resource that would otherwise go into the FPGA, opening up the > resource for other use. > > Alternatively, in keeping with the direction that SKA-SA is heading > w.r.t. external ADCs (Jason's comments below), the Skarab 4 x 40 GbE > Mezzanine can actually run any high speed SERDES protocol between the > QSFP+ connectors and the FPGA (there are no Ethernet-specific PHYs in > between - PHY is fully implemented in the FPGA). So while SKA-SA prefers > 40 GbE interfacing everywhere, it may be possible to use other SERDES > protocols to attach directly to external ADCs (e.g. JESD, ESIStream) > over the QSFP+ connectors - you can essentially see the mezzanine as > capable of carrying up to 160 Gbps over 16 configurable SERDES lanes. > > Let me know if you have any specific questions around the platform. > > Kind regards, > Clifford > > On 11/8/2017 9:24 AM, Jason Manley wrote: > > Hi Dale > > > > The CASPER packetised correlator (that PAPER/HERA/KAT-7/MeerKAT etc are > all using) would scale simply to your needs. You could easily build it out > of ROACH2s or SKARABs at your scales. It would be a bit more expensive than > the SWARM-type solution, in that it'd need additional hardware because it's > not as resource-efficient as the SWARM design (you're going to struggle to > beat SWARM with any other design!). A ROACH2 packetised solution would > allow a number of different ADC options, including the the same one that > SWARM uses. > > > > SKA-SA is moving in a slightly different direction with ADCs for our > current and future systems. We have moved the ADC off the digital > processing boards in the interests of reducing noise, and increasing > overall system performance (constrain analogue bits to the antennas, with > digital backhaul and processing to improve linearity and dynamic range, for > example). We are not alone in our thinking here, but many in the CASPER > community have not caught on to the benefits of this yet. And, > unfortunately, it is a more expensive solution overall. At the moment, > there are no universal/community off-board ADCs. Everyone's got really > expensive, custom-designed ones (a possible gap in the CASPER lineup, > here?!). SKARAB (and the upcoming SKARAB-2) fit into this model, and so we > haven't really designed any ADCs to mate directly with them. At the moment, > there is actually one ADC that plugs directly into a SKARAB, but I'm not > sure it'd be appropriate for your needs, and it has no yellow block yet > (though Peralex promised to deliver one at the last CASPER workshop if a > customer wanted it). > > > > Another consideration is that the newer platforms (SKARAB, SNAP etc) are > using the new JASPER toolflow, which is under active development and has > support from the bigger CASPER developers. The older CASPER flow is still > supported on ROACH2s, but new features and bug-fixes are not being > back-ported and development there has stagnated. Since Xilinx will not > support Virtex 6 in Vivado, ROACH2 (and earlier boards) can never be > supported by all the new tools. I don't know of anyone using JASPER with > ISE (though, it was possible at one time). > > > > If you wanted to build your correlator out of ROACH2s or SKARABs (they > have similar processing capacities), a quick back-of-the-envelope, > worst-case calculation suggests that, for a 16 dual-pol, 4k channel 8-tap, > 2GHz BW correlator (I can almost hear the infomercial already: "act now, > and we'll throw in a free beamformer"; you'd fit a beam or two in the spare > capacity within the X-engine boards, FWIW), a packetised design would need > something like: > > > > 32x F-engine boards (though I'd say there's a good chance you could > squeeze it into 16 boards). > > 32x X-engine boards (30 if you don't want to process the band edges) > > 1x Arista 7250QX-64 or similar ~64 port 40G switch (or just a 32-port > switch with some loopback trickery, which would be much cheaper). > > > > It looks like you will be BRAM limited in both cases (otherwise you > could halve the board counts). You could also opt for some BRAM-saving > tradeoffs to ease fitment of two polarisations onto an F-engine. For > example you could drop down to a 4-tap PFB, or reduce the delay-correction > resolution (MeerKAT's specs, upon which I based the numbers above, are > overkill for most applications). If you're using a single network switch, > you might also be able to reduce packet buffer requirements, which > currently use BRAM, too. > > > > The beauty of this design is in its flexibility. You can access the raw, > intermediate data streams on the switch, swap out portions of the design > for computers/GPUs (eg LEDA and HERA), add more antennas incrementally, > increase or decreased processed bandwidth, change spectral resolution etc. > all with a quick parameter change and a recompile. > > > > Jason Manley > > Functional Manager: DSP > > SKA-SA > > > > Cell: +27 82 662 7726 > > Work: +27 21 506 7300 > > > > On 07 Nov 2017, at 20:02, Jonathan Weintroub <jweintr...@cfa.harvard.edu> > wrote: > > > >> Hi Dale, > >> > >> I’ll offer a few bullets on SWARM, the new SMA system. > >> > >> 1. SWARM is all open source and shared via CASPER and you are welcome > to use it as is, or develop it further to adapt it to a new application, > indeed it would be very pleasing to see the design used in some other > instrument. > >> > >> 2. There is a paper which is worth reading to understand what SWARM is > and what it does. Take a careful look if you are contemplating using the > design. > >> http://www.worldscientific.com/doi/pdf/10.1142/S2251171716410063 > >> You can get insight the paper without looking at the gory details of > source codes, both bitcodes and associated software, but if you want to dig > even deeper, sources are all shared here: > >> http://www.github.com/sma-wideband. > >> > >> 3. You are correct SWARM processes 2 GHz blocks of *usable” > bandwidth. The Nyquist band is somewhat wider, 2.288 GHz. That Nyquist > band is divided into 16,384 channels (not 1024), so in fact it exceeds > (rather than falls short of) your requirement for at least 4096 channels. > >> > >> 4. With all of the above the positive aspects, now comes the > cautionary remark: it is by no means trivial to expand SWARM from 8 dual > polarization antennas to 16 antennas. The X-engine would then have to > process roughly 4x the number of baselines as for SWARM. This may well > push the ROACH2 too far—we struggled to meet timing on the highly utilized > ROACH2 for SWARM (286 MHz FPGA fabric clock). > >> > >> We are also looking at porting SWARM to newer platforms, primarily to > expand bandwidth in the SMA’s case, rather than number of antennas. We > have also studied application of CASPER-like methods to ALMA, which of > course has far more than 8 antennas, but those studies were on paper, we > have yet to reduce to real design. Taking SWARM as-is (8 antennas 2 GHz > 16384 channels on ROACH2) is fairly simple. Expanding SWARM to 16 antennas > and/or porting to a new FPGA platform will be a significant project—the > SWARM design may be an excellent starting point, but even so. > >> > >> SKARAB is an interesting platform but doesn’t presently support the > appropriate ADC. Not sure about SNAP2 I’ll leave that assessment to others. > >> > >> Best wishes. > >> > >> Jonathan > >> > >> > >> > >>> On Nov 7, 2017, at 12:29 PM, Gary, Dale E. <dale.e.g...@njit.edu> > wrote: > >>> > >>> Dear Jonathan (and the rest of the CASPER list, in case anyone has > additional comments), > >>> > >>> I am looking into a new project that would require processing around 2 > GHz of bandwidth on of-order 10 (but more than 8) dual-polarization > antennas. Our science case calls for at least 4096 frequency channels. My > understanding is that the SMA correlator design is for a similar bandwidth, > for 8 dual-pol antennas, but 1024 channels or something similar. We do not > want to spend a lot of resources on correlator design, so my question is > whether it is possible and would it make sense to adapt the SMA design to a > 16-antenna, dual-pol, 4096-channel system, or whether it is better (or > necessary) to leave the ROACH-2 designs behind and move to one of the newer > platforms? If the latter, what digitizer bandwidths are available, and > which board (SNAP2, Scarab, others?) would be most appropriate to a new > project of this scope? > >>> > >>> Thanks, > >>> Dale > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups "casper@lists.berkeley.edu" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an email to casper+unsubscr...@lists.berkeley.edu. > >> To post to this group, send email to casper@lists.berkeley.edu. > > > -- You received this message because you are subscribed to the Google Groups "casper@lists.berkeley.edu" group. To unsubscribe from this group and stop receiving emails from it, send an email to casper+unsubscr...@lists.berkeley.edu. To post to this group, send email to casper@lists.berkeley.edu.
