Hi Dale The CASPER packetised correlator (that PAPER/HERA/KAT-7/MeerKAT etc are all using) would scale simply to your needs. You could easily build it out of ROACH2s or SKARABs at your scales. It would be a bit more expensive than the SWARM-type solution, in that it'd need additional hardware because it's not as resource-efficient as the SWARM design (you're going to struggle to beat SWARM with any other design!). A ROACH2 packetised solution would allow a number of different ADC options, including the the same one that SWARM uses.
SKA-SA is moving in a slightly different direction with ADCs for our current and future systems. We have moved the ADC off the digital processing boards in the interests of reducing noise, and increasing overall system performance (constrain analogue bits to the antennas, with digital backhaul and processing to improve linearity and dynamic range, for example). We are not alone in our thinking here, but many in the CASPER community have not caught on to the benefits of this yet. And, unfortunately, it is a more expensive solution overall. At the moment, there are no universal/community off-board ADCs. Everyone's got really expensive, custom-designed ones (a possible gap in the CASPER lineup, here?!). SKARAB (and the upcoming SKARAB-2) fit into this model, and so we haven't really designed any ADCs to mate directly with them. At the moment, there is actually one ADC that plugs directly into a SKARAB, but I'm not sure it'd be appropriate for your needs, and it has no yellow block yet (though Peralex promised to deliver one at the last CASPER workshop if a customer wanted it). Another consideration is that the newer platforms (SKARAB, SNAP etc) are using the new JASPER toolflow, which is under active development and has support from the bigger CASPER developers. The older CASPER flow is still supported on ROACH2s, but new features and bug-fixes are not being back-ported and development there has stagnated. Since Xilinx will not support Virtex 6 in Vivado, ROACH2 (and earlier boards) can never be supported by all the new tools. I don't know of anyone using JASPER with ISE (though, it was possible at one time). If you wanted to build your correlator out of ROACH2s or SKARABs (they have similar processing capacities), a quick back-of-the-envelope, worst-case calculation suggests that, for a 16 dual-pol, 4k channel 8-tap, 2GHz BW correlator (I can almost hear the infomercial already: "act now, and we'll throw in a free beamformer"; you'd fit a beam or two in the spare capacity within the X-engine boards, FWIW), a packetised design would need something like: 32x F-engine boards (though I'd say there's a good chance you could squeeze it into 16 boards). 32x X-engine boards (30 if you don't want to process the band edges) 1x Arista 7250QX-64 or similar ~64 port 40G switch (or just a 32-port switch with some loopback trickery, which would be much cheaper). It looks like you will be BRAM limited in both cases (otherwise you could halve the board counts). You could also opt for some BRAM-saving tradeoffs to ease fitment of two polarisations onto an F-engine. For example you could drop down to a 4-tap PFB, or reduce the delay-correction resolution (MeerKAT's specs, upon which I based the numbers above, are overkill for most applications). If you're using a single network switch, you might also be able to reduce packet buffer requirements, which currently use BRAM, too. The beauty of this design is in its flexibility. You can access the raw, intermediate data streams on the switch, swap out portions of the design for computers/GPUs (eg LEDA and HERA), add more antennas incrementally, increase or decreased processed bandwidth, change spectral resolution etc. all with a quick parameter change and a recompile. Jason Manley Functional Manager: DSP SKA-SA Cell: +27 82 662 7726 Work: +27 21 506 7300 On 07 Nov 2017, at 20:02, Jonathan Weintroub <[email protected]> wrote: > Hi Dale, > > I’ll offer a few bullets on SWARM, the new SMA system. > > 1. SWARM is all open source and shared via CASPER and you are welcome to use > it as is, or develop it further to adapt it to a new application, indeed it > would be very pleasing to see the design used in some other instrument. > > 2. There is a paper which is worth reading to understand what SWARM is and > what it does. Take a careful look if you are contemplating using the design. > http://www.worldscientific.com/doi/pdf/10.1142/S2251171716410063 > You can get insight the paper without looking at the gory details of source > codes, both bitcodes and associated software, but if you want to dig even > deeper, sources are all shared here: > http://www.github.com/sma-wideband. > > 3. You are correct SWARM processes 2 GHz blocks of *usable” bandwidth. The > Nyquist band is somewhat wider, 2.288 GHz. That Nyquist band is divided > into 16,384 channels (not 1024), so in fact it exceeds (rather than falls > short of) your requirement for at least 4096 channels. > > 4. With all of the above the positive aspects, now comes the cautionary > remark: it is by no means trivial to expand SWARM from 8 dual polarization > antennas to 16 antennas. The X-engine would then have to process roughly 4x > the number of baselines as for SWARM. This may well push the ROACH2 too > far—we struggled to meet timing on the highly utilized ROACH2 for SWARM (286 > MHz FPGA fabric clock). > > We are also looking at porting SWARM to newer platforms, primarily to expand > bandwidth in the SMA’s case, rather than number of antennas. We have also > studied application of CASPER-like methods to ALMA, which of course has far > more than 8 antennas, but those studies were on paper, we have yet to reduce > to real design. Taking SWARM as-is (8 antennas 2 GHz 16384 channels on > ROACH2) is fairly simple. Expanding SWARM to 16 antennas and/or porting to a > new FPGA platform will be a significant project—the SWARM design may be an > excellent starting point, but even so. > > SKARAB is an interesting platform but doesn’t presently support the > appropriate ADC. Not sure about SNAP2 I’ll leave that assessment to others. > > Best wishes. > > Jonathan > > > >> On Nov 7, 2017, at 12:29 PM, Gary, Dale E. <[email protected]> wrote: >> >> Dear Jonathan (and the rest of the CASPER list, in case anyone has >> additional comments), >> >> I am looking into a new project that would require processing around 2 GHz >> of bandwidth on of-order 10 (but more than 8) dual-polarization antennas. >> Our science case calls for at least 4096 frequency channels. My >> understanding is that the SMA correlator design is for a similar bandwidth, >> for 8 dual-pol antennas, but 1024 channels or something similar. We do not >> want to spend a lot of resources on correlator design, so my question is >> whether it is possible and would it make sense to adapt the SMA design to a >> 16-antenna, dual-pol, 4096-channel system, or whether it is better (or >> necessary) to leave the ROACH-2 designs behind and move to one of the newer >> platforms? If the latter, what digitizer bandwidths are available, and >> which board (SNAP2, Scarab, others?) would be most appropriate to a new >> project of this scope? >> >> Thanks, >> Dale > > > -- > You received this message because you are subscribed to the Google Groups > "[email protected]" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected].
