Hi Jason, Thanks for the quick estimate. If we are targeting 10 antennas, can't the correlator be made to work with only 10 F engines? The packets of the missing F engines would just not arrive at the X-engine, and hence the associated baselines would be zeroed or just ignored. We might also design the X-engine not to send blank data, and hence reduce the size of the switch needed. This might even allow us to halve the number of X-engine boards, but I am not sure about that.
A 4-tap PFB is fine, and we will do fine delay correction off the boards (phase correction in software), so only coarse delay steps are needed. This is what we do for EOVSA, very successfully. We will probably include spectral kurtosis calculation, though, which means accumulating power-squared. Still, I think we have a shot at 1 F-engine board per antenna. Regards, Dale On Wed, Nov 8, 2017 at 2:24 AM, Jason Manley <[email protected]> wrote: > Hi Dale > > The CASPER packetised correlator (that PAPER/HERA/KAT-7/MeerKAT etc are > all using) would scale simply to your needs. You could easily build it out > of ROACH2s or SKARABs at your scales. It would be a bit more expensive than > the SWARM-type solution, in that it'd need additional hardware because it's > not as resource-efficient as the SWARM design (you're going to struggle to > beat SWARM with any other design!). A ROACH2 packetised solution would > allow a number of different ADC options, including the the same one that > SWARM uses. > > SKA-SA is moving in a slightly different direction with ADCs for our > current and future systems. We have moved the ADC off the digital > processing boards in the interests of reducing noise, and increasing > overall system performance (constrain analogue bits to the antennas, with > digital backhaul and processing to improve linearity and dynamic range, for > example). We are not alone in our thinking here, but many in the CASPER > community have not caught on to the benefits of this yet. And, > unfortunately, it is a more expensive solution overall. At the moment, > there are no universal/community off-board ADCs. Everyone's got really > expensive, custom-designed ones (a possible gap in the CASPER lineup, > here?!). SKARAB (and the upcoming SKARAB-2) fit into this model, and so we > haven't really designed any ADCs to mate directly with them. At the moment, > there is actually one ADC that plugs directly into a SKARAB, but I'm not > sure it'd be appropriate for your needs, and it has no yellow block yet > (though Peralex promised to deliver one at the last CASPER workshop if a > customer wanted it). > > Another consideration is that the newer platforms (SKARAB, SNAP etc) are > using the new JASPER toolflow, which is under active development and has > support from the bigger CASPER developers. The older CASPER flow is still > supported on ROACH2s, but new features and bug-fixes are not being > back-ported and development there has stagnated. Since Xilinx will not > support Virtex 6 in Vivado, ROACH2 (and earlier boards) can never be > supported by all the new tools. I don't know of anyone using JASPER with > ISE (though, it was possible at one time). > > If you wanted to build your correlator out of ROACH2s or SKARABs (they > have similar processing capacities), a quick back-of-the-envelope, > worst-case calculation suggests that, for a 16 dual-pol, 4k channel 8-tap, > 2GHz BW correlator (I can almost hear the infomercial already: "act now, > and we'll throw in a free beamformer"; you'd fit a beam or two in the spare > capacity within the X-engine boards, FWIW), a packetised design would need > something like: > > 32x F-engine boards (though I'd say there's a good chance you could > squeeze it into 16 boards). > 32x X-engine boards (30 if you don't want to process the band edges) > 1x Arista 7250QX-64 or similar ~64 port 40G switch (or just a 32-port > switch with some loopback trickery, which would be much cheaper). > > It looks like you will be BRAM limited in both cases (otherwise you could > halve the board counts). You could also opt for some BRAM-saving tradeoffs > to ease fitment of two polarisations onto an F-engine. For example you > could drop down to a 4-tap PFB, or reduce the delay-correction resolution > (MeerKAT's specs, upon which I based the numbers above, are overkill for > most applications). If you're using a single network switch, you might also > be able to reduce packet buffer requirements, which currently use BRAM, too. > > The beauty of this design is in its flexibility. You can access the raw, > intermediate data streams on the switch, swap out portions of the design > for computers/GPUs (eg LEDA and HERA), add more antennas incrementally, > increase or decreased processed bandwidth, change spectral resolution etc. > all with a quick parameter change and a recompile. > > Jason Manley > Functional Manager: DSP > SKA-SA > > Cell: +27 82 662 7726 > Work: +27 21 506 7300 > > On 07 Nov 2017, at 20:02, Jonathan Weintroub <[email protected]> > wrote: > > > Hi Dale, > > > > I’ll offer a few bullets on SWARM, the new SMA system. > > > > 1. SWARM is all open source and shared via CASPER and you are welcome > to use it as is, or develop it further to adapt it to a new application, > indeed it would be very pleasing to see the design used in some other > instrument. > > > > 2. There is a paper which is worth reading to understand what SWARM is > and what it does. Take a careful look if you are contemplating using the > design. > > http://www.worldscientific.com/doi/pdf/10.1142/S2251171716410063 > > You can get insight the paper without looking at the gory details of > source codes, both bitcodes and associated software, but if you want to dig > even deeper, sources are all shared here: > > http://www.github.com/sma-wideband. > > > > 3. You are correct SWARM processes 2 GHz blocks of *usable” bandwidth. > The Nyquist band is somewhat wider, 2.288 GHz. That Nyquist band is > divided into 16,384 channels (not 1024), so in fact it exceeds (rather than > falls short of) your requirement for at least 4096 channels. > > > > 4. With all of the above the positive aspects, now comes the cautionary > remark: it is by no means trivial to expand SWARM from 8 dual polarization > antennas to 16 antennas. The X-engine would then have to process roughly > 4x the number of baselines as for SWARM. This may well push the ROACH2 too > far—we struggled to meet timing on the highly utilized ROACH2 for SWARM > (286 MHz FPGA fabric clock). > > > > We are also looking at porting SWARM to newer platforms, primarily to > expand bandwidth in the SMA’s case, rather than number of antennas. We > have also studied application of CASPER-like methods to ALMA, which of > course has far more than 8 antennas, but those studies were on paper, we > have yet to reduce to real design. Taking SWARM as-is (8 antennas 2 GHz > 16384 channels on ROACH2) is fairly simple. Expanding SWARM to 16 antennas > and/or porting to a new FPGA platform will be a significant project—the > SWARM design may be an excellent starting point, but even so. > > > > SKARAB is an interesting platform but doesn’t presently support the > appropriate ADC. Not sure about SNAP2 I’ll leave that assessment to others. > > > > Best wishes. > > > > Jonathan > > > > > > > >> On Nov 7, 2017, at 12:29 PM, Gary, Dale E. <[email protected]> > wrote: > >> > >> Dear Jonathan (and the rest of the CASPER list, in case anyone has > additional comments), > >> > >> I am looking into a new project that would require processing around 2 > GHz of bandwidth on of-order 10 (but more than 8) dual-polarization > antennas. Our science case calls for at least 4096 frequency channels. My > understanding is that the SMA correlator design is for a similar bandwidth, > for 8 dual-pol antennas, but 1024 channels or something similar. We do not > want to spend a lot of resources on correlator design, so my question is > whether it is possible and would it make sense to adapt the SMA design to a > 16-antenna, dual-pol, 4096-channel system, or whether it is better (or > necessary) to leave the ROACH-2 designs behind and move to one of the newer > platforms? If the latter, what digitizer bandwidths are available, and > which board (SNAP2, Scarab, others?) would be most appropriate to a new > project of this scope? > >> > >> Thanks, > >> Dale > > > > > > -- > > You received this message because you are subscribed to the Google > Groups "[email protected]" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected]. > > To post to this group, send email to [email protected]. > > -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected].
