I can think of three ways to operate non-2^N antennas... 1) "fake" F-engine inputs: This is what MeerKAT does right now when it has fewer than 2^N antennas. We just multicast copies of redundant the data into unused F-engine inputs. It doesn't save you any hardware and is wasteful, because you're still calculating everything even though you're not using it.
2) omit unused X-engine inputs: Saves you the unnecessary F-engine boards over option 1. MeerKAT operates like this during a failure (eg an F-engine board fails during an observation). The system will continue to run with missing boards; you'll just lose the relevant baselines or frequency channels. Right now, if you try'n run the system with missing F-engines, a number of error counters will tick over to indicate faults. You would probably have to fiddle a bit with the registers and/or monitoring&control software to tweak it for your needs and avoid reporting these "false" failures if you plan to operate it like this nominally. 3) By far the neatest solution (the one the engineer in me begs for!) would be to edit the design to natively support non-2^N antennas. There are a few things that would need to be changed where we've taken "shortcuts" by just slicing out binary bits to figure out address mappings (eg IP address generation, buffer address lookups etc) which you'd have to calculate properly for non-2^N powers. But it's definitely do-able if you're up for a bit of development work. In fact, the CASPER X-engine core itself is not bound by 2^N powers. You can already ask it to calculate baselines for any number of antennas. The reorder block that follows it, though, is locked to 2^N antennas, as are packet buffers. The F-engine's output packetiser (which distributes data to the X-engines) also employs a bit-slicing mechanism and would need to be modified. These are actually not big changes for someone familiar with the design. SKA-SA may tackle #3 anyway, if we end up building a digital backend for SKA-1 (it'll have 190-odd antennas). Jason On 08 Nov 2017, at 14:12, Gary, Dale E. <[email protected]> wrote: > Hi Jason, > > Thanks for the quick estimate. If we are targeting 10 antennas, can't the > correlator be made to work with only 10 F engines? The packets of the > missing F engines would just not arrive at the X-engine, and hence the > associated baselines would be zeroed or just ignored. We might also design > the X-engine not to send blank data, and hence reduce the size of the switch > needed. This might even allow us to halve the number of X-engine boards, but > I am not sure about that. > > A 4-tap PFB is fine, and we will do fine delay correction off the boards > (phase correction in software), so only coarse delay steps are needed. This > is what we do for EOVSA, very successfully. We will probably include > spectral kurtosis calculation, though, which means accumulating > power-squared. Still, I think we have a shot at 1 F-engine board per > antenna. > > Regards, > Dale > > On Wed, Nov 8, 2017 at 2:24 AM, Jason Manley <[email protected]> wrote: > Hi Dale > > The CASPER packetised correlator (that PAPER/HERA/KAT-7/MeerKAT etc are all > using) would scale simply to your needs. You could easily build it out of > ROACH2s or SKARABs at your scales. It would be a bit more expensive than the > SWARM-type solution, in that it'd need additional hardware because it's not > as resource-efficient as the SWARM design (you're going to struggle to beat > SWARM with any other design!). A ROACH2 packetised solution would allow a > number of different ADC options, including the the same one that SWARM uses. > > SKA-SA is moving in a slightly different direction with ADCs for our current > and future systems. We have moved the ADC off the digital processing boards > in the interests of reducing noise, and increasing overall system performance > (constrain analogue bits to the antennas, with digital backhaul and > processing to improve linearity and dynamic range, for example). We are not > alone in our thinking here, but many in the CASPER community have not caught > on to the benefits of this yet. And, unfortunately, it is a more expensive > solution overall. At the moment, there are no universal/community off-board > ADCs. Everyone's got really expensive, custom-designed ones (a possible gap > in the CASPER lineup, here?!). SKARAB (and the upcoming SKARAB-2) fit into > this model, and so we haven't really designed any ADCs to mate directly with > them. At the moment, there is actually one ADC that plugs directly into a > SKARAB, but I'm not sure it'd be appropriate for your needs, and it has no > yellow block yet (though Peralex promised to deliver one at the last CASPER > workshop if a customer wanted it). > > Another consideration is that the newer platforms (SKARAB, SNAP etc) are > using the new JASPER toolflow, which is under active development and has > support from the bigger CASPER developers. The older CASPER flow is still > supported on ROACH2s, but new features and bug-fixes are not being > back-ported and development there has stagnated. Since Xilinx will not > support Virtex 6 in Vivado, ROACH2 (and earlier boards) can never be > supported by all the new tools. I don't know of anyone using JASPER with ISE > (though, it was possible at one time). > > If you wanted to build your correlator out of ROACH2s or SKARABs (they have > similar processing capacities), a quick back-of-the-envelope, worst-case > calculation suggests that, for a 16 dual-pol, 4k channel 8-tap, 2GHz BW > correlator (I can almost hear the infomercial already: "act now, and we'll > throw in a free beamformer"; you'd fit a beam or two in the spare capacity > within the X-engine boards, FWIW), a packetised design would need something > like: > > 32x F-engine boards (though I'd say there's a good chance you could squeeze > it into 16 boards). > 32x X-engine boards (30 if you don't want to process the band edges) > 1x Arista 7250QX-64 or similar ~64 port 40G switch (or just a 32-port switch > with some loopback trickery, which would be much cheaper). > > It looks like you will be BRAM limited in both cases (otherwise you could > halve the board counts). You could also opt for some BRAM-saving tradeoffs to > ease fitment of two polarisations onto an F-engine. For example you could > drop down to a 4-tap PFB, or reduce the delay-correction resolution > (MeerKAT's specs, upon which I based the numbers above, are overkill for most > applications). If you're using a single network switch, you might also be > able to reduce packet buffer requirements, which currently use BRAM, too. > > The beauty of this design is in its flexibility. You can access the raw, > intermediate data streams on the switch, swap out portions of the design for > computers/GPUs (eg LEDA and HERA), add more antennas incrementally, increase > or decreased processed bandwidth, change spectral resolution etc. all with a > quick parameter change and a recompile. > > Jason Manley > Functional Manager: DSP > SKA-SA > > Cell: +27 82 662 7726 > Work: +27 21 506 7300 > > On 07 Nov 2017, at 20:02, Jonathan Weintroub <[email protected]> > wrote: > > > Hi Dale, > > > > I’ll offer a few bullets on SWARM, the new SMA system. > > > > 1. SWARM is all open source and shared via CASPER and you are welcome to > > use it as is, or develop it further to adapt it to a new application, > > indeed it would be very pleasing to see the design used in some other > > instrument. > > > > 2. There is a paper which is worth reading to understand what SWARM is and > > what it does. Take a careful look if you are contemplating using the > > design. > > http://www.worldscientific.com/doi/pdf/10.1142/S2251171716410063 > > You can get insight the paper without looking at the gory details of source > > codes, both bitcodes and associated software, but if you want to dig even > > deeper, sources are all shared here: > > http://www.github.com/sma-wideband. > > > > 3. You are correct SWARM processes 2 GHz blocks of *usable” bandwidth. > > The Nyquist band is somewhat wider, 2.288 GHz. That Nyquist band is > > divided into 16,384 channels (not 1024), so in fact it exceeds (rather than > > falls short of) your requirement for at least 4096 channels. > > > > 4. With all of the above the positive aspects, now comes the cautionary > > remark: it is by no means trivial to expand SWARM from 8 dual polarization > > antennas to 16 antennas. The X-engine would then have to process roughly > > 4x the number of baselines as for SWARM. This may well push the ROACH2 too > > far—we struggled to meet timing on the highly utilized ROACH2 for SWARM > > (286 MHz FPGA fabric clock). > > > > We are also looking at porting SWARM to newer platforms, primarily to > > expand bandwidth in the SMA’s case, rather than number of antennas. We > > have also studied application of CASPER-like methods to ALMA, which of > > course has far more than 8 antennas, but those studies were on paper, we > > have yet to reduce to real design. Taking SWARM as-is (8 antennas 2 GHz > > 16384 channels on ROACH2) is fairly simple. Expanding SWARM to 16 antennas > > and/or porting to a new FPGA platform will be a significant project—the > > SWARM design may be an excellent starting point, but even so. > > > > SKARAB is an interesting platform but doesn’t presently support the > > appropriate ADC. Not sure about SNAP2 I’ll leave that assessment to others. > > > > Best wishes. > > > > Jonathan > > > > > > > >> On Nov 7, 2017, at 12:29 PM, Gary, Dale E. <[email protected]> wrote: > >> > >> Dear Jonathan (and the rest of the CASPER list, in case anyone has > >> additional comments), > >> > >> I am looking into a new project that would require processing around 2 GHz > >> of bandwidth on of-order 10 (but more than 8) dual-polarization antennas. > >> Our science case calls for at least 4096 frequency channels. My > >> understanding is that the SMA correlator design is for a similar > >> bandwidth, for 8 dual-pol antennas, but 1024 channels or something > >> similar. We do not want to spend a lot of resources on correlator design, > >> so my question is whether it is possible and would it make sense to adapt > >> the SMA design to a 16-antenna, dual-pol, 4096-channel system, or whether > >> it is better (or necessary) to leave the ROACH-2 designs behind and move > >> to one of the newer platforms? If the latter, what digitizer bandwidths > >> are available, and which board (SNAP2, Scarab, others?) would be most > >> appropriate to a new project of this scope? > >> > >> Thanks, > >> Dale > > > > > > -- > > You received this message because you are subscribed to the Google Groups > > "[email protected]" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [email protected]. > > To post to this group, send email to [email protected]. > > -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected].
