On Sun, 16 Aug 2020, 11:59 am Neil Salmon, <[email protected]> wrote:
> Hi Dan, > > > > I really appreciate the good points you make. > > > > Certainly, NLogN versus N^2 in processing power for the two approaches can > make a big difference. Some of the active (radar) security screening portal > systems currently use GPUs to the FT’s. It is challenging, as in the > near-field region it’s a 3-D imaging problem. I’m trying to figure the most > efficient algorithm to do this passively. > > > > For security screening portals I’m looking at hundreds of channels, if not > 1000. I think that number should be sufficient, and beyond that I think the > costs for the commercial security screening market will be too high. > Certainly for a next demo system I’d be looking at several hundred > channels. The forward simulation I have to generate the radiometric > E-fields from scenes can at least trial possible creation algorithms. Some > of the Chinese groups are building systems with 1000 channels. > > > > However, with the GPUs I understood these to be good at high data rate, > but only in a burst mode. If you had a sustained high data rate the > specifications of the PCI express (something like 60 gbps) bus could not be > realised over an extended period. On the basis of this I considered the > GPUs not optimal for real-time processing? Is this still the case? > Hi Neil, For "small" numbers of inputs, there's no reason not to be getting close to a sustained 90+ gbps of throughput for a bandwidth bound problem like correlation on a big GPU. This is certainly achievable in a radio astronomy setting where integration on the GPU is often long enough that the returning data rate is negligible. I'd recommend pulling xGPU from GitHub and running a quick benchmark if you have a gpu to hand. Keeping in mind that the code hasn't been modified in a couple of years, I think you might be pleasantly surprised. If you don't have a gpu to hand I'd be happy to take a minute to run a benchmark for you on my (ever-growing) collection of cards if you give me the system parameters. > > > Concerning the computation of a floating point correlation, I gather some > are using more efficient integer arithmetic for this, then perhaps storing > the result in a floating point accumulator? > Depends on the codebase / user. The 8-bit integer optimized xgpu outputs accumulated data as 32 bit ints, similar to Casper's xengine FPGA module. Can't speak for the various other implementations folks use. Cheers Jack > > Best wishes, > > Neil > > > > *From:* Dan Werthimer <[email protected]> > *Sent:* 15 August 2020 18:31 > *To:* CASPER Mailing List <[email protected]> > *Cc:* Danny Price <[email protected]> > *Subject:* Re: [casper] references to recent cross-correlator technology > developments > > > > > > Hi Neil, > > > > regarding using GPU's or FPGA's for the X engine of an FX correlator: > > > > You probably appreciate GPU's are best used in correlators with a large > number of antennas. > > The problem with using GPU's for a small number of antennas is the GPU is > limited by input data rate. > > > > The correlator input data rate is proportional to Nantenna, and the > correlator X engine computation rate is proportional to Nantenna^2. > > I think with modern GPUs you need about 1024 antennas before you aren't > I/O bound and can take advantage of the GPU's computational power; > > otherwise the GPU will be idle waiting for data. > > > > One way out of this problem is to find an application which can use the > GPU for additional tasks besides correlation - tasks that don't need much > I/O. > > Another possibility is to use cheap GPU's that can't do much computation, > but then the host computer dominates the cost and power comsumption of the > Xengine. > > > > FPGA's have a lot more I/O capacity and are lower power than GPU's, but as > you know, they are harder to program. > > > > I think most modern GPU's can do integer arithmetic with a selectable > number of bits, similar to FPGA's. > > But because floating point is easier, and many correlator GPU applications > are I/O bound, > > most people send their data as small fixed point numbers into the GPU for > correlation (eg: 4 bit real, 4 bit imag), but then compute the correlation > in floating point. > > > > Best Wishes, > > > > Dan > > > > > > On Fri, Aug 14, 2020 at 9:37 PM Neil Salmon <[email protected]> wrote: > > Hi Danny, > > > > GPU’s may be starting to rival FPGA’s in processing power for correlators. > However, are GPU’s restricted to long word correlation, say 32-bit floating > point, whereas in FPGA I’m assuming the bit length is variable, so you may > choose say 4-bit integer correlation – which would match to a 4-bit ADC. > > > > If you can’t adapt the word length for GPU correlators, then the FPGA > technology still has an edge, because as far as correlation is concerned, > I’m not sure if going to word lengths longer than 4-bit is resourceful use > of silicon real estate? > > > > What is your view on that? > > > > I did include a section on correlators in the document on security > screening imaging https://ieeexplore.ieee.org/document/9154708 and some > of your selected publications in the references – thank you. > > > > Cheers, > > Neil > > > > *From:* Danny Price <[email protected]> > *Sent:* 20 July 2020 14:44 > *To:* Neil Salmon <[email protected]>; [email protected] > *Subject:* RE: [casper] references to recent cross-correlator technology > developments > > > > Hi Neil, > > > > The correlation is indeed done in real time using stream processing > frameworks for most interferometer telescopes. Conversion from (very > sparse) visibilities to images is generally done offline (this can be very > time consuming!). > > > > There are a few real-time imaging systems: the EPIC correlator that Jack > mentioned, and the realfast system on the VLA ( > https://science.nrao.edu/facilities/vla/observing/realfast) are good > examples. > > > > Cheers, > > Danny > > On 20 July 2020 at 9:55:06 pm, Neil Salmon ([email protected]) wrote: > > Hi Danny, > > > > Thank you for these references. > > > > For security screening systems the name of the game is real-time, ie an > image in less than 1 second. However, I see a great many references to GPU > based correlators. I was used to seeing these devices as off-line > correlators, as in software correlators. Are the GPUs being used by the > radio astronomy community as real-time correlators, or as software > correlators? > > > > Many thanks, > > Neil > > > > *From:* Danny Price <[email protected]> > *Sent:* 20 July 2020 12:21 > *To:* [email protected] > *Subject:* Re: [casper] references to recent cross-correlator technology > developments > > > > Hi Neil, > > > > To add to Jack's post, allow me to plug some overview articles that may be > of interest. The first, https://arxiv.org/abs/1702.00442, was for an > introduction for a special issue of JAI on DSP in radio astronomy in 2016. > Table 1 summarises some of the larger correlators: the references therein > may be of use. Jack (et al)'s CASPER article in said JAI special issue is > also a font of references: https://arxiv.org/abs/1611.01826. The full > special issue article listing is up here: > https://www.worldscientific.com/toc/jai/05/04. > > > > More recently, here's my book chapter on real-time stream processing in > radio astronomy, https://arxiv.org/abs/1912.09041, which delves a bit > deeper into technical details for common approaches. > > > > In terms of cutting edge, there are various groups working with the Xilinx > RFSoC components for next-gen systems -- you will no doubt have seen some > traffic on this list. The ASKAP telescope group have plans to use an Alveo > Xilinx U280 accelerator card for high time resolution imaging + > dedispersion, which is an alternative to the GPU correlator. > > > > GPU correlators are still the most widespread for O(100) antennas. There's > some discussion on GPU correlator performance in J. Kocz et al 2014 ( > https://arxiv.org/abs/1401.8288); for O(100) inputs a GPU correlator will > likely be memory bandwidth bound. > > > > Cheers, > > Danny > > On 18 July 2020 at 7:54:49 pm, Neil Salmon ([email protected]) wrote: > > I need references on recent developments in cross-correlator technology > for an IEEE paper on the subject of aperture synthesis imaging in the area > of security screening of people for concealed weapons. Typical requirements > for this application are cross-correlators that can process in real-time > signals from hundreds of receiver channels with around 1 GHz of RF > bandwidth. As none of this technology is commercially available > off-the-shelf I’m dependent on the radio astronomy community to get the > latest information of correlator development. This might be just technical > knowhow on the building of correlators, or communities who would be willing > to supply for a fee correlators to a security screening technology > development company. > > > > Could anyone provide me with any references of papers on recent correlator > development that I could include in this paper? > > > > Many thanks, > > Neil > > "Before acting on this email or opening any attachments you should read > the Manchester Metropolitan University email disclaimer available on its > website http://www.mmu.ac.uk/emaildisclaimer " > > -- > You received this message because you are subscribed to the Google Groups " > [email protected]" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/7aff8f2f6ed3482a8283e2994bbd9fc6%40ASEX03.ad.mmu.ac.uk > <https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/7aff8f2f6ed3482a8283e2994bbd9fc6%40ASEX03.ad.mmu.ac.uk?utm_medium=email&utm_source=footer> > . > > -- > You received this message because you are subscribed to the Google Groups " > [email protected]" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAAtMgq%3D_nce72fbSUWBZ62EiCxW5ojUgy%2ByAbPXHMNRd%3DXbLDw%40mail.gmail.com > <https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAAtMgq%3D_nce72fbSUWBZ62EiCxW5ojUgy%2ByAbPXHMNRd%3DXbLDw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > -- > You received this message because you are subscribed to the Google Groups " > [email protected]" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/46372fbec8c34c909df5b72793ec1d09%40ASEX01.ad.mmu.ac.uk > <https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/46372fbec8c34c909df5b72793ec1d09%40ASEX01.ad.mmu.ac.uk?utm_medium=email&utm_source=footer> > . > > -- > You received this message because you are subscribed to the Google Groups " > [email protected]" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/f058e58f91134f8bacc604ae28c844d9%40ASEX01.ad.mmu.ac.uk > <https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/f058e58f91134f8bacc604ae28c844d9%40ASEX01.ad.mmu.ac.uk?utm_medium=email&utm_source=footer> > . > > -- > You received this message because you are subscribed to the Google Groups " > [email protected]" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAGHS_vGLW%3DsUTb3-tkm%2BLZ-3%3DUEGraF3%2BxDb5mfmZcF0-1HboA%40mail.gmail.com > <https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAGHS_vGLW%3DsUTb3-tkm%2BLZ-3%3DUEGraF3%2BxDb5mfmZcF0-1HboA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > -- > You received this message because you are subscribed to the Google Groups " > [email protected]" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/fd9a3dff541f4b05a9c133c6c1e19dc0%40ASEX01.ad.mmu.ac.uk > <https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/fd9a3dff541f4b05a9c133c6c1e19dc0%40ASEX01.ad.mmu.ac.uk?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAG1GKSnPEBigfB%2Bd5TKEfpJhYTijPU-p62ajn-waAc5YsYJgdw%40mail.gmail.com.
