Hi Jack, Again that’s for advice and the offer to run the benchmark. Let me learn more about GPU’s in the meantime and I’d hope to get back to you on this.
Best wishes, Neil From: Jack Hickish <[email protected]> Sent: 16 August 2020 12:18 To: casper <[email protected]> Cc: Danny Price <[email protected]> Subject: Re: [casper] references to recent cross-correlator technology developments On Sun, 16 Aug 2020, 11:59 am Neil Salmon, <[email protected]<mailto:[email protected]>> wrote: Hi Dan, I really appreciate the good points you make. Certainly, NLogN versus N^2 in processing power for the two approaches can make a big difference. Some of the active (radar) security screening portal systems currently use GPUs to the FT’s. It is challenging, as in the near-field region it’s a 3-D imaging problem. I’m trying to figure the most efficient algorithm to do this passively. For security screening portals I’m looking at hundreds of channels, if not 1000. I think that number should be sufficient, and beyond that I think the costs for the commercial security screening market will be too high. Certainly for a next demo system I’d be looking at several hundred channels. The forward simulation I have to generate the radiometric E-fields from scenes can at least trial possible creation algorithms. Some of the Chinese groups are building systems with 1000 channels. However, with the GPUs I understood these to be good at high data rate, but only in a burst mode. If you had a sustained high data rate the specifications of the PCI express (something like 60 gbps) bus could not be realised over an extended period. On the basis of this I considered the GPUs not optimal for real-time processing? Is this still the case? Hi Neil, For "small" numbers of inputs, there's no reason not to be getting close to a sustained 90+ gbps of throughput for a bandwidth bound problem like correlation on a big GPU. This is certainly achievable in a radio astronomy setting where integration on the GPU is often long enough that the returning data rate is negligible. I'd recommend pulling xGPU from GitHub and running a quick benchmark if you have a gpu to hand. Keeping in mind that the code hasn't been modified in a couple of years, I think you might be pleasantly surprised. If you don't have a gpu to hand I'd be happy to take a minute to run a benchmark for you on my (ever-growing) collection of cards if you give me the system parameters. Concerning the computation of a floating point correlation, I gather some are using more efficient integer arithmetic for this, then perhaps storing the result in a floating point accumulator? Depends on the codebase / user. The 8-bit integer optimized xgpu outputs accumulated data as 32 bit ints, similar to Casper's xengine FPGA module. Can't speak for the various other implementations folks use. Cheers Jack Best wishes, Neil From: Dan Werthimer <[email protected]<mailto:[email protected]>> Sent: 15 August 2020 18:31 To: CASPER Mailing List <[email protected]<mailto:[email protected]>> Cc: Danny Price <[email protected]<mailto:[email protected]>> Subject: Re: [casper] references to recent cross-correlator technology developments Hi Neil, regarding using GPU's or FPGA's for the X engine of an FX correlator: You probably appreciate GPU's are best used in correlators with a large number of antennas. The problem with using GPU's for a small number of antennas is the GPU is limited by input data rate. The correlator input data rate is proportional to Nantenna, and the correlator X engine computation rate is proportional to Nantenna^2. I think with modern GPUs you need about 1024 antennas before you aren't I/O bound and can take advantage of the GPU's computational power; otherwise the GPU will be idle waiting for data. One way out of this problem is to find an application which can use the GPU for additional tasks besides correlation - tasks that don't need much I/O. Another possibility is to use cheap GPU's that can't do much computation, but then the host computer dominates the cost and power comsumption of the Xengine. FPGA's have a lot more I/O capacity and are lower power than GPU's, but as you know, they are harder to program. I think most modern GPU's can do integer arithmetic with a selectable number of bits, similar to FPGA's. But because floating point is easier, and many correlator GPU applications are I/O bound, most people send their data as small fixed point numbers into the GPU for correlation (eg: 4 bit real, 4 bit imag), but then compute the correlation in floating point. Best Wishes, Dan On Fri, Aug 14, 2020 at 9:37 PM Neil Salmon <[email protected]<mailto:[email protected]>> wrote: Hi Danny, GPU’s may be starting to rival FPGA’s in processing power for correlators. However, are GPU’s restricted to long word correlation, say 32-bit floating point, whereas in FPGA I’m assuming the bit length is variable, so you may choose say 4-bit integer correlation – which would match to a 4-bit ADC. If you can’t adapt the word length for GPU correlators, then the FPGA technology still has an edge, because as far as correlation is concerned, I’m not sure if going to word lengths longer than 4-bit is resourceful use of silicon real estate? What is your view on that? I did include a section on correlators in the document on security screening imaging https://ieeexplore.ieee.org/document/9154708 and some of your selected publications in the references – thank you. Cheers, Neil From: Danny Price <[email protected]<mailto:[email protected]>> Sent: 20 July 2020 14:44 To: Neil Salmon <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: RE: [casper] references to recent cross-correlator technology developments Hi Neil, The correlation is indeed done in real time using stream processing frameworks for most interferometer telescopes. Conversion from (very sparse) visibilities to images is generally done offline (this can be very time consuming!). There are a few real-time imaging systems: the EPIC correlator that Jack mentioned, and the realfast system on the VLA (https://science.nrao.edu/facilities/vla/observing/realfast) are good examples. Cheers, Danny On 20 July 2020 at 9:55:06 pm, Neil Salmon ([email protected]<mailto:[email protected]>) wrote: Hi Danny, Thank you for these references. For security screening systems the name of the game is real-time, ie an image in less than 1 second. However, I see a great many references to GPU based correlators. I was used to seeing these devices as off-line correlators, as in software correlators. Are the GPUs being used by the radio astronomy community as real-time correlators, or as software correlators? Many thanks, Neil From: Danny Price <[email protected]<mailto:[email protected]>> Sent: 20 July 2020 12:21 To: [email protected]<mailto:[email protected]> Subject: Re: [casper] references to recent cross-correlator technology developments Hi Neil, To add to Jack's post, allow me to plug some overview articles that may be of interest. The first, https://arxiv.org/abs/1702.00442, was for an introduction for a special issue of JAI on DSP in radio astronomy in 2016. Table 1 summarises some of the larger correlators: the references therein may be of use. Jack (et al)'s CASPER article in said JAI special issue is also a font of references: https://arxiv.org/abs/1611.01826. The full special issue article listing is up here: https://www.worldscientific.com/toc/jai/05/04. More recently, here's my book chapter on real-time stream processing in radio astronomy, https://arxiv.org/abs/1912.09041, which delves a bit deeper into technical details for common approaches. In terms of cutting edge, there are various groups working with the Xilinx RFSoC components for next-gen systems -- you will no doubt have seen some traffic on this list. The ASKAP telescope group have plans to use an Alveo Xilinx U280 accelerator card for high time resolution imaging + dedispersion, which is an alternative to the GPU correlator. GPU correlators are still the most widespread for O(100) antennas. There's some discussion on GPU correlator performance in J. Kocz et al 2014 (https://arxiv.org/abs/1401.8288); for O(100) inputs a GPU correlator will likely be memory bandwidth bound. Cheers, Danny On 18 July 2020 at 7:54:49 pm, Neil Salmon ([email protected]<mailto:[email protected]>) wrote: I need references on recent developments in cross-correlator technology for an IEEE paper on the subject of aperture synthesis imaging in the area of security screening of people for concealed weapons. Typical requirements for this application are cross-correlators that can process in real-time signals from hundreds of receiver channels with around 1 GHz of RF bandwidth. As none of this technology is commercially available off-the-shelf I’m dependent on the radio astronomy community to get the latest information of correlator development. This might be just technical knowhow on the building of correlators, or communities who would be willing to supply for a fee correlators to a security screening technology development company. Could anyone provide me with any references of papers on recent correlator development that I could include in this paper? Many thanks, Neil "Before acting on this email or opening any attachments you should read the Manchester Metropolitan University email disclaimer available on its website http://www.mmu.ac.uk/emaildisclaimer " -- You received this message because you are subscribed to the Google Groups "[email protected]<mailto:[email protected]>" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To view this discussion on the web visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/7aff8f2f6ed3482a8283e2994bbd9fc6%40ASEX03.ad.mmu.ac.uk<https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/7aff8f2f6ed3482a8283e2994bbd9fc6%40ASEX03.ad.mmu.ac.uk?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "[email protected]<mailto:[email protected]>" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To view this discussion on the web visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAAtMgq%3D_nce72fbSUWBZ62EiCxW5ojUgy%2ByAbPXHMNRd%3DXbLDw%40mail.gmail.com<https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAAtMgq%3D_nce72fbSUWBZ62EiCxW5ojUgy%2ByAbPXHMNRd%3DXbLDw%40mail.gmail.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "[email protected]<mailto:[email protected]>" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To view this discussion on the web visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/46372fbec8c34c909df5b72793ec1d09%40ASEX01.ad.mmu.ac.uk<https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/46372fbec8c34c909df5b72793ec1d09%40ASEX01.ad.mmu.ac.uk?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "[email protected]<mailto:[email protected]>" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To view this discussion on the web visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/f058e58f91134f8bacc604ae28c844d9%40ASEX01.ad.mmu.ac.uk<https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/f058e58f91134f8bacc604ae28c844d9%40ASEX01.ad.mmu.ac.uk?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "[email protected]<mailto:[email protected]>" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To view this discussion on the web visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAGHS_vGLW%3DsUTb3-tkm%2BLZ-3%3DUEGraF3%2BxDb5mfmZcF0-1HboA%40mail.gmail.com<https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAGHS_vGLW%3DsUTb3-tkm%2BLZ-3%3DUEGraF3%2BxDb5mfmZcF0-1HboA%40mail.gmail.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "[email protected]<mailto:[email protected]>" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To view this discussion on the web visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/fd9a3dff541f4b05a9c133c6c1e19dc0%40ASEX01.ad.mmu.ac.uk<https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/fd9a3dff541f4b05a9c133c6c1e19dc0%40ASEX01.ad.mmu.ac.uk?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "[email protected]<mailto:[email protected]>" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To view this discussion on the web visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAG1GKSnPEBigfB%2Bd5TKEfpJhYTijPU-p62ajn-waAc5YsYJgdw%40mail.gmail.com<https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAG1GKSnPEBigfB%2Bd5TKEfpJhYTijPU-p62ajn-waAc5YsYJgdw%40mail.gmail.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/7d9794f07d36464793c10f6cd91f0950%40ASEX01.ad.mmu.ac.uk.
