[casper] building 300-receiver channel cross-correlator

[Neil Salmon]

Anyone help?

I'm working in academia and need to build a 300-receiver channel single-bit 
digitiser / cross-correlator with a single frequency channel having a bandwidth 
of 300 MHz, centre frequency ~3 GHz. The single bit digitisers sample I 
giving a total data rate of 180 Gbps and using XOR gates to do the 
cross-correlations, the total computation rate is 54 T XOR operations per 
second. I need to accumulate cross-correlations typically for times ranging 
from 10 ms to a few seconds. The system would comprise an array of single bit 
digitisers linked via a high speed data bus to FPGA boards for the 
cross-correlation/accumulation. I've no skills in board design but could 
probably learn VHDL. I don't have funding to commission a design and build but 
wondered if anyone in this community could advise how I should go about 
building this system at our university.

Thank you for any help you can provide.

Neil
"Before acting on this email or opening any attachments you should read the 
Manchester Metropolitan University email disclaimer available on its website 
http://www.mmu.ac.uk/emaildisclaimer "

Re: [casper] building 300-receiver channel cross-correlator

[Jack Hickish]

Hi Neil,

A bit more information would be useful, but it sounds like if you could
construct a ZDOK card that interfaced some (40, one per differential pair?)
of your digitizers to a ROACH board you could use a handful of ROACH boards
to perform all of the cross multiplication and accumulation and interface
with CPU data recorders / post-processors.

Jack

On Fri, 18 Dec 2015 at 14:26 James Smith  wrote:

> Hello Neil,
>
> CASPER tools could probably do what you're looking for, but I found your
> description a bit confusing. You're going to need to clarify somewhat.
>
> Regards,
> James
>
>
> On Fri, Dec 18, 2015 at 4:15 PM, Neil Salmon  wrote:
>
>> Anyone help?
>>
>>
>>
>> I’m working in academia and need to build a 300-receiver channel
>> single-bit digitiser / cross-correlator with a single frequency channel
>> having a bandwidth of 300 MHz, centre frequency ~3 GHz. The single bit
>> digitisers sample I giving a total data rate of 180 Gbps and using XOR
>> gates to do the cross-correlations, the total computation rate is 54 T XOR
>> operations per second. I need to accumulate cross-correlations typically
>> for times ranging from 10 ms to a few seconds. The system would comprise an
>> array of single bit digitisers linked via a high speed data bus to FPGA
>> boards for the cross-correlation/accumulation. I’ve no skills in board
>> design but could probably learn VHDL. I don’t have funding to commission a
>> design and build but wondered if anyone in this community could advise how
>> I should go about building this system at our university.
>>
>>
>>
>> Thank you for any help you can provide.
>>
>>
>>
>> Neil
>> "Before acting on this email or opening any attachments you should read
>> the Manchester Metropolitan University email disclaimer available on its
>> website http://www.mmu.ac.uk/emaildisclaimer "
>>
>
>

Re: [casper] building 300-receiver channel cross-correlator

[John Ford]

> Anyone help?
>
> I'm working in academia and need to build a 300-receiver channel
> single-bit digitiser / cross-correlator with a single frequency channel
> having a bandwidth of 300 MHz, centre frequency ~3 GHz. The single bit
> digitisers sample I giving a total data rate of 180 Gbps and using XOR
> gates to do the cross-correlations, the total computation rate is 54 T XOR
> operations per second. I need to accumulate cross-correlations typically
> for times ranging from 10 ms to a few seconds. The system would comprise
> an array of single bit digitisers linked via a high speed data bus to FPGA
> boards for the cross-correlation/accumulation. I've no skills in board
> design but could probably learn VHDL. I don't have funding to commission a
> design and build but wondered if anyone in this community could advise how
> I should go about building this system at our university.

Hi Neal.  I think the main problem you face is the I/O.  I'm assuming your
sample rate is 300 ms/s?  Clocking the FPGA at 300 MHz is pushing it, but
not completely insane.  The 180 Gbps is above the capacity of the Roach-2,
but maybe R-3 can do it.  There are other off the shelf hardware platforms
you could look at, like pico computing that might have enough I/O.

Dumping the output at 10 ms shouldn't be a problem, as that's only 3.6
Mbps at a 4 bit width, or ~29 Mbps at 32 bits wide.

Do you already have the digitizers?  That's probably a nice bit of
engineering in itself.  There has been some work on using the Xilinx I/O
block bits as single-bit ADCs.  I don't remember the details.

John

>
> Thank you for any help you can provide.
>
> Neil
> "Before acting on this email or opening any attachments you should read
> the Manchester Metropolitan University email disclaimer available on its
> website http://www.mmu.ac.uk/emaildisclaimer "
>

Re: [casper] building 300-receiver channel cross-correlator

[Neil Salmon]

Jack,
Thanks for help. Do you have any idea of the I/O capacity of a single Roach2 
board – just trying to figure out how many I may need?
Thank you,
Neil

From: Jack Hickish [mailto:jackhick...@gmail.com]
Sent: 18 December 2015 15:08
To: James Smith; Neil Salmon
Cc: casper@lists.berkeley.edu
Subject: Re: [casper] building 300-receiver channel cross-correlator

Hi Neil,

A bit more information would be useful, but it sounds like if you could 
construct a ZDOK card that interfaced some (40, one per differential pair?) of 
your digitizers to a ROACH board you could use a handful of ROACH boards to 
perform all of the cross multiplication and accumulation and interface with CPU 
data recorders / post-processors.

Jack

On Fri, 18 Dec 2015 at 14:26 James Smith 
> wrote:
Hello Neil,

CASPER tools could probably do what you're looking for, but I found your 
description a bit confusing. You're going to need to clarify somewhat.

Regards,
James


On Fri, Dec 18, 2015 at 4:15 PM, Neil Salmon 
> wrote:
Anyone help?

I’m working in academia and need to build a 300-receiver channel single-bit 
digitiser / cross-correlator with a single frequency channel having a bandwidth 
of 300 MHz, centre frequency ~3 GHz. The single bit digitisers sample I 
giving a total data rate of 180 Gbps and using XOR gates to do the 
cross-correlations, the total computation rate is 54 T XOR operations per 
second. I need to accumulate cross-correlations typically for times ranging 
from 10 ms to a few seconds. The system would comprise an array of single bit 
digitisers linked via a high speed data bus to FPGA boards for the 
cross-correlation/accumulation. I’ve no skills in board design but could 
probably learn VHDL. I don’t have funding to commission a design and build but 
wondered if anyone in this community could advise how I should go about 
building this system at our university.

Thank you for any help you can provide.

Neil
"Before acting on this email or opening any attachments you should read the 
Manchester Metropolitan University email disclaimer available on its website 
http://www.mmu.ac.uk/emaildisclaimer "

"Before acting on this email or opening any attachments you should read the 
Manchester Metropolitan University email disclaimer available on its website 
http://www.mmu.ac.uk/emaildisclaimer "

Re: [casper] building 300-receiver channel cross-correlator

[Dan Werthimer]

hi neil,

as john pointed out, you can use the differential inputs of an FPGA
as a 1 bit ADC if you'd like.
there's a nice write up on this by paul horowitz and his students
who used xilinx fpga's as adc's.

if you want to use the fpga as the digiitzers,
you will need 600 LVDS inputs at 300 Msps, which means 600 pairs, or 1200
pins.
i think some of the larger fpga's will do this - you'll need to check.
 (eg: a virtex ultrascale).
or you could split up the data over several chips (see below):

a larger problem is that you need 300^2/2 complex 1 bit multipliers and
accumulators.
that probably will not fit on a single FPGA.   you might need to break this
into four FPGA's.
i suggest you develop a complex multiplier accumulator in simulink or HDL,
and plunk down 300^2/2 of them, and compile them into a large ultrascale
FGPA
and see if it will fit into 1 fgpa?  or  4?   or 9?   or 16?

best wishes,

dan


On Fri, Dec 18, 2015 at 8:23 AM, Neil Salmon  wrote:

> Thank you for your response. The system is part of a generic
> microwave/mm-wave aperture synthesis imaging system, so there’s an array of
> front-end heterodyne receivers with an IF earmarked at a centre frequency
> of 3 GHz (away from Wifi mobile comms), but the bandwidth is 300 MHz.
> Front-end initially may be receiving at a centre frequency of 20 GHz, but I
> could change this to 10 GHz or go up to 35 GHz. (I’ll be taking a single
> polarisation say horizontal or right-hand circular – I’ve not decided yet)
>
>
>
> Either way, I’ll need to digitise this 300 MHz bandwidth on each channel,
> and I’m quite happy with the loss in SNR in using a single bit
> digitisation, so satisfying the Nyquist criterion there will be I & Q
> channels, each generating a data stream at 300 M samples per second, ie a
> total of 600 Mbps for both I per receiver channel, giving the total data
> rate of 180 Gbps. (sampling clock and mixing LO’s will be synched to a
> master oscillator)
>
>
>
> (as for the 3 GHz centre frequency the I digitisation could be bandpass
> sampling/digital down conversion or a second analogue downshift using a
> matched pair of mixers and then comparators in each section to generate I
> and Q digits)
>
>
>
> So there will be this huge rate of I & Q data from 300 channels that needs
> to be cross-correlated in real-time with 95% duty cycle to avoid loss of
> SNR. (software correlation would generate just too much data for harddisk
> and a GPU PCIe bus solution couldn’t cope with the data rate – or at least
> I’d be uncomfortable about working close to data rate ceilings of PCIe.)
> That leaves the FPGA solution. So I need some high speed data bus to get
> the data into the FPGAs for cross-correlation. As I’m working single bits
> XOR gates will do nicely for the cross-multiplies and I want to store the
> four components of the cross-multiply in separate registers, just for
> diagnostic / trouble shooting. This gives the XOR op rate 54 T ops/sec and
> the requirement for the 180,000 accumulation registers.
>
>
>
> For me the challenges with be getting the arrays of single bit digitisers
> and linking them to the cross-correlators and doing the cross-correlation
> at this huge rate. Build of analogue front end heterodyne array and image
> formation algorithms I’ve done before. It’s just the digital hardware I
> need to sort. I’ve got a few researchers and postgrads around me in the
> engineering department who have general educational interests in FPGA
> technologies and the Xilinx and Altera University representative support
> under the university agreement. So I’m just wondering if I can do this with
> the Casper tools or others if necessary.
>
>
>
> Hope this extra information help.  Many thanks for your help.
>
> Neil
>
>
>
> *From:* James Smith [mailto:jsm...@ska.ac.za]
> *Sent:* 18 December 2015 14:25
> *To:* Neil Salmon
> *Cc:* casper@lists.berkeley.edu
> *Subject:* Re: [casper] building 300-receiver channel cross-correlator
>
>
>
> Hello Neil,
>
>
>
> CASPER tools could probably do what you're looking for, but I found your
> description a bit confusing. You're going to need to clarify somewhat.
>
>
>
> Regards,
>
> James
>
>
>
>
>
> On Fri, Dec 18, 2015 at 4:15 PM, Neil Salmon  wrote:
>
> Anyone help?
>
>
>
> I’m working in academia and need to build a 300-receiver channel
> single-bit digitiser / cross-correlator with a single frequency channel
> having a bandwidth of 300 MHz, centre frequency ~3 GHz. The single bit
> digitisers sample I giving a total data rate of 180 Gbps and using XOR
> gates to do the cross-correlations, the total computation rate is 54 T XOR
> operations per second. I need to accumulate cross-correlations typically
> for times ranging from 10 ms to a few seconds. The system would comprise an
> array of single bit digitisers linked via a high speed data bus to FPGA
> boards for the cross-correlation/accumulation. I’ve no skills in board
> design but could probably learn VHDL. I 

Re: [casper] building 300-receiver channel cross-correlator

[Neil Salmon]

John, yes sample rate is 300 Msps on both I & Q sections, ie total 600 Msps per 
channel, a massive rate when you've got 300 channels. As I understand if you 
put a comparator in front to the FPGA serial link it cleans up the 
digitisation. But no I don't yet have these. I need to get the basic building 
blocks right first then fill in the gaps. I'll check out the pico computing.
Thanks for help.
Neil

-Original Message-
From: John Ford [mailto:jf...@nrao.edu]
Sent: 18 December 2015 15:06
To: Neil Salmon
Cc: casper@lists.berkeley.edu
Subject: Re: [casper] building 300-receiver channel cross-correlator

> Anyone help?
>
> I'm working in academia and need to build a 300-receiver channel
> single-bit digitiser / cross-correlator with a single frequency
> channel having a bandwidth of 300 MHz, centre frequency ~3 GHz. The
> single bit digitisers sample I giving a total data rate of 180 Gbps
> and using XOR gates to do the cross-correlations, the total
> computation rate is 54 T XOR operations per second. I need to
> accumulate cross-correlations typically for times ranging from 10 ms
> to a few seconds. The system would comprise an array of single bit
> digitisers linked via a high speed data bus to FPGA boards for the
> cross-correlation/accumulation. I've no skills in board design but
> could probably learn VHDL. I don't have funding to commission a design
> and build but wondered if anyone in this community could advise how I should 
> go about building this system at our university.

Hi Neal.  I think the main problem you face is the I/O.  I'm assuming your 
sample rate is 300 ms/s?  Clocking the FPGA at 300 MHz is pushing it, but not 
completely insane.  The 180 Gbps is above the capacity of the Roach-2, but 
maybe R-3 can do it.  There are other off the shelf hardware platforms you 
could look at, like pico computing that might have enough I/O.

Dumping the output at 10 ms shouldn't be a problem, as that's only 3.6 Mbps at 
a 4 bit width, or ~29 Mbps at 32 bits wide.

Do you already have the digitizers?  That's probably a nice bit of engineering 
in itself.  There has been some work on using the Xilinx I/O block bits as 
single-bit ADCs.  I don't remember the details.

John

>
> Thank you for any help you can provide.
>
> Neil
> "Before acting on this email or opening any attachments you should
> read the Manchester Metropolitan University email disclaimer available
> on its website http://www.mmu.ac.uk/emaildisclaimer "
>


"Before acting on this email or opening any attachments you should read the 
Manchester Metropolitan University email disclaimer available on its website 
http://www.mmu.ac.uk/emaildisclaimer "

Re: [casper] building 300-receiver channel cross-correlator

[Neil Salmon]

Thank you for your response. The system is part of a generic microwave/mm-wave 
aperture synthesis imaging system, so there’s an array of front-end heterodyne 
receivers with an IF earmarked at a centre frequency of 3 GHz (away from Wifi 
mobile comms), but the bandwidth is 300 MHz. Front-end initially may be 
receiving at a centre frequency of 20 GHz, but I could change this to 10 GHz or 
go up to 35 GHz. (I’ll be taking a single polarisation say horizontal or 
right-hand circular – I’ve not decided yet)

Either way, I’ll need to digitise this 300 MHz bandwidth on each channel, and 
I’m quite happy with the loss in SNR in using a single bit digitisation, so 
satisfying the Nyquist criterion there will be I & Q channels, each generating 
a data stream at 300 M samples per second, ie a total of 600 Mbps for both I 
per receiver channel, giving the total data rate of 180 Gbps. (sampling clock 
and mixing LO’s will be synched to a master oscillator)

(as for the 3 GHz centre frequency the I digitisation could be bandpass 
sampling/digital down conversion or a second analogue downshift using a matched 
pair of mixers and then comparators in each section to generate I and Q digits)

So there will be this huge rate of I & Q data from 300 channels that needs to 
be cross-correlated in real-time with 95% duty cycle to avoid loss of SNR. 
(software correlation would generate just too much data for harddisk and a GPU 
PCIe bus solution couldn’t cope with the data rate – or at least I’d be 
uncomfortable about working close to data rate ceilings of PCIe.) That leaves 
the FPGA solution. So I need some high speed data bus to get the data into the 
FPGAs for cross-correlation. As I’m working single bits XOR gates will do 
nicely for the cross-multiplies and I want to store the four components of the 
cross-multiply in separate registers, just for diagnostic / trouble shooting. 
This gives the XOR op rate 54 T ops/sec and the requirement for the 180,000 
accumulation registers.

For me the challenges with be getting the arrays of single bit digitisers and 
linking them to the cross-correlators and doing the cross-correlation at this 
huge rate. Build of analogue front end heterodyne array and image formation 
algorithms I’ve done before. It’s just the digital hardware I need to sort. 
I’ve got a few researchers and postgrads around me in the engineering 
department who have general educational interests in FPGA technologies and the 
Xilinx and Altera University representative support under the university 
agreement. So I’m just wondering if I can do this with the Casper tools or 
others if necessary.

Hope this extra information help.  Many thanks for your help.
Neil

From: James Smith [mailto:jsm...@ska.ac.za]
Sent: 18 December 2015 14:25
To: Neil Salmon
Cc: casper@lists.berkeley.edu
Subject: Re: [casper] building 300-receiver channel cross-correlator

Hello Neil,

CASPER tools could probably do what you're looking for, but I found your 
description a bit confusing. You're going to need to clarify somewhat.

Regards,
James


On Fri, Dec 18, 2015 at 4:15 PM, Neil Salmon 
> wrote:
Anyone help?

I’m working in academia and need to build a 300-receiver channel single-bit 
digitiser / cross-correlator with a single frequency channel having a bandwidth 
of 300 MHz, centre frequency ~3 GHz. The single bit digitisers sample I 
giving a total data rate of 180 Gbps and using XOR gates to do the 
cross-correlations, the total computation rate is 54 T XOR operations per 
second. I need to accumulate cross-correlations typically for times ranging 
from 10 ms to a few seconds. The system would comprise an array of single bit 
digitisers linked via a high speed data bus to FPGA boards for the 
cross-correlation/accumulation. I’ve no skills in board design but could 
probably learn VHDL. I don’t have funding to commission a design and build but 
wondered if anyone in this community could advise how I should go about 
building this system at our university.

Thank you for any help you can provide.

Neil
"Before acting on this email or opening any attachments you should read the 
Manchester Metropolitan University email disclaimer available on its website 
http://www.mmu.ac.uk/emaildisclaimer "

"Before acting on this email or opening any attachments you should read the 
Manchester Metropolitan University email disclaimer available on its website 
http://www.mmu.ac.uk/emaildisclaimer "

Re: [casper] building 300-receiver channel cross-correlator

[Rich Lacasse]

Hi Neal,

Some of this may be obvious to you, but I pass it along for what it's worth.


Please pardon my ASCII.  I'd like this to make it through people's spam 
filters!


As I'm sure you're aware, the general approach to correlating many 
antennas together

tends to favor a triangular processing architecture as shown below.

 Grp 0  Grp 1  ...  Grp M  Grp N
   |  |   | |
   |  |   | |
 _  _   _  _ ___
Grp 0 - | C00 | -> | C01 | -> ...  ->  | COM | -> | C0N | --> |
|_||_| |_||_| |
   |  |   | |
 __|__  __|__   __|__ |
Grp 1 - | C00 | -> | C01 | -> ...  ->  | COM |--> |
|_||_| |_| |
   |  | |
   ......... | DATA
 __|__  __|__ | ACCUM
Grp M - | C00 | -> | C01 | --> |
|_||_| |
 __|__ |
Grp N - | C00 | --> |
|_| |

M = N-1 above, for notational simplicity

You certainly have too much processing to do in a single ROACH. You will
have to divide and conquer somehow.  Grouping the samples from several 
antennas
into groups and grouping of smaller correlators into a (at least 
conceptually)

triangular array, as shown above would be one approach.

You will probably have to add delay stages for each antenna ahead of
the correlators to align the wavefront.  You will also have to find a 
way to

initially distribute the signals.
300 MHz is a little fast; you may have to back off to 150 MHz at least
initially by parallelizing the data.

The CASPER way of passing signals around is to use a big Ethernet 
switch.  That

would take care of all of your interconnects in the triangle above and also
to the data accumulator to the right above.

As an alternative architecture you may want to have a look at powerMx.
See www.powermx.org

Rich

Re: [casper] building 300-receiver channel cross-correlator

[James Smith]

Hello Neil,

CASPER tools could probably do what you're looking for, but I found your
description a bit confusing. You're going to need to clarify somewhat.

Regards,
James


On Fri, Dec 18, 2015 at 4:15 PM, Neil Salmon  wrote:

> Anyone help?
>
>
>
> I’m working in academia and need to build a 300-receiver channel
> single-bit digitiser / cross-correlator with a single frequency channel
> having a bandwidth of 300 MHz, centre frequency ~3 GHz. The single bit
> digitisers sample I giving a total data rate of 180 Gbps and using XOR
> gates to do the cross-correlations, the total computation rate is 54 T XOR
> operations per second. I need to accumulate cross-correlations typically
> for times ranging from 10 ms to a few seconds. The system would comprise an
> array of single bit digitisers linked via a high speed data bus to FPGA
> boards for the cross-correlation/accumulation. I’ve no skills in board
> design but could probably learn VHDL. I don’t have funding to commission a
> design and build but wondered if anyone in this community could advise how
> I should go about building this system at our university.
>
>
>
> Thank you for any help you can provide.
>
>
>
> Neil
> "Before acting on this email or opening any attachments you should read
> the Manchester Metropolitan University email disclaimer available on its
> website http://www.mmu.ac.uk/emaildisclaimer "
>