Re: [casper] ROACH2 / katADC million channel spectrometer

[jborsen]

  Hi all

    I do agree. We are routinely doing 16 million channels FFT using 
ROACH2+GPU (note you need a 6giga GPU memory to to this), basically to 
track RFI. The fact GPU works in floating point simplifies greatly the 
design, since you end with huge narrow features. Not easy to deal with 
in a fixed point environment.


  Another concern may be first stage channel isolation,depending on 
what you have in, and what you are looking for. No such concern in a one 
shot direct FFT. CUDA has been found efficient for very very long FFT.



Regards

Jean Borsenberger


On 10/03/2020 18:10, Dan Werthimer wrote:


hi molly,

regarding making a large FFT from small FFT's:

as jason pointed out, there's a way to compute a million point 
spectrum from lots of small FFT's.
the technique jason described is to implement a course channelization 
polyphase filter bank spectrometer to divide the band into 1024 sub-bands,
and then implement a fine channelizer FFT or PFB to divide each of the 
sub-bands up into another 1024 channels.

jason also pointed out this technique results in scalloping.

there's also a way to compute a 1M point FFT without scalloping, 
mathematically identical to a 1M point FFT,

but composed of 2048  1Kpoint FFT's:
a) fill a 1024 by 1024 complex array with 2^20 complex points
b) do 1024 1K point FFT's on the columns
c) multiply all million points by twiddle factors  (complex 
coefficients).

d) do 1024 1K point FFT's on the rows

we used this technique for serendip 3 and 4 seti spectrometers.

there are several variations on this theme of computing large FFT's 
from small ones -

i can dig probably up some references if you want...

but i think danny prices and my suggestion of shipping raw ADC samples 
from your roach2 to a CPU/GPU for spectral analysis
(done in floating point, easy to implement), will make your life more 
enjoyable.

there's some good casper open source code for FPGA/GPU spectrometers...

best wishes,

dan


On Mon, Mar 9, 2020 at 11:47 PM Jason Manley > wrote:


On FPGAs, ~million channel spectrometers are typically built by
doing a 1024ch PFB, followed by a matrix transpose (to collect
1024 samples of a single channel together) and another FFT.
Mostly, you'll find you run out of memory if you try to do a
million-channel FFT directly on an FPGA.

There are implications for this 2-D approach, though. If you use a
critically-sampled filterbank in the first stage, your channels
cross-over at -6dB, and so if you input a signal near this
intersection point, you'll get some fraction of the power in each
of the two adjacent channels. After you then do a second FFT on
these two channels, you get a spectrum that has some power in one
channel, and also power in a channel 1024 channels later. So it
now looks like you've got two CW signals going in, 1024 channels
apart, even if you only actually had one CW tone.

There was a million channel spectrometer built this way for a
ROACH1 tutorial at the 2009 CASPER workshop. That used to live on
the CASPER SVN server, but is no longer maintained. On a ROACH2,
with about 1GHz bandwidth, I'd expect you might possibly achieve
something closer to 16M channels, if you wanted (4k by 4k). No
doubt this won't work in modern Simulink toolflows, but it might
be a useful reference... try here:
https://casper.ssl.berkeley.edu/svn/trunk/projects/roach_mspec/

Jason Manley
DSP Manager
SKA-SA

Cell: +27 82 662 7726
Work: +27 21 506 7300

> On 10 Mar 2020, at 08:23, James Smith mailto:jsm...@ska.ac.za>> wrote:
>
> Hello Molly,
>
> We have done something broadly similar using ROACH2 and katADC,
but only 1024 channels:
>
https://github.com/ska-sa/AVNRoachGateware/tree/master/WideBandSpectrometer
>
> What frequency are you looking at? The only thing I'm thinking
is that you might struggle to get the design to meet timing if
you're clocking the board too fast.
>
> Shout if you get stuck.
>
> Regards,
> James
>
>
> On Mon, Mar 9, 2020 at 8:34 PM Molly Smith mailto:molsm...@ucdavis.edu>> wrote:
> Hello,
>
> I'm new to CASPER and am trying to implement a million channel
spectrometer on ROACH2 with the katADC. Does anyone know of any
useful resources or has done something similar?
>
> I have experience using Vivado for newer FPGA development and am
comfortable with Verilog & VHDL. I'm not familiar with the
Simulink/ISE workflow and have been relying on the tutorials to
come up to speed.
>
> I'm currently trying to modify Tutorial 3 (wideband
spectrometer) to use the katADC instead.
>
> Thanks,
> Molly
>
> --
> You received this message because you are subscribed to the
Google Groups "casper@lists.berkeley.edu
" group.
> To unsubscribe 

Re: [casper] ROACH2 / katADC million channel spectrometer

[Dan Werthimer]

hi molly,

regarding making a large FFT from small FFT's:

as jason pointed out, there's a way to compute a million point spectrum
from lots of small FFT's.
the technique jason described is to implement a course channelization
polyphase filter bank spectrometer to divide the band into 1024 sub-bands,
and then implement a fine channelizer FFT or PFB to divide each of the
sub-bands up into another 1024 channels.
jason also pointed out this technique results in scalloping.

there's also a way to compute a 1M point FFT without scalloping,
mathematically identical to a 1M point FFT,
but composed of 2048  1Kpoint FFT's:
a) fill a 1024 by 1024 complex array with 2^20 complex points
b) do 1024 1K point FFT's on the columns
c) multiply all million points by twiddle factors  (complex coefficients).
d) do 1024 1K point FFT's on the rows

we used this technique for serendip 3 and 4 seti spectrometers.

there are several variations on this theme of computing large FFT's from
small ones -
i can dig probably up some references if you want...

but i think danny prices and my suggestion of shipping raw ADC samples from
your roach2 to a CPU/GPU for spectral analysis
(done in floating point, easy to implement), will make your life more
enjoyable.
there's some good casper open source code for FPGA/GPU spectrometers...

best wishes,

dan


On Mon, Mar 9, 2020 at 11:47 PM Jason Manley  wrote:

> On FPGAs, ~million channel spectrometers are typically built by doing a
> 1024ch PFB, followed by a matrix transpose (to collect 1024 samples of a
> single channel together) and another FFT. Mostly, you'll find you run out
> of memory if you try to do a million-channel FFT directly on an FPGA.
>
> There are implications for this 2-D approach, though. If you use a
> critically-sampled filterbank in the first stage, your channels cross-over
> at -6dB, and so if you input a signal near this intersection point, you'll
> get some fraction of the power in each of the two adjacent channels. After
> you then do a second FFT on these two channels, you get a spectrum that has
> some power in one channel, and also power in a channel 1024 channels later.
> So it now looks like you've got two CW signals going in, 1024 channels
> apart, even if you only actually had one CW tone.
>
> There was a million channel spectrometer built this way for a ROACH1
> tutorial at the 2009 CASPER workshop. That used to live on the CASPER SVN
> server, but is no longer maintained. On a ROACH2, with about 1GHz
> bandwidth, I'd expect you might possibly achieve something closer to 16M
> channels, if you wanted (4k by 4k). No doubt this won't work in modern
> Simulink toolflows, but it might be a useful reference... try here:
> https://casper.ssl.berkeley.edu/svn/trunk/projects/roach_mspec/
>
> Jason Manley
> DSP Manager
> SKA-SA
>
> Cell: +27 82 662 7726
> Work: +27 21 506 7300
>
> > On 10 Mar 2020, at 08:23, James Smith  wrote:
> >
> > Hello Molly,
> >
> > We have done something broadly similar using ROACH2 and katADC, but only
> 1024 channels:
> >
> https://github.com/ska-sa/AVNRoachGateware/tree/master/WideBandSpectrometer
> >
> > What frequency are you looking at? The only thing I'm thinking is that
> you might struggle to get the design to meet timing if you're clocking the
> board too fast.
> >
> > Shout if you get stuck.
> >
> > Regards,
> > James
> >
> >
> > On Mon, Mar 9, 2020 at 8:34 PM Molly Smith  wrote:
> > Hello,
> >
> > I'm new to CASPER and am trying to implement a million channel
> spectrometer on ROACH2 with the katADC. Does anyone know of any useful
> resources or has done something similar?
> >
> > I have experience using Vivado for newer FPGA development and am
> comfortable with Verilog & VHDL. I'm not familiar with the Simulink/ISE
> workflow and have been relying on the tutorials to come up to speed.
> >
> > I'm currently trying to modify Tutorial 3 (wideband spectrometer) to use
> the katADC instead.
> >
> > Thanks,
> > Molly
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "casper@lists.berkeley.edu" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to casper+unsubscr...@lists.berkeley.edu.
> > To view this discussion on the web visit
> https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/ad7c8afb-5775-401a-b53f-e769ba5d53e1%40lists.berkeley.edu
> .
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "casper@lists.berkeley.edu" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to casper+unsubscr...@lists.berkeley.edu.
> > To view this discussion on the web visit
> https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAG67D35z5XjBi4mu1s8e7EqCKnnV3NDf7r4UUT%2BQHr9LfF6EkQ%40mail.gmail.com
> .
>
> --
> You received this message because you are subscribed to the Google Groups "
> casper@lists.berkeley.edu" group.
> To unsubscribe from this group and stop receiving 

Re: [casper] ROACH2 / katADC million channel spectrometer

[Jason Manley]

On FPGAs, ~million channel spectrometers are typically built by doing a 1024ch 
PFB, followed by a matrix transpose (to collect 1024 samples of a single 
channel together) and another FFT. Mostly, you'll find you run out of memory if 
you try to do a million-channel FFT directly on an FPGA.

There are implications for this 2-D approach, though. If you use a 
critically-sampled filterbank in the first stage, your channels cross-over at 
-6dB, and so if you input a signal near this intersection point, you'll get 
some fraction of the power in each of the two adjacent channels. After you then 
do a second FFT on these two channels, you get a spectrum that has some power 
in one channel, and also power in a channel 1024 channels later. So it now 
looks like you've got two CW signals going in, 1024 channels apart, even if you 
only actually had one CW tone.

There was a million channel spectrometer built this way for a ROACH1 tutorial 
at the 2009 CASPER workshop. That used to live on the CASPER SVN server, but is 
no longer maintained. On a ROACH2, with about 1GHz bandwidth, I'd expect you 
might possibly achieve something closer to 16M channels, if you wanted (4k by 
4k). No doubt this won't work in modern Simulink toolflows, but it might be a 
useful reference... try here: 
https://casper.ssl.berkeley.edu/svn/trunk/projects/roach_mspec/

Jason Manley
DSP Manager
SKA-SA

Cell: +27 82 662 7726
Work: +27 21 506 7300

> On 10 Mar 2020, at 08:23, James Smith  wrote:
> 
> Hello Molly,
> 
> We have done something broadly similar using ROACH2 and katADC, but only 1024 
> channels:
> https://github.com/ska-sa/AVNRoachGateware/tree/master/WideBandSpectrometer
> 
> What frequency are you looking at? The only thing I'm thinking is that you 
> might struggle to get the design to meet timing if you're clocking the board 
> too fast.
> 
> Shout if you get stuck.
> 
> Regards,
> James
> 
> 
> On Mon, Mar 9, 2020 at 8:34 PM Molly Smith  wrote:
> Hello,
> 
> I'm new to CASPER and am trying to implement a million channel spectrometer 
> on ROACH2 with the katADC. Does anyone know of any useful resources or has 
> done something similar? 
> 
> I have experience using Vivado for newer FPGA development and am comfortable 
> with Verilog & VHDL. I'm not familiar with the Simulink/ISE workflow and have 
> been relying on the tutorials to come up to speed. 
> 
> I'm currently trying to modify Tutorial 3 (wideband spectrometer) to use the 
> katADC instead. 
> 
> Thanks,
> Molly
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "casper@lists.berkeley.edu" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to casper+unsubscr...@lists.berkeley.edu.
> To view this discussion on the web visit 
> https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/ad7c8afb-5775-401a-b53f-e769ba5d53e1%40lists.berkeley.edu.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "casper@lists.berkeley.edu" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to casper+unsubscr...@lists.berkeley.edu.
> To view this discussion on the web visit 
> https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAG67D35z5XjBi4mu1s8e7EqCKnnV3NDf7r4UUT%2BQHr9LfF6EkQ%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"casper@lists.berkeley.edu" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to casper+unsubscr...@lists.berkeley.edu.
To view this discussion on the web visit 
https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/8C7473F8-95E1-4A6D-A2E5-E7A4A314EE3D%40ska.ac.za.

Re: [casper] ROACH2 / katADC million channel spectrometer

[Danny Price]

Hi Molly,

There is an old tutorial with a million channel spectrometer up here:
https://casper.ssl.berkeley.edu/wiki/Old_Tutorials
This was on the ROACH1, and used a 1024-point real FFT, a 1024x1024
transpose using QDR RAM, followed by a second 1024-point complex FFT.

If you have access to a computer with a 10 GbE card + GPU, a nice solution
would be send the data over 10 GbE and do the FFT on the GPU. You can
transfer 500 MHz of bandwidth @ 8 bit resolution over a 10 GbE link, and
the NVIDIA cuFFT can do large FFTs without breaking a sweat. There are
quite a few groups who use FPGA --> GPU, so should be some help on the
mailing list should you take this route.

Cheers,
Danny

On 10 March 2020 at 5:23:38 pm, James Smith (jsm...@ska.ac.za) wrote:

Hello Molly,

We have done something broadly similar using ROACH2 and katADC, but only
1024 channels:
https://github.com/ska-sa/AVNRoachGateware/tree/master/WideBandSpectrometer

What frequency are you looking at? The only thing I'm thinking is that you
might struggle to get the design to meet timing if you're clocking the
board too fast.

Shout if you get stuck.

Regards,
James


On Mon, Mar 9, 2020 at 8:34 PM Molly Smith  wrote:

> Hello,
>
> I'm new to CASPER and am trying to implement a million channel
> spectrometer on ROACH2 with the katADC. Does anyone know of any useful
> resources or has done something similar?
>
> I have experience using Vivado for newer FPGA development and am
> comfortable with Verilog & VHDL. I'm not familiar with the Simulink/ISE
> workflow and have been relying on the tutorials to come up to speed.
>
> I'm currently trying to modify Tutorial 3 (wideband spectrometer) to use
> the katADC instead.
>
> Thanks,
> Molly
> --
> You received this message because you are subscribed to the Google Groups "
> casper@lists.berkeley.edu" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to casper+unsubscr...@lists.berkeley.edu.
> To view this discussion on the web visit
> https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/ad7c8afb-5775-401a-b53f-e769ba5d53e1%40lists.berkeley.edu
> 
> .
>
--
You received this message because you are subscribed to the Google Groups "
casper@lists.berkeley.edu" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to casper+unsubscr...@lists.berkeley.edu.
To view this discussion on the web visit
https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAG67D35z5XjBi4mu1s8e7EqCKnnV3NDf7r4UUT%2BQHr9LfF6EkQ%40mail.gmail.com

.

-- 
You received this message because you are subscribed to the Google Groups 
"casper@lists.berkeley.edu" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to casper+unsubscr...@lists.berkeley.edu.
To view this discussion on the web visit 
https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAAtMgqmMh9jWgGrmuHvXo63q74XvFPxM9FwpfWmYfAaSmso6Lg%40mail.gmail.com.

Re: [casper] ROACH2 / katADC million channel spectrometer

[James Smith]

Hello Molly,

We have done something broadly similar using ROACH2 and katADC, but only
1024 channels:
https://github.com/ska-sa/AVNRoachGateware/tree/master/WideBandSpectrometer

What frequency are you looking at? The only thing I'm thinking is that you
might struggle to get the design to meet timing if you're clocking the
board too fast.

Shout if you get stuck.

Regards,
James


On Mon, Mar 9, 2020 at 8:34 PM Molly Smith  wrote:

> Hello,
>
> I'm new to CASPER and am trying to implement a million channel
> spectrometer on ROACH2 with the katADC. Does anyone know of any useful
> resources or has done something similar?
>
> I have experience using Vivado for newer FPGA development and am
> comfortable with Verilog & VHDL. I'm not familiar with the Simulink/ISE
> workflow and have been relying on the tutorials to come up to speed.
>
> I'm currently trying to modify Tutorial 3 (wideband spectrometer) to use
> the katADC instead.
>
> Thanks,
> Molly
>
> --
> You received this message because you are subscribed to the Google Groups "
> casper@lists.berkeley.edu" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to casper+unsubscr...@lists.berkeley.edu.
> To view this discussion on the web visit
> https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/ad7c8afb-5775-401a-b53f-e769ba5d53e1%40lists.berkeley.edu
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"casper@lists.berkeley.edu" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to casper+unsubscr...@lists.berkeley.edu.
To view this discussion on the web visit 
https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAG67D35z5XjBi4mu1s8e7EqCKnnV3NDf7r4UUT%2BQHr9LfF6EkQ%40mail.gmail.com.

[casper] ROACH2 / katADC million channel spectrometer

[Molly Smith]

Hello,

I'm new to CASPER and am trying to implement a million channel spectrometer 
on ROACH2 with the katADC. Does anyone know of any useful resources or has 
done something similar? 

I have experience using Vivado for newer FPGA development and am 
comfortable with Verilog & VHDL. I'm not familiar with the Simulink/ISE 
workflow and have been relying on the tutorials to come up to speed. 

I'm currently trying to modify Tutorial 3 (wideband spectrometer) to use 
the katADC instead. 

Thanks,
Molly

-- 
You received this message because you are subscribed to the Google Groups 
"casper@lists.berkeley.edu" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to casper+unsubscr...@lists.berkeley.edu.
To view this discussion on the web visit 
https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/ad7c8afb-5775-401a-b53f-e769ba5d53e1%40lists.berkeley.edu.