hi andy and others interested: i suggest we create a section on the casper wiki that documents the various open source programs that people have developed for receiving and processing10Gbit data on computers and GPU's.
there's PSRDADA and GUPPI/VEGAS/HASHPIPE: PSRDADA was developed by matthew bailes' group in australia. GUPPI was developed at NRAO, mostly by paul demorest, for pulsar instrumentation. VEGAS and HASHPIPE are berkeley derivatives of GUPPI for spectrometers and correlators. these opensource codes typically have one thread that moves data from a NIC to a circular buffer, another thread the moves the data from a circular buffer to a GPU or CPU based compute process. another thread the moves the output from GPU/CPU to a circular output buffer, and another thread that write the output buffers to disk. optionally, there's a fifth thread that gathers meta data from a telescope (eg: pointing and frequency information) and puts this info into the circular buffers. and there are optional several processes that perform monitor and control to make sure everything is working, display data, etc. best wishes, dan On Thu, Mar 15, 2012 at 6:24 AM, Andrew Lutomirski <a...@luto.us> wrote: > > On Mar 14, 2012 3:26 PM, "Dan Werthimer" <d...@ssl.berkeley.edu> wrote: > > > > > > > > hi laura, > > > > i second john's remarks: > > > > an inexpensive 10Gbit nic card in your computer > > would make your data streaming task relatively easy, > > as there are tutorials and several instrument designs > > that stream data from a roach over 10Gbe into a computer. > > > > there's also a lot of good open source software you can use > > to receive 10gBE data on a computer and send it to a disk drive.... > > > > I have a program to do exactly this on the roach. MIT should allow me to > open-source it any day now. > > --Andy > > > best wishes, > > > > dan > > > > > > On Wed, Mar 14, 2012 at 3:18 PM, John Ford <jf...@nrao.edu> wrote: > >> > >> > Hey Andrew, > >> > > >> > Thanks for the reply - we are looking forward to the ROACH 2 in the > >> > future. > >> > > >> > I haven't been able to open all of the tutorials, but I have seen a > few. > >> > So I apologize if I may have missed something obvious, and I would > even > >> > really appreciate an answer that says "hey you, go read about this." > I > >> > see > >> > that data can be saved using a snap block, and I could save a > concatenated > >> > set of samples as a single 64 bit integer, and I can save a maximum of > >> > 2^16 > >> > of these. Is there a better way of using the roach memory to > guarantee > >> > that I can read and save all of the data continuously? I'm wondering > if > >> > there is perhaps a way of constantly reading from a snap register fast > >> > enough to insure that we have captured the data continuously and > saved it > >> > to our external computer. > >> > >> Hi Laura. In addition to what Dan just said, I want to point out that > the > >> 10m/100m/1g bps port (on at least many ROACH I's) is not reliable at 1 > >> gbps. > >> > >> So you would be streaming 10 MB/s (80 mb/s plus framing) over a 100 mb > >> link. Pretty risky... > >> > >> The 10 gbe solution is better in so many ways. If your computer is > within > >> a few meters of the ROACH, I highly recommend you think about buying a > 10 > >> gb card for the computer. It'll only cost ~$500.00 plus a $100.00 > cable, > >> and will save you a lot of sleepless nights, I think! > >> > >> John > >> > >> > > >> > --Laura > >> > > >> > > >> > On Mon, Mar 12, 2012 at 11:59 PM, Andrew Martens <and...@ska.ac.za> > wrote: > >> > > >> >> Hi Laura > >> >> > >> >> The ROACH does not have a direct 1Ge connection to the FPGA, any data > >> >> must exit via the PPC if it is not going through the 10Ge links. > >> >> > >> >> The ROACH2 however, has a 1Ge link directly to the FPGA. The yellow > >> >> block has been completed and has been designed to act the same as the > >> >> 10Ge links (basically the same interface, can be accessed from the > PPC > >> >> if needed, ARP table managed from PPC etc) except that the data rate > is > >> >> lower (8 bit data paths instead of 64 bit). > >> >> > >> >> We should be finalising these yellow blocks and making an > announcement > >> >> soon so that people who are interested can start designing for > ROACH2. > >> >> > >> >> Regards > >> >> Andrew > >> >> > >> >> On Mon, 2012-03-12 at 17:22 -0700, Laura Vertatschitsch wrote: > >> >> > Hey Casperites, > >> >> > > >> >> > I see a lot of data about reliable streaming using the 10GbE ports > and > >> >> > a lovely simulink block to boot. Is there an analogous method for > >> >> > streaming data out over the 1Gbps ethernet? > >> >> > > >> >> > Not sure if someone has written some python control scripts to > >> >> > accomplish this - I may have just missed it on the wiki. We would > >> >> > love to stream 10MHz time-domain data off our board. > >> >> > > >> >> > --Laura > >> >> > >> >> > >> >> > >> > > >> > >> > >> > > >
