hi laura, some info on NIC's is at: https://casper.berkeley.edu/wiki/Recommended_10_GbE_Hardware but these part numbers are a bit old - there are several newer model NIC's available from both myricom and chelsio.
i recommend myricom's NIC's. make sure you get one with CX4 port(s). chelsio works well too. best wishes, dan On Fri, Mar 16, 2012 at 3:54 PM, Laura Vertatschitsch <[email protected]>wrote: > Any specific recommendations for a 10Ge NIC? Our sys admin is warning us > about the compatibility with Red Hat (I think we are running red hat 6 on > the computer we would like to stream to.) He had suggested that > this<http://www.cdw.com/shop/products/Myricom-Myri-10G-network-adapter/2034587.aspx>may > work, but I would be grateful for input from others that are currently > using a RedHat machine with a 10GbE NIC. > > --Laura > > On Thu, Mar 15, 2012 at 9:45 AM, Dan Werthimer <[email protected]>wrote: > >> >> hi andy and others interested: >> >> i suggest we create a section on the casper wiki >> that documents the various open source programs that >> people have developed for receiving and processing10Gbit data >> on computers and GPU's. >> >> there's PSRDADA and GUPPI/VEGAS/HASHPIPE: >> PSRDADA was developed by matthew bailes' group in australia. >> GUPPI was developed at NRAO, mostly by paul demorest, for pulsar >> instrumentation. >> VEGAS and HASHPIPE are berkeley derivatives of GUPPI for spectrometers >> and correlators. >> these opensource codes typically have one thread that moves data from a >> NIC to a circular buffer, >> another thread the moves the data from a circular buffer to a GPU or CPU >> based compute process. >> another thread the moves the output from GPU/CPU to a circular output >> buffer, >> and another thread that write the output buffers to disk. >> >> optionally, there's a fifth thread that gathers meta data from a >> telescope >> (eg: pointing and frequency information) and puts this info into the >> circular buffers. >> and there are optional several processes that perform monitor and control >> to make sure >> everything is working, display data, etc. >> >> >> best wishes, >> >> dan >> >> >> >> On Thu, Mar 15, 2012 at 6:24 AM, Andrew Lutomirski <[email protected]> wrote: >> >>> >>> On Mar 14, 2012 3:26 PM, "Dan Werthimer" <[email protected]> wrote: >>> > >>> > >>> > >>> > hi laura, >>> > >>> > i second john's remarks: >>> > >>> > an inexpensive 10Gbit nic card in your computer >>> > would make your data streaming task relatively easy, >>> > as there are tutorials and several instrument designs >>> > that stream data from a roach over 10Gbe into a computer. >>> > >>> > there's also a lot of good open source software you can use >>> > to receive 10gBE data on a computer and send it to a disk drive.... >>> > >>> >>> I have a program to do exactly this on the roach. MIT should allow me >>> to open-source it any day now. >>> >>> --Andy >>> >>> > best wishes, >>> > >>> > dan >>> > >>> > >>> > On Wed, Mar 14, 2012 at 3:18 PM, John Ford <[email protected]> wrote: >>> >> >>> >> > Hey Andrew, >>> >> > >>> >> > Thanks for the reply - we are looking forward to the ROACH 2 in the >>> >> > future. >>> >> > >>> >> > I haven't been able to open all of the tutorials, but I have seen a >>> few. >>> >> > So I apologize if I may have missed something obvious, and I would >>> even >>> >> > really appreciate an answer that says "hey you, go read about >>> this." I >>> >> > see >>> >> > that data can be saved using a snap block, and I could save a >>> concatenated >>> >> > set of samples as a single 64 bit integer, and I can save a maximum >>> of >>> >> > 2^16 >>> >> > of these. Is there a better way of using the roach memory to >>> guarantee >>> >> > that I can read and save all of the data continuously? I'm >>> wondering if >>> >> > there is perhaps a way of constantly reading from a snap register >>> fast >>> >> > enough to insure that we have captured the data continuously and >>> saved it >>> >> > to our external computer. >>> >> >>> >> Hi Laura. In addition to what Dan just said, I want to point out >>> that the >>> >> 10m/100m/1g bps port (on at least many ROACH I's) is not reliable at 1 >>> >> gbps. >>> >> >>> >> So you would be streaming 10 MB/s (80 mb/s plus framing) over a 100 mb >>> >> link. Pretty risky... >>> >> >>> >> The 10 gbe solution is better in so many ways. If your computer is >>> within >>> >> a few meters of the ROACH, I highly recommend you think about buying >>> a 10 >>> >> gb card for the computer. It'll only cost ~$500.00 plus a $100.00 >>> cable, >>> >> and will save you a lot of sleepless nights, I think! >>> >> >>> >> John >>> >> >>> >> > >>> >> > --Laura >>> >> > >>> >> > >>> >> > On Mon, Mar 12, 2012 at 11:59 PM, Andrew Martens <[email protected]> >>> wrote: >>> >> > >>> >> >> Hi Laura >>> >> >> >>> >> >> The ROACH does not have a direct 1Ge connection to the FPGA, any >>> data >>> >> >> must exit via the PPC if it is not going through the 10Ge links. >>> >> >> >>> >> >> The ROACH2 however, has a 1Ge link directly to the FPGA. The yellow >>> >> >> block has been completed and has been designed to act the same as >>> the >>> >> >> 10Ge links (basically the same interface, can be accessed from >>> the PPC >>> >> >> if needed, ARP table managed from PPC etc) except that the data >>> rate is >>> >> >> lower (8 bit data paths instead of 64 bit). >>> >> >> >>> >> >> We should be finalising these yellow blocks and making an >>> announcement >>> >> >> soon so that people who are interested can start designing for >>> ROACH2. >>> >> >> >>> >> >> Regards >>> >> >> Andrew >>> >> >> >>> >> >> On Mon, 2012-03-12 at 17:22 -0700, Laura Vertatschitsch wrote: >>> >> >> > Hey Casperites, >>> >> >> > >>> >> >> > I see a lot of data about reliable streaming using the 10GbE >>> ports and >>> >> >> > a lovely simulink block to boot. Is there an analogous method >>> for >>> >> >> > streaming data out over the 1Gbps ethernet? >>> >> >> > >>> >> >> > Not sure if someone has written some python control scripts to >>> >> >> > accomplish this - I may have just missed it on the wiki. We >>> would >>> >> >> > love to stream 10MHz time-domain data off our board. >>> >> >> > >>> >> >> > --Laura >>> >> >> >>> >> >> >>> >> >> >>> >> > >>> >> >>> >> >>> >> >>> > >>> >> >> >
