Hi Danny,
Thanks very much for the references and suggestion. Our correlator is 192 inputs, 125 MHz bandwidth with 1024 frequency channel. And I want to do some test about how fast the computer can capture packet from the NIC. Now I have one NIC with dual 10GbE port made by Mellanox. Best Wishes, Lin At 2015-04-01 23:50:22, "Danny Price" <[email protected]> wrote: >Hi Shu Lean > >The short answer is: it depends, but you should probably buy a good Xeon >CPU if you are not sure. How many antennas and what bandwidth will you >have? > >The LEDA correlator (that I work with) uses xGPU for the X-engine, >and PSRDADA for packet capture / shared memory buffers. Here's some >papers for reference: >http://psrdada.sourceforge.net/ >http://www.worldscientific.com/doi/abs/10.1142/S2251171715500038?src=recsys >http://www.worldscientific.com/doi/abs/10.1142/S2251171714500020 > >For reference, our server specifications are: > >Ubuntu 12.04.2 LTS (GNU/Linux 3.5.0-23-generic x86_64) >Supermicro 1027TR-TQF >2x Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz >128 GB RAM >2x NVIDIA K20X GPU >1x Mellanox ConnectX-3 40GbE NIC > >Each server (11 total) is capturing 21.4 Gb/s, for 512 inputs, with 2.6 >MHz bandwidth per server. > >The MWA correlator (http://arxiv.org/abs/1501.05992) and the PAPER >correlator also use xGPU. >I believe PAPER are using HASHPIPE instead of PSRDADA for their pipeline >https://github.com/david-macmahon/hashpipe > >The CHIME project have recently posted some pretty darn impressive info >on their correlator: >http://arxiv.org/abs/1503.06189 >http://arxiv.org/abs/1503.06203 >http://arxiv.org/abs/1503.06202 > >CHIME are not using xGPU as they are using AMD GPUs, which don't support >CUDA. I think LOFAR >might be doing something with AMD GPUs also. From the CHIME papers, >they seem to be getting amazing packet capture results by using NTOP >PF_RING ZC (Zero Copy): >http://www.ntop.org/products/pf_ring/pf_ring-zc-zero-copy/ > >Most NIC vendors now have "offload engines" that try and reduce load by >bypassing the kernel. For example, Mellanox has libvma, and Solarflare >has TCP OE. If you're about to build something big, I would highly >recommend you purchase some cards from different vendors and do some lab >tests. It may turn out that one card in particular has a killer feature. >Please let us all know if you do! > >- Danny > >shu.lean wrote: >> Hi all, >> >> if using GPU to do X Enigne, what the specification of computer should >> be, like the performance of CPU, DDR memory? So the CPU can transfer >> the data NIC recieved to GPU in time? >> >> Thanks very much. >> >> Lin >> >>
