Hi louis,
Please see the below link for packet capture https://staff.washington.edu/corey/gulp/ Thanks Indrajit On 2016-07-01 18:52, casper-requ...@lists.berkeley.edu wrote: > Send casper mailing list submissions to > casper@lists.berkeley.edu > > To subscribe or unsubscribe via the World Wide Web, visit > https://calmail.berkeley.edu/manage/list/listinfo/casper@lists.berkeley.edu > [1] > > or, via email, send a message with subject or body 'help' to > casper-requ...@lists.berkeley.edu > > You can reach the person managing the list at > casper-ow...@lists.berkeley.edu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of casper digest..." > > Today's Topics: > > 1. Re: packet capture (Daniel C Price) > 2. Re: packet capture (David MacMahon) > 3. Fwd: SECOND ANNOUNCEMENT : RFI 2016 - Abstract submission > August 1 (Andrew Siemion) > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 30 Jun 2016 14:52:25 -0700 > From: Daniel C Price <dan...@berkeley.edu> > Subject: Re: [casper] packet capture > To: David George <dgeorges...@gmail.com> > Cc: casper list <casper@lists.berkeley.edu> > Message-ID: > <caatmgqnsj4k3x24519ufrqvrlsjmabm2ykmhsihwtf9ksj9...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hey Louis > > For 10GbE, you *should* be able to capture everything just by tuning your > linux install. Jonathon Kocz et. al. spent quite a lot of time fine tuning > parameters, some of which are detailed in http://arxiv.org/abs/1411.3751 [2] > > The two main components to the problem are 1) high-speed packet capture is > really hard using only sockets, and 2) capturing and arranging packets into > a ring buffer that can then be the start of a pipeline (or just write it to > disk). > > ## Capture to a ring buffer > > For capture to a ring buffer, there are a few pipelines that might be of > help. For LEDA, we used PSRDADA (http://psrdada.sourceforge.net [3]). VEGAS > uses HASHPIPE (https://github.com/david-macmahon/hashpipe/tree/master/src > [4]). > CHIME called their pipeline kotekan (http://arxiv.org/pdf/1503.06189v1.pdf [5] > ). > > We are actually currently working on a friendlier pipeline and are making > some good progress, and I believe Andrew Jameson has made some good > progress on a similar path. Hopefully these come to fruition and can help > form a "CASPER toolflow for GPUs". This is probably still a bit far off for > your immediate use though :( > > ## High-speed packet capture > > Returning to the packet capture problem, there are now a few competing > "accelerator frameworks" out there for high-speed packet capture. Most of > it comes down to doing a zero-copy from the NIC to the CPU by bypassing the > kernel. > > * If you're using a Mellanox NIC, you can use libvma. > http://www.mellanox.com/page/software_vma?mtag=vma [6] > > This can be used in two ways: > 1) Not changing your code: use a LD_LIBRARY_PRELOAD to load libvma to > intercept socket calls in your C code. We haven't had much luck making this > improve our code. > 2) Changing your code: using the "VMA Extra API" and using VMA offload > instead of sockets (see Chapter 7 of > http://www.mellanox.com/related-docs/prod_acceleration_software/VMA_8_0_4_User_Manual_DOC-00393.pdf > [7]). > I haven't personally used the API yet, but have heard some success stories. > > * CHIME used NTOP's PF_RING ZC and got very impressive results > http://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/ [8] > > * And as Dave mentioned, there is DPDK as well (http://dpdk.org [9]). > > Of all of those, I would suggest trying DPDK first, should fine tuning not > get you 100% of the way there. > > Good luck! > - Danny > > On Thu, Jun 30, 2016 at 3:18 AM, David George <dgeorges...@gmail.com> wrote: > Use a NIC that can do dpdk: http://dpdk.org/doc/nics [10] then look on the > web for dpdk packet capture utils. David On 30 June 2016 at 10:30, Andrew > Siemion <siem...@berkeley.edu> wrote: Hi Louis, We found this paper quite > useful: http://www.ieee-hpec.org/2014/CD/index_htm_files/FinalPapers/95.pdf > [11] Cheers, Andrew On Jun 30, 2016, at 6:58 AM, Louis P. Dartez > <louisdar...@gmail.com> wrote: Hi all, Can anyone guide me on what tools are > being used for packet capture purposes right now? I?ve been working on a > baseband recorder using a ROACH1 using the 10GbE lines but I can?t seem to > avoid losing packets when writing data to memory or disk. I wrote a threaded > packet sniffer in C with a pretty tight loop that write directly to memory in > Ubuntu Linux. I?m wondering if there are any optimizations that I could be > missing out. Any advice would be greatly appreciated, Thanks in advance! LD > Louis P. Dartez Ph.D. Candidate STARGATE Center for Advanced Radio Astronomy > University of Te xas Rio Grande Valley (956) 372-5812 -- Danny Price dan...@berkeley.edu | +1 617-386-3700 -------------- next part -------------- An HTML attachment scrubbed and removed. HTML attachments are only available in MIME digests. ------------------------------ Message: 2 Date: Thu, 30 Jun 2016 15:20:54 -0700 From: David MacMahon <dav...@berkeley.edu> Subject: Re: [casper] packet capture To: "Louis P. Dartez" <louisdar...@gmail.com> Cc: casper list <casper@lists.berkeley.edu> Message-ID: <48f3b4d9-bcca-4e7f-9ef4-b4c4048c9...@berkeley.edu> Content-Type: text/plain; charset="utf-8" Hi, Louis, In addition to the DPDK and the libvma options that others have discussed, you might want to consider using memory mapped packet sockets. This is very agnostic to the NIC you are using and requires no special driver magic to work, so in that sense it is simpler than the other options. The latest Hashpipe program includes a user friendly API for setting up and using sockets of this type to receive UDP packets. Once setup, the kernel will DMA packets directly from the NIC into your application's memory space and set an in-memory flag which you can poll to detect the arrival of new packets. It may not be quite as fast/optimized as the DPDK or libvma options, but it is very much faster than receiving packets from the kernel?s IP stack via recv(). Since the barrier to entry is lower, it might be worth exploring this option first. If it doesn?t meet your needs, then you can try the other more involved approaches. I agree with the suggestion to measure where your code is spending time. If you are getting bogged down in disk I/O or some other non-network part of your code then no amount of speeding up the network code will fix things. You can read about the mmap?d packet sockets here: https://www.kernel.org/doc/Documentation/networking/packet_mmap.txt [13] You can find the latest Hashpipe code here: https://github.com/david-macmahon/hashpipe [14]Hope this helps, Dave > On Jun 29, 2016, at 9:58 PM, Louis P. Dartez <louisdar...@gmail.com> wrote: > Hi all, Can anyone guide me on what tools are being used for packet capture > purposes right now? I?ve been working on a baseband recorder using a ROACH1 > using the 10GbE lines but I can?t seem to avoid losing packets when writing > data to memory or disk. I wrote a threaded packet sniffer in C with a pretty > tight loop that write directly to memory in Ubuntu Linux. I?m wondering if > there are any optimizations that I could be missing out. Any advice would be > greatly appreciated, Thanks in advance! LD > >> Louis P. Dartez Ph.D. Candidate STARGATE Center for Advanced Radio Astronomy >> University of Texas Rio Grande Valley (956) 372-5812 -------------- next part -------------- An HTML attachment scrubbed and removed. HTML attachments are only available in MIME digests. ------------------------------ Message: 3 Date: Fri, 1 Jul 2016 15:21:44 +0200 From: Andrew Siemion <siem...@berkeley.edu> Subject: [casper] Fwd: SECOND ANNOUNCEMENT : RFI 2016 - Abstract submission August 1 To: casper list <casper@lists.berkeley.edu> Message-ID: <18bc72d0-b3c2-4d45-ab28-325631e8d...@berkeley.edu> Content-Type: text/plain; charset="utf-8" > Dear colleague, It is my pleasure to announce the following conference: RFI > 2016 ? Coexisting with Radio Frequency Interference October 17-20, 2016 Macey > Center ? New Mexico Tech Socorro, NM, USA http://go.nrao.edu/rfi2016 [12] > <http://go.nrao.edu/rfi2016 [12]> RFI 2016 will bring together scientists and > engineers from all horizons to discuss the impact of Radio Frequency > Interference in Astronomy, Earth Remote Sensing, and other passive or active > fields exploiting the electromagnetic spectrum. This conference is a great > opportunity to debate on the current and future status of interference > contamination, and report recent advances made in RFI mitigation from a > theoretical and applied point of view. A particular effort will be made to > foster exchanges and collaborations between attendees. RFI 2016 will be held > in October 17-20, 2016 in Socorro, NM, USA, a few steps away from the > Operation Center of the famous NRAO Very Large Array. Prospective authors are > invited to submit a half- to full-page abstract before August 1, 2016 in the following topics: ? Defining and quantifying RFI ? Spectrum management and frequency allocations ? Radio Quiet Zones, electromagnetic interference ? SETI : separating terrestrial and extra-terrestrial transmissions ? RFI detection and prediction ? Calibration, direction-of-arrival estimation ? RFI in passive and active microwave remote sensing ? Mono and multi antenna signal processing ? Progress in flagging and excision techniques ? Signal subtraction approaches ? Time/frequency/spatial filtering ? Pre- and post-correlation techniques ? Towards a real-time implementation of identification and mitigation ? Future sources of RFI Presenting authors will be required to register before September 15, 2016. Travel grants will be made available for students, as well as a registration fee discount. Further information can be found at http://go.nrao.edu/rfi2016 [12] <http://go.nrao.edu/rfi2016 [12]> or by contacting our organizing com mittee at rfi2...@nrao.edu <mailto:rfi2...@nrao.edu> We look forward to seeing you in Socorro. Kind regards, Greg Hellbourg Important dates: Abstract submission : August 1, 2016 Paper acceptance notification : September 4, 2016 Author registration : September 15, 2016 Conference : October 17-20, 2016 Proceeding paper submission : November 11, 2016 -------------- next part -------------- An HTML attachment scrubbed and removed. HTML attachments are only available in MIME digests. -------------- next part -------------- A non-text attachment was scrubbed and removed. Name: image002.png Type: image/png Size: 7456 bytes Desc: not available Non-text attachments are only available in MIME digests. -------------- next part -------------- An HTML attachment scrubbed and removed. HTML attachments are only available in MIME digests. -------------- next part -------------- A non-text attachment was scrubbed and removed. Name: RFI2016_CfP2.pdf Type: application/pdf Size: 579681 bytes Desc: not available Non-text attachments are only available in MIME digests. -------------- next part -------------- An HTML attachment scrubbed and removed. HTML attachments are only available in MIME digests. End of casper Digest, Vol 104, Issue 1 ************************************** Links: ------ [1] https://calmail.berkeley.edu/manage/list/listinfo/casper@lists.berkeley.edu [2] http://arxiv.org/abs/1411.3751 [3] http://psrdada.sourceforge.net [4] https://github.com/david-macmahon/hashpipe/tree/master/src [5] http://arxiv.org/pdf/1503.06189v1.pdf [6] http://www.mellanox.com/page/software_vma?mtag=vma [7] http://www.mellanox.com/related-docs/prod_acceleration_software/VMA_8_0_4_User_Manual_DOC-00393.pdf [8] http://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/ [9] http://dpdk.org [10] http://dpdk.org/doc/nics [11] http://www.ieee-hpec.org/2014/CD/index_htm_files/FinalPapers/95.pdf [12] http://go.nrao.edu/rfi2016 [13] https://www.kernel.org/doc/Documentation/networking/packet_mmap.txt [14] https://github.com/david-macmahon/hashpipe
