memory mapped packet capturing - bpf replacement ?
Traffic analysis applications running on high-speed networks are using BPF to capture packet. However, on heavy traffic frequent context switches caused by read(2) from a BPF device is a overhead. I implemented a prototype module that stores captured packets in user mmap-able area. I want to describe the prototype and discuss benchmark results. The module is a netgraph node, called ng_mmq. mmq stands for memory-mapped queue. The node has one hook, called input. When this hook is connected, o memory buffer is allocated. size is controlled by the debug.mmq_size sysctl. o a device /dev/mmqX is created, where X is a node ID o /dev/mmqX is mmap-able by the user, mmap() returns an allocated buffer o when packet arrives on hook, it is copied to the buffer, which is actually a ringbuffer. The ringbuffer's head is advanced. o user spins until tail != head, which means new data arrived. Then it reads from ringbuffer, and advances the tail. o no mutexes are used The code is at So this is the basic idea. I connected ng_mmq node to my rl0: ethernet node via the ng_hub, and benchmarked it against the pcap, using the same pcap callback function. Packet processing was simulated by the delay() function that just takes some CPU cycles. What I have found is: 1. bpf seems to be faster, i.e. it drops less packets than mmq 2. mmq seems to capture more packets. This is sample output from the benchmark utility: # ./benchmark rl0 /dev/mmq5 1000 pcap: rcvd: 15061, dropped: 14047, seen: 1000 mmq: rcvd: 23172, dropped: 21789, seen: 1000 Now, the questions: 1. is my interpretation of benchmark results correct? 2. if they are correct, why bpf is faster? 3. is it OK to have no mutexes for ringbuffer operations ? The ng_mmq code, as well as the benchmark code, are at http://oasis.uptsoft.com/~devnull/ng_mmq/ Setup instructions are at http://oasis.uptsoft.com/~devnull/ng_mmq/README -- Sergey Lyubka, Network Security Consultant NetFort Technologies Ltd, Galway, Ireland ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: How to build lint libraries, e.g. llib-lc.ln?
Alexander Mueller writes: when writing C applications I'd like to check them using lint. Unfortunately FreeBSD comes with only llib-lposix.ln and llib-lstdc.ln in /usr/libdata/lint. Is there an easy, i.e. automated, way to create lint libraries (e.g. llib-lc.ln) for all the libs in /usr/src/lib? yes. In current use WANT_LINT= yes in /etc/make.conf. This won't work in STABLE; I haven't MFCed it. M -- Mark Murray iumop ap!sdn w,I idlaH ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
freebsd asm
Hello to all, I'm proud to join this mailing, having posibilities to learn some new features of freebsd system. I have to mention i did some tests in asm from freebsd-devel and i get surprised, look at this: [demon]~$ uname -a FreeBSD demon.noconname.org 5.2.1-RELEASE-p4 FreeBSD 5.2.1-RELEASE-p4 #0: Tue Apr 6 19:35:49 CEST 2004 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/NocONName i386 [demon]~$ cat hello.asm %include 'system.inc' section .data holadb 'Hola', 0Ah hbytes equ $-hola section .text global _start _start: pushdword hbytes pushdword hola pushdword stdout sys.write pushdword 0 sys.exit [demon]~$ nasm -f elf hello.asm hello.asm:1: fatal: unable to open include file `system.inc' ¿Where is that file?... the -current port of nasm is incomplete ? Ok... we take some modifications... %include 'system.inc' pushdword stdout , and we replace it with pushdword 1 ok. compile... linking... [demon]~$ nasm -f elf hello.asm [demon]~$ ld -s -o hello hello.o [demon]~$ ./hello Bus error (core dumped) what's happening??? -Nico ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: memory mapped packet capturing - bpf replacement ?
The module is a netgraph node, called ng_mmq. mmq stands for memory-mapped queue. The node has one hook, called input. When this hook is connected, o memory buffer is allocated. size is controlled by the debug.mmq_size sysctl. o a device /dev/mmqX is created, where X is a node ID o /dev/mmqX is mmap-able by the user, mmap() returns an allocated buffer o when packet arrives on hook, it is copied to the buffer, which is actually a ringbuffer. The ringbuffer's head is advanced. o user spins until tail != head, which means new data arrived. Then it reads from ringbuffer, and advances the tail. o no mutexes are used The code is at So this is the basic idea. I connected ng_mmq node to my rl0: ethernet node via the ng_hub, and benchmarked it against the pcap, using the same pcap callback function. Packet processing was simulated by the delay() function that just takes some CPU cycles. What I have found is: 1. bpf seems to be faster, i.e. it drops less packets than mmq 2. mmq seems to capture more packets. This is sample output from the benchmark utility: # ./benchmark rl0 /dev/mmq5 1000 pcap: rcvd: 15061, dropped: 14047, seen: 1000 mmq: rcvd: 23172, dropped: 21789, seen: 1000 Now, the questions: 1. is my interpretation of benchmark results correct? 2. if they are correct, why bpf is faster? 3. is it OK to have no mutexes for ringbuffer operations ? Hello Sergey. I haven't looked at your code, but I'll provide some comments, having implemented a mmaped ringbuffer BPF replacement myself. First off, you should be able to do significantly better than vanilla BPF. Gigabit line rate is doable for reasonable sized packets and good hardware. Watch how much time you spend in your simulated packet processing. I also needed to add a delay to my benchmarking, because without it I'd run into the hardware limit (i.e. 1gbps), hiding the effects of further tweaking. However, if it's too great it will overwhelm the bpf/ringbuffer overhead, making your results less useful. I did my benchmark by increasing the packet rate until I found the point at which packets started to be dropped. In my testing I found the call to microtime() to be quite expensive. (It will vary depending on which timecounter is being used.) Is this in a SMP or uniprocesor environment? I think your gain from a ringbuffer interface will be more significant in the SMP case. Does the ng_hub cause the packet to be copied? If so you've still got the same number of copies as vanilla BPF. Are you using the same snap length (or copying the entire packet) in each case? As for question 3, be careful that you're atomically modifying the head and tail indices/pointers. But yes, you can do it without a mutex. -ed ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: indent bugfix / added features
Followup: I was off most of the weekend, but I did find one bug and have corrected it. The new patch can be found at http://telekinesis.org/indent/indent.patch. I whipped up a simple script to test it over /usr/src and that is at http://telekinesis.org/indent/test.sh. I found that my patched version (when run with no options) produced identical output to the existing version. -chip -- personal: chip norkus; renaissance hacker;[EMAIL PROTECTED] work: systems engineer @ lunarpages, inc.;[EMAIL PROTECTED] info: finger [EMAIL PROTECTED] for plan or keys; http://telekinesis.org ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: freebsd asm
On Mon, Jun 14, 2004 at 01:59:11PM +0200, Jos? Nicol?s Castellano wrote: Hello to all, I'm proud to join this mailing, having posibilities to learn some new features of freebsd system. I have to mention i did some tests in asm from freebsd-devel and i get surprised, look at this: [demon]~$ uname -a FreeBSD demon.noconname.org 5.2.1-RELEASE-p4 FreeBSD 5.2.1-RELEASE-p4 #0: Tue Apr 6 19:35:49 CEST 2004 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/NocONName i386 [demon]~$ cat hello.asm %include 'system.inc' section .data holadb 'Hola', 0Ah hbytes equ $-hola section .text global _start _start: pushdword hbytes pushdword hola pushdword stdout sys.write pushdword 0 sys.exit [demon]~$ nasm -f elf hello.asm hello.asm:1: fatal: unable to open include file `system.inc' ?Where is that file?... the -current port of nasm is incomplete ? I assume you are referring to the system.inc file mentioned in http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/x86-portable-code.html If so, note that this page says that you need to create this file yourself, it is neither part of nasm nor part of the standard FreeBSD distribution :) See the last paragraph on the page - Go ahead, enter it into your editor and save it as system.inc. :) G'luck, Peter -- Peter Pentchev [EMAIL PROTECTED][EMAIL PROTECTED][EMAIL PROTECTED] PGP key:http://people.FreeBSD.org/~roam/roam.key.asc Key fingerprint FDBA FD79 C26F 3C51 C95E DF9E ED18 B68D 1619 4553 yields falsehood, when appended to its quotation. yields falsehood, when appended to its quotation. pgpCPU36CpqdH.pgp Description: PGP signature
Re: memory mapped packet capturing - bpf replacement ?
Hi Ed, I did my benchmark by increasing the packet rate until I found the point at which packets started to be dropped. A bit offtopic - what traffic generator you use ? In my testing I found the call to microtime() to be quite expensive. (It will vary depending on which timecounter is being used.) I haven't added the timestamp to the header yet, so what would you recommed to use ? Is this in a SMP or uniprocesor environment? I think your gain from a ringbuffer interface will be more significant in the SMP case. I gonna test it much more on both SMP and UP machines Does the ng_hub cause the packet to be copied? If so you've still got the same number of copies as vanilla BPF. I think ng_hub does not do a copy, instead it passes mbufs through. Originally, I wanted this architecture: ng_ether / \ (lower) (upper) \/ ng_hub | ng_bpf | ng_mmq This way, you intercept all Ethernet traffic trough ng_hub. Then, ng_bpf does BPF filtering, if any. If no filtering is needed, then ng_bpf node may be omitted. And, at last, ng_mmq does queuing. Are you using the same snap length (or copying the entire packet) in each case? Hmm not sure what are you mean here. I am copying whole mbuf chain the same way BPF does. mbuf chain comes from the hook, and it can arrive to the hook from whatever source. As for question 3, be careful that you're atomically modifying the head and tail indices/pointers. But yes, you can do it without a mutex. Any points how to do that ? the only thing I can think of is having atomic variable in shared memory chunk, and using spinlock + atomic change. sergey ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: freebsd asm
[EMAIL PROTECTED] said this stuff: [...] [demon]~$ cat hello.asm %include 'system.inc' section .data holadb 'Hola', 0Ah hbytes equ $-hola section .text global _start _start: pushdword hbytes pushdword hola pushdword stdout sys.write pushdword 0 sys.exit [demon]~$ nasm -f elf hello.asm hello.asm:1: fatal: unable to open include file `system.inc' ?Where is that file?... the -current port of nasm is incomplete ? system.inc is not a part of nasm. Ok... we take some modifications... %include 'system.inc' pushdword stdout , and we replace it with pushdword 1 nasm also has no knowledge of sys.func style directives, so they are ignored. The directives in question would be located in the system.inc file that you don't have, and your program may as well be: holadb 'Hola', 0Ah hbytes equ $-hola section .text global _start _start: pushdword hbytes pushdword hola pushdword 1 pushdword 0 ... which doesn't exit, therefore your program accesses memory addresses that aren't meant to supply program code, and it crashes. Freebsd system calls are generally accessed using interrupt vector 0x80. The function that deals with this interrupt in the kernel expects the number of the system call in eax, and it expects the program to have called a function along the way. Thus, it's looking for the following stack structure: [][][][] : return address, inserted by 'call' instruction : descriptor vector : string address : number of bytes to write. To get this, you can try something like the following: holadb 'Hola', 0Ah hbytes equ $-hola section .text global _start _start: pushdword hbytes pushdword hola pushdword 1 mov eax,4 ; SYS_write calldoint pushdword 0 mov eax,1 ; SYS_exit calldoint doint: int 0x80 ret You can find the necessary system call numbers in /usr/include/sys/syscall.h . ari ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: freebsd asm
- Original Message - ... which doesn't exit, therefore your program accesses memory addresses that aren't meant to supply program code, and it crashes. Freebsd system calls are generally accessed using interrupt vector 0x80. The function that deals with this interrupt in the kernel expects the number of the system call in eax, and it expects the program to have called a function along the way. Thus, it's looking for the following stack structure: [][][][] : return address, inserted by 'call' instruction : descriptor vector : string address : number of bytes to write. ... You can find the necessary system call numbers in /usr/include/sys/syscall.h . Tnaks a lot! i understand all perfectly ;-) ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: memory mapped packet capturing - bpf replacement ?
On Mon, 14 Jun 2004, Ed Maste wrote: Now, the questions: 1. is my interpretation of benchmark results correct? 2. if they are correct, why bpf is faster? 3. is it OK to have no mutexes for ringbuffer operations ? don't forget that bpf is not copying the entire packet, just the header.. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: memory mapped packet capturing - bpf replacement ?
On Mon, Jun 14, 2004 at 10:15:14AM -0700, Julian Elischer wrote: don't forget that bpf is not copying the entire packet, just the header.. If you look at Sergey's benchmark.c, you will see that the snaplen was set to 32KB. BMS ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: memory mapped packet capturing - bpf replacement ?
On Mon, Jun 14, 2004 at 08:38:57AM -0400, Ed Maste wrote: Hello Sergey. I haven't looked at your code, but I'll provide some comments, having implemented a mmaped ringbuffer BPF replacement myself. We've had some prior interest in this. Do you have patches? If so, I'd be more than happy to look at them. Linux has something similar, but when I looked at the mechanism involved, I was loathe to adopt the same logic because the buffer(s) involved were allocated from userland and then mapped accordingly; we generally can't afford to take a page fault in that path, for mutex related reasons. Regards, BMS ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
FreeBSD or other BSD for no-MMU ARM processor ?
Does anyone know if there is a port of FreeBSD, or any of the other BSDs (e.g. NetBSD) for that matter, which will run on an ARM processor which does NOT have an MMU (Memory Management Unit)? Thanks, Gary ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: indent bugfix / added features
Chip Norkus [EMAIL PROTECTED] wrote I was off most of the weekend, but I did find one bug and have corrected it. The new patch can be found at http://telekinesis.org/indent/indent.patch. I whipped up a simple script to test it over /usr/src and that is at http://telekinesis.org/indent/test.sh. I found that my patched version (when run with no options) produced identical output to the existing version. don't know if you submit a PR, if not, man send-pr. Cyrille Lefevre. -- home: mailto:[EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
