Hi all, Hope you're doing fine. I was able to add packet sockets and the functions provided by Hashpipe in hashpipe_pktsock.h, but I get permission issues when trying to capture packets as a non-root user.
The method I am trying to use to overcome this is owning the plugins/executables as root and using the setuid flag to give root privileges to hashpipe. At this point, I still get an 'operation not permitted' when trying to open the socket. Then when trying to use the CAP_NET_RAW privilege (setcap cap_net_raw=pe 'program'), I'm told that the operation is not supported. Just to be clear, I don't have any of these issues when running the process as root, but I'd rather have non-root users running hashpipe. How were you able to overcome the permission issues when trying to capture raw packets with hashpipe as a non-root user? If you were running it as a non-root user. Let me know whether you need any more information or whether I'm not stating anything clearly. Thanks a lot for the help. Mark Ruzindana On Tue, Mar 31, 2020 at 5:08 PM Mark Ruzindana <ruziem...@gmail.com> wrote: > Thanks a lot for the quick responses John and David! I really appreciate > it. > > I will definitely be updating the version of Hashpipe that I currently > have on the server as well as ensure that the network tuning is good. > > I'm currently using the standard "socket()" function, and a switch to > packet sockets, with the description that you gave, seems like it will > definitely be beneficial. > > I also currently pin the threads to the desired cores with a "-c #" on the > command line, but thank you for mentioning it, I might have not been doing > so. The NUMA info is also very helpful. I'll make sure that the > architecture is as optimal as it should be. > > Thanks again! This was very helpful and I'll update you with the progress > that I make. > > Mark > > > > > On Tue, Mar 31, 2020 at 4:38 PM David MacMahon <dav...@berkeley.edu> > wrote: > >> Just to expand on John's excellent tips, Hashpipe does lock its shared >> memory buffers with mlock. These buffers will have the NUMA node affinity >> of the thread that created them so be sure to pin the threads to the >> desired core or cores by preceding the thread names on the command line >> with a -c # (set thread affinity to a single core) or -m # (set thread >> affinity to multiple cores) option. Alternatively (or additional) you can >> run the entire hashpipe process with numactl. For example... >> >> numactl --cpunodebind=1 --membind=1 hashpipe [...] >> >> ...will restrict hashpipe and all its threads to run on NUMA node 1 and >> all memory allocations will (to the extent possible) be made within memory >> that is affiliated with NUMA node 1. You can use various tools to find out >> which hardware is associated with which NUMA node such as "numactl >> --hardware" or "lstopo". Hashpipe includes its own such utility: >> "hashpipe_topology.sh". >> >> On NUMA (i.e. multi-socket) systems, each PCIe slot is associated with a >> specific NUMA node. It can be beneficial to have relevant peripherals >> (e.g. NIC and GPU) be in PCIe slots that are on the same NUMA node. >> >> Of course, if you have as single socket mainboard, then all this NUMA >> stuff is irrelevant. :P >> >> Cheers, >> Dave >> >> On Mar 31, 2020, at 15:04, John Ford <jmfor...@gmail.com> wrote: >> >> >> >> Hi Mark. Since the newer version has a script called >> "hashpipe_irqaffinity.sh" I would think that the most expedient thing to do >> is to upgrade to the newer version. It's likely to fix some or all of this. >> >> That said, there are a lot of things that you can check, and not only the >> irq affinity, but also make sure that your network tuning is good, that >> your network card irqs are attached to processes where the memory is local >> to that processor, and that the hashpipe threads are mapped to processor >> cores that are also local to that memory. Sometimes it's >> counterproductive to map processes to processor cores by themselves if they >> need data that is produced by a different core that's far away, NUMA-wise. >> And lock all the memory in core with mlockall() or one of his friends. >> >> Good luck with it! >> >> John >> >> >> >> >> On Tue, Mar 31, 2020 at 12:09 PM Mark Ruzindana <ruziem...@gmail.com> >> wrote: >> >>> Hi all, >>> >>> I am fairly new to asking questions on a forum so if I need to provide >>> more details, please let me know. >>> >>> Worth noting that just as I was about to send this out, I checked and I >>> don't have the most recent version of HASHPIPE with hashpipe_irqaffinity.sh >>> among other additions and modifications. So this might fix my problem, but >>> maybe not and someone else has more insight. I will update everyone if it >>> does. >>> >>> I am trying to reduce the number of packets lost/dropped when running >>> HASHPIPE on a 32 core RHEL 7 server. I have run enough tests and >>> diagnostics to be confident that the problem is not any HASHPIPE thread >>> running for too long. Also, the percentage of packets dropped on any given >>> scan is between about 0.3 and 0.8%. Approx. 5,000 packets in a 30 second >>> scan with a total of 1,650,000 packets. So while it's a small percentage, >>> the number of packets lost is still quite large. I have also done enough >>> tests with 'top', 'iostat' as well as timing HASHPIPE in between time >>> windows where there are no packets dropped to diagnose the issue further. I >>> (as well as my colleagues) have come to the conclusion that the kernel is >>> allowing processes to interrupt HASHPIPE as it is running. >>> >>> So I have researched and run tests involving 'niceness' and I am >>> currently trying to configure smp affinities and irq balancing, but the >>> changes that I make to the smp_affinity files aren't doing anything. My >>> plan was to have the interrupts run on the 20 cores that aren't being used >>> by HASHPIPE. Also, disabling 'irqbalance' didn't do anything either. I also >>> restarted the machine to see whether the changes made are permanent, but >>> the system reverts back to what it was. >>> >>> I might be missing something, or trying the wrong things. Has anyone >>> experienced this? And could you point me in the right direction if you have >>> any insight? >>> >>> If you need anymore details, please let me know. I didn't add as much as >>> I could because I wanted this to be a reasonably sized message. >>> >>> Thanks, >>> >>> Mark Ruzindana >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "casper@lists.berkeley.edu" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to casper+unsubscr...@lists.berkeley.edu. >>> To view this discussion on the web visit >>> https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CA%2B41hpxcwSQT-EsjuyqXpGmmBzykDeLt6JbfUUg_ZYpkXyat2w%40mail.gmail.com >>> <https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CA%2B41hpxcwSQT-EsjuyqXpGmmBzykDeLt6JbfUUg_ZYpkXyat2w%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "casper@lists.berkeley.edu" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to casper+unsubscr...@lists.berkeley.edu. >> To view this discussion on the web visit >> https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CABmH8B_4MoNDsO4yZNYH608u6DVtbSPkKz0YBS8%2Bb%3DffqS%3DwaA%40mail.gmail.com >> <https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CABmH8B_4MoNDsO4yZNYH608u6DVtbSPkKz0YBS8%2Bb%3DffqS%3DwaA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "casper@lists.berkeley.edu" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to casper+unsubscr...@lists.berkeley.edu. >> To view this discussion on the web visit >> https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/F1F8AB17-9030-4875-8792-69CBA99F816F%40berkeley.edu >> <https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/F1F8AB17-9030-4875-8792-69CBA99F816F%40berkeley.edu?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "casper@lists.berkeley.edu" group. To unsubscribe from this group and stop receiving emails from it, send an email to casper+unsubscr...@lists.berkeley.edu. To view this discussion on the web visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CA%2B41hpwo%2B2p%3Dohrb9jPOUyuhDs6uErNowE%2BvM4%3D67eu_Q_0KiQ%40mail.gmail.com.
