Enrico what do you want to do exactly? The way ntop has been designed is to have one processing thread per interface, with each packet updating immediately the ntop datastructures. This is very costly but it gives you an updated (not delayed) view of the system. If you want to improve the performance you need to use external probes, such as nProbe, that preprocess the traffic on ntop's behalf
Luca On Nov 20, 2011, at 3:55 PM, Enrico wrote: > ideas? > > On 11/13/2011 08:37 AM, Enrico wrote: >> On 11/12/2011 05:12 PM, Burton Strauss III wrote: >>> You need the actual THREAD id >>> - check the man page on ps. >> >> i know, but even if i can abviously list the created threads in many >> different ways with 'ps', >> i would not identity the data collection thread ID using simply ps output at >> runtime. >> >>> But it clearly >>> shows both the child and parent PID. >> >> please show me where it shows in the ntop output? >> i still see only the parent PID. In the lines produced by ntop on the very >> end of a few lines you can see [p1434] that in my case is the parent ID. >> I cannot see any other number that can possible be related it the child >> threads. >> >> After the 'THREADMGMT' string there are additional values like >> 't140309304494336' >> do you mean this? This is not a thread PID. >> >> What i need is the thread PID, obviously after it has been created by the >> father process, to pass it as a 'taskset' argument, and set the CPU affinity >> only on this thread. >> I cannot do this, by passing the father PID to 'taskset', because for newly >> created threads the system will assign the same cpu affinity. >> >> There might be other methods of finding out the thread ID, for example >> looking in detail at the IO for each thread using linux system monitoring >> tools, >> but i need to find it automatically when ntop starts and after that apply >> the cpu affinity. >> >>> >>> It depends on your OS, some create threads from fork() some full processes. >>> That's why we show both. >> >> i think this is not the problem here. >> i'm on a RHEL 6.1 with the usual pthread library which ntop uses. >> >>> >>> -----Original Message----- >>> From: [email protected] >>> [mailto:[email protected]] On Behalf Of Enrico >>> Sent: Friday, November 11, 2011 3:15 PM >>> To: [email protected] >>> Subject: Re: [Ntop] ntop - set cpu affinity for data collection thread >>> >>> this is simply not true, at least in my system. >>> as i have written in my first post the PID showed is the one of the father >>> process. >>> >>> grep -i threadmgmt /var/log/messages >>> Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309304494336]: Now running >>> as a daemon >>> Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309268899584]: SFP: >>> Started thread for fingerprinting >>> Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309268899584]: SFP: >>> Fingerprint scan thread starting [p1434] >>> Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309260506880]: SIH: Idle >>> host scan thread starting [p1434] >>> Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309260506880]: SIH: >>> Started thread for idle hosts detection >>> Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309252114176]: DNSAR(1): >>> Started thread for DNS address resolution >>> Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309252114176]: DNSAR(1): >>> Address resolution thread running >>> Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309243721472]: DNSAR(2): >>> Address resolution thread running >>> Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309243721472]: DNSAR(2): >>> Started thread for DNS address resolution >>> Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309235328768]: DNSAR(3): >>> Started thread for DNS address resolution >>> Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309235328768]: DNSAR(3): >>> Address resolution thread running >>> Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309226936064]: INITWEB: >>> Started thread for web server >>> Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309226936064]: WEB: Server >>> connection thread starting [p1434] >>> Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309226936064]: WEB: Server >>> connection thread running [p1434] >>> Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT: RRD: Started thread >>> (t140308864038656) for data collection >>> Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140308864038656]: RRD: Data >>> collection thread starting [p1434] >>> Nov 11 16:38:22 ids01 ntop[1434]: THREADMGMT[t140309304494336]: ntop >>> RUNSTATE: INITNONROOT(3) >>> Nov 11 16:38:22 ids01 ntop[1434]: THREADMGMT[t140309304494336]: ntop >>> RUNSTATE: RUN(4) >>> Nov 11 16:38:22 ids01 ntop[1434]: THREADMGMT[t140308855645952]: NPS(1): >>> Started thread for network packet sniffing [eth0] >>> Nov 11 16:38:22 ids01 ntop[1434]: THREADMGMT[t140308855645952]: NPS(eth0): >>> pcapDispatch thread starting [p1434] >>> Nov 11 16:38:22 ids01 ntop[1434]: THREADMGMT[t140309268899584]: SFP: >>> Fingerprint scan thread running [p1434] >>> Nov 11 16:38:22 ids01 ntop[1434]: THREADMGMT[t140309260506880]: SIH: Idle >>> host scan thread running [p1434] >>> Nov 11 16:38:22 ids01 ntop[1434]: THREADMGMT[t140308855645952]: NPS(eth0): >>> pcapDispatch thread running [p1434] >>> Nov 11 16:38:32 ids01 ntop[1434]: THREADMGMT[t140308847253248]: RRD: >>> Started thread for throughput data collection >>> Nov 11 16:38:32 ids01 ntop[1434]: THREADMGMT[t140308864038656]: RRD: Data >>> collection thread running [p1434] >>> Nov 11 16:38:32 ids01 ntop[1434]: THREADMGMT[t140308847253248]: RRD: >>> Throughput data collection: Thread starting [p1434] >>> Nov 11 16:38:32 ids01 ntop[1434]: THREADMGMT[t140308847253248]: RRD: >>> Throughput data collection: Thread running [p1434] >>> >>> ps -efL | grep 1434 >>> ntop 1434 1 1434 0 10 16:38 ? 00:00:00 /usr/local/bin/ntop >>> @/usr/local/etc/ntop/eth0-start --protocols /usr/local/etc/ntop/eth0-proto >>> -d >>> ntop 1434 1 1436 0 10 16:38 ? 00:00:00 /usr/local/bin/ntop >>> @/usr/local/etc/ntop/eth0-start --protocols /usr/local/etc/ntop/eth0-proto >>> -d >>> ntop 1434 1 1437 0 10 16:38 ? 00:00:01 /usr/local/bin/ntop >>> @/usr/local/etc/ntop/eth0-start --protocols /usr/local/etc/ntop/eth0-proto >>> -d >>> ntop 1434 1 1438 0 10 16:38 ? 00:00:00 /usr/local/bin/ntop >>> @/usr/local/etc/ntop/eth0-start --protocols /usr/local/etc/ntop/eth0-proto >>> -d >>> ntop 1434 1 1439 0 10 16:38 ? 00:00:00 /usr/local/bin/ntop >>> @/usr/local/etc/ntop/eth0-start --protocols /usr/local/etc/ntop/eth0-proto >>> -d >>> ntop 1434 1 1440 0 10 16:38 ? 00:00:00 /usr/local/bin/ntop >>> @/usr/local/etc/ntop/eth0-start --protocols /usr/local/etc/ntop/eth0-proto >>> -d >>> ntop 1434 1 1443 0 10 16:38 ? 00:00:05 /usr/local/bin/ntop >>> @/usr/local/etc/ntop/eth0-start --protocols /usr/local/etc/ntop/eth0-proto >>> -d >>> ntop 1434 1 1445 0 10 16:38 ? 00:00:00 /usr/local/bin/ntop >>> @/usr/local/etc/ntop/eth0-start --protocols /usr/local/etc/ntop/eth0-proto >>> -d >>> ntop 1434 1 1448 7 10 16:38 ? 00:25:55 /usr/local/bin/ntop >>> @/usr/local/etc/ntop/eth0-start --protocols /usr/local/etc/ntop/eth0-proto >>> -d >>> ntop 1434 1 1477 0 10 16:38 ? 00:00:00 /usr/local/bin/ntop >>> @/usr/local/etc/ntop/eth0-start --protocols /usr/local/etc/ntop/eth0-proto >>> -d >>> root 4308 3626 4308 0 1 22:16 pts/0 00:00:00 grep 1434 >>> >>> still waiting for a reply. Eventually is accept suggestions for modifying >>> the code that forks this particular thread. >>> >>> thanks. >>> >>> >>> On 11/11/2011 06:41 PM, Burton Strauss III wrote: >>>> Read the log - look for the THREADMGMT messages. They show thread >>>> creation and WHAT each thread is. >>>> >>>> >>>> >>>> -----Burton >>>> >>>> -----Original Message----- >>>> From: [email protected] >>>> [mailto:[email protected]] On Behalf Of Enrico >>>> Sent: Thursday, November 10, 2011 8:26 AM >>>> To: [email protected] >>>> Subject: [Ntop] ntop - set cpu affinity for data collection thread >>>> >>>> Hello >>>> >>>> we have seen that the snort-daq module, that comes with pfring, can >>>> bind the data acquistion process to a particular cpu subset with the >>>> 'sched_setaffinity (2)' function. >>>> >>>> This is really useful for reducing interrupts handling time in our >>>> system >>>> >>>> We would like to be able to do the same thing also for ntop. >>>> Unlike snort we see that it is an high parallelized application and >>>> usually creates almost 10 threads when starts. >>>> >>>> Using Taskset would bind all the 10 threads to a single cpu, but >>>> brings to resources competition between ntop threads on the same core >>>> which increases a lot the context switching time. >>>> >>>> Is there a way to know what is the data acquisition/collection thread >>>> for a ntop instance that actually handles the packets coming from the >>> kernel? >>>> >>>> Looking at ntop logs for example only shows what is the father process >>>> creating that thread. >>>> >>>> Any suggestion? >>>> >>>> Thanks in advance. >>>> >>>> Enrico. >>>> >>>> _______________________________________________ >>>> Ntop mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>> >>> >>> _______________________________________________ >>> Ntop mailing list >>> [email protected] >>> http://listgateway.unipi.it/mailman/listinfo/ntop > > _______________________________________________ > Ntop mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop --- Bildung ist kein Verbrechen _______________________________________________ Ntop mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop
