Enrico
you're not asking much: go ahead and send us a patch we'll include in
ntop. We're busy recoding selected ntop sections
Luca
On 11/29/2011 11:40 AM, Enrico wrote:
am i asking too much? :)
On 11/20/2011 05:36 PM, Enrico wrote:
Hello Luca,
i would like to bind the data processing thread to a specific core.
For this i need to know the thread ID.
I know that there is only one thread for this task per instance and
other threads are doing different things.
As i have explained in the first post, in our systems we have the
NICs interrupts that are assigned to the same core that runs the
Snort Instance.
This is not a performance problem, but doing so, we increase the
process locality per core, and we optimize the use of the cache,
keeping the processing of the data acquired from the NIC on the same
core where the hardware interrupt for the data chunk is handled.
In our system we have seen great improvements, also with pfring.
We use 'taskset' to do this in the case of snort.
But ntop is multitheaded and this would bind all the 10 threads
created by ntop on the same core, creating a mess.
I need to know automatically the thread ID (the thread PID) after it
has been created.
One alternative would be modifying the ntop code and put a parameter
that takes input the cpumask and then use sched_setaffinity in the
code like for the snort_daq.
It it more clear now?
On 11/20/2011 05:07 PM, Luca Deri wrote:
Enrico
what do you want to do exactly? The way ntop has been designed is to
have one processing thread per interface, with each packet updating
immediately the ntop datastructures. This is very costly but it
gives you an updated (not delayed) view of the system. If you want
to improve the performance you need to use external probes, such as
nProbe, that preprocess the traffic on ntop's behalf
Luca
On Nov 20, 2011, at 3:55 PM, Enrico wrote:
ideas?
On 11/13/2011 08:37 AM, Enrico wrote:
On 11/12/2011 05:12 PM, Burton Strauss III wrote:
You need the actual THREAD id
- check the man page on ps.
i know, but even if i can abviously list the created threads in
many different ways with 'ps',
i would not identity the data collection thread ID using simply ps
output at runtime.
But it clearly
shows both the child and parent PID.
please show me where it shows in the ntop output?
i still see only the parent PID. In the lines produced by ntop on
the very end of a few lines you can see [p1434] that in my case is
the parent ID.
I cannot see any other number that can possible be related it the
child threads.
After the 'THREADMGMT' string there are additional values like
't140309304494336'
do you mean this? This is not a thread PID.
What i need is the thread PID, obviously after it has been created
by the father process, to pass it as a 'taskset' argument, and set
the CPU affinity only on this thread.
I cannot do this, by passing the father PID to 'taskset', because
for newly created threads the system will assign the same cpu
affinity.
There might be other methods of finding out the thread ID, for
example looking in detail at the IO for each thread using linux
system monitoring tools,
but i need to find it automatically when ntop starts and after
that apply the cpu affinity.
It depends on your OS, some create threads from fork() some full
processes.
That's why we show both.
i think this is not the problem here.
i'm on a RHEL 6.1 with the usual pthread library which ntop uses.
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Enrico
Sent: Friday, November 11, 2011 3:15 PM
To: [email protected]
Subject: Re: [Ntop] ntop - set cpu affinity for data collection
thread
this is simply not true, at least in my system.
as i have written in my first post the PID showed is the one of
the father
process.
grep -i threadmgmt /var/log/messages
Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309304494336]:
Now running as a daemon
Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309268899584]:
SFP: Started thread for fingerprinting
Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309268899584]:
SFP: Fingerprint scan thread starting [p1434]
Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309260506880]:
SIH: Idle host scan thread starting [p1434]
Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309260506880]:
SIH: Started thread for idle hosts detection
Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309252114176]:
DNSAR(1): Started thread for DNS address resolution
Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309252114176]:
DNSAR(1): Address resolution thread running
Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309243721472]:
DNSAR(2): Address resolution thread running
Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309243721472]:
DNSAR(2): Started thread for DNS address resolution
Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309235328768]:
DNSAR(3): Started thread for DNS address resolution
Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309235328768]:
DNSAR(3): Address resolution thread running
Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309226936064]:
INITWEB: Started thread for web server
Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309226936064]:
WEB: Server connection thread starting [p1434]
Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140309226936064]:
WEB: Server connection thread running [p1434]
Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT: RRD: Started thread
(t140308864038656) for data collection
Nov 11 16:38:21 ids01 ntop[1434]: THREADMGMT[t140308864038656]:
RRD: Data collection thread starting [p1434]
Nov 11 16:38:22 ids01 ntop[1434]: THREADMGMT[t140309304494336]:
ntop RUNSTATE: INITNONROOT(3)
Nov 11 16:38:22 ids01 ntop[1434]: THREADMGMT[t140309304494336]:
ntop RUNSTATE: RUN(4)
Nov 11 16:38:22 ids01 ntop[1434]: THREADMGMT[t140308855645952]:
NPS(1): Started thread for network packet sniffing [eth0]
Nov 11 16:38:22 ids01 ntop[1434]: THREADMGMT[t140308855645952]:
NPS(eth0): pcapDispatch thread starting [p1434]
Nov 11 16:38:22 ids01 ntop[1434]: THREADMGMT[t140309268899584]:
SFP: Fingerprint scan thread running [p1434]
Nov 11 16:38:22 ids01 ntop[1434]: THREADMGMT[t140309260506880]:
SIH: Idle host scan thread running [p1434]
Nov 11 16:38:22 ids01 ntop[1434]: THREADMGMT[t140308855645952]:
NPS(eth0): pcapDispatch thread running [p1434]
Nov 11 16:38:32 ids01 ntop[1434]: THREADMGMT[t140308847253248]:
RRD: Started thread for throughput data collection
Nov 11 16:38:32 ids01 ntop[1434]: THREADMGMT[t140308864038656]:
RRD: Data collection thread running [p1434]
Nov 11 16:38:32 ids01 ntop[1434]: THREADMGMT[t140308847253248]:
RRD: Throughput data collection: Thread starting [p1434]
Nov 11 16:38:32 ids01 ntop[1434]: THREADMGMT[t140308847253248]:
RRD: Throughput data collection: Thread running [p1434]
ps -efL | grep 1434
ntop 1434 1 1434 0 10 16:38 ? 00:00:00 /usr/local/bin/ntop
@/usr/local/etc/ntop/eth0-start --protocols
/usr/local/etc/ntop/eth0-proto -d
ntop 1434 1 1436 0 10 16:38 ? 00:00:00 /usr/local/bin/ntop
@/usr/local/etc/ntop/eth0-start --protocols
/usr/local/etc/ntop/eth0-proto -d
ntop 1434 1 1437 0 10 16:38 ? 00:00:01 /usr/local/bin/ntop
@/usr/local/etc/ntop/eth0-start --protocols
/usr/local/etc/ntop/eth0-proto -d
ntop 1434 1 1438 0 10 16:38 ? 00:00:00 /usr/local/bin/ntop
@/usr/local/etc/ntop/eth0-start --protocols
/usr/local/etc/ntop/eth0-proto -d
ntop 1434 1 1439 0 10 16:38 ? 00:00:00 /usr/local/bin/ntop
@/usr/local/etc/ntop/eth0-start --protocols
/usr/local/etc/ntop/eth0-proto -d
ntop 1434 1 1440 0 10 16:38 ? 00:00:00 /usr/local/bin/ntop
@/usr/local/etc/ntop/eth0-start --protocols
/usr/local/etc/ntop/eth0-proto -d
ntop 1434 1 1443 0 10 16:38 ? 00:00:05 /usr/local/bin/ntop
@/usr/local/etc/ntop/eth0-start --protocols
/usr/local/etc/ntop/eth0-proto -d
ntop 1434 1 1445 0 10 16:38 ? 00:00:00 /usr/local/bin/ntop
@/usr/local/etc/ntop/eth0-start --protocols
/usr/local/etc/ntop/eth0-proto -d
ntop 1434 1 1448 7 10 16:38 ? 00:25:55 /usr/local/bin/ntop
@/usr/local/etc/ntop/eth0-start --protocols
/usr/local/etc/ntop/eth0-proto -d
ntop 1434 1 1477 0 10 16:38 ? 00:00:00 /usr/local/bin/ntop
@/usr/local/etc/ntop/eth0-start --protocols
/usr/local/etc/ntop/eth0-proto -d
root 4308 3626 4308 0 1 22:16 pts/0 00:00:00 grep 1434
still waiting for a reply. Eventually is accept suggestions for
modifying
the code that forks this particular thread.
thanks.
On 11/11/2011 06:41 PM, Burton Strauss III wrote:
Read the log - look for the THREADMGMT messages. They show thread
creation and WHAT each thread is.
-----Burton
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Enrico
Sent: Thursday, November 10, 2011 8:26 AM
To: [email protected]
Subject: [Ntop] ntop - set cpu affinity for data collection thread
Hello
we have seen that the snort-daq module, that comes with pfring, can
bind the data acquistion process to a particular cpu subset with
the
'sched_setaffinity (2)' function.
This is really useful for reducing interrupts handling time in our
system
We would like to be able to do the same thing also for ntop.
Unlike snort we see that it is an high parallelized application and
usually creates almost 10 threads when starts.
Using Taskset would bind all the 10 threads to a single cpu, but
brings to resources competition between ntop threads on the same
core
which increases a lot the context switching time.
Is there a way to know what is the data acquisition/collection
thread
for a ntop instance that actually handles the packets coming
from the
kernel?
Looking at ntop logs for example only shows what is the father
process
creating that thread.
Any suggestion?
Thanks in advance.
Enrico.
_______________________________________________
Ntop mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop
_______________________________________________
Ntop mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop
_______________________________________________
Ntop mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop
---
Bildung ist kein Verbrechen
_______________________________________________
Ntop mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop
_______________________________________________
Ntop mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop
