the fact that this patch does solve a *specific* problem is the problem. I don't want to see into the code fixes for specific problems. Futhermore if somebody has a specific problem I don't see why everyone must be affected by this. In addition, why this patch is only for NetFlow? Why not for the basic ntop, for sFlow and others? The same applies to the white/black list addition.
Other things that I dislike is the .rpm-only format. You can release ntop 2.2 whenever you want, but I won't bless you until there are a .deb, .pkg (Solaris+FreeBSD), .dmg (OSX) and .rpm (RH) available or at least a way to do "make XXX" and build a XXX package.
Concerning the development ideas I read below all I can tell you is: 1. ntop needs to be futher simplified 2. ntop needs to be able to scale at Gbit speed 3. data persistency should be improved
Luca
Burton M. Strauss III wrote:
With the exception that the patch DOES solve a specific problem, basically I agree.
I would like to elaborate upon and revisit the proposal I made when we shipped 2.1...
Split ntop into modules:
A* A capture module - lean and mean and non-stop (failure proof). B* A filtering and processing module - takes the output from above interprets it and passes it to: C* An in-memory database module - which owns all of the current ntop internal table structures and can be accessed by: D* A display module (which can be the ntop web interface or intop's or something else)
We also may need
E* which is a timer driven processor to compute interval statistics.
A* is really a family - it can be nprobe, netFlow, sFlow, the current pcap stuff, a file read, or whatever, as long as it can capture the packet(s) and forward them to B*
B* is the descendent of the current pbuf.c, sessions.c etc. The basic B* flow is a couple of nested switch statements, but with pluggable protocol modules, allowing the user to install only what s/he wants or needs and to allow others to write small clean, focused modules (i.e. takes in a packet and returns an array of record(s) to be inserted into C*).:
switch (proto) { tcp: udp: default: (If plugin registered, call it)
And then, say
switch (tcp.port) { dns: sniff_dns ftp: sniff_ftp default: (If tcp.portx plugin registered, call it) }
We write the key ones and open things up for people with a particular interest (e.g. IPsec) to write small bits of code to enhance ntop.
C* becomes the owner of the data structures, can be enhanced to do more intelligent purges, use a database as a backing store, etc. The key here is that we have a real understanding of the data structures. We also gain the ability to load/unload all data, giving ntop 3.0 true persistence. I am NOT suggesting that C* be a SQL database - I think the overhead would kill us. But there are good, proven choices (e.g. Berkeley DB Transactional Data Store) which gain us the benefits (transaction based, integrity) without the overhead.
D* is the same old interface we know and love, except it can be a different one. Instead of groping through the tables like a charging Elephant, it nicely asks the C* module through defined interfaces for the data.
Again, we put a plugin strategy here, where plugins register themselves to appear on a specific interface tab | entry (e.g register_interface_plugin("Admin", "title", "adminxyzplugin", function(), parameter1, parameter2) would add a line, "title" to the "Admin" tab menu that causes a request to adminxyzplugin with two parameters). The plugin runs in a separate thread, but has access to C* and can send back to the user whatever it wants, including a redirect...
I'd also like to put in place a plugin event-like strategy in the B* module.
* At various points in the code (session init, session term, new host, host purge, etc.) we invoke a switchboard module with all the appropriate data passed to it (the signature).
* That module invokes each plugin registered for that event. If more than one plugin that fits the signature, they all get a chance at the "event". It simply needs to call everybody that fits the signature with the data.
* We could even make timed calls into the switchboard for our own housekeeping (instead of E*).
* We can rewrite the netFlow and sFlow sender routines into this model.
* Setting the points in ntop becomes a macro (we coerce the parameters to (void*) - the plugin knows what it's expecting and so it casts back...)
#define ntopEventPoint(eventID, p1, p2, p3, p4, p5) callNtopEventSwitchboard(eventID, (void *)&p1 ...)
* The plugin has access to other data through C* so we only pass the most critical pieces.
This makes 'em lean and mean. But since they run in separate threads, perhaps at a lower priority, it doesn't impact the basic processing of ntop. Don't want to do dns sniffing? Don't install the plugin. It forces ntop to do it's own queries, but if you're an ISP monitoring a few network links and don't care about your customers traffic, it's the right thing to do...
In closing, I actually think we do split our efforts.
We leave the current cvs open for bug fixes to 2.2, but no future enhancements.
We create an ntopNG or whatever (someday to be 3.0), build the infrastructure UP FRONT and RIGHT and then put the features functions into it.
The beauty? Since we're using a robust message passing architecture between the *s, we don't even have to code things in the same old C. We can use C++ or Java or even COBOL - what ever is right for the function. Want to provide true i18n - you can fix up the D* module and leave the rest alone. Or do 'em piecemeal. Whatever...
Want ntop to be an IDS? Write a B* plugin to view all the packets and feed them to snort...
etc....
-----Burton
-----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Luca Deri Sent: Wednesday, April 02, 2003 1:37 AM To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: fork(ntop) - Was: Re: [Ntop-dev] New ntop commit (author burton)
Burton, I think that once version 2.2 has been released we must sit down, relax and think what we want to do. My opinion is that we need soon an ntop fork: if you won't do that I will. In the past I have rewritten large parts of the application just because they were implemented the wrong way, or ntop used to include a lot of crap like rules for detecting intrusions that are now gone. IMHO, the current application is too fat and complex. For instance the patch below adds more complexity and doesn't solve any problem.
Said that, I don't want to split our development efforts, just build a fresh CVS root for a clean application. We can't simply include everything people ask in the source tree.
Basically it's time to rethink everything.
Regards, Luca
On Tue, 1 Apr 2003 [EMAIL PROTECTED] wrote:
Update of /export/home/ntop/ntop/plugins In directory jabber:/tmp/cvs-serv14058/plugins
Modified Files: netflowPlugin.c Log Message:
When you process netFlow data, there's one major issue, which is that ntop doesn't see the contents of the packets, just the header.
Accordingly, ntop can't classify ftp traffic (where the ftp server and client agree to use a random pair of ports) since it doesn't see the PORT command and so there's nothing for the netFlow plugin to classify.
This patch adds a ./configure parameter:
--enable-netflowassumeftpdata
Which tells ntop to classify the unclassified data (that is not netflow and ports > 1023) as ftp-data.
If you really, really understand your data, this might be a good assumption. But for most situations, this is probably not a good choice.
Regardless, this patch adds some stats to the netFlow plugin which will show you what packets were thrown away.
-----Burton (ref 257)
_______________________________________________ Ntop-dev mailing list [EMAIL PROTECTED] http://listgateway.unipi.it/mailman/listinfo/ntop-dev
_______________________________________________ Ntop-dev mailing list [EMAIL PROTECTED] http://listgateway.unipi.it/mailman/listinfo/ntop-dev
_______________________________________________ Ntop-dev mailing list [EMAIL PROTECTED] http://listgateway.unipi.it/mailman/listinfo/ntop-dev
