Re: IP Acounting Idea for 2.5

[Mark van Walraven]

On Fri, Apr 20, 2001 at 02:51:55AM +, Ton Hospel wrote:
> Resettable counters are evil.

Perhaps "evil" should be reserved to describe counters which automatically
reset as a side effect of being read.

> I really think cisco got this right: from the commandline interface
> you can reset counters, and watch them, the SNMP counters however just
> keep going and going and going independently from this.

Except for the IP accounting table?  The 'checkpoint' operation copies and
*clears* the table ...

I don't really need snapshots, just hazard-free reads.  It's quite
easy to work out deltas in userspace, but I'd hate for things to be
unnecessarily painful for multiple readers.

Mark.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Mark van Walraven]

On Fri, Apr 20, 2001 at 02:51:55AM +, Ton Hospel wrote:
 Resettable counters are evil.

Perhaps evil should be reserved to describe counters which automatically
reset as a side effect of being read.

 I really think cisco got this right: from the commandline interface
 you can reset counters, and watch them, the SNMP counters however just
 keep going and going and going independently from this.

Except for the IP accounting table?  The 'checkpoint' operation copies and
*clears* the table ...

I don't really need snapshots, just hazard-free reads.  It's quite
easy to work out deltas in userspace, but I'd hate for things to be
unnecessarily painful for multiple readers.

Mark.
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Counters [Re: IP Acounting Idea for 2.5]

[Harald Welte]

On Tue, Apr 17, 2001 at 12:29:30PM +0200, Olaf Titz wrote:
> > Umm, no.  Counters can be resetable - you just specify that accounting
> > programs should not reset them, ever.
> >
> > The ability to reset counters is extremely useful if you're a human
> > looking at the output of iptables -L -v.  (I thus far know of no one
> > who can memorise the counter values for around 40 rules).
> 
> You'll get bogus accounting results unless you stop/restart the
> accounting programs every time you manually deal with the counters.
> This sounds dangerously easy to make mistakes to me.

mmh. But isn't that obvious? Every time you mess with anything (counters
or rules) of the chains/tables, your accounting program will get into
trouble, if it relies on those counters.

I don't think that it makes sense to do iptables-based accounting if you have
changes to the ruleset (counters and/or rules) at runtime. You'd have 
to be very cautious what you are doing.

> Olaf

-- 
Live long and prosper
- Harald Welte / [EMAIL PROTECTED]http://www.gnumonks.org

GCS/E/IT d- s-: a-- C+++ UL$ P+++ L$ E--- W- N++ o? K- w--- O- M- 
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Harald Welte]

On Tue, Apr 17, 2001 at 06:56:42AM +, Henning P. Schmiedehausen wrote:
> 
> Resettable counters in a security sensitive environment are just a
> call for trouble. That's why you can't reset the SNMP counters on any
> Cisco device I've encountered today. They learned their lesson. Maybe
> you will, too.

Well, I'm not sure about which SNTP counters you are talking, but I suppose
it is not about per-filtering-rule counters, but something like per-interface
counters, etc.

There's always a way for somebody with root access to reset the counters of
a rule: 

just delete and re-insert the rule.

If somebody wants to reset the counter, he can. If we remove the functionality
from iptables, people still can - but it's more difficult.

>   Regards
>   Henning

-- 
Live long and prosper
- Harald Welte / [EMAIL PROTECTED]http://www.gnumonks.org

GCS/E/IT d- s-: a-- C+++ UL$ P+++ L$ E--- W- N++ o? K- w--- O- M- 
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Counters] Re: IP Acounting Idea for 2.5

[Harald Welte]

On Tue, Apr 17, 2001 at 11:13:19AM +1000, Manfred Bartz wrote:

> I had a brief look at MRTG.  It seems to be a well written app and
> while it can handle counter reset (with potential loss of an unknown 
> amount of data), it does not actively reset counters.  It also doesn't
> use iptables.

Yes, that's the whole point. If you want to have some kind of per-ip / 
per-network accounting, I'd rather recommend using something else
(i.e. an accounting module attaching to the netfilter hook itself,
something using the ULOG target to do the logging in userspace, ...)

Just reading out the per-rule counters always introduces problems as soon
as your ruleset changes. And when do you have a really static ruleset?
There are always new services/... to configure.

> Agreed too.  Counters should not arbitrarily be equipped with a reset
> capability, there is hardly any benefit in that and it causes nothing
> but problems.

so what about iptables-save at shutdown time and iptables-restore at 
bootup time? Then you can have your counters persist even after kernel
upgrades / reboots / crashes / ... 

> As far as I can see, the counters in /proc/net/snmp don't have a
> reset, same with /proc/net/dev and possibly other counters elsewhere.

Yes, because it is per network device, not per some arbitrarily inserted
rule which can be changed all the time. As I've stated more than once
in this thread, and as you just continue to say: just delete and re-insert
the rule, and you have your counter reset.

> Ideally iptables would fall in line with that.  Rules can still be
> unloaded and reloaded, also causing counter reset and loss of data,
> but since that is a lot more involved, application authors would have
> an incentive to handle counters properly.

I don't think that the iptables kernel part should remove some feature
just because there are application programmers wrongly designing their
applications.


> Manfred Bartz

-- 
Live long and prosper
- Harald Welte / [EMAIL PROTECTED]http://www.gnumonks.org

GCS/E/IT d- s-: a-- C+++ UL$ P+++ L$ E--- W- N++ o? K- w--- O- M- 
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Counters] Re: IP Acounting Idea for 2.5

[Harald Welte]

On Tue, Apr 17, 2001 at 11:13:19AM +1000, Manfred Bartz wrote:

 I had a brief look at MRTG.  It seems to be a well written app and
 while it can handle counter reset (with potential loss of an unknown 
 amount of data), it does not actively reset counters.  It also doesn't
 use iptables.

Yes, that's the whole point. If you want to have some kind of per-ip / 
per-network accounting, I'd rather recommend using something else
(i.e. an accounting module attaching to the netfilter hook itself,
something using the ULOG target to do the logging in userspace, ...)

Just reading out the per-rule counters always introduces problems as soon
as your ruleset changes. And when do you have a really static ruleset?
There are always new services/... to configure.

 Agreed too.  Counters should not arbitrarily be equipped with a reset
 capability, there is hardly any benefit in that and it causes nothing
 but problems.

so what about iptables-save at shutdown time and iptables-restore at 
bootup time? Then you can have your counters persist even after kernel
upgrades / reboots / crashes / ... 

 As far as I can see, the counters in /proc/net/snmp don't have a
 reset, same with /proc/net/dev and possibly other counters elsewhere.

Yes, because it is per network device, not per some arbitrarily inserted
rule which can be changed all the time. As I've stated more than once
in this thread, and as you just continue to say: just delete and re-insert
the rule, and you have your counter reset.

 Ideally iptables would fall in line with that.  Rules can still be
 unloaded and reloaded, also causing counter reset and loss of data,
 but since that is a lot more involved, application authors would have
 an incentive to handle counters properly.

I don't think that the iptables kernel part should remove some feature
just because there are application programmers wrongly designing their
applications.


 Manfred Bartz

-- 
Live long and prosper
- Harald Welte / [EMAIL PROTECTED]http://www.gnumonks.org

GCS/E/IT d- s-: a-- C+++ UL$ P+++ L$ E--- W- N++ o? K- w--- O- M- 
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Harald Welte]

On Tue, Apr 17, 2001 at 06:56:42AM +, Henning P. Schmiedehausen wrote:
 
 Resettable counters in a security sensitive environment are just a
 call for trouble. That's why you can't reset the SNMP counters on any
 Cisco device I've encountered today. They learned their lesson. Maybe
 you will, too.

Well, I'm not sure about which SNTP counters you are talking, but I suppose
it is not about per-filtering-rule counters, but something like per-interface
counters, etc.

There's always a way for somebody with root access to reset the counters of
a rule: 

just delete and re-insert the rule.

If somebody wants to reset the counter, he can. If we remove the functionality
from iptables, people still can - but it's more difficult.

   Regards
   Henning

-- 
Live long and prosper
- Harald Welte / [EMAIL PROTECTED]http://www.gnumonks.org

GCS/E/IT d- s-: a-- C+++ UL$ P+++ L$ E--- W- N++ o? K- w--- O- M- 
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Counters [Re: IP Acounting Idea for 2.5]

[Harald Welte]

On Tue, Apr 17, 2001 at 12:29:30PM +0200, Olaf Titz wrote:
  Umm, no.  Counters can be resetable - you just specify that accounting
  programs should not reset them, ever.
 
  The ability to reset counters is extremely useful if you're a human
  looking at the output of iptables -L -v.  (I thus far know of no one
  who can memorise the counter values for around 40 rules).
 
 You'll get bogus accounting results unless you stop/restart the
 accounting programs every time you manually deal with the counters.
 This sounds dangerously easy to make mistakes to me.

mmh. But isn't that obvious? Every time you mess with anything (counters
or rules) of the chains/tables, your accounting program will get into
trouble, if it relies on those counters.

I don't think that it makes sense to do iptables-based accounting if you have
changes to the ruleset (counters and/or rules) at runtime. You'd have 
to be very cautious what you are doing.

 Olaf

-- 
Live long and prosper
- Harald Welte / [EMAIL PROTECTED]http://www.gnumonks.org

GCS/E/IT d- s-: a-- C+++ UL$ P+++ L$ E--- W- N++ o? K- w--- O- M- 
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Ton Hospel]

In article <[EMAIL PROTECTED]>,
Alan Cox <[EMAIL PROTECTED]> writes:
>> > No he isnt confused, you are trying to dictate policy.
>> 
>> What then *is* the policy?
> 
> The policy is not to have policy. It works as well in kernel design as politics.
> 
> Alan
> 
Since my job is in fact mainly this kind of apps, i really feel strongly
about this. 

Resettable counters are evil.

Having resettable counters may not sound like it, but it is in fact policy.
It forces all apps to add code to detect these resets (and then give 
warnings/errors, since there is just no way to do anything sensible with
them), since ignoring them will seemingly cause up to 2**32 counts suddenly.

It is also doing something in kernelspace which can be done in userspace,
which is normally considered a big nono.

Proposal: have a snapshot command, that remembers the current value of a
counter. Then have two interfaces: one that shows the continuous counter
and one that shows the subtraction of the current value from the snapshot.
The first can be used by used by serious applications (don't have to
add code to give warnings about dataloss), and the second can be used by 
users who want to watch the counters a bit to get a feel for what a rule
is doing.

I really think cisco got this right: from the commandline interface
you can reset counters, and watch them, the SNMP counters however just
keep going and going and going independently from this.

(I think this snapshotting belongs in the apps reading the counters
really, but if you really insist on a kernel based reset, this might be
reasonable).
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Ton Hospel]

In article [EMAIL PROTECTED],
Alan Cox [EMAIL PROTECTED] writes:
  No he isnt confused, you are trying to dictate policy.
 
 What then *is* the policy?
 
 The policy is not to have policy. It works as well in kernel design as politics.
 
 Alan
 
Since my job is in fact mainly this kind of apps, i really feel strongly
about this. 

Resettable counters are evil.

Having resettable counters may not sound like it, but it is in fact policy.
It forces all apps to add code to detect these resets (and then give 
warnings/errors, since there is just no way to do anything sensible with
them), since ignoring them will seemingly cause up to 2**32 counts suddenly.

It is also doing something in kernelspace which can be done in userspace,
which is normally considered a big nono.

Proposal: have a snapshot command, that remembers the current value of a
counter. Then have two interfaces: one that shows the continuous counter
and one that shows the subtraction of the current value from the snapshot.
The first can be used by used by serious applications (don't have to
add code to give warnings about dataloss), and the second can be used by 
users who want to watch the counters a bit to get a feel for what a rule
is doing.

I really think cisco got this right: from the commandline interface
you can reset counters, and watch them, the SNMP counters however just
keep going and going and going independently from this.

(I think this snapshotting belongs in the apps reading the counters
really, but if you really insist on a kernel based reset, this might be
reasonable).
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Jonathan Lundell]

At 10:16 AM +0200 2001-04-18, Kenneth Johansson wrote:
>Alan Cox wrote:
>
>> > > Fix your userspace applications to behave correctly.  If _you_
>> > > require your userspace applications to not clear counters, then fix
>> > > the application.
>> >
>> > You are confused.  What would you say if a close() by another,
>>
>> No he isnt confused, you are trying to dictate policy.
>
>Well it's not actually possible to do a fix in userspace for a odometer type of 
>counter that can be reset. I don't know what you mean about policy but this reset 
>"feature" is a shure way to get bad values. I have not seen one good reason to have a 
>reset other than it easier to read and that is something that can be fixed in 
>userspace.

Moreover, in a non-resettable-counter environment, any client of the counter can 
trivially achieve the effect of a reset by (locally) storing a snapshot and doing a 
subtract. Conversely, if the counter is truly reset, information is lost permanently.
-- 
/Jonathan Lundell.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: IP Acounting Idea for 2.5

[Michael Clark]

> I repeat myself, fighting is apparently so pleasant that
> you are stuck on
> fighting over dead-end technology:
>
>   I seriously suggest that for the primary (subject given) topic
>   you are SERIOUSLY OFF TARGET.  Look around, counting hits on
>   some fw rules is waste of time!  (And mightly inaccurate!)

I agree. We could all stop re-inventing the wheel and use a
RFC2724/RFC2722/RFC2720 compliant traffic meter such as NeTraMet -
which has already solved most of the mentioned problems - has a
flexible rule language for matching flows and managing counters -
support for multiple protocols; not just IP - a distributed
architecture - SNMP accessable meter and remote Manager and Controller
(NeMaC) which can concurrently read from multiiple meters (including
NetFlow meters).

>   You absolutely don't want to do any sort of counting
> aggeration policy
>   control within kernel ( = FW rules ).   You want to
> collect accounting
>   per flow, and send those data records to offline analysis.

Yes, the IP accounting effort could do well by creating a fastpath for
feeding packet headers (can't say I know how optimal libpcap is
currently on Linux) to a userspace meter (like NeTraMet) letting it
deal with all of the policy.

I remember the DOS version of NeTraMet performed much better than the
Linux version (some years ago) due to custom ethernet drivers (for
some cards) that only generate interupts when a ring buffer is full of
packet headers - maybe the same sort of infrastructure (some of the
Linux GigE drivers also avoid the interupt per packet performance hit)
could be added to Linux and integrated with libpcap and leave the rest
up to a userspace meter application.

I'm sure Neville (traffic metering god - Hi Neville) would be pleased
to have optimized support for NeTraMet in the Linux kernel.

>   No more fighting of when to clear counters, and when not.
>
>   Having used (with own custom analyzers) cisco netflow, I can say
>   that any sort of "count hits on access-list elements" things are
>   from stone-age:

NetFlow really sucks alot doesn't - I remeber having bad aliasing
problems (trying to generate 5min averages) due to its minumum flow
export interval of 1 minute. Is this still the case?

Michael Clark.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Kenneth Johansson]

Alan Cox wrote:

> > > Fix your userspace applications to behave correctly.  If _you_
> > > require your userspace applications to not clear counters, then fix
> > > the application.
> >
> > You are confused.  What would you say if a close() by another,
>
> No he isnt confused, you are trying to dictate policy.

Well it's not actually possible to do a fix in userspace for a odometer type of 
counter that can be reset. I don't know what you mean about policy but this reset 
"feature" is a shure way to get bad values. I have not seen one good reason to have a 
reset other than it easier to read and that is something that can be fixed in 
userspace.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Kenneth Johansson]

Alan Cox wrote:

   Fix your userspace applications to behave correctly.  If _you_
   require your userspace applications to not clear counters, then fix
   the application.
 
  You are confused.  What would you say if a close() by another,

 No he isnt confused, you are trying to dictate policy.

Well it's not actually possible to do a fix in userspace for a odometer type of 
counter that can be reset. I don't know what you mean about policy but this reset 
"feature" is a shure way to get bad values. I have not seen one good reason to have a 
reset other than it easier to read and that is something that can be fixed in 
userspace.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: IP Acounting Idea for 2.5

[Michael Clark]

 I repeat myself, fighting is apparently so pleasant that
 you are stuck on
 fighting over dead-end technology:

   I seriously suggest that for the primary (subject given) topic
   you are SERIOUSLY OFF TARGET.  Look around, counting hits on
   some fw rules is waste of time!  (And mightly inaccurate!)

I agree. We could all stop re-inventing the wheel and use a
RFC2724/RFC2722/RFC2720 compliant traffic meter such as NeTraMet -
which has already solved most of the mentioned problems - has a
flexible rule language for matching flows and managing counters -
support for multiple protocols; not just IP - a distributed
architecture - SNMP accessable meter and remote Manager and Controller
(NeMaC) which can concurrently read from multiiple meters (including
NetFlow meters).

   You absolutely don't want to do any sort of counting
 aggeration policy
   control within kernel ( = FW rules ).   You want to
 collect accounting
   per flow, and send those data records to offline analysis.

Yes, the IP accounting effort could do well by creating a fastpath for
feeding packet headers (can't say I know how optimal libpcap is
currently on Linux) to a userspace meter (like NeTraMet) letting it
deal with all of the policy.

I remember the DOS version of NeTraMet performed much better than the
Linux version (some years ago) due to custom ethernet drivers (for
some cards) that only generate interupts when a ring buffer is full of
packet headers - maybe the same sort of infrastructure (some of the
Linux GigE drivers also avoid the interupt per packet performance hit)
could be added to Linux and integrated with libpcap and leave the rest
up to a userspace meter application.

I'm sure Neville (traffic metering god - Hi Neville) would be pleased
to have optimized support for NeTraMet in the Linux kernel.

   No more fighting of when to clear counters, and when not.

   Having used (with own custom analyzers) cisco netflow, I can say
   that any sort of "count hits on access-list elements" things are
   from stone-age:

NetFlow really sucks alot doesn't - I remeber having bad aliasing
problems (trying to generate 5min averages) due to its minumum flow
export interval of 1 minute. Is this still the case?

Michael Clark.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Jonathan Lundell]

At 10:16 AM +0200 2001-04-18, Kenneth Johansson wrote:
Alan Cox wrote:

   Fix your userspace applications to behave correctly.  If _you_
   require your userspace applications to not clear counters, then fix
   the application.
 
  You are confused.  What would you say if a close() by another,

 No he isnt confused, you are trying to dictate policy.

Well it's not actually possible to do a fix in userspace for a odometer type of 
counter that can be reset. I don't know what you mean about policy but this reset 
"feature" is a shure way to get bad values. I have not seen one good reason to have a 
reset other than it easier to read and that is something that can be fixed in 
userspace.

Moreover, in a non-resettable-counter environment, any client of the counter can 
trivially achieve the effect of a reset by (locally) storing a snapshot and doing a 
subtract. Conversely, if the counter is truly reset, information is lost permanently.
-- 
/Jonathan Lundell.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Alan Cox]

> > No he isnt confused, you are trying to dictate policy.
> 
> What then *is* the policy?

The policy is not to have policy. It works as well in kernel design as politics.

Alan

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Manfred Bartz]

Alan Cox <[EMAIL PROTECTED]> writes:

> > > Fix your userspace applications to behave correctly.  If _you_
> > > require your userspace applications to not clear counters, then fix
> > > the application.
> > 
> > You are confused.  What would you say if a close() by another,
> 
> No he isnt confused, you are trying to dictate policy.

What then *is* the policy?

AFAIK, no counters in the Linux kernel have resets, except in netfilter.

-- 
Manfred Bartz
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Alan Cox]

> > Fix your userspace applications to behave correctly.  If _you_
> > require your userspace applications to not clear counters, then fix
> > the application.
> 
> You are confused.  What would you say if a close() by another,

No he isnt confused, you are trying to dictate policy.

> unrelated application closed all open descriptors for that file,
> including the one you just opened?  Just fix your applications?

That would be a bug. The standards and common sense say so, but your argument
is more akin to 'if I delete a file in one app it hurts when I try and use
it in another'.


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Manfred Bartz]

Leif Sawyer <[EMAIL PROTECTED]> writes:

> > > Jesse Pollard replies:

> > Removing/no-oping the reset code would make the module
> > SMALLER, and simpler.

> NO.  Don't remove the functionality that is required.  

Please explain where counter reset capability provides any 
functionality that is not already available without it.

You might want to read RFC2724 ``Traffic Flow Measurement''.
Search for ``reset''.

Counter resets have always caused problems, that's why mission 
critical counters never have a reset.  Have you ever seen an
electricity, gas or water meter with a reset?  The same reasoning
applies when you have accounting rules that are used to charge
customers for traffic volume.

> Fix your userspace applications to behave correctly.  If _you_
> require your userspace applications to not clear counters, then fix
> the application.

You are confused.  What would you say if a close() by another,
unrelated application closed all open descriptors for that file,
including the one you just opened?  Just fix your applications?

-- 
Manfred Bartz
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Manfred Bartz]

Leif Sawyer <[EMAIL PROTECTED]> writes:

> Jesse Pollard replies:
> to Leif Sawyer who wrote: 
> > > Besides, what would be gained in making the counters RO, if 
> > > they were cleared every time the module was loaded/unloaded?
> > 
> > 1. Knowlege that the module was reloaded.
> > 2. Knowlege that the data being measured is correct
> > 3. Having reliable measures
> > 4. being able to derive valid statistics

> Good.  Now that we have valid objectives to reach, which of these
> are NOT met by making the fixes entirely in userspace, ...

By this logic you could ask why we need separate user spaces,
just fix all applications so they cooperate nicely.

> They're still all met, right?

No.  Because they require well behaved, cooperative user programs.

> And we haven't had to fill the kernel with more cruft.

Nonsense.  Removing counter reset capabilities reduces kernel cruft.

-- 
Manfred Bartz
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: IP Acounting Idea for 2.5

[Leif Sawyer]

> > Jesse Pollard replies:
> > to Leif Sawyer who wrote: 
> > >> Besides, what would be gained in making the counters RO, if 
> > >> they were cleared every time the module was loaded/unloaded?
> > > 
> > > 1. Knowlege that the module was reloaded.
> > > 2. Knowlege that the data being measured is correct
> > > 3. Having reliable measures
> > > 4. being able to derive valid statistics
> > > 
> > 
> > Good.  Now that we have valid objectives to reach, which of these
> > are NOT met by making the fixes entirely in userspace, say by
> > incorporating a wrapper script to ensure that no external 
> applications
> > can flush the table counters?
> > 
> > They're still all met, right?
> 
> Nope - some of the applications that may be purchased do not have
> to go through the scripts to reset the counters. They just need access
> to the counters, and reset is built into the applications. 
> This violates
> all 4 objectives.
> 
> > And we haven't had to fill the kernel with more cruft.
> 
> Removing/no-oping the reset code would make the module
> SMALLER, and simpler.
> 

NO.  Don't remove the functionality that is required.  Fix your
userspace applications to behave correctly.  If _you_ require your
userspace applications to not clear counters, then fix the application.

The counters are not gospel.  The modules are not omnipresent.

Time and again, people beg and plead the kernel developers to change
the kernel to suit their personal biases and applications.  Historically
they have been vehemently against it.  Of what purpose does breaking EVERY
application out there by performing this kernel mod have?

As Matti Aarnio points out, this is nearly beaten to death.  Since none
of the 'Big Boys' have commented, my guess is that this is a moot argument,
and no changes to the kernel will take place.

/me steps off of soapbox
/me retires this thread

 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: IP Acounting Idea for 2.5

[Jesse Pollard]

-  Received message begins Here  -

> 
> Jesse Pollard replies:
> to Leif Sawyer who wrote: 
> >> Besides, what would be gained in making the counters RO, if 
> >> they were cleared every time the module was loaded/unloaded?
> > 
> > 1. Knowlege that the module was reloaded.
> > 2. Knowlege that the data being measured is correct
> > 3. Having reliable measures
> > 4. being able to derive valid statistics
> > 
> 
> Good.  Now that we have valid objectives to reach, which of these
> are NOT met by making the fixes entirely in userspace, say by
> incorporating a wrapper script to ensure that no external applications
> can flush the table counters?
> 
> They're still all met, right?

Nope - some of the applications that may be purchased do not have
to go through the scripts to reset the counters. They just need access
to the counters, and reset is built into the applications. This violates
all 4 objectives.

> And we haven't had to fill the kernel with more cruft.

Removing/no-oping the reset code would make the module
SMALLER, and simpler.

-
Jesse I Pollard, II
Email: [EMAIL PROTECTED]

Any opinions expressed are solely my own.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Matti Aarnio]

I repeat myself, fighting is apparently so pleasant that you are stuck on
fighting over dead-end technology:

  I seriously suggest that for the primary (subject given) topic
  you are SERIOUSLY OFF TARGET.  Look around, counting hits on
  some fw rules is waste of time!  (And mightly inaccurate!)

  You absolutely don't want to do any sort of counting aggeration policy
  control within kernel ( = FW rules ).   You want to collect accounting
  per flow, and send those data records to offline analysis.

  No more fighting of when to clear counters, and when not.

  Having used (with own custom analyzers) cisco netflow, I can say
  that any sort of "count hits on access-list elements" things are
  from stone-age:

http://www.cisco.com/warp/public/cc/pd/iosw/ioft/neflct/tech/napps_wp.htm


  Yet another nice thing to cook up -- if I had time ...

/Matti Aarnio
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: IP Acounting Idea for 2.5

[Leif Sawyer]

Jesse Pollard replies:
to Leif Sawyer who wrote: 
>> Besides, what would be gained in making the counters RO, if 
>> they were cleared every time the module was loaded/unloaded?
> 
> 1. Knowlege that the module was reloaded.
> 2. Knowlege that the data being measured is correct
> 3. Having reliable measures
> 4. being able to derive valid statistics
> 

Good.  Now that we have valid objectives to reach, which of these
are NOT met by making the fixes entirely in userspace, say by
incorporating a wrapper script to ensure that no external applications
can flush the table counters?

They're still all met, right?

And we haven't had to fill the kernel with more cruft.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: IP Acounting Idea for 2.5

[Jesse Pollard]

Leif Sawyer <[EMAIL PROTECTED]>:
> > And that introduces errors in measurement. It also depends on 
> > how frequently an uncontroled process is clearing the counters.
> > You may never be able to get a valid measurement.
> 
> This is true.  Which is why application programmers need to write
> code as if they are not the only [ab]users of data.
> 
> Which brings me back to my point.
> 
> Don't force the kernel to uphold your local application requirements
> of stable counters.
> 
> Enforce it in the userspace portion of the code.
> 
> 
> Yes, you could extend the proc filesystem (ugh) with a flag that could
> be read by the ip[chains|tables] user app to determine if clearing flags
> were allowed.  Then a simple
> 
> echo 1 > /proc/sys/net/ipv4/counters_locked
> 
> or some such cruft.  But I don't see this extension making into the
> standard kernel at this time.  It just seems to be wasteful.
> 
> 
> If you (at your site) really need this type of functionality, it's
> pretty darn simple to write a wrapper to ip[tables|chains] which
> silently (or not so) drops the option to clear the counters before
> calling the real version.
> 
> Besides, what would be gained in making the counters RO, if they were
> cleared every time the module was loaded/unloaded?

1. Knowlege that the module was reloaded.
2. Knowlege that the data being measured is correct
3. Having reliable measures
4. being able to derive valid statistics


-
Jesse I Pollard, II
Email: [EMAIL PROTECTED]

Any opinions expressed are solely my own.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: IP Acounting Idea for 2.5

[Leif Sawyer]

Jesse Pollard continues with:
> Leif Sawyer <[EMAIL PROTECTED]>:
>>> Ian Stirling [mailto:[EMAIL PROTECTED]]
 Manfred Bartz responded to
> Russell King <[EMAIL PROTECTED]> who writes:
>
> You just illustrated my point.  While there is a 
> reset capability people will use it and accounting/
> logging programs will get wrong data.  Resetable
> counters might be a minor convenience when debugging
> but the price is unreliable programs and the loss of the 
> ability of several programs to use the same counters.
 
 You of course, are commenting from the fact that your 
 applications are stupid, written poorly, and cannot handle
 'wrapped' data.  Take MRTG
 
 Similarly, if my InPackets are at 102345 at one read, and 
 2345 the next read, and I know that my counter is 32 bits,
 then I know i've wrapped and can do
>>> 
>>> I think the point being made is that if InPackets are at 
>>> 102345 at one read, and 2345 the next, and you know it's
>>> a 32 bit counter, it's completely unreliable to assume that
>>> you have in fact recieved 4294867295 packets, if the counter
>>> can be zeroed. You can say nothing other than at least 2345
>>> packets, at most 2345+n*2^32 have been got since you last
>>> checked.
>> 
>> Ah, yes.. I seem to have misplaced a bit of text in my reply.
>> 
>> The continuation of thought:
>> 
>> How the application derives the status of a wrapped counter or
>> a zero'ed counter is dependant on the device being monitored.
>> 
>> Yes, you have to know what your interface is capable of 
>> (maxbytes/sec) so that you can do a simple calculation where:
>> 
>> maximum_throughput = maxbytes_sec * (time_now - time_last_read)
>> 
>> and if your previous good counter + the maximum throughput wraps
>> the counter, you have a good chance that you've simply wrapped.
>> 
>> If not, then you can assume that your counters were cleared 
>> at some point, log the data you've got, and keep moving forward.
> 
> And that introduces errors in measurement. It also depends on 
> how frequently an uncontroled process is clearing the counters.
> You may never be able to get a valid measurement.

This is true.  Which is why application programmers need to write
code as if they are not the only [ab]users of data.

Which brings me back to my point.

Don't force the kernel to uphold your local application requirements
of stable counters.

Enforce it in the userspace portion of the code.


Yes, you could extend the proc filesystem (ugh) with a flag that could
be read by the ip[chains|tables] user app to determine if clearing flags
were allowed.  Then a simple

echo 1 > /proc/sys/net/ipv4/counters_locked

or some such cruft.  But I don't see this extension making into the
standard kernel at this time.  It just seems to be wasteful.


If you (at your site) really need this type of functionality, it's
pretty darn simple to write a wrapper to ip[tables|chains] which
silently (or not so) drops the option to clear the counters before
calling the real version.

Besides, what would be gained in making the counters RO, if they were
cleared every time the module was loaded/unloaded?


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: IP Acounting Idea for 2.5

[Jesse Pollard]

Leif Sawyer <[EMAIL PROTECTED]>:
> > From: Ian Stirling [mailto:[EMAIL PROTECTED]]
> > > Manfred Bartz responded to
> > > > Russell King <[EMAIL PROTECTED]> who writes:
> > 
> > > > You just illustrated my point.  While there is a reset capability
> > > > people will use it and accounting/logging programs will get wrong
> > > > data.  Resetable counters might be a minor convenience 
> > when debugging
> > > > but the price is unreliable programs and the loss of the 
> > ability of
> > > > several programs to use the same counters.
> > > 
> > > You of course, are commenting from the fact that your 
> > applications are
> > > stupid, written poorly, and cannot handle 'wrapped' data.  Take MRTG
> > 
> > > Similarly, if my InPackets are at 102345 at one read, and 
> > 2345 the next
> > > read,
> > > and I know that my counter is 32 bits, then I know i've 
> > wrapped and can do
> > 
> > I think the point being made is that if InPackets are at 
> > 102345 at one read,
> > and 2345 the next, and you know it's a 32 bit counter, it's completely
> > unreliable to assume that you have in fact recieved 4294867295
> > packets, if the counter can be zeroed.
> > You can say nothing other than at least 2345 packets, at most 
> > 2345+n*2^32 have been got since you last checked.
> 
> Ah, yes.. I seem to have misplaced a bit of text in my reply.
> 
> The continuation of thought:
> 
> How the application derives the status of a wrapped counter or
> a zero'ed counter is dependant on the device being monitored.
> 
> Yes, you have to know what your interface is capable of (maxbytes/sec)
> so that you can do a simple calculation where:
> 
> maximum_throughput = maxbytes_sec * (time_now - time_last_read)
> 
> and if your previous good counter + the maximum throughput wraps the
> counter, you have a good chance that you've simply wrapped.
> 
> If not, then you can assume that your counters were cleared at some point,
> log the data you've got, and keep moving forward.

And that introduces errors in measurement. It also depends on how frequently
an uncontroled process is clearing the counters. You may never be able to
get a valid measurement.

-
Jesse I Pollard, II
Email: [EMAIL PROTECTED]

Any opinions expressed are solely my own.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Olaf Titz]

> Umm, no.  Counters can be resetable - you just specify that accounting
> programs should not reset them, ever.
>
> The ability to reset counters is extremely useful if you're a human
> looking at the output of iptables -L -v.  (I thus far know of no one
> who can memorise the counter values for around 40 rules).

You'll get bogus accounting results unless you stop/restart the
accounting programs every time you manually deal with the counters.
This sounds dangerously easy to make mistakes to me.

Olaf

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Olaf Titz]

> Similarly, if my InPackets are at 102345 at one read, and 2345 the
> next read, and I know that my counter is 32 bits, then I know i've
> wrapped and can do my own math.

No. When you have resettable counters, you don't know if the counter
has wrapped or been reset. Either you have received 2345 packets in
between, or 2^32-102345. The difference is not negligible. ;-)

Unless you have a second counter which is incremented with every wrap
(_or_, perhaps better, with every reset) and can _not_ be reset
manually. This would be the date in your date/time example.

Olaf

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Olaf Titz]

 Similarly, if my InPackets are at 102345 at one read, and 2345 the
 next read, and I know that my counter is 32 bits, then I know i've
 wrapped and can do my own math.

No. When you have resettable counters, you don't know if the counter
has wrapped or been reset. Either you have received 2345 packets in
between, or 2^32-102345. The difference is not negligible. ;-)

Unless you have a second counter which is incremented with every wrap
(_or_, perhaps better, with every reset) and can _not_ be reset
manually. This would be the date in your date/time example.

Olaf

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Olaf Titz]

 Umm, no.  Counters can be resetable - you just specify that accounting
 programs should not reset them, ever.

 The ability to reset counters is extremely useful if you're a human
 looking at the output of iptables -L -v.  (I thus far know of no one
 who can memorise the counter values for around 40 rules).

You'll get bogus accounting results unless you stop/restart the
accounting programs every time you manually deal with the counters.
This sounds dangerously easy to make mistakes to me.

Olaf

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: IP Acounting Idea for 2.5

[Jesse Pollard]

Leif Sawyer [EMAIL PROTECTED]:
  From: Ian Stirling [mailto:[EMAIL PROTECTED]]
   Manfred Bartz responded to
Russell King [EMAIL PROTECTED] who writes:
  snip
You just illustrated my point.  While there is a reset capability
people will use it and accounting/logging programs will get wrong
data.  Resetable counters might be a minor convenience 
  when debugging
but the price is unreliable programs and the loss of the 
  ability of
several programs to use the same counters.
   
   You of course, are commenting from the fact that your 
  applications are
   stupid, written poorly, and cannot handle 'wrapped' data.  Take MRTG
  snip
   Similarly, if my InPackets are at 102345 at one read, and 
  2345 the next
   read,
   and I know that my counter is 32 bits, then I know i've 
  wrapped and can do
  
  I think the point being made is that if InPackets are at 
  102345 at one read,
  and 2345 the next, and you know it's a 32 bit counter, it's completely
  unreliable to assume that you have in fact recieved 4294867295
  packets, if the counter can be zeroed.
  You can say nothing other than at least 2345 packets, at most 
  2345+n*2^32 have been got since you last checked.
 
 Ah, yes.. I seem to have misplaced a bit of text in my reply.
 
 The continuation of thought:
 
 How the application derives the status of a wrapped counter or
 a zero'ed counter is dependant on the device being monitored.
 
 Yes, you have to know what your interface is capable of (maxbytes/sec)
 so that you can do a simple calculation where:
 
 maximum_throughput = maxbytes_sec * (time_now - time_last_read)
 
 and if your previous good counter + the maximum throughput wraps the
 counter, you have a good chance that you've simply wrapped.
 
 If not, then you can assume that your counters were cleared at some point,
 log the data you've got, and keep moving forward.

And that introduces errors in measurement. It also depends on how frequently
an uncontroled process is clearing the counters. You may never be able to
get a valid measurement.

-
Jesse I Pollard, II
Email: [EMAIL PROTECTED]

Any opinions expressed are solely my own.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: IP Acounting Idea for 2.5

[Leif Sawyer]

Jesse Pollard continues with:
 Leif Sawyer [EMAIL PROTECTED]:
 Ian Stirling [mailto:[EMAIL PROTECTED]]
 Manfred Bartz responded to
 Russell King [EMAIL PROTECTED] who writes:

 You just illustrated my point.  While there is a 
 reset capability people will use it and accounting/
 logging programs will get wrong data.  Resetable
 counters might be a minor convenience when debugging
 but the price is unreliable programs and the loss of the 
 ability of several programs to use the same counters.
 
 You of course, are commenting from the fact that your 
 applications are stupid, written poorly, and cannot handle
 'wrapped' data.  Take MRTG
 snip
 Similarly, if my InPackets are at 102345 at one read, and 
 2345 the next read, and I know that my counter is 32 bits,
 then I know i've wrapped and can do
 
 I think the point being made is that if InPackets are at 
 102345 at one read, and 2345 the next, and you know it's
 a 32 bit counter, it's completely unreliable to assume that
 you have in fact recieved 4294867295 packets, if the counter
 can be zeroed. You can say nothing other than at least 2345
 packets, at most 2345+n*2^32 have been got since you last
 checked.
 
 Ah, yes.. I seem to have misplaced a bit of text in my reply.
 
 The continuation of thought:
 
 How the application derives the status of a wrapped counter or
 a zero'ed counter is dependant on the device being monitored.
 
 Yes, you have to know what your interface is capable of 
 (maxbytes/sec) so that you can do a simple calculation where:
 
 maximum_throughput = maxbytes_sec * (time_now - time_last_read)
 
 and if your previous good counter + the maximum throughput wraps
 the counter, you have a good chance that you've simply wrapped.
 
 If not, then you can assume that your counters were cleared 
 at some point, log the data you've got, and keep moving forward.
 
 And that introduces errors in measurement. It also depends on 
 how frequently an uncontroled process is clearing the counters.
 You may never be able to get a valid measurement.

This is true.  Which is why application programmers need to write
code as if they are not the only [ab]users of data.

Which brings me back to my point.

Don't force the kernel to uphold your local application requirements
of stable counters.

Enforce it in the userspace portion of the code.

subtopic
Yes, you could extend the proc filesystem (ugh) with a flag that could
be read by the ip[chains|tables] user app to determine if clearing flags
were allowed.  Then a simple

echo 1  /proc/sys/net/ipv4/counters_locked

or some such cruft.  But I don't see this extension making into the
standard kernel at this time.  It just seems to be wasteful.
/subtopic

If you (at your site) really need this type of functionality, it's
pretty darn simple to write a wrapper to ip[tables|chains] which
silently (or not so) drops the option to clear the counters before
calling the real version.

Besides, what would be gained in making the counters RO, if they were
cleared every time the module was loaded/unloaded?


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: IP Acounting Idea for 2.5

[Jesse Pollard]

Leif Sawyer [EMAIL PROTECTED]:
  And that introduces errors in measurement. It also depends on 
  how frequently an uncontroled process is clearing the counters.
  You may never be able to get a valid measurement.
 
 This is true.  Which is why application programmers need to write
 code as if they are not the only [ab]users of data.
 
 Which brings me back to my point.
 
 Don't force the kernel to uphold your local application requirements
 of stable counters.
 
 Enforce it in the userspace portion of the code.
 
 subtopic
 Yes, you could extend the proc filesystem (ugh) with a flag that could
 be read by the ip[chains|tables] user app to determine if clearing flags
 were allowed.  Then a simple
 
 echo 1  /proc/sys/net/ipv4/counters_locked
 
 or some such cruft.  But I don't see this extension making into the
 standard kernel at this time.  It just seems to be wasteful.
 /subtopic
 
 If you (at your site) really need this type of functionality, it's
 pretty darn simple to write a wrapper to ip[tables|chains] which
 silently (or not so) drops the option to clear the counters before
 calling the real version.
 
 Besides, what would be gained in making the counters RO, if they were
 cleared every time the module was loaded/unloaded?

1. Knowlege that the module was reloaded.
2. Knowlege that the data being measured is correct
3. Having reliable measures
4. being able to derive valid statistics


-
Jesse I Pollard, II
Email: [EMAIL PROTECTED]

Any opinions expressed are solely my own.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: IP Acounting Idea for 2.5

[Leif Sawyer]

Jesse Pollard replies:
to Leif Sawyer who wrote: 
 Besides, what would be gained in making the counters RO, if 
 they were cleared every time the module was loaded/unloaded?
 
 1. Knowlege that the module was reloaded.
 2. Knowlege that the data being measured is correct
 3. Having reliable measures
 4. being able to derive valid statistics
 

Good.  Now that we have valid objectives to reach, which of these
are NOT met by making the fixes entirely in userspace, say by
incorporating a wrapper script to ensure that no external applications
can flush the table counters?

They're still all met, right?

And we haven't had to fill the kernel with more cruft.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Matti Aarnio]

I repeat myself, fighting is apparently so pleasant that you are stuck on
fighting over dead-end technology:

  I seriously suggest that for the primary (subject given) topic
  you are SERIOUSLY OFF TARGET.  Look around, counting hits on
  some fw rules is waste of time!  (And mightly inaccurate!)

  You absolutely don't want to do any sort of counting aggeration policy
  control within kernel ( = FW rules ).   You want to collect accounting
  per flow, and send those data records to offline analysis.

  No more fighting of when to clear counters, and when not.

  Having used (with own custom analyzers) cisco netflow, I can say
  that any sort of "count hits on access-list elements" things are
  from stone-age:

http://www.cisco.com/warp/public/cc/pd/iosw/ioft/neflct/tech/napps_wp.htm


  Yet another nice thing to cook up -- if I had time ...

/Matti Aarnio
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: IP Acounting Idea for 2.5

[Jesse Pollard]

-  Received message begins Here  -

 
 Jesse Pollard replies:
 to Leif Sawyer who wrote: 
  Besides, what would be gained in making the counters RO, if 
  they were cleared every time the module was loaded/unloaded?
  
  1. Knowlege that the module was reloaded.
  2. Knowlege that the data being measured is correct
  3. Having reliable measures
  4. being able to derive valid statistics
  
 
 Good.  Now that we have valid objectives to reach, which of these
 are NOT met by making the fixes entirely in userspace, say by
 incorporating a wrapper script to ensure that no external applications
 can flush the table counters?
 
 They're still all met, right?

Nope - some of the applications that may be purchased do not have
to go through the scripts to reset the counters. They just need access
to the counters, and reset is built into the applications. This violates
all 4 objectives.

 And we haven't had to fill the kernel with more cruft.

Removing/no-oping the reset code would make the module
SMALLER, and simpler.

-
Jesse I Pollard, II
Email: [EMAIL PROTECTED]

Any opinions expressed are solely my own.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: IP Acounting Idea for 2.5

[Leif Sawyer]

  Jesse Pollard replies:
  to Leif Sawyer who wrote: 
   Besides, what would be gained in making the counters RO, if 
   they were cleared every time the module was loaded/unloaded?
   
   1. Knowlege that the module was reloaded.
   2. Knowlege that the data being measured is correct
   3. Having reliable measures
   4. being able to derive valid statistics
   
  
  Good.  Now that we have valid objectives to reach, which of these
  are NOT met by making the fixes entirely in userspace, say by
  incorporating a wrapper script to ensure that no external 
 applications
  can flush the table counters?
  
  They're still all met, right?
 
 Nope - some of the applications that may be purchased do not have
 to go through the scripts to reset the counters. They just need access
 to the counters, and reset is built into the applications. 
 This violates
 all 4 objectives.
 
  And we haven't had to fill the kernel with more cruft.
 
 Removing/no-oping the reset code would make the module
 SMALLER, and simpler.
 

NO.  Don't remove the functionality that is required.  Fix your
userspace applications to behave correctly.  If _you_ require your
userspace applications to not clear counters, then fix the application.

The counters are not gospel.  The modules are not omnipresent.

Time and again, people beg and plead the kernel developers to change
the kernel to suit their personal biases and applications.  Historically
they have been vehemently against it.  Of what purpose does breaking EVERY
application out there by performing this kernel mod have?

As Matti Aarnio points out, this is nearly beaten to death.  Since none
of the 'Big Boys' have commented, my guess is that this is a moot argument,
and no changes to the kernel will take place.

/me steps off of soapbox
/me retires this thread

 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Manfred Bartz]

Leif Sawyer [EMAIL PROTECTED] writes:

 Jesse Pollard replies:
 to Leif Sawyer who wrote: 
   Besides, what would be gained in making the counters RO, if 
   they were cleared every time the module was loaded/unloaded?
  
  1. Knowlege that the module was reloaded.
  2. Knowlege that the data being measured is correct
  3. Having reliable measures
  4. being able to derive valid statistics

 Good.  Now that we have valid objectives to reach, which of these
 are NOT met by making the fixes entirely in userspace, ...

By this logic you could ask why we need separate user spaces,
just fix all applications so they cooperate nicely.

 They're still all met, right?

No.  Because they require well behaved, cooperative user programs.

 And we haven't had to fill the kernel with more cruft.

Nonsense.  Removing counter reset capabilities reduces kernel cruft.

-- 
Manfred Bartz
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Manfred Bartz]

Leif Sawyer [EMAIL PROTECTED] writes:

   Jesse Pollard replies:

  Removing/no-oping the reset code would make the module
  SMALLER, and simpler.

 NO.  Don't remove the functionality that is required.  

Please explain where counter reset capability provides any 
functionality that is not already available without it.

You might want to read RFC2724 ``Traffic Flow Measurement''.
Search for ``reset''.

Counter resets have always caused problems, that's why mission 
critical counters never have a reset.  Have you ever seen an
electricity, gas or water meter with a reset?  The same reasoning
applies when you have accounting rules that are used to charge
customers for traffic volume.

 Fix your userspace applications to behave correctly.  If _you_
 require your userspace applications to not clear counters, then fix
 the application.

You are confused.  What would you say if a close() by another,
unrelated application closed all open descriptors for that file,
including the one you just opened?  Just fix your applications?

-- 
Manfred Bartz
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Alan Cox]

  Fix your userspace applications to behave correctly.  If _you_
  require your userspace applications to not clear counters, then fix
  the application.
 
 You are confused.  What would you say if a close() by another,

No he isnt confused, you are trying to dictate policy.

 unrelated application closed all open descriptors for that file,
 including the one you just opened?  Just fix your applications?

That would be a bug. The standards and common sense say so, but your argument
is more akin to 'if I delete a file in one app it hurts when I try and use
it in another'.


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Manfred Bartz]

Alan Cox [EMAIL PROTECTED] writes:

   Fix your userspace applications to behave correctly.  If _you_
   require your userspace applications to not clear counters, then fix
   the application.
  
  You are confused.  What would you say if a close() by another,
 
 No he isnt confused, you are trying to dictate policy.

What then *is* the policy?

AFAIK, no counters in the Linux kernel have resets, except in netfilter.

-- 
Manfred Bartz
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Alan Cox]

  No he isnt confused, you are trying to dictate policy.
 
 What then *is* the policy?

The policy is not to have policy. It works as well in kernel design as politics.

Alan

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Manfred Bartz]

Harald Welte <[EMAIL PROTECTED]> writes:

> On Mon, Apr 16, 2001 at 12:07:31PM +1000, Manfred Bartz wrote:
> > Resetable counters guarantee that no two programs can co-exists if
> > they happen to reset the same counters.
> 
> That sounds like crap (sorry). 

Care to explain how two independent apps can co-exists if they happen
to reset the same counters?

> so why are your applications resetting counters then? 

Ask the authors, not me.  I wiped those apps from my system.
Its just a pity about the time wasted.

> Nobody forces you to reset them right now.

Why do authors then write apps (f.e. ipac) which reset counters?  
Perhaps only because they can, not because it makes sense.

-- 
Manfred Bartz
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Harald Welte]

On Mon, Apr 16, 2001 at 12:07:31PM +1000, Manfred Bartz wrote:
> 
> If there really is a performance issue with a few hundred rules, then
> it can be overcome by grouping rules in separate custom chains.  F.e.
> if you have 1024 rules create 32 custom chains with 32 rules each.
> Then have 32 rules in your main table which jump to the appropriate
> custom chain --> maximum rules traversed by each packet = 64.

still.. if you want to log multiple class B's it's a PITA, especially
if your ruleset changes and you have to atomically replace all chains
in one table to update the ruleset from userspace->kernel.

> There is another issue with logging in general:
> 
> *COUNTERS MUST NOT BE RESETABLE!!!*
> 
> Resetable counters guarantee that no two programs can co-exists if
> they happen to reset the same counters.

That sounds like crap (sorry). Counters are resettable, and will be.
If you run two applications resetting counters individually, you have
a problem with your applications.

> All logging counters should be implemented with 32bit or 64bit
> unsigned integers.  Any software using correct unsigned integer

they are uint32 right now.

> arithmetic can then simply subtract a previous value from the current
> value to get the difference.  This works reliably across counter
> wrap-arounds.  There is absolutely *no need for reset* !

so why are your applications resetting counters then? Nobody forces you
to reset them right now.

> Manfred Bartz
> ---

-- 
Live long and prosper
- Harald Welte / [EMAIL PROTECTED]http://www.gnumonks.org

GCS/E/IT d- s-: a-- C+++ UL$ P+++ L$ E--- W- N++ o? K- w--- O- M- 
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Harald Welte]

On Tue, Apr 17, 2001 at 07:53:28AM +1000, David Findlay wrote:

> In the 2.5 series of kernels, working towards 2.6, could you please make the 
> IP Accounting so that I can set a single rule that will make it watch all IP 
> traffic going from the local network, through the masquerading service to the 
> internet, and log local IP Addresses using it? This would allow me to set 1 
> rule, but have the information I want on a per IP address system.

:) Well... there was a discussion about this on the netfilter-devel
list some weeks ago. 

This is definitely not a 2.5 / 2.6 issue, as it could be easily implemented
by using the already existing flexibility of netfilter/iptables.

It's just a matter of somebody who needs it getting around actually 
implementing it. Could be either implemented as a 'raw' netfilter 
hook-attaching module or (more convenient) as a new iptables target, 
which allocates some internal tables for storing the accounting data.

Doesn't make sense to have a discussion about it on linux-kernel, and
it certainly doesn't belong into a 2.5 featurelist.

Please refer to the netfilter-devel mailinglist (subscription information
available at http://netfilter.samba.org/)

> David

-- 
Live long and prosper
- Harald Welte / [EMAIL PROTECTED]http://www.gnumonks.org

GCS/E/IT d- s-: a-- C+++ UL$ P+++ L$ E--- W- N++ o? K- w--- O- M- 
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Manfred Bartz]

Leif Sawyer <[EMAIL PROTECTED]> writes:

> Manfred Bartz responded to
> > Russell King <[EMAIL PROTECTED]> who writes:
> > 
> > > On Mon, Apr 16, 2001 at 12:07:31PM +1000, Manfred Bartz wrote:
> > > > There is another issue with logging in general:
> > > > 
> > > > *COUNTERS MUST NOT BE RESETABLE!!!*
> > > 
> > > Umm, no.  Counters can be resetable - you just specify that 
> > > accounting programs should not reset them, ever.
> > > 
> > > The ability to reset counters is extremely useful if you're a human
> > > looking at the output of iptables -L -v.  (I thus far know of no one
> > > who can memorise the counter values for around 40 rules).
> > 
> > You just illustrated my point.  While there is a reset capability
> > people will use it and accounting/logging programs will get wrong
> > data.  Resetable counters might be a minor convenience when debugging
> > but the price is unreliable programs and the loss of the ability of
> > several programs to use the same counters.
> 
> You of course, are commenting from the fact that your applications are
> stupid, written poorly, and cannot handle 'wrapped' data.  

Well, apps written by me *can handle* wrapped data; it is actually far
easier that resetting at each read.

> Take MRTG as one example of a good application, for it can handle
> counters that have been cleared between reads.

Regardless of how well a logging application is written, there is no
way that it does not loose data when someone or something resets its
counter.  F.e. if an app reads a counter every 10 minutes and a
counter reset occurs from elsewhere, then anywhere between 0 and 10
minutes worth of data is lost.

I had a brief look at MRTG.  It seems to be a well written app and
while it can handle counter reset (with potential loss of an unknown 
amount of data), it does not actively reset counters.  It also doesn't
use iptables.

> Applications need to get smarter.  

Agreed.

> Don't place arbitrary rules where they don't belong.

Agreed too.  Counters should not arbitrarily be equipped with a reset
capability, there is hardly any benefit in that and it causes nothing
but problems.

As far as I can see, the counters in /proc/net/snmp don't have a
reset, same with /proc/net/dev and possibly other counters elsewhere.

Ideally iptables would fall in line with that.  Rules can still be
unloaded and reloaded, also causing counter reset and loss of data,
but since that is a lot more involved, application authors would have
an incentive to handle counters properly.

-- 
Manfred Bartz

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: IP Acounting Idea for 2.5

[Leif Sawyer]

> From: Ian Stirling [mailto:[EMAIL PROTECTED]]
> > Manfred Bartz responded to
> > > Russell King <[EMAIL PROTECTED]> who writes:
> 
> > > You just illustrated my point.  While there is a reset capability
> > > people will use it and accounting/logging programs will get wrong
> > > data.  Resetable counters might be a minor convenience 
> when debugging
> > > but the price is unreliable programs and the loss of the 
> ability of
> > > several programs to use the same counters.
> > 
> > You of course, are commenting from the fact that your 
> applications are
> > stupid, written poorly, and cannot handle 'wrapped' data.  Take MRTG
> 
> > Similarly, if my InPackets are at 102345 at one read, and 
> 2345 the next
> > read,
> > and I know that my counter is 32 bits, then I know i've 
> wrapped and can do
> 
> I think the point being made is that if InPackets are at 
> 102345 at one read,
> and 2345 the next, and you know it's a 32 bit counter, it's completely
> unreliable to assume that you have in fact recieved 4294867295
> packets, if the counter can be zeroed.
> You can say nothing other than at least 2345 packets, at most 
> 2345+n*2^32 have been got since you last checked.

Ah, yes.. I seem to have misplaced a bit of text in my reply.

The continuation of thought:

How the application derives the status of a wrapped counter or
a zero'ed counter is dependant on the device being monitored.

Yes, you have to know what your interface is capable of (maxbytes/sec)
so that you can do a simple calculation where:

maximum_throughput = maxbytes_sec * (time_now - time_last_read)

and if your previous good counter + the maximum throughput wraps the
counter, you have a good chance that you've simply wrapped.

If not, then you can assume that your counters were cleared at some point,
log the data you've got, and keep moving forward.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Ian Stirling]

> 
> Manfred Bartz responded to
> > Russell King <[EMAIL PROTECTED]> who writes:

> > You just illustrated my point.  While there is a reset capability
> > people will use it and accounting/logging programs will get wrong
> > data.  Resetable counters might be a minor convenience when debugging
> > but the price is unreliable programs and the loss of the ability of
> > several programs to use the same counters.
> 
> You of course, are commenting from the fact that your applications are
> stupid, written poorly, and cannot handle 'wrapped' data.  Take MRTG

> Similarly, if my InPackets are at 102345 at one read, and 2345 the next
> read,
> and I know that my counter is 32 bits, then I know i've wrapped and can do

I think the point being made is that if InPackets are at 102345 at one read,
and 2345 the next, and you know it's a 32 bit counter, it's completely
unreliable to assume that you have in fact recieved 4294867295
packets, if the counter can be zeroed.
You can say nothing other than at least 2345 packets, at most 
2345+n*2^32 have been got since you last checked.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: IP Acounting Idea for 2.5

[Leif Sawyer]

Manfred Bartz responded to
> Russell King <[EMAIL PROTECTED]> who writes:
> 
> > On Mon, Apr 16, 2001 at 12:07:31PM +1000, Manfred Bartz wrote:
> > > There is another issue with logging in general:
> > > 
> > > *COUNTERS MUST NOT BE RESETABLE!!!*
> > 
> > Umm, no.  Counters can be resetable - you just specify that 
> > accounting programs should not reset them, ever.
> > 
> > The ability to reset counters is extremely useful if you're a human
> > looking at the output of iptables -L -v.  (I thus far know of no one
> > who can memorise the counter values for around 40 rules).
> 
> You just illustrated my point.  While there is a reset capability
> people will use it and accounting/logging programs will get wrong
> data.  Resetable counters might be a minor convenience when debugging
> but the price is unreliable programs and the loss of the ability of
> several programs to use the same counters.

You of course, are commenting from the fact that your applications are
stupid, written poorly, and cannot handle 'wrapped' data.  Take MRTG
as one example of a good application, for it can handle counters that
have been cleared between reads.
 
> Also, do you always reset the date and time everytime you make time 
> measurements?

No, but i realize that if my time has incremented, and my date has not
changed, then it's later.  If my time has _decremented_ and my date has not,
then I've time-warped.  If my time has decremented and my date has
incremented,
then it's simply the next-date-increment.

Similarly, if my InPackets are at 102345 at one read, and 2345 the next
read,
and I know that my counter is 32 bits, then I know i've wrapped and can do
my
own math.

Applications need to get smarter.  Don't place arbitrary rules where they
don't
belong.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Manfred Bartz]

Russell King <[EMAIL PROTECTED]> writes:

> On Mon, Apr 16, 2001 at 12:07:31PM +1000, Manfred Bartz wrote:
> > There is another issue with logging in general:
> > 
> > *COUNTERS MUST NOT BE RESETABLE!!!*
> 
> Umm, no.  Counters can be resetable - you just specify that accounting
> programs should not reset them, ever.
> 
> The ability to reset counters is extremely useful if you're a human
> looking at the output of iptables -L -v.  (I thus far know of no one
> who can memorise the counter values for around 40 rules).

You just illustrated my point.  While there is a reset capability
people will use it and accounting/logging programs will get wrong
data.  Resetable counters might be a minor convenience when debugging
but the price is unreliable programs and the loss of the ability of
several programs to use the same counters.

Also, do you always reset the date and time everytime you make time 
measurements?

-- 
Manfred Bartz
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Andreas Ferber]

Hi,

On Tue, Apr 17, 2001 at 08:46:12AM +1000, David Findlay wrote:
> 
> I suppose, but it would be so much easier if the kernel did it automatically. 
> Having a rule to go through for each IP address to be logged would be slower 
> than implementing one rule that would log all of them. Doing this in the 
> kernel would improve preformance.

You can use the iptables ULOG facility (see netfilter CVS for the
patches). With this you can copy packets to userspace programs which
can do any further processing, like the accounting stuff. You can even
choose to only copy part of a packet (say only the IP header) to
userspace, to reduce the amount of data your application has to
handle.

Simply copy the headers of all IP packets going through your router to
userspace, and write a small application (possibly using libipulog
which comes with the netfilter userspace code) which does the actual
accounting. This has the additional benefit that further processing of
the packet in kernel (i.e. outputting it on the destination interface)
can continue while your application is processing the accounting data.

Andreas
-- 
You are in a maze of little twisting passages, all alike.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Russell King]

On Mon, Apr 16, 2001 at 12:07:31PM +1000, Manfred Bartz wrote:
> There is another issue with logging in general:
> 
> *COUNTERS MUST NOT BE RESETABLE!!!*

Umm, no.  Counters can be resetable - you just specify that accounting
programs should not reset them, ever.

The ability to reset counters is extremely useful if you're a human
looking at the output of iptables -L -v.  (I thus far know of no one
who can memorise the counter values for around 40 rules).

--
Russell King ([EMAIL PROTECTED])The developer of ARM Linux
 http://www.arm.linux.org.uk/personal/aboutme.html

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Russell King]

On Mon, Apr 16, 2001 at 12:07:31PM +1000, Manfred Bartz wrote:
 There is another issue with logging in general:
 
 *COUNTERS MUST NOT BE RESETABLE!!!*

Umm, no.  Counters can be resetable - you just specify that accounting
programs should not reset them, ever.

The ability to reset counters is extremely useful if you're a human
looking at the output of iptables -L -v.  (I thus far know of no one
who can memorise the counter values for around 40 rules).

--
Russell King ([EMAIL PROTECTED])The developer of ARM Linux
 http://www.arm.linux.org.uk/personal/aboutme.html

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Andreas Ferber]

Hi,

On Tue, Apr 17, 2001 at 08:46:12AM +1000, David Findlay wrote:
 
 I suppose, but it would be so much easier if the kernel did it automatically. 
 Having a rule to go through for each IP address to be logged would be slower 
 than implementing one rule that would log all of them. Doing this in the 
 kernel would improve preformance.

You can use the iptables ULOG facility (see netfilter CVS for the
patches). With this you can copy packets to userspace programs which
can do any further processing, like the accounting stuff. You can even
choose to only copy part of a packet (say only the IP header) to
userspace, to reduce the amount of data your application has to
handle.

Simply copy the headers of all IP packets going through your router to
userspace, and write a small application (possibly using libipulog
which comes with the netfilter userspace code) which does the actual
accounting. This has the additional benefit that further processing of
the packet in kernel (i.e. outputting it on the destination interface)
can continue while your application is processing the accounting data.

Andreas
-- 
You are in a maze of little twisting passages, all alike.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Manfred Bartz]

Russell King [EMAIL PROTECTED] writes:

 On Mon, Apr 16, 2001 at 12:07:31PM +1000, Manfred Bartz wrote:
  There is another issue with logging in general:
  
  *COUNTERS MUST NOT BE RESETABLE!!!*
 
 Umm, no.  Counters can be resetable - you just specify that accounting
 programs should not reset them, ever.
 
 The ability to reset counters is extremely useful if you're a human
 looking at the output of iptables -L -v.  (I thus far know of no one
 who can memorise the counter values for around 40 rules).

You just illustrated my point.  While there is a reset capability
people will use it and accounting/logging programs will get wrong
data.  Resetable counters might be a minor convenience when debugging
but the price is unreliable programs and the loss of the ability of
several programs to use the same counters.

Also, do you always reset the date and time everytime you make time 
measurements?

-- 
Manfred Bartz
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: IP Acounting Idea for 2.5

[Leif Sawyer]

Manfred Bartz responded to
 Russell King [EMAIL PROTECTED] who writes:
 
  On Mon, Apr 16, 2001 at 12:07:31PM +1000, Manfred Bartz wrote:
   There is another issue with logging in general:
   
   *COUNTERS MUST NOT BE RESETABLE!!!*
  
  Umm, no.  Counters can be resetable - you just specify that 
  accounting programs should not reset them, ever.
  
  The ability to reset counters is extremely useful if you're a human
  looking at the output of iptables -L -v.  (I thus far know of no one
  who can memorise the counter values for around 40 rules).
 
 You just illustrated my point.  While there is a reset capability
 people will use it and accounting/logging programs will get wrong
 data.  Resetable counters might be a minor convenience when debugging
 but the price is unreliable programs and the loss of the ability of
 several programs to use the same counters.

You of course, are commenting from the fact that your applications are
stupid, written poorly, and cannot handle 'wrapped' data.  Take MRTG
as one example of a good application, for it can handle counters that
have been cleared between reads.
 
 Also, do you always reset the date and time everytime you make time 
 measurements?

No, but i realize that if my time has incremented, and my date has not
changed, then it's later.  If my time has _decremented_ and my date has not,
then I've time-warped.  If my time has decremented and my date has
incremented,
then it's simply the next-date-increment.

Similarly, if my InPackets are at 102345 at one read, and 2345 the next
read,
and I know that my counter is 32 bits, then I know i've wrapped and can do
my
own math.

Applications need to get smarter.  Don't place arbitrary rules where they
don't
belong.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Ian Stirling]

 
 Manfred Bartz responded to
  Russell King [EMAIL PROTECTED] who writes:
snip
  You just illustrated my point.  While there is a reset capability
  people will use it and accounting/logging programs will get wrong
  data.  Resetable counters might be a minor convenience when debugging
  but the price is unreliable programs and the loss of the ability of
  several programs to use the same counters.
 
 You of course, are commenting from the fact that your applications are
 stupid, written poorly, and cannot handle 'wrapped' data.  Take MRTG
snip
 Similarly, if my InPackets are at 102345 at one read, and 2345 the next
 read,
 and I know that my counter is 32 bits, then I know i've wrapped and can do

I think the point being made is that if InPackets are at 102345 at one read,
and 2345 the next, and you know it's a 32 bit counter, it's completely
unreliable to assume that you have in fact recieved 4294867295
packets, if the counter can be zeroed.
You can say nothing other than at least 2345 packets, at most 
2345+n*2^32 have been got since you last checked.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: IP Acounting Idea for 2.5

[Leif Sawyer]

 From: Ian Stirling [mailto:[EMAIL PROTECTED]]
  Manfred Bartz responded to
   Russell King [EMAIL PROTECTED] who writes:
 snip
   You just illustrated my point.  While there is a reset capability
   people will use it and accounting/logging programs will get wrong
   data.  Resetable counters might be a minor convenience 
 when debugging
   but the price is unreliable programs and the loss of the 
 ability of
   several programs to use the same counters.
  
  You of course, are commenting from the fact that your 
 applications are
  stupid, written poorly, and cannot handle 'wrapped' data.  Take MRTG
 snip
  Similarly, if my InPackets are at 102345 at one read, and 
 2345 the next
  read,
  and I know that my counter is 32 bits, then I know i've 
 wrapped and can do
 
 I think the point being made is that if InPackets are at 
 102345 at one read,
 and 2345 the next, and you know it's a 32 bit counter, it's completely
 unreliable to assume that you have in fact recieved 4294867295
 packets, if the counter can be zeroed.
 You can say nothing other than at least 2345 packets, at most 
 2345+n*2^32 have been got since you last checked.

Ah, yes.. I seem to have misplaced a bit of text in my reply.

The continuation of thought:

How the application derives the status of a wrapped counter or
a zero'ed counter is dependant on the device being monitored.

Yes, you have to know what your interface is capable of (maxbytes/sec)
so that you can do a simple calculation where:

maximum_throughput = maxbytes_sec * (time_now - time_last_read)

and if your previous good counter + the maximum throughput wraps the
counter, you have a good chance that you've simply wrapped.

If not, then you can assume that your counters were cleared at some point,
log the data you've got, and keep moving forward.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Manfred Bartz]

Leif Sawyer [EMAIL PROTECTED] writes:

 Manfred Bartz responded to
  Russell King [EMAIL PROTECTED] who writes:
  
   On Mon, Apr 16, 2001 at 12:07:31PM +1000, Manfred Bartz wrote:
There is another issue with logging in general:

*COUNTERS MUST NOT BE RESETABLE!!!*
   
   Umm, no.  Counters can be resetable - you just specify that 
   accounting programs should not reset them, ever.
   
   The ability to reset counters is extremely useful if you're a human
   looking at the output of iptables -L -v.  (I thus far know of no one
   who can memorise the counter values for around 40 rules).
  
  You just illustrated my point.  While there is a reset capability
  people will use it and accounting/logging programs will get wrong
  data.  Resetable counters might be a minor convenience when debugging
  but the price is unreliable programs and the loss of the ability of
  several programs to use the same counters.
 
 You of course, are commenting from the fact that your applications are
 stupid, written poorly, and cannot handle 'wrapped' data.  

Well, apps written by me *can handle* wrapped data; it is actually far
easier that resetting at each read.

 Take MRTG as one example of a good application, for it can handle
 counters that have been cleared between reads.

Regardless of how well a logging application is written, there is no
way that it does not loose data when someone or something resets its
counter.  F.e. if an app reads a counter every 10 minutes and a
counter reset occurs from elsewhere, then anywhere between 0 and 10
minutes worth of data is lost.

I had a brief look at MRTG.  It seems to be a well written app and
while it can handle counter reset (with potential loss of an unknown 
amount of data), it does not actively reset counters.  It also doesn't
use iptables.

 Applications need to get smarter.  

Agreed.

 Don't place arbitrary rules where they don't belong.

Agreed too.  Counters should not arbitrarily be equipped with a reset
capability, there is hardly any benefit in that and it causes nothing
but problems.

As far as I can see, the counters in /proc/net/snmp don't have a
reset, same with /proc/net/dev and possibly other counters elsewhere.

Ideally iptables would fall in line with that.  Rules can still be
unloaded and reloaded, also causing counter reset and loss of data,
but since that is a lot more involved, application authors would have
an incentive to handle counters properly.

-- 
Manfred Bartz

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Harald Welte]

On Tue, Apr 17, 2001 at 07:53:28AM +1000, David Findlay wrote:

 In the 2.5 series of kernels, working towards 2.6, could you please make the 
 IP Accounting so that I can set a single rule that will make it watch all IP 
 traffic going from the local network, through the masquerading service to the 
 internet, and log local IP Addresses using it? This would allow me to set 1 
 rule, but have the information I want on a per IP address system.

:) Well... there was a discussion about this on the netfilter-devel
list some weeks ago. 

This is definitely not a 2.5 / 2.6 issue, as it could be easily implemented
by using the already existing flexibility of netfilter/iptables.

It's just a matter of somebody who needs it getting around actually 
implementing it. Could be either implemented as a 'raw' netfilter 
hook-attaching module or (more convenient) as a new iptables target, 
which allocates some internal tables for storing the accounting data.

Doesn't make sense to have a discussion about it on linux-kernel, and
it certainly doesn't belong into a 2.5 featurelist.

Please refer to the netfilter-devel mailinglist (subscription information
available at http://netfilter.samba.org/)

 David

-- 
Live long and prosper
- Harald Welte / [EMAIL PROTECTED]http://www.gnumonks.org

GCS/E/IT d- s-: a-- C+++ UL$ P+++ L$ E--- W- N++ o? K- w--- O- M- 
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Harald Welte]

On Mon, Apr 16, 2001 at 12:07:31PM +1000, Manfred Bartz wrote:
 
 If there really is a performance issue with a few hundred rules, then
 it can be overcome by grouping rules in separate custom chains.  F.e.
 if you have 1024 rules create 32 custom chains with 32 rules each.
 Then have 32 rules in your main table which jump to the appropriate
 custom chain -- maximum rules traversed by each packet = 64.

still.. if you want to log multiple class B's it's a PITA, especially
if your ruleset changes and you have to atomically replace all chains
in one table to update the ruleset from userspace-kernel.

 There is another issue with logging in general:
 
 *COUNTERS MUST NOT BE RESETABLE!!!*
 
 Resetable counters guarantee that no two programs can co-exists if
 they happen to reset the same counters.

That sounds like crap (sorry). Counters are resettable, and will be.
If you run two applications resetting counters individually, you have
a problem with your applications.

 All logging counters should be implemented with 32bit or 64bit
 unsigned integers.  Any software using correct unsigned integer

they are uint32 right now.

 arithmetic can then simply subtract a previous value from the current
 value to get the difference.  This works reliably across counter
 wrap-arounds.  There is absolutely *no need for reset* !

so why are your applications resetting counters then? Nobody forces you
to reset them right now.

 Manfred Bartz
 ---

-- 
Live long and prosper
- Harald Welte / [EMAIL PROTECTED]http://www.gnumonks.org

GCS/E/IT d- s-: a-- C+++ UL$ P+++ L$ E--- W- N++ o? K- w--- O- M- 
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Manfred Bartz]

Harald Welte [EMAIL PROTECTED] writes:

 On Mon, Apr 16, 2001 at 12:07:31PM +1000, Manfred Bartz wrote:
  Resetable counters guarantee that no two programs can co-exists if
  they happen to reset the same counters.
 
 That sounds like crap (sorry). 

Care to explain how two independent apps can co-exists if they happen
to reset the same counters?

 so why are your applications resetting counters then? 

Ask the authors, not me.  I wiped those apps from my system.
Its just a pity about the time wasted.

 Nobody forces you to reset them right now.

Why do authors then write apps (f.e. ipac) which reset counters?  
Perhaps only because they can, not because it makes sense.

-- 
Manfred Bartz
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Dax Kelson]

David Findlay said once upon a time (Tue, 17 Apr 2001):

> I am using the kernel IP Accounting in Linux to record the amount of data
> transfered via my Linux internet gateway from individual IP addresses. This
> currently requires me to set up an accounting rule for each IP address that I
> want to record accounting info for. If I had 200 machines to individually log
> this would require me to set 200 rules.

In 1999 I setup a Linux kernel 2.2 box to monitor traffic for about 2500
hosts.  I created two rules for each host (to monitor inbound and
outbound).  I had over 5000 total rules on a Pentium II 300 Mhz box.  I
wrote some perl glue so I could accounts on a per customer basis (each
customer had one or more IPs, not necessarily contiguous) and graph the
results in MRTG.

There was ZERO performance problems on the box.  The box is still
runningly happily today.  I don't there is a "problem" to fix.

Dax

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Manfred Bartz]

David Findlay <[EMAIL PROTECTED]> writes:

> On Monday 16 April 2001 10:40, you wrote:

> > Perhaps I misunderstand what it is exactly you are trying to do,
> > but I would think that this could be done entirely in userland by
> > software that just adds rules for you instead of you having to do
> > it manually.
> 
> I suppose, but it would be so much easier if the kernel did it
> automatically. 

Yes, but by what criteria?  You want to log on a per host basis.
Someone else wants to log by service (www, ftp, mail, news), or
any number of other criteria.

> Having a rule to go through for each IP address to be logged would
> be slower than implementing one rule that would log all of
> them. Doing this in the kernel would improve preformance.

If there really is a performance issue with a few hundred rules, then
it can be overcome by grouping rules in separate custom chains.  F.e.
if you have 1024 rules create 32 custom chains with 32 rules each.
Then have 32 rules in your main table which jump to the appropriate
custom chain --> maximum rules traversed by each packet = 64.

There is another issue with logging in general:

*COUNTERS MUST NOT BE RESETABLE!!!*

Resetable counters guarantee that no two programs can co-exists if
they happen to reset the same counters.

All logging counters should be implemented with 32bit or 64bit
unsigned integers.  Any software using correct unsigned integer
arithmetic can then simply subtract a previous value from the current
value to get the difference.  This works reliably across counter
wrap-arounds.  There is absolutely *no need for reset* !

-- 
Manfred Bartz
---
ipchainsLogAnalyzer, NetCalc, whois at: 
 NEW: 

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[swds . mlowe]

No, one rule would be MUCH faster. What's do you think would be faster of the two:

if ((ipaddr>=3232235521)&&(ipaddr<=3232235774))
return 1;
else
return 0;

or

for (a=0;a<(3232235774-3232235521);a++)
if (ipaddr==a)
return 1;
return 0;

Obviously it compares the 192.168.0.1 - 192.168.0.255 range, but I think we both
know which one will be faster. Not to mention countless other redundant checks
will be added in on both, but on the second one the time of the checks is
multiplied by the number of times you have too loop.

But, I'm just a newbie so I don't really know :P Just taking up bandwith :(

L8ers,
Matt

"Mike A. Harris" wrote:

> On Tue, 17 Apr 2001, David Findlay wrote:
>
> >> Perhaps I misunderstand what it is exactly you are trying to do,
> >> but I would think that this could be done entirely in userland by
> >> software that just adds rules for you instead of you having to do
> >> it manually.
> >
> >I suppose, but it would be so much easier if the kernel did it automatically.
> >Having a rule to go through for each IP address to be logged would be slower
> >than implementing one rule that would log all of them. Doing this in the
> >kernel would improve preformance.
>
> I don't think it would, but then only benchmarking it both ways
> would know for sure.  Even with incredibly large rulesets,
> ipchains &&/|| netfilter works admirably well.  Rusty?
>
> --
> Mike A. Harris  -  Linux advocate  -  Free Software advocate
>   This message is copyright 2001, all rights reserved.
>   Views expressed are my own, not necessarily shared by my employer.
> --
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [EMAIL PROTECTED]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Matti Aarnio]

On Tue, Apr 17, 2001 at 08:46:12AM +1000, David Findlay wrote:
> On Monday 16 April 2001 10:40, you wrote:
> > Perhaps I misunderstand what it is exactly you are trying to do,
> > but I would think that this could be done entirely in userland by
> > software that just adds rules for you instead of you having to do
> > it manually.
> 
> I suppose, but it would be so much easier if the kernel did it automatically. 
> Having a rule to go through for each IP address to be logged would be slower 
> than implementing one rule that would log all of them. Doing this in the 
> kernel would improve preformance.

Perhaps "netflow" would be an answer ?

It is cisco idea for collecting accounting data for
network flows.   System has a cache of flows for
which it collects data, and once the cache overflows
(or flow times out), data is sent to designated flow
collection server(s).

http://www.cisco.com/warp/public/cc/pd/iosw/ioft/neflct/tech/napps_wp.htm

> David

/Matti Aarnio
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Mike A. Harris]

On Tue, 17 Apr 2001, David Findlay wrote:

>> Perhaps I misunderstand what it is exactly you are trying to do,
>> but I would think that this could be done entirely in userland by
>> software that just adds rules for you instead of you having to do
>> it manually.
>
>I suppose, but it would be so much easier if the kernel did it automatically.
>Having a rule to go through for each IP address to be logged would be slower
>than implementing one rule that would log all of them. Doing this in the
>kernel would improve preformance.

I don't think it would, but then only benchmarking it both ways
would know for sure.  Even with incredibly large rulesets,
ipchains &&/|| netfilter works admirably well.  Rusty?


--
Mike A. Harris  -  Linux advocate  -  Free Software advocate
  This message is copyright 2001, all rights reserved.
  Views expressed are my own, not necessarily shared by my employer.
--

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[David Findlay]

On Monday 16 April 2001 10:40, you wrote:
> Perhaps I misunderstand what it is exactly you are trying to do,
> but I would think that this could be done entirely in userland by
> software that just adds rules for you instead of you having to do
> it manually.

I suppose, but it would be so much easier if the kernel did it automatically. 
Having a rule to go through for each IP address to be logged would be slower 
than implementing one rule that would log all of them. Doing this in the 
kernel would improve preformance.

David
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Mike A. Harris]

On Tue, 17 Apr 2001, David Findlay wrote:

>I am using the kernel IP Accounting in Linux to record the amount of data
>transfered via my Linux internet gateway from individual IP addresses. This
>currently requires me to set up an accounting rule for each IP address that I
>want to record accounting info for. If I had 200 machines to individually log
>this would require me to set 200 rules.
>
>In the 2.5 series of kernels, working towards 2.6, could you please make the
>IP Accounting so that I can set a single rule that will make it watch all IP
>traffic going from the local network, through the masquerading service to the
>internet, and log local IP Addresses using it? This would allow me to set 1
>rule, but have the information I want on a per IP address system.
>
>One other person I have talked to would like to see this too, and he
>basically says we need a software version of the Cisco IP Accounting
>server/router.
>
>Could you please add this to the next kernel? Please CC me your responses as
>I am not a member of the kernel mailing list. Thanks,

Perhaps I misunderstand what it is exactly you are trying to do,
but I would think that this could be done entirely in userland by
software that just adds rules for you instead of you having to do
it manually.

Just a thought.

--
Mike A. Harris  -  Linux advocate  -  Free Software advocate
  This message is copyright 2001, all rights reserved.
  Views expressed are my own, not necessarily shared by my employer.
--
There are two major products that come out of Berkeley: LSD and BSD.
We don't believe this to be a coincidence. -- Jeremy S. Anderson

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: IP Acounting Idea for 2.5

[Michael Clark]

> In the 2.5 series of kernels, working towards 2.6, could you please make the
> IP Accounting so that I can set a single rule that will make it watch all IP
> traffic going from the local network, through the masquerading service to the
> internet, and log local IP Addresses using it? This would allow me to set 1
> rule, but have the information I want on a per IP address system.

You could try using a mature userspace traffic meter like 'NeTraMet' (uses
libpcap).

ftp://ftp.auckland.ac.nz/pub/iawg/NeTraMet/

> One other person I have talked to would like to see this too, and he
> basically says we need a software version of the Cisco IP Accounting
> server/router.

NeTraMet can also account using Cisco Netflow accounting records.

> Could you please add this to the next kernel? Please CC me your responses as
> I am not a member of the kernel mailing list. Thanks,
>
> David
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [EMAIL PROTECTED]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>
>

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

IP Acounting Idea for 2.5

[David Findlay]

I am using the kernel IP Accounting in Linux to record the amount of data 
transfered via my Linux internet gateway from individual IP addresses. This 
currently requires me to set up an accounting rule for each IP address that I 
want to record accounting info for. If I had 200 machines to individually log 
this would require me to set 200 rules.

In the 2.5 series of kernels, working towards 2.6, could you please make the 
IP Accounting so that I can set a single rule that will make it watch all IP 
traffic going from the local network, through the masquerading service to the 
internet, and log local IP Addresses using it? This would allow me to set 1 
rule, but have the information I want on a per IP address system.

One other person I have talked to would like to see this too, and he 
basically says we need a software version of the Cisco IP Accounting 
server/router.

Could you please add this to the next kernel? Please CC me your responses as 
I am not a member of the kernel mailing list. Thanks,

David
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

IP Acounting Idea for 2.5

[David Findlay]

I am using the kernel IP Accounting in Linux to record the amount of data 
transfered via my Linux internet gateway from individual IP addresses. This 
currently requires me to set up an accounting rule for each IP address that I 
want to record accounting info for. If I had 200 machines to individually log 
this would require me to set 200 rules.

In the 2.5 series of kernels, working towards 2.6, could you please make the 
IP Accounting so that I can set a single rule that will make it watch all IP 
traffic going from the local network, through the masquerading service to the 
internet, and log local IP Addresses using it? This would allow me to set 1 
rule, but have the information I want on a per IP address system.

One other person I have talked to would like to see this too, and he 
basically says we need a software version of the Cisco IP Accounting 
server/router.

Could you please add this to the next kernel? Please CC me your responses as 
I am not a member of the kernel mailing list. Thanks,

David
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: IP Acounting Idea for 2.5

[Michael Clark]

 In the 2.5 series of kernels, working towards 2.6, could you please make the
 IP Accounting so that I can set a single rule that will make it watch all IP
 traffic going from the local network, through the masquerading service to the
 internet, and log local IP Addresses using it? This would allow me to set 1
 rule, but have the information I want on a per IP address system.

You could try using a mature userspace traffic meter like 'NeTraMet' (uses
libpcap).

ftp://ftp.auckland.ac.nz/pub/iawg/NeTraMet/

 One other person I have talked to would like to see this too, and he
 basically says we need a software version of the Cisco IP Accounting
 server/router.

NeTraMet can also account using Cisco Netflow accounting records.

 Could you please add this to the next kernel? Please CC me your responses as
 I am not a member of the kernel mailing list. Thanks,

 David
 -
 To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
 the body of a message to [EMAIL PROTECTED]
 More majordomo info at  http://vger.kernel.org/majordomo-info.html
 Please read the FAQ at  http://www.tux.org/lkml/



-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Mike A. Harris]

On Tue, 17 Apr 2001, David Findlay wrote:

I am using the kernel IP Accounting in Linux to record the amount of data
transfered via my Linux internet gateway from individual IP addresses. This
currently requires me to set up an accounting rule for each IP address that I
want to record accounting info for. If I had 200 machines to individually log
this would require me to set 200 rules.

In the 2.5 series of kernels, working towards 2.6, could you please make the
IP Accounting so that I can set a single rule that will make it watch all IP
traffic going from the local network, through the masquerading service to the
internet, and log local IP Addresses using it? This would allow me to set 1
rule, but have the information I want on a per IP address system.

One other person I have talked to would like to see this too, and he
basically says we need a software version of the Cisco IP Accounting
server/router.

Could you please add this to the next kernel? Please CC me your responses as
I am not a member of the kernel mailing list. Thanks,

Perhaps I misunderstand what it is exactly you are trying to do,
but I would think that this could be done entirely in userland by
software that just adds rules for you instead of you having to do
it manually.

Just a thought.

--
Mike A. Harris  -  Linux advocate  -  Free Software advocate
  This message is copyright 2001, all rights reserved.
  Views expressed are my own, not necessarily shared by my employer.
--
There are two major products that come out of Berkeley: LSD and BSD.
We don't believe this to be a coincidence. -- Jeremy S. Anderson

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[David Findlay]

On Monday 16 April 2001 10:40, you wrote:
 Perhaps I misunderstand what it is exactly you are trying to do,
 but I would think that this could be done entirely in userland by
 software that just adds rules for you instead of you having to do
 it manually.

I suppose, but it would be so much easier if the kernel did it automatically. 
Having a rule to go through for each IP address to be logged would be slower 
than implementing one rule that would log all of them. Doing this in the 
kernel would improve preformance.

David
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Mike A. Harris]

On Tue, 17 Apr 2001, David Findlay wrote:

 Perhaps I misunderstand what it is exactly you are trying to do,
 but I would think that this could be done entirely in userland by
 software that just adds rules for you instead of you having to do
 it manually.

I suppose, but it would be so much easier if the kernel did it automatically.
Having a rule to go through for each IP address to be logged would be slower
than implementing one rule that would log all of them. Doing this in the
kernel would improve preformance.

I don't think it would, but then only benchmarking it both ways
would know for sure.  Even with incredibly large rulesets,
ipchains /|| netfilter works admirably well.  Rusty?


--
Mike A. Harris  -  Linux advocate  -  Free Software advocate
  This message is copyright 2001, all rights reserved.
  Views expressed are my own, not necessarily shared by my employer.
--

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Matti Aarnio]

On Tue, Apr 17, 2001 at 08:46:12AM +1000, David Findlay wrote:
 On Monday 16 April 2001 10:40, you wrote:
  Perhaps I misunderstand what it is exactly you are trying to do,
  but I would think that this could be done entirely in userland by
  software that just adds rules for you instead of you having to do
  it manually.
 
 I suppose, but it would be so much easier if the kernel did it automatically. 
 Having a rule to go through for each IP address to be logged would be slower 
 than implementing one rule that would log all of them. Doing this in the 
 kernel would improve preformance.

Perhaps "netflow" would be an answer ?

It is cisco idea for collecting accounting data for
network flows.   System has a cache of flows for
which it collects data, and once the cache overflows
(or flow times out), data is sent to designated flow
collection server(s).

http://www.cisco.com/warp/public/cc/pd/iosw/ioft/neflct/tech/napps_wp.htm

 David

/Matti Aarnio
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[swds . mlowe]

No, one rule would be MUCH faster. What's do you think would be faster of the two:

if ((ipaddr=3232235521)(ipaddr=3232235774))
return 1;
else
return 0;

or

for (a=0;a(3232235774-3232235521);a++)
if (ipaddr==a)
return 1;
return 0;

Obviously it compares the 192.168.0.1 - 192.168.0.255 range, but I think we both
know which one will be faster. Not to mention countless other redundant checks
will be added in on both, but on the second one the time of the checks is
multiplied by the number of times you have too loop.

But, I'm just a newbie so I don't really know :P Just taking up bandwith :(

L8ers,
Matt

"Mike A. Harris" wrote:

 On Tue, 17 Apr 2001, David Findlay wrote:

  Perhaps I misunderstand what it is exactly you are trying to do,
  but I would think that this could be done entirely in userland by
  software that just adds rules for you instead of you having to do
  it manually.
 
 I suppose, but it would be so much easier if the kernel did it automatically.
 Having a rule to go through for each IP address to be logged would be slower
 than implementing one rule that would log all of them. Doing this in the
 kernel would improve preformance.

 I don't think it would, but then only benchmarking it both ways
 would know for sure.  Even with incredibly large rulesets,
 ipchains /|| netfilter works admirably well.  Rusty?

 --
 Mike A. Harris  -  Linux advocate  -  Free Software advocate
   This message is copyright 2001, all rights reserved.
   Views expressed are my own, not necessarily shared by my employer.
 --

 -
 To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
 the body of a message to [EMAIL PROTECTED]
 More majordomo info at  http://vger.kernel.org/majordomo-info.html
 Please read the FAQ at  http://www.tux.org/lkml/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Manfred Bartz]

David Findlay [EMAIL PROTECTED] writes:

 On Monday 16 April 2001 10:40, you wrote:

  Perhaps I misunderstand what it is exactly you are trying to do,
  but I would think that this could be done entirely in userland by
  software that just adds rules for you instead of you having to do
  it manually.
 
 I suppose, but it would be so much easier if the kernel did it
 automatically. 

Yes, but by what criteria?  You want to log on a per host basis.
Someone else wants to log by service (www, ftp, mail, news), or
any number of other criteria.

 Having a rule to go through for each IP address to be logged would
 be slower than implementing one rule that would log all of
 them. Doing this in the kernel would improve preformance.

If there really is a performance issue with a few hundred rules, then
it can be overcome by grouping rules in separate custom chains.  F.e.
if you have 1024 rules create 32 custom chains with 32 rules each.
Then have 32 rules in your main table which jump to the appropriate
custom chain -- maximum rules traversed by each packet = 64.

There is another issue with logging in general:

*COUNTERS MUST NOT BE RESETABLE!!!*

Resetable counters guarantee that no two programs can co-exists if
they happen to reset the same counters.

All logging counters should be implemented with 32bit or 64bit
unsigned integers.  Any software using correct unsigned integer
arithmetic can then simply subtract a previous value from the current
value to get the difference.  This works reliably across counter
wrap-arounds.  There is absolutely *no need for reset* !

-- 
Manfred Bartz
---
ipchainsLogAnalyzer, NetCalc, whois at: http://logi.cc/linux/
 NEW: http://logi.cc/linux/NetfilterLogAnalyzer.php3

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: IP Acounting Idea for 2.5

[Dax Kelson]

David Findlay said once upon a time (Tue, 17 Apr 2001):

 I am using the kernel IP Accounting in Linux to record the amount of data
 transfered via my Linux internet gateway from individual IP addresses. This
 currently requires me to set up an accounting rule for each IP address that I
 want to record accounting info for. If I had 200 machines to individually log
 this would require me to set 200 rules.

In 1999 I setup a Linux kernel 2.2 box to monitor traffic for about 2500
hosts.  I created two rules for each host (to monitor inbound and
outbound).  I had over 5000 total rules on a Pentium II 300 Mhz box.  I
wrote some perl glue so I could accounts on a per customer basis (each
customer had one or more IPs, not necessarily contiguous) and graph the
results in MRTG.

There was ZERO performance problems on the box.  The box is still
runningly happily today.  I don't there is a "problem" to fix.

Dax

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/