Re: [RADIATOR] logging (radiator and authlog) and accounting to ElasticSearch
Hi Heikki, attached is what I just wrote, feedback welcome! Feel free to include it in the Radiator dist with an extended copyright, different name, ... Best regards, Alex On 2014-04-04 14:42, Heikki Vatiainen wrote: > On 04/03/2014 12:28 PM, Hartmaier Alexander wrote: > >> I just checked, LogFormat is indeed defined in LogFILE.pm, I thought it >> is shared by different logging classes and defined in a base class. > You are correct, there's no common class for LogFILE but the common > class is LogGeneric instead. And since logs can be pushed to for > example, SQL, LogFormat is only defined in LogFILE. > >> I don't care so much where it is defined but how. >> Do you think configuring a Perl sub that returns a serialized string is >> the way to go for structured logging? > I guess that would be like a in-line Hook? Maybe it could be configured > as a hook type of parameter with a default value. See for example, > SqlDb.pm and how ConnectionAttemptFailedHook is initialised with a > default hook code. > >> I was thinking about writing a Log module that uses Log::Any, Log4perl >> or Log::Dispatch so that you can do everything these modules support. >> My only concern is that the logging module affects Radiator and if e.g. >> RabbitMQ is down and logs can't be stored Radiator stops working. >> This might be intended if logging is an absolute requirement, but others >> might prefer availability over logging. Ideally this should be configurable. > Indeed. As noted below, LogSQL can be configured with so that it won't > make everything stop unless required so. > >> What happens for the LogSQL class if the database goes down? > It uses the functions defined in SqlDb.pm which for example, AuthBy SQL > uses. Because of this, LogSQL respects FailureBackoffTime, Timeout and > other common parameters AuthBy SQL uses. > >> Is there some documentation how to write a Log class? I looks like It >> only has to have the ConfigKeywords hash, subclass from >> Radius::LogGeneric and an initialize and log sub. > I'd say the existing Log* files are the best example. > >> Is there anything special I need to do to allow a ConfigKeyword to hold >> a Perl sub? > Maybe create it as a Hook? That should help not duplicating existing > functionality. > > Thanks, > Heikki > > >> BR Alex >> >>> >>> Thanks, >>> Heikki >>> >> >> >> *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"* >> T-Systems Austria GesmbH Rennweg 97-99, 1030 Wien >> Handelsgericht Wien, FN 79340b >> *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"* >> Notice: This e-mail contains information that is confidential and may be >> privileged. >> If you are not the intended recipient, please notify the sender and then >> delete this e-mail immediately. >> *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"* >> > LogStructured.pm Description: Perl program ___ radiator mailing list radiator@open.com.au http://www.open.com.au/mailman/listinfo/radiator
Re: [RADIATOR] logging (radiator and authlog) and accounting to ElasticSearch
On 04/03/2014 12:28 PM, Hartmaier Alexander wrote: > I just checked, LogFormat is indeed defined in LogFILE.pm, I thought it > is shared by different logging classes and defined in a base class. You are correct, there's no common class for LogFILE but the common class is LogGeneric instead. And since logs can be pushed to for example, SQL, LogFormat is only defined in LogFILE. > I don't care so much where it is defined but how. > Do you think configuring a Perl sub that returns a serialized string is > the way to go for structured logging? I guess that would be like a in-line Hook? Maybe it could be configured as a hook type of parameter with a default value. See for example, SqlDb.pm and how ConnectionAttemptFailedHook is initialised with a default hook code. > I was thinking about writing a Log module that uses Log::Any, Log4perl > or Log::Dispatch so that you can do everything these modules support. > My only concern is that the logging module affects Radiator and if e.g. > RabbitMQ is down and logs can't be stored Radiator stops working. > This might be intended if logging is an absolute requirement, but others > might prefer availability over logging. Ideally this should be configurable. Indeed. As noted below, LogSQL can be configured with so that it won't make everything stop unless required so. > What happens for the LogSQL class if the database goes down? It uses the functions defined in SqlDb.pm which for example, AuthBy SQL uses. Because of this, LogSQL respects FailureBackoffTime, Timeout and other common parameters AuthBy SQL uses. > Is there some documentation how to write a Log class? I looks like It > only has to have the ConfigKeywords hash, subclass from > Radius::LogGeneric and an initialize and log sub. I'd say the existing Log* files are the best example. > Is there anything special I need to do to allow a ConfigKeyword to hold > a Perl sub? Maybe create it as a Hook? That should help not duplicating existing functionality. Thanks, Heikki > BR Alex > >> >> >> Thanks, >> Heikki >> > > > > *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"* > T-Systems Austria GesmbH Rennweg 97-99, 1030 Wien > Handelsgericht Wien, FN 79340b > *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"* > Notice: This e-mail contains information that is confidential and may be > privileged. > If you are not the intended recipient, please notify the sender and then > delete this e-mail immediately. > *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"* > -- Heikki Vatiainen Radiator: the most portable, flexible and configurable RADIUS server anywhere. SQL, proxy, DBM, files, LDAP, NIS+, password, NT, Emerald, Platypus, Freeside, TACACS+, PAM, external, Active Directory, EAP, TLS, TTLS, PEAP, TNC, WiMAX, RSA, Vasco, Yubikey, MOTP, HOTP, TOTP, DIAMETER etc. Full source on Unix, Windows, MacOSX, Solaris, VMS, NetWare etc. ___ radiator mailing list radiator@open.com.au http://www.open.com.au/mailman/listinfo/radiator
Re: [RADIATOR] logging (radiator and authlog) and accounting to ElasticSearch
On 2014-04-02 20:57, Heikki Vatiainen wrote: > On 04/01/2014 02:59 PM, Hartmaier Alexander wrote: > >> I think extending LogFormat is the right way to go because one might >> want to log to a file or database in json or yaml as well. >> What I still haven't figured out is a config format. >> Enabling to pass a Perl sub to LogFormat would be the most flexible >> option but not the most user friendly. >> >> Ideas? > Wouldn't subclassing Log FILE work. Or maybe subclassing LogGeneric > directly? I'd think keeping Log FILE as simple and straight forward > would be a good goal since sometimes (Trace 4) a lot of traffic will > pass through it. I just checked, LogFormat is indeed defined in LogFILE.pm, I thought it is shared by different logging classes and defined in a base class. I don't care so much where it is defined but how. Do you think configuring a Perl sub that returns a serialized string is the way to go for structured logging? I was thinking about writing a Log module that uses Log::Any, Log4perl or Log::Dispatch so that you can do everything these modules support. My only concern is that the logging module affects Radiator and if e.g. RabbitMQ is down and logs can't be stored Radiator stops working. This might be intended if logging is an absolute requirement, but others might prefer availability over logging. Ideally this should be configurable. What happens for the LogSQL class if the database goes down? Is there some documentation how to write a Log class? I looks like It only has to have the ConfigKeywords hash, subclass from Radius::LogGeneric and an initialize and log sub. Is there anything special I need to do to allow a ConfigKeyword to hold a Perl sub? BR Alex > > > Thanks, > Heikki > *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"* T-Systems Austria GesmbH Rennweg 97-99, 1030 Wien Handelsgericht Wien, FN 79340b *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"* Notice: This e-mail contains information that is confidential and may be privileged. If you are not the intended recipient, please notify the sender and then delete this e-mail immediately. *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"* ___ radiator mailing list radiator@open.com.au http://www.open.com.au/mailman/listinfo/radiator
Re: [RADIATOR] logging (radiator and authlog) and accounting to ElasticSearch
On 04/01/2014 02:59 PM, Hartmaier Alexander wrote: > I think extending LogFormat is the right way to go because one might > want to log to a file or database in json or yaml as well. > What I still haven't figured out is a config format. > Enabling to pass a Perl sub to LogFormat would be the most flexible > option but not the most user friendly. > > Ideas? Wouldn't subclassing Log FILE work. Or maybe subclassing LogGeneric directly? I'd think keeping Log FILE as simple and straight forward would be a good goal since sometimes (Trace 4) a lot of traffic will pass through it. Thanks, Heikki -- Heikki Vatiainen Radiator: the most portable, flexible and configurable RADIUS server anywhere. SQL, proxy, DBM, files, LDAP, NIS+, password, NT, Emerald, Platypus, Freeside, TACACS+, PAM, external, Active Directory, EAP, TLS, TTLS, PEAP, TNC, WiMAX, RSA, Vasco, Yubikey, MOTP, HOTP, TOTP, DIAMETER etc. Full source on Unix, Windows, MacOSX, Solaris, VMS, NetWare etc. ___ radiator mailing list radiator@open.com.au http://www.open.com.au/mailman/listinfo/radiator
Re: [RADIATOR] logging (radiator and authlog) and accounting to ElasticSearch
On 2014-03-28 09:02, Hartmaier Alexander wrote:
> On 2014-03-27 20:43, Heikki Vatiainen wrote:
>> On 03/27/2014 05:22 PM, Hartmaier Alexander wrote:
>>
>>> Did you have time to work on this feature?
>> We have worked on EAP-SIM, Diameter and other RADIUS functionality, but
>> not this. It's still on the ideas to explore list, though.
>>
>>> I've started tring to get all Radiator logs into Elasticsearch via
>>> RabbitMQ and found out that also closes the pipe in case of
>>> Filename |/etc/radiator/log_to_rabbitmq.pl which is a performance problem.
>> It makes the log files easy to rotate, but I can see why it's not good
>> in this case.
> Because the logging script has some quite a 'high' startup time because
> it has to connect to RabbitMQ and also disconnects immediatly afterwards
> causing quite some overhead and RabbitMQ logging.
>>> The comment in Radius::Util::append says:
>>> # Current implementation opens, writes and closes
>>> # Future implementations might hold the file open, and reopen on
>>> # signal, or perhaps pipe to a daemon
>>>
>>> I need a feature that keeps the spawned (forked?) process open and pipe
>>> to it.
>>> I'd also need to escape the log message so my JSON doesn't break in case
>>> the message contains one or more ".
>>>
>>> LogFormat { "timestamp":"%Y-%m-%d %H:%M:%S", "priority":"%1",
>>> "message":"%2" }
>> Hmm, maybe a new class that could be configured to support the above? It
>> would need to be able to quote the log messages, maybe a configurable
>> method for different kinds of destinations (JSON, etc), hold the fd open
>> while supporting FarmSize and possibly something else too. I'd say
>> extending Log FILE may not be a good idea but to have a new logging class.
>>
>> If you already have something that does what you require on the > ...> side, please get back to me directly.
> Yeah, I was thinking about writing a LogMessagePassing or Log4perl class
> but am not sure to which api I have to follow as its internal and as far
> as I know undocumented.
> For now I've created a named pipe which I configured Radiator to log to
> and have a Message::Passing DSL script that runs as daemon using
> Daemon::Control which uses the TailFile input and the AMQP output.
> I wanted to not have to run that extra daemon because it's one more
> thing that can go wrong.
I need a solution that outputs structed log messages serialized as JSON
fast.
My quick solution is:
# this is a named pipe created using mkfifo
Filename %L/radiator.fifo
LogFormat { "timestamp":"%Y-%m-%dT%H:%M:%SZ", "source_host":"%h",
"priority":"%1", "type":"radiator",
"customer":"%{OSC-Customer-Identifier}", "message":"%2" }
Trace 3
The problem is that some messages contain linefeeds (openssl library
errors when using EAP-TLS for example) which don't get escaped as per
JSON spec and make the JSON parsing fail.
I think extending LogFormat is the right way to go because one might
want to log to a file or database in json or yaml as well.
What I still haven't figured out is a config format.
Enabling to pass a Perl sub to LogFormat would be the most flexible
option but not the most user friendly.
Ideas?
>
>> Thanks,
>> Heikki
>>
>>
>
>
> *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*
> T-Systems Austria GesmbH Rennweg 97-99, 1030 Wien
> Handelsgericht Wien, FN 79340b
> *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*
> Notice: This e-mail contains information that is confidential and may be
> privileged.
> If you are not the intended recipient, please notify the sender and then
> delete this e-mail immediately.
> *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*
> ___
> radiator mailing list
> radiator@open.com.au
> http://www.open.com.au/mailman/listinfo/radiator
___
radiator mailing list
radiator@open.com.au
http://www.open.com.au/mailman/listinfo/radiator
Re: [RADIATOR] logging (radiator and authlog) and accounting to ElasticSearch
On 2014-03-27 20:43, Heikki Vatiainen wrote:
> On 03/27/2014 05:22 PM, Hartmaier Alexander wrote:
>
>> Did you have time to work on this feature?
> We have worked on EAP-SIM, Diameter and other RADIUS functionality, but
> not this. It's still on the ideas to explore list, though.
>
>> I've started tring to get all Radiator logs into Elasticsearch via
>> RabbitMQ and found out that also closes the pipe in case of
>> Filename |/etc/radiator/log_to_rabbitmq.pl which is a performance problem.
> It makes the log files easy to rotate, but I can see why it's not good
> in this case.
Because the logging script has some quite a 'high' startup time because
it has to connect to RabbitMQ and also disconnects immediatly afterwards
causing quite some overhead and RabbitMQ logging.
>
>> The comment in Radius::Util::append says:
>> # Current implementation opens, writes and closes
>> # Future implementations might hold the file open, and reopen on
>> # signal, or perhaps pipe to a daemon
>>
>> I need a feature that keeps the spawned (forked?) process open and pipe
>> to it.
>> I'd also need to escape the log message so my JSON doesn't break in case
>> the message contains one or more ".
>>
>> LogFormat { "timestamp":"%Y-%m-%d %H:%M:%S", "priority":"%1",
>> "message":"%2" }
> Hmm, maybe a new class that could be configured to support the above? It
> would need to be able to quote the log messages, maybe a configurable
> method for different kinds of destinations (JSON, etc), hold the fd open
> while supporting FarmSize and possibly something else too. I'd say
> extending Log FILE may not be a good idea but to have a new logging class.
>
> If you already have something that does what you require on the ...> side, please get back to me directly.
Yeah, I was thinking about writing a LogMessagePassing or Log4perl class
but am not sure to which api I have to follow as its internal and as far
as I know undocumented.
For now I've created a named pipe which I configured Radiator to log to
and have a Message::Passing DSL script that runs as daemon using
Daemon::Control which uses the TailFile input and the AMQP output.
I wanted to not have to run that extra daemon because it's one more
thing that can go wrong.
>
> Thanks,
> Heikki
>
>
*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*
T-Systems Austria GesmbH Rennweg 97-99, 1030 Wien
Handelsgericht Wien, FN 79340b
*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*
Notice: This e-mail contains information that is confidential and may be
privileged.
If you are not the intended recipient, please notify the sender and then
delete this e-mail immediately.
*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*
___
radiator mailing list
radiator@open.com.au
http://www.open.com.au/mailman/listinfo/radiator
Re: [RADIATOR] logging (radiator and authlog) and accounting to ElasticSearch
On 03/27/2014 05:22 PM, Hartmaier Alexander wrote:
> Did you have time to work on this feature?
We have worked on EAP-SIM, Diameter and other RADIUS functionality, but
not this. It's still on the ideas to explore list, though.
> I've started tring to get all Radiator logs into Elasticsearch via
> RabbitMQ and found out that also closes the pipe in case of
> Filename |/etc/radiator/log_to_rabbitmq.pl which is a performance problem.
It makes the log files easy to rotate, but I can see why it's not good
in this case.
> The comment in Radius::Util::append says:
> # Current implementation opens, writes and closes
> # Future implementations might hold the file open, and reopen on
> # signal, or perhaps pipe to a daemon
>
> I need a feature that keeps the spawned (forked?) process open and pipe
> to it.
> I'd also need to escape the log message so my JSON doesn't break in case
> the message contains one or more ".
>
> LogFormat { "timestamp":"%Y-%m-%d %H:%M:%S", "priority":"%1",
> "message":"%2" }
Hmm, maybe a new class that could be configured to support the above? It
would need to be able to quote the log messages, maybe a configurable
method for different kinds of destinations (JSON, etc), hold the fd open
while supporting FarmSize and possibly something else too. I'd say
extending Log FILE may not be a good idea but to have a new logging class.
If you already have something that does what you require on the side, please get back to me directly.
Thanks,
Heikki
--
Heikki Vatiainen
Radiator: the most portable, flexible and configurable RADIUS server
anywhere. SQL, proxy, DBM, files, LDAP, NIS+, password, NT, Emerald,
Platypus, Freeside, TACACS+, PAM, external, Active Directory, EAP, TLS,
TTLS, PEAP, TNC, WiMAX, RSA, Vasco, Yubikey, MOTP, HOTP, TOTP,
DIAMETER etc. Full source on Unix, Windows, MacOSX, Solaris, VMS,
NetWare etc.
___
radiator mailing list
radiator@open.com.au
http://www.open.com.au/mailman/listinfo/radiator
Re: [RADIATOR] logging (radiator and authlog) and accounting to ElasticSearch
On 2013-09-20 12:15, Hartmaier Alexander wrote:
> On 2013-09-20 11:44, Heikki Vatiainen wrote:
>> On 09/20/2013 11:35 AM, Alexander Hartmaier wrote:
>>
>>> @Radiator guys: are you interessted in supporting Message::Passing,
>>> Log::Log4perl or Log::Any?
>>> They support a lot of outputs which would be a great feature addition!
>> Sounds interesting. So this would be for Accounting, at least first, or
>> do you see need for other passing other information through these too?
> I'd prefer having this support for all types of logs.
> Maybe extending to handle authentication, accounting and general
> radiator logs would make sense.
>
> AuthLog would become
> (or all other currently available like )
> Auth 1
> Acct 0
> Other 0
> ...
>
>
> If you don't want to change the config DSL so much the Log and AuthLog
> stanzas would both need to support each output.
> Haven't looked at the code yet but I guess there is much too share
> between them and AuthLog could internally become just an alias for
> Auth 1, Acct 0, Other 0.
>
> Accounting logging is currently handled entirely different as there is
> no .
Did you have time to work on this feature?
I've started tring to get all Radiator logs into Elasticsearch via
RabbitMQ and found out that also closes the pipe in case of
Filename |/etc/radiator/log_to_rabbitmq.pl which is a performance problem.
The comment in Radius::Util::append says:
# Current implementation opens, writes and closes
# Future implementations might hold the file open, and reopen on
# signal, or perhaps pipe to a daemon
I need a feature that keeps the spawned (forked?) process open and pipe
to it.
I'd also need to escape the log message so my JSON doesn't break in case
the message contains one or more ".
LogFormat { "timestamp":"%Y-%m-%d %H:%M:%S", "priority":"%1",
"message":"%2" }
Thanks, Alex
>> As always, any additional ideas and comments from the list members would
>> be appreciated too.
>>
> Yes, please, speak up everybody!
>
>
> *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*
> T-Systems Austria GesmbH Rennweg 97-99, 1030 Wien
> Handelsgericht Wien, FN 79340b
> *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*
> Notice: This e-mail contains information that is confidential and may be
> privileged.
> If you are not the intended recipient, please notify the sender and then
> delete this e-mail immediately.
> *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*
> ___
> radiator mailing list
> radiator@open.com.au
> http://www.open.com.au/mailman/listinfo/radiator
___
radiator mailing list
radiator@open.com.au
http://www.open.com.au/mailman/listinfo/radiator
Re: [RADIATOR] logging (radiator and authlog) and accounting to ElasticSearch
Hi Alexander, On 09/20/2013 10:35 AM, Alexander Hartmaier wrote: > I expected that Radiator executes the configured program in a forked > process once and expects it to read from STDIN in an event loop. > Seems the program is executed for every log message. That's right, it's executed for every log message. > What are your experiences with scaling and performance? I never noticed any performance problem. I just counted the radius accounting log messages (generated by this script on one server) we have per day: it's about 700 000. Max per second is around 60. On this server (with two multi-core CPUs) we have 8 radiator instances running (load balancing with iptables). But I agree, it would be better to have the possibility to log accounting messages to syslog one per line from inside radiator. :) Regards Klara -- Karlsruher Institut für Technologie (KIT) Steinbuch Centre for Computing (SCC) Klara Mall Netze und Telekommunikation (NET) Hermann-von-Helmholtz-Platz 1 76344 Eggenstein-Leopoldshafen Telefon: +49 721 608-28630 Telefon: +49 721 608-48946 E-Mail: klara.m...@kit.edu Web: http://www.scc.kit.edu KIT - Universität des Landes Baden-Württemberg und nationales Forschungszentrum in der Helmholtz-Gemeinschaft ___ radiator mailing list radiator@open.com.au http://www.open.com.au/mailman/listinfo/radiator
Re: [RADIATOR] logging (radiator and authlog) and accounting to ElasticSearch
On 2013-09-20 11:44, Heikki Vatiainen wrote: > On 09/20/2013 11:35 AM, Alexander Hartmaier wrote: > >> @Radiator guys: are you interessted in supporting Message::Passing, >> Log::Log4perl or Log::Any? >> They support a lot of outputs which would be a great feature addition! > Sounds interesting. So this would be for Accounting, at least first, or > do you see need for other passing other information through these too? I'd prefer having this support for all types of logs. Maybe extending to handle authentication, accounting and general radiator logs would make sense. AuthLog would become (or all other currently available like ) Auth 1 Acct 0 Other 0 ... If you don't want to change the config DSL so much the Log and AuthLog stanzas would both need to support each output. Haven't looked at the code yet but I guess there is much too share between them and AuthLog could internally become just an alias for Auth 1, Acct 0, Other 0. Accounting logging is currently handled entirely different as there is no . > > As always, any additional ideas and comments from the list members would > be appreciated too. > Yes, please, speak up everybody! *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"* T-Systems Austria GesmbH Rennweg 97-99, 1030 Wien Handelsgericht Wien, FN 79340b *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"* Notice: This e-mail contains information that is confidential and may be privileged. If you are not the intended recipient, please notify the sender and then delete this e-mail immediately. *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"* ___ radiator mailing list radiator@open.com.au http://www.open.com.au/mailman/listinfo/radiator
Re: [RADIATOR] logging (radiator and authlog) and accounting to ElasticSearch
On 09/20/2013 11:35 AM, Alexander Hartmaier wrote: > @Radiator guys: are you interessted in supporting Message::Passing, > Log::Log4perl or Log::Any? > They support a lot of outputs which would be a great feature addition! Sounds interesting. So this would be for Accounting, at least first, or do you see need for other passing other information through these too? As always, any additional ideas and comments from the list members would be appreciated too. -- Heikki Vatiainen Radiator: the most portable, flexible and configurable RADIUS server anywhere. SQL, proxy, DBM, files, LDAP, NIS+, password, NT, Emerald, Platypus, Freeside, TACACS+, PAM, external, Active Directory, EAP, TLS, TTLS, PEAP, TNC, WiMAX, RSA, Vasco, Yubikey, MOTP, HOTP, TOTP, DIAMETER etc. Full source on Unix, Windows, MacOSX, Solaris, VMS, NetWare etc. ___ radiator mailing list radiator@open.com.au http://www.open.com.au/mailman/listinfo/radiator
Re: [RADIATOR] logging (radiator and authlog) and accounting to ElasticSearch
Hi Klara, thanks for the script! I expected that Radiator executes the configured program in a forked process once and expects it to read from STDIN in an event loop. Seems the program is executed for every log message. What are your experiences with scaling and performance? @Radiator guys: are you interessted in supporting Message::Passing, Log::Log4perl or Log::Any? They support a lot of outputs which would be a great feature addition! On 2013-09-19 19:56, Klara Mall wrote: > Hi Alexander, > > On 09/19/2013 04:57 PM, Alexander Hartmaier wrote: >> Since quite some time I'm looking for a way to customize the accounting >> log file format but the problem I'm having with it is that there seems >> to be no way to log all key/value pairs contained in the accounting >> packet without specifying each name. >> The default format is nice to read but hard to search with e.g. ack or grep. >> I've read that using pipe followed by a program as AcctLogFileName works >> but passing data serialized one log per line to it would also be easier >> for the program to parse the log and pass it on (e.g. JSON serialized). > We ran into the same problem and wrote a perl script which we pipe the > Accounting Log to. It's attached. > > radiator config: > AcctLogFileName | /usr/local/bin/radacclog.pl > > Regards > Klara > -- Best regards, Alexander Hartmaier T-Systems Austria GesmbH TSS Security Services Network Security & Monitoring Engineer phone: +43(0)57057-4320 fax: +43(0)57057-954320 *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"* T-Systems Austria GesmbH Rennweg 97-99, 1030 Wien Handelsgericht Wien, FN 79340b *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"* Notice: This e-mail contains information that is confidential and may be privileged. If you are not the intended recipient, please notify the sender and then delete this e-mail immediately. *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"* ___ radiator mailing list radiator@open.com.au http://www.open.com.au/mailman/listinfo/radiator
[RADIATOR] logging (radiator and authlog) and accounting to ElasticSearch
After pushing all our network device syslogs into ElasticSearch I'm looking into doing the same for our applications starting with Radiator. The Radiator application logs should be fairly trivial by using . The same goes for where the format could be e.g. key/value pair JSON serialized. What I'm missing is the same for accounting logs. Since quite some time I'm looking for a way to customize the accounting log file format but the problem I'm having with it is that there seems to be no way to log all key/value pairs contained in the accounting packet without specifying each name. The default format is nice to read but hard to search with e.g. ack or grep. I've read that using pipe followed by a program as AcctLogFileName works but passing data serialized one log per line to it would also be easier for the program to parse the log and pass it on (e.g. JSON serialized). Is there some feature I've overlooked? -- Best regards, Alexander Hartmaier T-Systems Austria GesmbH TSS Security Services Network Security & Monitoring Engineer phone: +43(0)57057-4320 fax: +43(0)57057-954320 *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"* T-Systems Austria GesmbH Rennweg 97-99, 1030 Wien Handelsgericht Wien, FN 79340b *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"* Notice: This e-mail contains information that is confidential and may be privileged. If you are not the intended recipient, please notify the sender and then delete this e-mail immediately. *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"* ___ radiator mailing list radiator@open.com.au http://www.open.com.au/mailman/listinfo/radiator
Re: [RADIATOR] logging (radiator and authlog) and accounting to ElasticSearch
Hi Alexander, On 09/19/2013 04:57 PM, Alexander Hartmaier wrote: > Since quite some time I'm looking for a way to customize the accounting > log file format but the problem I'm having with it is that there seems > to be no way to log all key/value pairs contained in the accounting > packet without specifying each name. > The default format is nice to read but hard to search with e.g. ack or grep. > I've read that using pipe followed by a program as AcctLogFileName works > but passing data serialized one log per line to it would also be easier > for the program to parse the log and pass it on (e.g. JSON serialized). We ran into the same problem and wrote a perl script which we pipe the Accounting Log to. It's attached. radiator config: AcctLogFileName | /usr/local/bin/radacclog.pl Regards Klara -- Karlsruher Institut für Technologie (KIT) Steinbuch Centre for Computing (SCC) Klara Mall Netze und Telekommunikation (NET) Hermann-von-Helmholtz-Platz 1 76344 Eggenstein-Leopoldshafen Telefon: +49 721 608-28630 Telefon: +49 721 608-48946 E-Mail: klara.m...@kit.edu Web: http://www.scc.kit.edu KIT - Universität des Landes Baden-Württemberg und nationales Forschungszentrum in der Helmholtz-Gemeinschaft radacclog.pl Description: Perl program ___ radiator mailing list radiator@open.com.au http://www.open.com.au/mailman/listinfo/radiator
