Last hours have been quite busy!
After reading your comments, here's where we are. Please, advice and
comment.
On some remote hosts, logs are readed from files. Each remote will have
a configuration like:
# remote.conf
global(
MaxMessageSize="32k"
)
template(name="json" type="string"
string="%timestamp:::date-rfc3339% %hostname%
logs/$!data!group/$!data!app $!data")
module(load="omrelp")
ruleset(name="relp"){
action(
port="20514"
target="server"
template="json"
type="omrelp"
)
}
ruleset(name="myapp1") {
set data!app="myapp1"
$!data!file="$!metadata!filename"
$!data!group="mygroup"
$!data!msg="%msg"
call relp
}
input(type="imfile" file="/logs/apps/app1/app1.log"
tag="mygroup/myapp1" addMetadata="on" ruleset="myapp1"
startmsg.regex="^[[:digit:]]{2} [[:alpha:]]{3} [[:digit:]]{4}"
readTimeout="5" PersistStateInterval="1")
ruleset(name="myapp2") {
set data!app="myapp2"
$!data!file="$!metadata!filename"
$!data!group="mygroup"
$!data!msg="%msg"
call relp
}
input(type="imfile" file="/logs/apps/app2/app2.log"
tag="mygroup/myapp2" addMetadata="on" ruleset="myapp2"
readTimeout="5" PersistStateInterval="1")
As depicted above, logs will be sent to server using RELP. We plan to
make a config-generator-script, to add imfile+ruleset for each file.
If you know a better way for having multiple files, each having their
own information, just let me know.
We also played a bit with mmpstrucdata, but seems json is easier (once
you need it for elastic)
On the server side, we would like to accept and index any groups
applications, but each app config should be on their own file, making it
easier to change, understand...
Our current approach is based on copying multiple files to rsyslog.d
directory, being all configurations loaded. This is what we got so far:
# core.conf
global(
MaxMessageSize="32k"
parser.escapeControlCharactersOnReceive="off"
)
module(load="imrelp")
input(
port="20514"
type="imrelp"
name="imrelp"
)
template(name="json" type="list") {
constant(value="{")
property(name="$!msg")
constant(value="}")
}
module(load="omelasticsearch")
ruleset(name="index"){
action(
type="omelasticsearch"
template="json"
searchIndex="$!index"
)
}
module(load="mmnormalize")
# app1.conf
# Perhaps something like $!group == "mygroup" could work
if $syslogtag startswith "logs/group/" then {
rule=:%[
{"type":"ipv4", "name":"ip"},
{"type":"literal", "text:" "},
{"type":"literal", "text:"-"},
{"type":"rest", "name":"r"}
]%
set index="$!app_$$year-$$month-$$day"
action(type="mmnormalize" rule="$rule" version="2")
call index
stop
}
# app2.conf
if $!app == "myapp2" then {
rule=:%[
{"type":"ipv4", "name":"ip"},
{"type":"literal", "text:" "},
{"type":"literal", "text:"-"},
{"type":"rest", "name":"r"}
]%
action(type="mmnormalize" rule="$rule" version="2")
call index
stop
}
*Does it makes sense for you? Any improvements? Anything that can't be
done?*
@bobthemighty: feedback!
Thanks in advance
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
