ruleseets only apply inside the instance or rsyslog that is running them. Once
you send the logs to a new machine, you now have a separate problem. How will
you identify the log you want to treat differently.
If they are arriving via the same port as other logs, this can be difficult. I
deal with this on my systems by having the sending machine send the logs in a
json format that can include additional metadata (like what file the log came
from) while still beingable to easily recreate the original log message.
If I am understanding your problem correctly it seems like what you want is:
you have files on machine A that you want to move to machine B in real-time
This can be done without defining any rulesets, but it will require you create
some way to identify the logs.
since you are reading files from disk that do not have any facility or severity
as part of the message being read, you could use those to encode what is what
(local0 is file1, local1 is file2, using severity is possible, but more
difficult), this gives you up to 64 combinations to work with, but is a pain to
keep straight.
another option would be to use the syslogtag field, and then just know on the
far side that if you receive one of the special syslogtag values, you need to
formt the write to disk without using that field. the syslogtag field cannot
include a / and is limited to 32 characters, so you do have some limitations
or you could make a custom format for your output that puts the file path as the
first thing after the syslogtag and then parse it out on the receiving side.
I go a step further in making a more complex, but more flexible solution where I
create a json message to send that has a field 'msg' that is the original
message, and a tree of objects 'trusted-<company abbriv>' that I have contain
the metadata that I want to add. This includes the filename if it's read from a
file, the name and timestamps related to any relays that it goes through, what
environment this is from (dev/qa/prod/etc) for cases where people like to re-use
names, and anything else that comes up in the future. On the receiving side,
it's a json message that gets parsed, then I look at the data in $!trusted-foo!*
and can make decisions on what to do at that point.
David Lang
On Tue, 2 Mar 2021, odrzen wrote:
Date: Tue, 02 Mar 2021 22:57:37 +0000
From: odrzen <[email protected]>
To: David Lang <[email protected]>
Cc: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] The right way to include more log files?
So, as I understand it - after your very good explanation, it's very important
to define a `ruleset` in case we want rsyslog to handle/manage additional logs.
And it need a new `ruleset`, with a specific `action`, `template`, `queue` and
`Target`.
In your opinion, if a service writes its own logs to separate log files and
rsyslog handles these logs by default, if this service generates a lot of logs,
would you also still create a separate `ruleset` for it?
On the side of the machine in which I have defined in the way you describe
which additional log files the rsyslog handles ( with its own `ruleset` ), I
see that I actually have more information about the messages from these logs
using the `impstats` module (at the moment I don't know how else I can get more
information about them).
For example:
```
Wed Mar 3 00:30:33 2021: global: origin=dynstats msg_per_host.ops_overflow=0
msg_per_host.new_metric_add=0 msg_per_host.no_metric=0
msg_per_host.metrics_purged=0 msg_per_host.ops_ignored=0
msg_per_host.purge_triggered=107
Wed Mar 3 00:30:33 2021: imuxsock: origin=imuxsock submitted=0
ratelimit.discarded=0 ratelimit.numratelimiters=0
Wed Mar 3 00:30:33 2021: action 0: origin=core.action processed=55295 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Mar 3 00:30:33 2021: action 1: origin=core.action processed=4511 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Mar 3 00:30:33 2021: action 2: origin=core.action processed=49706 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Mar 3 00:30:33 2021: action 3: origin=core.action processed=15 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Mar 3 00:30:33 2021: action 4: origin=core.action processed=1063 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Mar 3 00:30:33 2021: action 5: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Mar 3 00:30:33 2021: action 6: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Mar 3 00:30:33 2021: action 7: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Mar 3 00:30:33 2021: msg_per_host: origin=dynstats.bucket
Wed Mar 3 00:30:33 2021: apache: origin=core.action processed=6405 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Mar 3 00:30:33 2021: resource-usage: origin=impstats utime=25597640
stime=23465292 maxrss=17348 minflt=18258 majflt=0 inblock=656 oublock=125552
nvcsw=522977 nivcsw=115
Wed Mar 3 00:30:33 2021: apache queue[DA]: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Mar 3 00:30:33 2021: apache queue: origin=core.queue size=0 enqueued=6405
full=0 discarded.full=0 discarded.nf=0 maxqsize=3
Wed Mar 3 00:30:33 2021: main Q: origin=core.queue size=0 enqueued=61700
full=0 discarded.full=0 discarded.nf=0 maxqsize=10
```
But now, from the side of the central machine to which I send the logs, can or should I
set a separate "ruleset" for these messages ?
So that I can be sure that the messages were successfully processed and stored
on the central machine as well ?
Thank you very much for the explanations and your time.
The way you describe them is very nice and simple. You helped me a lot to better
understand why we need "rulesets".
Sorry if I had to figure this out on some page of the documentation, but I
didn't see it described that way.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, February 25, 2021 12:33 AM, David Lang <[email protected]> wrote:
On Wed, 24 Feb 2021, odrzen wrote:
I am more wondering to understand what is the right way and why to use
rulestes, actions or quests and especially in this case to send particular log
files to a central rsyslog.
There are a few reasons to use a ruleset
1. when you have an input (say a network port) that is very different than
other
inputs and you only want to have a subset of the rules processed for logs
that
arrive on this input
2. a varient of #1, if you want to make sure that logs arriving from one input
cannot be blocked if the queue builds up processing other inputs, you
configure
as #1 and add a queue to the ruleset
3. if you want to put a queue on a group of actions, say sending to one of a
couple different destinations (failover), if you put a queue on each
action, it
will 'succeed' by putting the message in the queue, even if it's not sent.
But
you can put a queue on the ruleset to buffer things at that level, then have
actions that don't have a queue and can fail (which you can detect)
4. avoiding duplicate writers to one destination. If you are writing to the
same
file/sending to the same remote machine and have 10 different actions in
your
rule that all have the same output, they will all be trying to output at the
same time (opening multiple connections to remote systems), if you put the
action in a ruleset and call it from all of those destinations, you ony
have one
connection
5. making the ruleset easier to understand. Just like functions in programming
languages, it may be easier to understand a config file that calls rulesets
that
hide the details rather than having all the statements inline.
David Lang
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
