On Thu, 1 Oct 2015, Dave Caplinger wrote:
If you process inbound UDP and TCP syslog exactly the same way, you *could*
just add the TCP input and point it to the same initial ruleset, such as:
input(type="imtcp"
address="192.168.1.1"
port="514"
ruleset="ruleset_eth0_514")
(note I changed the name of that common ruleset from "ruleset_eth0_514_udp"; do
that also on the imudp input and the ruleset itself of course.)
The result would be two inputs feeding a common queue, which then calls two
output actions (with their own associated output queues).
We typically don't do this; we'd have a separate input queue (ruleset) for UDP
and TCP (and we use imptcp rather than imtcp). This helps us better isolate
the impact of traffic bursts of one input type over the other. For example, a
large, persistent volume of inbound TCP could potentially (eventually) overrun
the shared ruleset/queue and impact reception of UDP (which could in turn lead
to the kernel dropping logs it already successfully received). Particularly
if you are forwarding to a remote destination, your output queue is more
likely to drain more slowly than the input rate, especially if you ever enter
disk-assist buffering. (Consider disk-assist mode a "last resort" for
short-term bursts or intermittent connectivity problems only.)
There is a queue parameter that tells rsyslog that if the queue is larger than
X, stop accepting inputs that can be delayed (like TCP) so that there is space
left for a burst of traffic from inputs that can't be delayed (like UDP)
I would probably just do one ruleset.
David Lang
--
Dave Caplinger, Director, Technical Product Management | 402.361.3063 |
Solutionary — An NTT Group Security Company
On Oct 1, 2015, at 12:13 PM, Randy Baca <[email protected]> wrote:
That looks similar to what we are doing, except we are using legacy commands.
Also, we have two inputs, tcp and udp 514. When we create the new input for
TCP, do we also have to create a new output for it, or will the outputs you
note below cover both inputs?
Regards,
Randy Baca
________________________________________
From: [email protected] [[email protected]] on
behalf of Dave Caplinger [[email protected]]
Sent: Thursday, October 01, 2015 5:36 AM
To: rsyslog-users
Subject: Re: [rsyslog] Complex forwarding and spoofing question
No; you'll need to explicitly turn on compression and disk-assisted queueing.
It's a little more complex this way, but if you're likely to do any kind of
filtering or selective forwarding, I'd recommend a config that:
1) Defines your input (in this case UDP 514) and connects it to a named ruleset
2) That ruleset/queue then forwards to two destinations by calling additional
rulesets
3) Define rulesets for each destination
4) Give everything descriptive names so you can implement impstats,
troubleshoot later, etc.
For example:
# 514/UDP
input(type="imudp"
address="192.168.1.1"
port="514"
ruleset="ruleset_eth0_514_udp")
# In: 514/UDP
# Out: UDP-spoof to the other local syslog and also forward to remote rsyslog
ruleset(name="ruleset_eth0_514_udp"
queue.type="LinkedList") {
call action.local.udp515
call action.fwd.remotebox
stop
}
# Fwd to localhost:udp/515
ruleset(name="action.local.udp515") {
action(type="omudpspoof"
name="omudpspoof.local515"
target="127.0.0.1"
port="515"
)
}
# Fwd to remote (10.11.12.13) via tcp/514 w/ compression and local disk queueing
ruleset(name="action.fwd.remotebox") {
action(type="omfwd"
name="omfwd.remotebox"
queue.type="LinkedList"
queue.filename="omfwd_remotebox"
queue.size="10000"
queue.maxdiskspace="2G"
queue.saveonshutdown="on"
action.resumeretrycount="-1"
target="10.11.12.13"
port="514"
protocol="tcp"
compression.mode="stream:always"
)
}
Note that the receiving remote rsyslog server will also need compression
enabled, and you cannot receive both compressed and non-compressed log feeds on
the same input. So if you need that, use multiple rulesets again: set up the
compressed feed on one input/port/ruleset, and non-compressed on another.
--
Dave Caplinger, Director, Technical Product Management | 402.361.3063 |
Solutionary — An NTT Group Security Company
On Sep 30, 2015, at 9:58 PM, Randy Baca <[email protected]> wrote:
We need the spoofing because rsyslog adds a timestamp and an IP (127.0.0.1) to
the beginning of each message forwarded and it messes up the legacy parsing.
Does @@remotesystem perform compression and will it spool messages by default
in case the link goes down?
RB
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of David Lang
Sent: Wednesday, September 30, 2015 6:14 PM
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Complex forwarding and spoofing question
On Thu, 1 Oct 2015, Randy Baca wrote:
I have a complex issue to resolve and I would like to use rsyslog for it.
Hopefully someone has done this.
Currently we have a number of remote sites and each site has a single syslog
server collecting events from multiple hosts. We need to keep the flow going
to the legacy syslog servers and also forward all events to a central server at
one of the sites. The legacy syslog servers need to stay running until several
other projects get completed; maybe a year.
This is what we want to do. We have pieces of it working but can't get
everything at once.
* Set up rsyslog on the same server that is running the legacy service.
* Change the legacy service to run on udp/515.
* Configure rsyslog to receive on existing ports udp/tcp 514. We are
essentially injecting it between the sending hosts and the legacy service.
* Configure rsyslog to forward and spoof (omudpspoof) to 127.0.0.1 on
udp/515. We need the exact original message because of all the custom parsers
written over the years.
* Add another action to forward all messages to the central location. Some
sites are not high bandwidth, so we will need compression, TCP for reliability,
and spooling in case the link goes down or one of the endpoints reboots.
* The server at the central location will itself then forward to multiple
services locally.
Is this do-able? Can someone get me started with an example config?
it's doable, but you shouldn't need to do the spoofing. Spoofing is only needed
if the receiving system ignores the text of the log message and instead uses
the IP address in the packet.
It's as simple as two outputs
@localhost:515
@remotesystem
or @@remotesystem if you want to use TCP forwarding
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
