On Wed, 11 Mar 2015, [email protected] wrote:
Hi David,
Thanks for your answer.
Waiting to hear you again :)
Actually my main issue is to avoid to spool on the source server and send all
my logs to the spooling server.
The key question you need to think about is:
When things go badly wrong with logging long enough, which would you rather have
happen, loose logs or have your servers and applications stop?
If you are not willing to loose logs, and don't want things queueing on the
servers generating the logs, then you need to make the systems you are sending
to redundant with auto-failover, and even then you are going to get some short
delays.
You really do want to have some spooling on the client sending to your local
server, but you don't need a lot.
I like to put a syslog relay/spooling server on each subnet so that there are no
firewalls or ACLs between the systems generating the logs and the relay/spooling
boxes. In this situation, simple UDP communications is very reliable (no
bottlenecks where UDP is going to be at risk), and then I use TCP or RELP to
relay from there to my central systems.
what I do on my relay boxes is currently:
# gather stats every 10 min. Process them independently of normal logs so that
# if the normal log flow gets backed up, these stats will not be affected
module(load="impstats" interval="600" resetCounters="on" format="legacy"
ruleset="high_p")
module(load="imuxsock" SysSock.Annotate="on" SysSock.ParseTrusted="on")
module(load="imklog")
module(load="imudp")
input(type="imudp" port="514")
module(load="imtcp" MaxSessions="1000")
input(type="imtcp" port="514")
module(load="mmjsonparse")
action(type="mmjsonparse")
if $fromhost-ip != "127.0.0.1" then {
# if the log is being received from another machine,
# add metadata to the log
set $!trusted!origserver = $fromhost-ip;
set $!trusted!edge!time = $timegenerated;
set $!trusted!edge!relay = $$myhostname;
set $!trusted!edge!input = $inputname;
} else {
set $!trusted!local!input = $inputname;
}
set $!trusted!environment = "Dev network";
$template structured_forwarding,"<%pri%>%timereported% %hostname% %syslogtag%
@cee:%$!%\n"
action(type="omfile" File="/var/log/messages" name="local_messages")
action(type="omfile" File="/var/log/messages-full" template="structured_forwarding"
name="cee_messages")
action(type="omfwd" Target="10.1.1.1" Port="514" Protocol="tcp" queue.type="FixedArray"
template="structured_forwarding" name="send_remote")
# for high priority messages (the stats) write them locally and send them to the
# central server. Define queues for the ruleset and for the remote send to
# decouple them from being affected or affecting other logs
ruleset(name="high_p" queue.type="FixedArray"){
set $!trusted!local!input = $inputname;
action(type="mmjsonparse")
action(type="omfile" file="/var/log/pstats" name="pstats_local")
action(name="send_HP" type="omfwd" target="10.1.50.85" port="514"
protocol="tcp" queue.type="FixedArray" template="structured_forwarding")
}
On my central server I do:
module(load="impstats" interval="600" resetCounters="on" format="legacy"
ruleset="high_p")
module(load="imuxsock" SysSock.Annotate="on" SysSock.ParseTrusted="on")
module(load="imklog")
module(load="imtcp" MaxSessions="1000")
module(load="imudp" timerequery="4" )
module(load="mmnormalize")
input(type="imtcp" port="514")
input(type="imudp" port="514")
module(load="mmjsonparse")
# define templates
# send JSON message
$template structured_forwarding,"<%pri%>%timereported% %hostname% %syslogtag%
@cee:%$!%\n"
# local traditional format
$template stdmsg,"%timereported% %hostname% %syslogtag% %$!msg%"
$template std,"%$.stdmsg%\n"
$template unknown,"%$!extracted!originalmsg%\n"
# forward traditional format
$template std-fwd,"<%pri%>%timereported% %hostname% %syslogtag% %$!msg%\n"
# define a high priority queue that will send impstats immediately rather
# than going into the main queue and possibly being delayed by other logs
ruleset(name="high_p" queue.type="FixedArray"){
set $!trusted!local!input = $inputname;
action(type="mmjsonparse")
action(type="omfile" file="/var/log/pstats-local-messages"
name="pstats_local")
action(name="send_HP" type="omfwd" target="10.1.0.1" port="514" protocol="udp"
template="structured_forwarding" queue.type="FixedArray")
}
# parse JSON messages to variables
action(type="mmjsonparse")
# if the message we got was in JSON from the beginning, there won't be a $!msg
variable
if $!msg == "" then set $!msg = $msg;
# parse the origional message to extract fields
set $.stdmsg = exec_template("stdmsg");
action(type="mmnormalize" path="$!extracted" variable="$.stdmsg"
ruleBase="/root/rsyslog.rulebase")
# if the message was extracted add timestamp and hostname as part of the
# extracted data
if $!extracted!originalmsg == '' then {
set $!extracted!timestamp = $timestamp;
set $!extracted!hostname = $hostname;
}
# if we failed to parse a cisco message, log what we failed to parse
if $!extracted!originalmsg != '' and $programname startswith '%ASA-' then {
/var/log/cisco-unknown-messages;unknown
}
# if this is a local log, send it to an edge relay. We do this so that it will
# end up being delivered to all destinations.
if $fromhost-ip == "127.0.0.1" then {
if $programname startswith 'rsyslogd' then {
/var/log/rsyslog-local-messages
}
set $!trusted!local!input = $inputname;
@10.1.0.1
stop
}
# write logs in the traditional format without metadata
/var/log/messages;std
# write messages with full metadata and high precision timestamp
/var/log/messages-full
# forward messages to something that understands the JSON format and can use
the metadata
action(type="omfwd" name="smart-out" target="10.1.1.2" port="514" protocol="udp"
template="structured_forwarding")
# forward messages to something that only understands the traditional format
action(type="omfwd" name="legacy-out" target="10.1.1.3" port="514" protocol="udp"
template="std-fwd")
David Lang
Regards,
Smana
----- Mail original -----
De: "David Lang" <[email protected]>
À: "rsyslog-users" <[email protected]>
Envoyé: Mercredi 11 Mars 2015 15:45:22
Objet: Re: [rsyslog] Spooling server per datacenter
here are some things to get you started. When I get to work today I can give you
examples of my live configs.
https://www.usenix.org/publications/login/august-2013-volume-38-number-4/enterprise-logging
https://www.usenix.org/publications/login/october-2013-volume-38-number-5/log-filtering-rsyslog
to handle the problem of network interruptions backing things up, you will need
to create some additional queues (lookup action queues and rulesets). I'll post
more later.
You are on the right track.
David Lang
On Wed, 11 Mar 2015, [email protected] wrote:
Date: Wed, 11 Mar 2015 15:37:19 +0100 (CET)
From: [email protected]
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Spooling server per datacenter
Please let me know i you need more info.
OS : debian wheezy
rsyslog version : 8.8.0.ad1-1
Regards,
Smana
----- Mail original -----
De: [email protected]
À: "rsyslog-users" <[email protected]>
Envoyé: Mercredi 11 Mars 2015 09:44:45
Objet: [rsyslog] Spooling server per datacenter
Hi guys,
Could you please help me to find out the proper configuration for the following
use case ?
* We have multiple datacenters
* All our logs are sent to a central analytic platform
* In each dc we'd like to have a spooling server which will keep to logs in
case of network failure.
* All the logs from the sources servers have to be sent to the spooling server
(no spooling on source servers)
* Relp if it's possible
To summarize :
source servers -> spooling server -> analytics plateform
I tried to use relp but when the destination (analytics pf) is unreachable all
the log flow slows down, even on source servers.
With tcp the source server keeps to send but i don't see my spooling space
growing. I presume i'm loosing data (i'll do further tests)
When i use the option "action.resumeRetryCount="-1" when the destination is
uncheachable the log flow stops completely...
Here is my current configuration
Source server:
module(load="impstats"
format="json"
interval="60"
log.syslog="off"
log.file="/var/log/rsyslog-stats.log"
severity="7")
module(load="imtcp")
input(type="imtcp" port="514")
if $programname startswith 'foo.' then @@bar.domain.tld:514
Spooling server:
module(load="imtcp")
input(type="imtcp" port="514")
module(load="impstats"
format="json"
interval="60"
log.syslog="off"
log.file="/var/log/rsyslog-stats.log"
severity="7")
if $programname startswith 'foo.' then {
action(type="omfwd"
action.resumeRetryCount="-1"
name="spooling"
target="analytics"
port="514"
protocol="tcp"
queue.filename="eggforward"
queue.spoolDirectory="/var/spool/rsyslog"
queue.type="LinkedList")
}
Thanks for your help
Smana
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
