templates define how rsyslog is going to output a line, but you are having
problems because the data you are receiving is not formatted correctly and so
the 'best effort' parsing is not giving you what you want.
log with the format RSYSLOG_DebugFormat and post an example of the message that
is malformed. Look at the rawmsg section of that output. I'll bet that it will
be fairly obvious what's wrong with the data you are being sent.
David Lang
On Fri, 18 Aug
2017, Don M Subscriptions via rsyslog wrote:
Date: Fri, 18 Aug 2017 13:47:30 -0400
From: Don M Subscriptions via rsyslog <rsyslog@lists.adiscon.com>
To: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: Don M Subscriptions <donmrdch.subscripti...@gmail.com>
Subject: [rsyslog] Malfored syslog records missing host and extra date
Greetings – Posting again, don’t recall seeing if anyone could offer a
solution.
This time – index to the message.
1. Background – marked with *** BACKGROUND ***
2. Input sample – shows what we are receiving w/ a comparison line marked
with *** SAMPLE ***
3. rsyslog file (my receiver’s rsyslog file) – *** RSYSLOG ***
4. applicable template file (from third party vendor) *** TEMPLATE ***
*** BACKGROUND ***
We have a cloud based desktop analysis package that sends data when it
recognizes something
suspicious (suspicious connection, hit from a virus total feed, etc.). To
that end, the cloud system sends in data over TLS (which is working). I am
seeing a very odd pattern in the 'messages' file. I suspect that this
pattern or issue has to do w/ the template or definition assigned to data
received because there are multiple dates and no ‘sender’ or ‘hostname’,
like in the *** SAMPLE *** section.
IPtables firewall line, which looks fine. Meaning all of the normal fields
are in place (key identifiers changed though with reasonable alternates.
Below that iptables line is a line received from the external host. I see
multiple dates, and I do not see the hostname in the second line, like I do
for My_Relay_Hostname (the obfuscated name of the rsyslog server itself).
What can cause multiple dates? Is there some way that I can enforce storing
the sending IP and/or the sending system host name, which is missing
(second line, again)?
*** SAMPLE ***
IPtables local firewall output – date, localhostname, and then the message.
IPtables firewall line, which looks fine. Meaning all of the normal fields
are in place (key identifiers changed though with reasonable alternates.
Aug 7 13:08:54 My_Relay_Hostname kernel: iptables: IN=eth0 OUT=
MAC=00:0c:29:3c:b6:61:58:49:3b:14:5d:14:08:00 SRC=AA.BB.CC.30
DST=DD.EE.FF.176 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=18235 DF PROTO=TCP
SPT=58752 DPT=10514 WINDOW=17922 RES=0x00 SYN URGP=0
Cloud monitoring solution – date, date repeated, no local hostname (missing)
Aug 7 13:08:55 2017-08-07 16: 50:16 [4166] <warning> reason=watchlist.hit
type=event process_guid=00000007-0000-0f8c-01d3-0f95444b7a13 segment_id=1
host='victim_hostname_here' comms_ip='GGG.HHH.III.202'
interface_ip='DD.JJ.KK.112' sensor_id=7 watchlist_id=280
watchlist_name='Notepad' timestamp='1502124615.77'
start_time='2017-08-07T15:53:09.421Z' group='testing_data_group_name_here'
process_md5='sum_value_here' process_name='not.exe'
process_path='c:\windows\system32\not.exe'
last_update='2017-08-07T16:46:35.818Z'
*** TEMPLATE *** From what I can tell from the rsyslog file, this is the
definition for received data external to the rsyslog receiving host
# this module manages the delivery of syslog calls from a logging process
to rsyslog
#$ModLoad imuxsock.so
# imklog: Reads messages from the kernel log and submits them to the syslog
engine.
#$ModLoad imklog.so
#$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
#*.info;mail.none;authpriv.none;cron.none /var/log/messages
# Remote logging
#$ModLoad imudp
#$UDPServerAddress 0.0.0.0
#$UDPServerRun 514
#Dyanmic folders
$template DYNmessages,"/var/log/VENDOR/devices/%fromhost-ip%/messages"
$template DYNsecure,"/var/log/VENDOR/devices/%fromhost-ip%/secure"
$template DYNmaillog,"/var/log/VENDOR/devices/%fromhost-ip%/maillog"
$template DYNcron,"/var/log/VENDOR/devices/%fromhost-ip%/cron"
$template DYNspooler,"/var/log/VENDOR/devices/%fromhost-ip%/spooler"
$template DYNboot,"/var/log/VENDOR/devices/%fromhost-ip%/boot.log"
$template DYNlog,"/var/log/VENDOR/devices/%fromhost-ip%/%fromhost-ip%.log"
if \
$fromhost-ip != '127.0.0.1' \
and \
$fromhost-ip != '172.17.0.29' \
then ?DYNlog
*** RSYSLOG *** Then there is the default rsyslog.conf file, a wee bit
sanitized.
$MaxMessageSize 64k
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog # provides kernel logging support
#$ModLoad immark # provides --MARK-- message capability
# provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514
# provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514
###########################
#### GLOBAL DIRECTIVES ####
###########################
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
#
# Set the default permissions for all log files.
#
$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog
#
# Include all config files in /etc/rsyslog.d/ <<< invokes the *** TEMPLATE
***
#
$IncludeConfig /etc/rsyslog.d/*.conf
###############
#### RULES ####
###############
#
# First some standard log files. Log by facility.
#
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log
daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
user.* -/var/log/user.log
#
# Logging for the mail system. Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info -/var/log/mail.info
mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err
#
# Logging for INN news system.
#
news.crit /var/log/news/news.crit
news.err /var/log/news/news.err
news.notice -/var/log/news/news.notice
#
# Some "catch-all" log files.
#
*.=debug;\
auth,authpriv.none;\
news.none;mail.none -/var/log/debug
*.=info;*.=notice;*.=warn;\
auth,authpriv.none;\
cron,daemon.none;\
mail,news.none -/var/log/messages
#
# Emergencies are sent to everybody logged in.
#
*.emerg :omusrmsg:*
daemon.*;mail.*;\
news.err;\
*.=debug;*.=info;\
*.=notice;*.=warn |/dev/xconsole
# rsyslog zasec.conf
# logs not from 127.0.0.1
if not ($fromhost-ip == '127.0.0.1') then -/var/log/vendor/asec_unk.log
if not ($fromhost-ip == '127.0.0.1') then ~
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
