My debug shows that the log is not being parsed
but I am calling the .rb file in my conf
or do I need to bring in on on TCP and apply a ruleset on that port?
Debug line with all properties:
FROMHOST: 'dencfw01.contournetworks.net', fromhost-ip: '10.17.0.2', HOSTNAME:
'dencfw01.contournetworks.net', PRI: 133,
syslogtag 'dencfw01:', programname: 'dencfw01', APP-NAME: 'dencfw01', PROCID:
'-', MSGID: '-',
TIMESTAMP: 'Sep 7 13:35:14', STRUCTURED-DATA: '-',
msg: ' NetScreen device_id=dencfw01 [Root]system-notification-00257(traffic):
start_time="2018-09-07 11:35:09" duration=4 policy_id=741 service=dns proto=17
src zone=PRIVATE_INET dst zone=Untrust action=Permit sent=93 rcvd=148
src=10.82.16.23 dst=207.155.165.4 src_port=3974 dst_port=53 src-xlated
ip=207.155.165.87 port=31915 dst-xlated ip=10.17.65.11 port=53 session_id=93684
reason=Close - RESP'
escaped msg: ' NetScreen device_id=dencfw01
[Root]system-notification-00257(traffic): start_time="2018-09-07 11:35:09"
duration=4 policy_id=741 service=dns proto=17 src zone=PRIVATE_INET dst
zone=Untrust action=Permit sent=93 rcvd=148 src=10.82.16.23 dst=207.155.165.4
src_port=3974 dst_port=53 src-xlated ip=207.155.165.87 port=31915 dst-xlated
ip=10.17.65.11 port=53 session_id=93684 reason=Close - RESP'
inputname: imudp rawmsg: '<133>dencfw01: NetScreen device_id=dencfw01
[Root]system-notification-00257(traffic): start_time="2018-09-07 11:35:09"
duration=4 policy_id=741 service=dns proto=17 src zone=PRIVATE_INET dst
zone=Untrust action=Permit sent=93 rcvd=148 src=10.82.16.23 dst=207.155.165.4
src_port=3974 dst_port=53 src-xlated ip=207.155.165.87 port=31915 dst-xlated
ip=10.17.65.11 port=53 session_id=93684 reason=Close - RESP'
$!:{ "originalmsg": "<133>dencfw01: NetScreen device_id=dencfw01
[Root]system-notification-00257(traffic): start_time=\"2018-09-07 11:35:09\"
duration=4 policy_id=741 service=dns proto=17 src zone=PRIVATE_INET dst
zone=Untrust action=Permit sent=93 rcvd=148 src=10.82.16.23 dst=207.155.165.4
src_port=3974 dst_port=53 src-xlated ip=207.155.165.87 port=31915 dst-xlated
ip=10.17.65.11 port=53 session_id=93684 reason=Close - RESP", "unparsed-data":
"<133>dencfw01: NetScreen device_id=dencfw01
[Root]system-notification-00257(traffic): start_time=\"2018-09-07 11:35:09\"
duration=4 policy_id=741 service=dns proto=17 src zone=PRIVATE_INET dst
zone=Untrust action=Permit sent=93 rcvd=148 src=10.82.16.23 dst=207.155.165.4
src_port=3974 dst_port=53 src-xlated ip=207.155.165.87 port=31915 dst-xlated
ip=10.17.65.11 port=53 session_id=93684 reason=Close - RESP" }
$.:
$/:
________________________________
From: rsyslog <rsyslog-boun...@lists.adiscon.com> on behalf of David Lang
<da...@lang.hm>
Sent: Friday, September 7, 2018 11:30 AM
To: rsyslog-users
Subject: Re: [rsyslog] Parsed data not loading to database all zeros
just before and just after the mmnormalize call, write the log to a file with
the RSYSLOG_DebugFormat template, that will show what you are getting, and what
you have after the call.
I would also suggest that you switch all multi-line items to the new action()
syntax, it makes what's going on much clearer.
David Lang
On Fri, 7 Sep 2018, Jason Prouty wrote:
> Date: Fri, 7 Sep 2018 17:07:38 +0000
> From: Jason Prouty <jpro...@cctus.com>
> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] Parsed data not loading to database all zeros
>
> Thank you for your help it look like
>
> my logs are not being parsed,
>
>
> I know my ruleset.rb is good from lognormalizer
>
>
> I have updated my conf file
>
>
> when I run it I get
>
>
> rsyslog internal message (3,-2218): The error statement was:
> starttime",host",device",fw_type",device_id",filler1",fwstart_time",duration",policy_id",service",proto",src_zoeason":"",}e",action",sent",rcvd",src_ip",dst_ip",src_port",dst_port",srx_xtranslateip",srxxlatedport",dst_xtranslateip",dstxlatedport",session_id",
> [v8.37.0 try http://www.rsyslog.com/e/2218 ]
> 9974.288918567:main Q:Reg/w0 : ../action.c:
> actionCallCommitTransaction[action-3-ommysql] state: datafail mod
> commitTransaction returned -2218
> 9974.288928415:main Q:Reg/w0 : errmsg.c: Called LogMsg, msg: action
> 'action-3-ommysql' (module 'ommysql') message lost, could not be processed.
> Check for additional error messages before this one.
>
>
>
>
>
> # rsyslog configuration file
>
> # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
> # If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
>
> #### MODULES ####
>
> # The imjournal module below is now used as a message source instead of
> imuxsock.
> $ModLoad imuxsock # provides support for local system logging (e.g. via
> logger command)
> $ModLoad imjournal # provides access to the systemd journal
> #$ModLoad imklog # reads kernel messages (the same are read from journald)
> $ModLoad immark # provides --MARK-- message capability
>
> # Load the MySQL Module
> module(load="ommysql")
> # Load the JSON Parser Module
> module(load="mmjsonparse")
> # Load the Normalizer Module
> module(load="mmnormalize")
> $ModLoad omruleset
> module(load="imfile" PollingInterval="10") #needs to be done just once
>
> # Provides UDP syslog reception
> $ModLoad imudp
> $UDPServerRun 514
>
> # Provides TCP syslog reception
> $ModLoad imtcp
> $InputTCPServerRun 514
>
> #### GLOBAL DIRECTIVES ####
>
> ### Normalize directives ####
> #input(type="imfile"
> # File="/var/log/policyid1623.log"
> #)
> $mmnormalizeUseRawMSG on
> $mmnormalizeRuleBase /rsyslog/rulebase.rb
> *.* :mmnormalize:
>
> # Where to place auxiliary files
> $WorkDirectory /var/lib/rsyslog
>
> # Use default timestamp format
> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>
> # File syncing capability is disabled by default. This feature is usually not
> required,
> # not useful and an extreme performance hit
> #$ActionFileEnableSync on
>
> # Include all config files in /etc/rsyslog.d/
> $IncludeConfig /etc/rsyslog.d/*.conf
>
> # Turn off message reception via local log socket;
> # local messages are retrieved through imjournal now.
> $OmitLocalLogging on
>
> # File to store the position in the journal
> $IMJournalStateFile imjournal.state
>
> # Database directive
> #*.* :ommysql:127.0.0.1,Syslog,rsyslog,crazylog2018;database
>
> #### RULES ####
>
> # Message containg policy id 1623
> :msg, contains, "policy_id=1623" /var/log/policyid1623.log
> :ommysql:127.0.0.1,Syslog,rsyslog,crazylog2018;database
>
>
> template (name="database" type="list" option.sql="on") {
> constant(value="starttime") property(name="$!start_time"
> dateFormat="rfc3164") constant(value="\",")
> constant(value="host") property(name="$!host" format="json")
> constant(value="\",")
> constant(value="device") property(name="$!device" format="json")
> constant(value="\",")
> constant(value="fw_type") property(name="$!fw_type" format="json")
> constant(value="\",")
> constant(value="device_id") property(name="$!device_id" format="json")
> constant(value="\",")
> constant(value="filler1") property(name="$!filler1" format="json")
> constant(value="\",")
> constant(value="fwstart_time") property(name="$!fwstart_time"
> format="json") constant(value="\",")
> constant(value="duration") property(name="$!duration" format="json")
> constant(value="\",")
> constant(value="policy_id") property(name="$!policy_id" format="json")
> constant(value="\",")
> constant(value="service") property(name="$!service" format="json")
> constant(value="\",")
> constant(value="proto") property(name="$!proto" format="json")
> constant(value="\",")
> constant(value="src_zone") property(name="$!src_zone" format="json")
> constant(value="\",")
> constant(value="dst_zone") property(name="$!dst_zone" format="json")
> constant(value="\",")
> constant(value="action") property(name="$!action" format="json")
> constant(value="\",")
> constant(value="sent") property(name="$!sent" format="json")
> constant(value="\",")
> constant(value="rcvd") property(name="$!rcvd" format="json")
> constant(value="\",")
> constant(value="src_ip") property(name="$!src_ip" format="json")
> constant(value="\",")
> constant(value="dst_ip") property(name="$!dst_ip" format="json")
> constant(value="\",")
> constant(value="src_port") property(name="$!src_port" format="json")
> constant(value="\",")
> constant(value="dst_port") property(name="$!dst_port" format="json")
> constant(value="\",")
> constant(value="srx_xtranslateip") property(name="$!srx_xtranslateip%"
> format="json") constant(value="\",")
> constant(value="srxxlatedport") property(name="$!srxxlatedport%"
> format="json") constant(value="\",")
> constant(value="dst_xtranslateip") property(name="$!dst_xtranslateip"
> format="json") constant(value="\",")
> constant(value="dstxlatedport") property(name="$!dstxlatedport%"
> format="json") constant(value="\",")
> constant(value="session_id") property(name="$!session_id%"
> format="json") constant(value="\",")
> constant(value="\reason\":\"") property(name="$!reason" format="json")
> constant(value="\",")
> constant(value="}\n")
> }
>
> *.* /var/log/test.log
>
> *.* :ommysql:127.0.0.1,Syslog,rsyslog,crazylog2018;database
>
>
> # Log all kernel messages to the console.
> # Logging much else clutters up the screen.
> #kern.* /dev/console
>
>
> *.info;mail.none;authpriv.none;cron.none /var/log/messages
>
>
> authpriv.* /var/log/secure
> mail.* -/var/log/maillog
>
> cron.* /var/log/cron
>
>
> *.emerg :omusrmsg:*
> uucp,news.crit /var/log/spooler
>
> # Save boot messages also to boot.log
> local7.* /var/log/boot.log
>
>
>
>
> ________________________________
> From: rsyslog <rsyslog-boun...@lists.adiscon.com> on behalf of Jason Prouty
> <jpro...@cctus.com>
> Sent: Thursday, September 6, 2018 2:26 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Parsed data not loading to database all zeros
>
> My Log files are empty
>
>
> ________________________________
> From: rsyslog <rsyslog-boun...@lists.adiscon.com> on behalf of David Lang
> <da...@lang.hm>
> Sent: Wednesday, September 5, 2018 5:34 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Parsed data not loading to database all zeros
>
> write the data to a file with the template that you are using to send it to
> the
> database and see if you can see a problem there.
>
> if that's not showing valid data, write with the template RSYSLOG_DebugFormat
> and see what the variable contents are.
>
> David Lang
>
> On Thu, 6 Sep 2018, Jason Prouty wrote:
>
>> Date: Thu, 6 Sep 2018 00:13:33 +0000
>> From: Jason Prouty <jpro...@cctus.com>
>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
>> Subject: [rsyslog] Parsed data not loading to database all zeros
>>
>> I have created a parser to parse my netscreen firewall logs and I am tring
>> to load it to a mysql database table
>>
>> Below is my conf file
>>
>> sample record I do see it being parsed and the rules file
>>
>>
>> I am on centos 7 rsyslog is rpm install of 8.24
>>
>>
>>
>> # The imjournal module below is now used as a message source instead of
>> imuxsock.
>> $ModLoad imuxsock # provides support for local system logging (e.g. via
>> logger command)
>> $ModLoad imjournal # provides access to the systemd journal
>> #$ModLoad imklog # reads kernel messages (the same are read from journald)
>> $ModLoad immark # provides --MARK-- message capability
>>
>> # Load the MySQL Module
>> module(load="ommysql")
>> # Load the JSON Parser Module
>> module(load="mmjsonparse")
>> # Load the Normalizer Module
>> module(load="mmnormalize")
>>
>> # Provides UDP syslog reception
>> $ModLoad imudp
>> $UDPServerRun 514
>>
>> # Provides TCP syslog reception
>> $ModLoad imtcp
>> $InputTCPServerRun 514
>>
>> #### GLOBAL DIRECTIVES ####
>> #### Normalize directives ####
>> $mmnormalizeUseRawMSG on
>> $mmnormalizeRuleBase /rsyslog/rulebase.rb
>> *.* :mmnormalize:
>>
>> # Where to place auxiliary files
>> $WorkDirectory /var/lib/rsyslog
>>
>> # Use default timestamp format
>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>
>> # File syncing capability is disabled by default. This feature is usually
>> not required,
>> # not useful and an extreme performance hit
>> #$ActionFileEnableSync on
>>
>> # Include all config files in /etc/rsyslog.d/
>> $IncludeConfig /etc/rsyslog.d/*.conf
>> # File to store the position in the journal
>> $IMJournalStateFile imjournal.state
>>
>> # Database directive
>> *.* :ommysql:127.0.0.1,Syslog,rsyslog,crazylog2018;database
>>
>> #### RULES ####
>>
>> # Message containg policy id 1623
>> :msg, contains, "policy_id=1623" /var/log/policyid1623.log
>>
>> $template database,"insert into nagalert (source_ip, start_date, sent, rcvd)
>> values ('$!src_ip', '$!fwstart_time', '$!sent', '$!rcvd')",SQL
>>
>>
>>
>> Sample record
>> # Aug 30 15:58:28 dencfw01.contournetworks.net dencfw01: NetScreen
>> device_id=dencfw01 [Root]system-notification-00257(traffic):
>> start_time="2018-08-30 13:58:26" duration=2 policy_id=1623 service=https
>> proto=6 src zone=private_atm dst zone=Untrust action=Permit sent=136 rcvd=68
>> src=10.82.8.20 dst=172.217.4.46 src_port=58024 dst_port=443 src-xlated
>> ip=74.115.157.233 port=49293 dst-xlated ip=172.217.4.46 port=443
>> session_id=86783 reason=Close - TCP RST
>>
>> rule=:%start_time:date-rfc3164% %host:word% %device:word% %fw_type:word%
>> device_id=%device_id:word% %filler1:char-to:\x3A%:
>> start_time="%fwstart_time:char-to:\x22%" duration=%duration:number%
>> policy_id=%policy_id:number% service=%service:word% proto=%proto:word% src
>> zone=%src_zone:word% dst zone=%dst_zone:word% action=%action:word%
>> sent=%sent:number% rcvd=%rcvd:number% src=%src_ip:ipv4% dst=%dst_ip:ipv4%
>> src_port=%src_port:number% dst_port=%dst_port:number% src-xlated
>> ip=%srx_xtranslateip:ipv4% port=%srxxlatedport:number% dst-xlated
>> ip=%dst_xtranslateip:ipv4% port=%dstxlatedport:number%
>> session_id=%session_id:number% reason=%reason:rest%
>>
>>
>> lognormalizer -r /rsyslog/rulebase.rb < /tmp/1a.log
>>
>> { "reason": "Close - TCP RST", "session_id": "86783", "dstxlatedport":
>> "443", "dst_xtranslateip": "172.217.4.46", "srxxlatedport": "49293",
>> "srx_xtranslateip": "74.115.157.233", "dst_port": "443", "src_port":
>> "58024", "dst_ip": "172.217.4.46", "src_ip": "10.82.8.20", "rcvd": "68",
>> "sent": "136", "action": "Permit", "dst_zone": "Untrust", "src_zone":
>> "private_atm", "proto": "6", "service": "https", "policy_id": "1623",
>> "duration": "2", "fwstart_time": "2018-08-30 13:58:26", "filler1": "
>> [Root]system-notification-00257(traffic)", "device_id": "dencfw01",
>> "fw_type": "NetScreen", "device": "dencfw01:", "host":
>> "dencfw01.contournetworks.net", "start_time": "Aug 30 15:58:28" }
>>
>>
>>
>>
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
