list members,
i never received responses from the below, and i looked in my junk mail
folders, so i am replying to my own message.
on the central cluster servers, i made a couple changes. I added a
template for the SQL format, and specified that format in the ommysql
action stanza...
# Define the MariaDB/SQL insert template
$template bpk2_SqlWithSeparatePID,"insert into SystemEvents (Message,
Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
InfoUnitID, SysLogTag, processid) values ('%msg%', %syslogfacility%,
'%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%',
'%timegenerated:::date-mysql%', %iut%, '%programname%', '%PROCID%')",SQL
# Provides MySQL syslog transmission
# for parameters see http://www.rsyslog.com/doc/ommysql.html
module(load="ommysql") # needs to be done just once
action(type="ommysql" server="database.bpk2.com" serverport="3306"
db="Syslog" uid="Syslog" pwd="SooperSekretString"
template="bpk2_SqlWithSeparatePID")
i now get the SysLogTag name without the PID, from those cluster
servers, but i need to have the client's data parsed in the same way. i
tried creating a format on the client and specifying the format in the
omrelp stanza, but that does not seem to be everything i need to do, if
it will even work.
i need to figure out how to have the clients syslogtag parsed, to
separate the "programname" and "procid" strings. this needs to be done,
seemingly on the server with a property replacer, and then mapped to the
fields used in the database template, so they are inserted properly into
the database. are there any pointers on how this is best/properly done?
thank you,
brendan
On 2/25/22 10:52 AM, Brendan Kearney wrote:
list members,
i am looking to separate the PID from the SysLogTag in all messages,
and have separate fields in the LogAnalyser database. i have several
hosts that send messages via RELP to a central cluster of receivers,
and those cluster members then submit the messages into MariaDB.
as i understand it, i need to:
1) modify each source hosts messages to split out the PID from the
SysLogTag field
2) modify the central clusters inserts to MariaDB, to specify the PID
independent of the SysLogTag
3) modify the view used in LogAnalyzer to show the PID column
for the first bullet, my typical syslog source has the following config:
global(workDirectory="/var/lib/rsyslog")
module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")
module(load="imuxsock"
SysSock.Use="off")
module(load="imjournal"
StateFile="imjournal.state")
include(file="/etc/rsyslog.d/*.conf" mode="optional")
module(load="omrelp") # needs to be done just once
action(type="omrelp" target="relp.bpk2.com" port="20514")
&~
i believe i need to create my own Template, and specify that instead
of "RSYSLOG_TraditionalFileFormat". Being that the source hosts do not
directly insert the messages into the database, i dont need to specify
an insert at this point. i believe RSYSLOG_TraditionalFileFormat has
the following fields:
$template TraditionalFileFormat,"%TIMESTAMP% %HOSTNAME%
%syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
what fields do i need to change, to use the raw SysLogTag, and PID as
separate fields to send to the central receivers?
for the second bullet, i insert messages to the database using the
following config:
global(workDirectory="/var/lib/rsyslog")
module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")
include(file="/etc/rsyslog.d/*.conf" mode="optional")
module(load="imuxsock"
SysSock.Use="off")
module(load="imjournal"
StateFile="imjournal.state")
module(load="imudp")
input(type="imudp" port="514")
module(load="imtcp")
input(type="imtcp" port="514")
module(load="imrelp")
input(type="imrelp" port="20514")
module(load="ommysql")
action(type="ommysql" server="database.domain.tld"
serverport="3306" db="Syslog" uid="" pwd="")
&~
i figure i need to specify another template here, and use that for the
inserts. i found the following, and want to sanity check it, as i
dont know alot about RegEx, which seems to be used:
$template dbFormat,"insert into SystemEvents (Message, Facility,
FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID,
SysLogTag, processid) values ('%msg%', %syslogfacility%,
'%HOSTNAME%',%syslogpriority%, '%timereported:::date-mysql%',
'%timegenerated:::date-mysql%', %iut%,
'%syslogtag:R,ERE,1,FIELD:(.+)(\[[0-9]{1,10}\]).*--end%',
'%syslogtag:R,ERE,1,BLANK:\[([0-9]{1,10})\]--end%')",sql
with the above Template, called dbFormat, i would modify the "action"
of my ommysql module, to be:
...
action(type="ommysql" server="database.domain.tld"
serverport="3306" db="Syslog" uid="" pwd="" template="dbFormat")
am i tracking properly on this? the database does have the processid
field defined already, so it seems that i just need to populate that
field.
for the third bullet, i have installed the helper pieces for the
phpLogCon/LogAnalyzer, so i just need to update the configs in the
database to show the ProcessID field in the view i defined and set as
the default. again, am i thinking about this in the right terms? are
there any steps i am overlooking?
thanks in advance,
brendan kearney
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
