Re: [rsyslog] Altering forwarded logfile names

Fri, 26 Mar 2021 01:08:03 -0700 

ahh, I missed that. A bit longer explination:
before the date, the message should start out with '<number>' or '<number>1' depending on which RFC it's following (from the format of the date, this should have the 1 as required by RFC5424) without that it's triggering the non-standard extension that rsyslog supports that allows you to send embedded newlines in the message. 


with it starting with 2021 (which would be detected to be the message length), 
that would explain why the newlines are being ignored.


disabling this feature as rainer suggests may fix it.


still a problem with the sender violating the RFC, just combined with a 
extension of the syslog protocol that rsyslog support to make a set of odd 
symptoms.


David Lang



On Fri, 26 Mar 2021, Rainer Gerhards wrote:


Date: Fri, 26 Mar 2021 08:55:02 +0100
From: Rainer Gerhards <[email protected]>
To: Scott Slattery <[email protected]>
Cc: David Lang <[email protected]>, rsyslog-users <[email protected]>,
    [email protected]
Subject: Re: [rsyslog] Altering forwarded logfile names

I think I guess what is going on. Probably a message starts, without pri,
with a number. If so, that triggers octet-counted framing, because that's
what it really indicates. So the number becomes the message length.

Try param

supportOctedtCountedFraming="off"

Doc:

https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html

Rainer

El jue, 25 mar 2021 a las 23:37, Scott Slattery (<
[email protected]>) escribió:


Upon further review, I'm seeing packet lengths from 4434 down to 111. It
appears once the packet length exceeds 2919 the trailing 0x0a is missing
from any remaining packets larger than 2919.

If I sort by packet size, it jumps from 2919 to 2974 then all further line
feed character is missing.

*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: [email protected]




On Thu, Mar 25, 2021 at 3:24 PM Scott Slattery <
[email protected]> wrote:



*Rsyslogd version:*
rsyslogd 8.24.0-52.amzn2, compiled with:
PLATFORM: x86_64-koji-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
Number of Bits in RainerScript integers: 64

Here is my config:
$MaxMessageSize 64k
$EscapeControlCharactersOnReceive on

module(load="imuxsock")
module(load="imjournal" StateFile="imjournal.state")
module(load="imklog")
module(load="immark")
module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat"
DirCreateMode="0705" FileCreateMode="0704")
module(load="impstats" interval="600" log.syslog="off" severity="7"
log.file="/var/log/impstats.log")

$DynaFileCacheSize 100
global(workDirectory="/var/lib/rsyslog" localHostname="cloudlogmaster01")

$MainMsgQueueSize 200000
$MainMsgQueueHighWaterMark 120000
$MainMsgQueueDiscardMark 160000
$MainMsgQueueWorkerThreads 100
$RulesetCreateMainQueue on

$imjournalRatelimitInterval 0
$imjournalRatelimitBurst 0
#### RULES ####

template(name="debug" type="string" string="Debug line with all
properties:\nFROMHOST: '%FROMHOST%'\n, FROMHOST-IP: '%fromhost-ip%'\n,
HOSTNAME: '%HOSTNAME%'\n, PRI: %PRI%,\nSYSLOGTAG '%syslogtag%'\n,
PROGRAMNAME: '%programname%'\n, APP-NAME: '%app-name%'\n, PROCID:
'%PROCID%'\n, MSGID: '%MSGID%'\n, TIMESTAMP: '%TIMESTAMP%'\n,
STRUCTURED-DATA: '%STRUCTURED-DATA%'\n, msg: '%msg%'\nRAWMSG:
'%rawmsg%',\n\nRAWMSG-AFTER-PRI: '%rawmsg-after-pri%'\nJSONMESG:
'%jsonmesg%'\n")

template(name="DynRemoteLogFile" type="string"
string="/splunklog/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")

template(name="MalformedMsgFormatter" type="string"
string="%timegenerated% %fromhost% %rawmsg:::drop-last-lf%\n")
template(name="DynRemoteMalformed" type="string"
string="/splunklog/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-oag-audit.log")

template(name="DynRemoteLogFile" type="list") {
constant(value="/splunklog/remote/")
property(name="FROMHOST")
constant(value="-")
property(name="FROMHOST-IP")
constant(value="/")
property(name="$YEAR")
constant(value="-")
property(name="$MONTH")
constant(value="-")
property(name="$DAY")
constant(value="-")
property(name="syslogtag" position.from="1" position.to="32")
constant(value="-")
property(name="app-name" controlcharacters="drop")
constant(value=".log")
}

ruleset(name="RemoteServer"){
if ($fromhost-ip startswith '10.41.102') then {
action(type="omfile" dynafile="DynRemoteMalformed"
template="MalformedMsgFormatter")
stop
}

 action(type="omfile" dynaFile="DynRemoteLogFile")
}


# module imtcp
module(load="imtcp" KeepAlive="on")
input(type="imtcp" port="10514" ruleset="RemoteServer")


#kern.*                                         /dev/console
*.info;mail.none;authpriv.none;cron.none        /var/log/messages
authpriv.*                                      /var/log/secure
mail.*                                          /var/log/maillog
cron.*                                          /var/log/cron
*.emerg                                         :omusrmsg:*
uucp,news.crit                                  /var/log/spooler
local7.*                                        /var/log/boot.log

*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: [email protected]




On Thu, Mar 25, 2021 at 3:20 PM David Lang <[email protected]> wrote:


what version of rsyslog are you running. can you post your full config?

if you are receiving via TCP and it's not splitting the logs based on
newlines,
something very odd is happening.

David Lang

  On Thu, 25 Mar 2021, Scott Slattery wrote:


Date: Thu, 25 Mar 2021 15:07:22 -0700
From: Scott Slattery <[email protected]>
To: David Lang <[email protected]>
Cc: Rainer Gerhards <[email protected]>,
    rsyslog-users <[email protected]>,

[email protected]

Subject: Re: [rsyslog] Altering forwarded logfile names

So I'm looking at the pcap and on the first log record the header
information looks fine and the record terminated by 0x0A line feed.

When I look at the record containing this record written to the rsyslog
file, the record length is 65586 (64k) long which is what I think you

were

referring to earlier. Why would rsyslog write such a long record? I
initially thought my data may be getting truncated so I updated
'$MaxMessageSize 64k'. Any suggestion how I fix this behavior.

I can confirm that each log record in the pcap is terminated by 0x0A

and

looks in tact so must be something I need to do on rsyslog to fix this.

*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: [email protected]




On Thu, Mar 25, 2021 at 1:15 PM David Lang <[email protected]> wrote:


you may want to capture with -X so that it decodes it into hex and

you can

see
newlines vs linefeeds

David Lang

On Thu, 25 Mar 2021, Scott Slattery wrote:


Date: Thu, 25 Mar 2021 12:47:43 -0700
From: Scott Slattery <[email protected]>
To: David Lang <[email protected]>
Cc: Rainer Gerhards <[email protected]>,
    rsyslog-users <[email protected]>,

[email protected]

Subject: Re: [rsyslog] Altering forwarded logfile names

Thanks David, I just grabbed my first capture and am looking at it.

I'm

using TCP vs UDP but your comment makes sense. I'll certainly share

with

you my observations once I have some data to refer to. This has been

a

learning experience. It's also the first time I've ever seen such an

issue

with rsyslog so it's a bit of a puzzle.

*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: [email protected]




On Thu, Mar 25, 2021 at 12:32 PM David Lang <[email protected]> wrote:


the rawmsg field in the debugformat output shows exactly what

rsyslog is

seeing.

the reason I asked you to check multiple entries is that if rsyslog

does

not see
the separator (due to either multiple messages in one UDP packet, or
missing
newlines in a TCP stream) it will combine what are intended as

multiple

messages
together into one message as far as rsyslog is concerned, and only

break

it into
multiple messages when it hits the configured maxmessagesize size.

That

breaks
things at arbitrary points in the message, so as you look at

multiple

messages,
they will not all start the same way.

a tcpdump -s0 -A -n (with the appropriate port/ip filter) will also

show

exactly
what's happening over the wire.

David Lang

On Thu, 25 Mar 2021, Scott Slattery wrote:


Date: Thu, 25 Mar 2021 12:23:03 -0700
From: Scott Slattery <[email protected]>
To: Rainer Gerhards <[email protected]>
Cc: rsyslog-users <[email protected]>, David Lang <

[email protected]

,
    [email protected]
Subject: Re: [rsyslog] Altering forwarded logfile names

Hey Guys, before I consume more of your time, I'm capturing the

packets

from the source device to determine if any of the payload is lost

on

its

way to the rsyslog collector. After I analyze further, I'll share

my

observations. I'm hoping nothing is getting dropped over the wire.

Thanks

for your patience and help so far. I expect to have additional

details

shortly.

*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: [email protected]




On Thu, Mar 25, 2021 at 12:51 AM Rainer Gerhards <

[email protected]>

wrote:


Sorry if I didn't catch it, but did you  post the configuration?

If

not, it would probably be useful.

Looking at the message properties, this looks like the message was
received via UDP but had a LF inside it which was intended as a

frame

separator. If so, that is another bug in the vendor's

implementation.

Framing is used for TCP, but not UDP. For UDP it is strictly one
message per package.

HTH,
Rainer

El mié, 24 mar 2021 a las 22:23, Scott Slattery via rsyslog
(<[email protected]>) escribió:


From what I can discern since there are lots of transactions,

there

seem

to

be multiple payload formats. For all that I have so far

analyzed, the

date

truncation issue is the same regardless of the payload received.

Like

you,

I've never used mmnormalize but I will look at it as an option.

I've

reached out to the product support team to better understand if

there

is

a

appliance-side misconfiguration and something that can be

corrected

there

or if I need to make adjustments on the rsyslog side to clean up

the

records.

What I meant is that there are multiple message formats (payload

data)

but

there seems to be a consistent truncation of the date field just

as

you

saw

in the app-name tag field.

I appreciate your ongoing feedback and help. This issue is new

to me

and

will need a creative solution if the vendor has no answers.

Many thanks,


*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: [email protected]




On Wed, Mar 24, 2021 at 2:14 PM David Lang <[email protected]>

wrote:



when you say 'each is incorrect but the same format'

does that mean that every log has the year missing? or that

every

log

is

combining the logs together?

I'll  note that it's possible to define a custom parser using

the

mmnormalize
library and add it to the parsing stack. I helped define the

need

for

such

a
capability, but haven't used it yet.

David Lang

  On Wed, 24 Mar 2021, Scott Slattery wrote:


Date: Wed, 24 Mar 2021 08:58:07 -0700
From: Scott Slattery <[email protected]>
To: [email protected]
Cc: rsyslog-users <[email protected]>, David Lang <

[email protected]


Subject: Re: [rsyslog] Altering forwarded logfile names

Yes, all the logs are coming in this way. There are three

separate

logs

being sent and each is incorrect in its own right but the same

format

behavior is consistent across each of the three logs. I will

look

further

into the suggestions you have provided and have already

reached out

to

the

appliance support team and will raise the nonconformance with

the

RFC.


This does provide me some valuable guidance and I thank each

of you

for

your feedback.

*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: [email protected]




On Tue, Mar 23, 2021 at 11:22 PM <[email protected]>

wrote:



There's something strange with this log entry. Notice that it

looks

as

if

it were more than one log entries joined with LF character

(escaped

on

input as #012). And all "further" entries start with a - still
nonconformant to the RFC but "fuller" - timestamp containing

year.It's

the

leading event that seems to be trimmed in the front somehow.

Do

all

events

arrive like this?Wysłano z telefonu Samsung<div>
</div><div>
</div><!-- originalMessage --><div>-------- Original message
--------</div><div>From: Scott Slattery via rsyslog <
[email protected]> </div><div>Date: 24/03/2021  01:52
(GMT+01:00) </div><div>To: David Lang <[email protected]>

</div><div>Cc:

Scott Slattery <[email protected]>, Scott

Slattery

via

rsyslog <[email protected]> </div><div>Subject: Re:

[rsyslog]

Altering forwarded logfile names </div><div>
</div>

Hi David, fortunately I had already done this. I'm including

an

actual

log

entry but have anonymized the data to keep the actual user and

email

address confidential:

Debug line with all properties:
FROMHOST: 'ause1oagatst02.aws.mycompany.com'
, fromhost-ip: '10.41.102.143'
, HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
, PRI: 13,
syslogtag '03-23T16:'
, programname: '03-23T16'
, APP-NAME: '03-23T16'
, PROCID: '-'
, MSGID: '-'
, TIMNESTAMP: 'Mar 23 21:47:13'
, STRUCTURED-DATA: '-'
, *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com

ACCESS_GATEWAY

ACCESS AUTHZ SESSION INFO USER_SESSION
[SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"

SUBJECT="

[email protected]" APP="Ignio Uat OAG"
APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
apigniodashboard-uat.mycompany.com"
RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"

REMOTE_IP="10.44.65.38"

USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192

Safari/537.36"]

SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
10.44.65.38#0122021-03-23T16:47:20.708-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ

POLICY

INFO

USER_AUTHZ

[SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"

SUBJECT="[email protected]"

RESOURCE="/_dash-dependencies"

METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED"

DURATION="0"

APP="Ignio

Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
apigniodashboard-uat.mycompany.com" RESULT="ALLOW"

REASON="N/A -

SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0

oag_username=

[email protected] RelayDomain=

apigniodashboard-uat.mycompany.com

[email protected]











SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport

RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel

Mac

OS X

10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/88.0.4324.192

Safari/537.36 creationTime=1616536016811

maxInactiveInterval=3600000

maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;

Intel

Mac

OS X

10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/88.0.4324.192

Safari/537.36"] allow access to

resource#0122021-03-23T16:47:20.711-05:00

AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ

SESSION

INFO

USER_SESSION

[SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"

SUBJECT="Joe.User@mycompan'
*escaped msg*: '47:20.708-05:00

AUSE1OAGATST02.aws.mycompany.com

ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
[SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"

SUBJECT="

[email protected]" APP="Ignio Uat OAG"
APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
apigniodashboard-uat.mycompany.com"
RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"

REMOTE_IP="10.44.65.38"

USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192

Safari/537.36"]

SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
10.44.65.38#0122021-03-23T16:47:20.708-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ

POLICY

INFO

USER_AUTHZ

[SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"

SUBJECT="[email protected]"

RESOURCE="/_dash-dependencies"

METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED"

DURATION="0"

APP="Ignio

Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
apigniodashboard-uat.mycompany.com" RESULT="ALLOW"

REASON="N/A -

SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0

oag_username=

[email protected] RelayDomain=

apigniodashboard-uat.mycompany.com

[email protected]











SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport

RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel

Mac

OS X

10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/88.0.4324.192

Safari/537.36 creationTime=1616536016811

maxInactiveInterval=3600000

maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;

Intel

Mac

OS X

10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/88.0.4324.192

Safari/537.36"] allow access to

resource#0122021-03-23T16:47:20.711-05:00

AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ

SESSION

INFO

USER_SESSION

[SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"

SUBJECT="Joe.User@mycompan'
*rawmsg*: '03-23T16:47:20.708-05:00

AUSE1OAGATST02.aws.mycompany.com

ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
[SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"

SUBJECT="

[email protected]" APP="Ignio Uat OAG"
APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
apigniodashboard-uat.mycompany.com"
RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"

REMOTE_IP="10.44.65.38"

USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192

Safari/537.36"]

SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
10.44.65.38#0122021-03-23T16:47:20.708-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ

POLICY

INFO

USER_AUTHZ

[SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"

SUBJECT="[email protected]"

RESOURCE="/_dash-dependencies"

METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED"

DURATION="0"

APP="Ignio

Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
apigniodashboard-uat.mycompany.com" RESULT="ALLOW"

REASON="N/A -

SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0

oag_username=

[email protected] RelayDomain=

apigniodashboard-uat.mycompany.com

[email protected]











SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport

RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel

Mac

OS X

10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/88.0.4324.192

Safari/537.36 creationTime=1616536016811

maxInactiveInterval=3600000

maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh;

Intel

Mac

OS X

10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)

Chrome/88.0.4324.192

Safari/537.36"] allow access to

resource#0122021-03-23T16:47:20.711-05:00

AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ

SESSION

INFO

USER_SESSION

[SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"

SUBJECT="Joe.User@mycompan'


By inspecting the rawmsg, I can see that field four

(space-delimited)

indicates this is the ACCESS log. So if I were able to

extract the

log

identifier from the msg, I could then write all access logs

to the

same

daily file. There are other formats as well from the same

device

but the

idea is the same.

*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: [email protected]




On Tue, Mar 23, 2021 at 5:29 PM David Lang <[email protected]>

wrote:



the source logfile name is not included in the payload by the

syslog

spec.

It
may be in the case of your appliance, but we would need to

see a

sample

log to
understand ho to parse it.

based on your template, you are using app-name, which may be

listed

separtely if
it's a RFC5424 format log, or may be part of the syslog tag

if

it's a

RFC3164
format log over the wire (neither format has a way to

specify a

source

log

file
by default)

you can look at












https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=

and see the *-cc
options that you could apply to the app-name to eliminate

control

characters.

Again, we really need to see the original log message to

understand

what's

what.
Please log it with the templateRSYSLOG_DebugFormat so we can

see

exactly

what is
sent over the wire and how rsyslog has parsed it.

David Lang

  On Tue, 23 Mar 2021, Scott Slattery via rsyslog
wrote:


Date: Tue, 23 Mar 2021 16:05:45 -0700
From: Scott Slattery via rsyslog <[email protected]



To: John Chivian <[email protected]>
Cc: Scott Slattery <[email protected]>,
    rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Altering forwarded logfile names

Thanks, John, let me try to clarify what I mean.

Normally when I forward from a remote server to the central

log

server, I

can include a tag that can then be used to determine the

file

name I

want

on the central server. Since I have no real way to include

this

tag

from

the appliance, this is not an option.

I'm looking for a way of inspecting the incoming packets to

determining

the

source logfile name (which is included in the payload) and

use

that

filename on the target central server. Since there are

multiple

logs

being

sent (access, audit, monitor, etc.), I'd like to segregate

these

into

their

own files. I'm already using a template with the host

information

to

dynamically create the file names. I just don't know how I

can

go

beyond

this to also include the source logname.

Here's the template I'm using. It works for all other hosts

where

I

can

configure the tag but I get garbage names from the

appliance. I

had

hoped

that the appliance included some standard syslog tags but it

doesn't

seem

so.

template(name="DynRemoteLogFile" type="string"














string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")


*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.8226*

*E*: [email protected]




On Tue, Mar 23, 2021 at 3:30 PM John Chivian <

[email protected]>

wrote:



Your use of the term “file name” is confusing.  When

senders

deliver

to

rsyslog over the network there is no exchange of files or

filenames,

only

packets of information.  Those packets are expected to be

in a

format

that

syslog understands such that useful information (header

elements

and

message body) may be parsed from them.  If you as the

rsyslog

admin

choose

to use some of that header information to compose

filenames for

output

files, then yes you are sort of at the mercy of the senders

content

(especially if the sender doesn’t follow the syslog rules).

However,

there

are functions in the advanced syntax that can be used to

perform

the

type

of character replacements you’re talking about.

It is common practice to use the syslog header/rsyslog

property

element

called “hostname” for just such purposes.  Is this what

you’re

talking

about?  You’d have to provide your configuration for real

analysis,

at

least the part you perceive to be responsible for the

problem.


Regards,




On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <

[email protected]> wrote:


I have a configured central log collector using rsyslog.

A few

of

the

devices forwarding their logs are appliances that have no

configuration

options other than the IP forwarding address and

protocol. I

cannot

control

what file names are being sent.

Unfortunately, they are sending unintelligible file names

with

characters

that normally would be escaped. Is there any way I can

control

or

alter

the

incoming file name to normalize it to avoid these odd

characters?


For example, could I establish a character map that maps

the

unallowed

character to something acceptable?

thanks,

*Scott Slattery*

*Sr. Systems & Cloud Architect*

*Cloud, Compute, Information & Architecture Team*

motorolasolutions.com

*O: 602.529.822*

*E*: [email protected]

--


*For more information on how and why we collect your

personal

information, please visit our Privacy Policy
<















https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement

.*
_______________________________________________
rsyslog mailing list
















https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=

















https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=

What's up with rsyslog? Follow















https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=

NOTE WELL: This is a PUBLIC mailing list, posts are

ARCHIVED

by

a

myriad

of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT

POST

if

you

DON'T LIKE THAT.




--


*For more information on how and why we collect your

personal

information, please visit our Privacy Policy
<













https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement

.*
_______________________________________________
rsyslog mailing list














https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=















https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=

What's up with rsyslog? Follow













https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=

NOTE WELL: This is a PUBLIC mailing list, posts are

ARCHIVED by

a

myriad

of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT

POST

if

you

DON'T LIKE THAT.


--


*For more information on how and why we collect your personal
information, please visit our Privacy Policy
<










https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement

.*

_______________________________________________
rsyslog mailing list











https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e=












https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e=

What's up with rsyslog? Follow










https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e=

NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED

by a

myriad

of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT

POST if

you

DON'T LIKE THAT.





--


*For more information on how and why we collect your personal
information, please visit our Privacy Policy
<







https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement

.*
_______________________________________________
rsyslog mailing list








https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=7hc9b2gSfvsfQzXvsRVXqGeP4Rz_tJtrykpjqkGBIdk&e=









https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=bDybhvts9KqI-WEfMHo7O40ulQK5M6Tg4tnKNRRIhE8&e=

What's up with rsyslog? Follow







https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=zPhpcUPlK_CWLiU75_WwjyQGz5mFTiXc1yyBIfppkrI&e=

NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a

myriad

of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST

if you

DON'T LIKE THAT.














*For more information on how and why we collect your personal information,
please visit our Privacy Policy
<https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*




_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.




























					Reply via email to