when you say 'each is incorrect but the same format'
does that mean that every log has the year missing? or that every log is
combining the logs together?
I'll note that it's possible to define a custom parser using the
mmnormalize
library and add it to the parsing stack. I helped define the need for such
a
capability, but haven't used it yet.
David Lang
On Wed, 24 Mar 2021, Scott Slattery wrote:
Date: Wed, 24 Mar 2021 08:58:07 -0700
From: Scott Slattery <[email protected]>
To: [email protected]
Cc: rsyslog-users <[email protected]>, David Lang <[email protected]
Subject: Re: [rsyslog] Altering forwarded logfile names
Yes, all the logs are coming in this way. There are three separate logs
being sent and each is incorrect in its own right but the same format
behavior is consistent across each of the three logs. I will look further
into the suggestions you have provided and have already reached out to
the
appliance support team and will raise the nonconformance with the RFC.
This does provide me some valuable guidance and I thank each of you for
your feedback.
*Scott Slattery*
*Sr. Systems & Cloud Architect*
*Cloud, Compute, Information & Architecture Team*
motorolasolutions.com
*O: 602.529.8226*
*E*: [email protected]
On Tue, Mar 23, 2021 at 11:22 PM <[email protected]> wrote:
There's something strange with this log entry. Notice that it looks as
if
it were more than one log entries joined with LF character (escaped on
input as #012). And all "further" entries start with a - still
nonconformant to the RFC but "fuller" - timestamp containing year.It's
the
leading event that seems to be trimmed in the front somehow. Do all
events
arrive like this?Wysłano z telefonu Samsung<div>
</div><div>
</div><!-- originalMessage --><div>-------- Original message
--------</div><div>From: Scott Slattery via rsyslog <
[email protected]> </div><div>Date: 24/03/2021 01:52
(GMT+01:00) </div><div>To: David Lang <[email protected]> </div><div>Cc:
Scott Slattery <[email protected]>, Scott Slattery
via
rsyslog <[email protected]> </div><div>Subject: Re: [rsyslog]
Altering forwarded logfile names </div><div>
</div>
Hi David, fortunately I had already done this. I'm including an actual
log
entry but have anonymized the data to keep the actual user and email
address confidential:
Debug line with all properties:
FROMHOST: 'ause1oagatst02.aws.mycompany.com'
, fromhost-ip: '10.41.102.143'
, HOSTNAME: 'ause1oagatst02.aws.mycompany.com'
, PRI: 13,
syslogtag '03-23T16:'
, programname: '03-23T16'
, APP-NAME: '03-23T16'
, PROCID: '-'
, MSGID: '-'
, TIMNESTAMP: 'Mar 23 21:47:13'
, STRUCTURED-DATA: '-'
, *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
ACCESS_GATEWAY
ACCESS AUTHZ SESSION INFO USER_SESSION
[SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
[email protected]" APP="Ignio Uat OAG"
APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
apigniodashboard-uat.mycompany.com"
RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
REMOTE_IP="10.44.65.38"
USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36"]
SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
10.44.65.38#0122021-03-23T16:47:20.708-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
INFO
USER_AUTHZ [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="[email protected]" RESOURCE="/_dash-dependencies"
METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
APP="Ignio
Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
[email protected] RelayDomain=
apigniodashboard-uat.mycompany.com
[email protected]
SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac
OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36"] allow access to
resource#0122021-03-23T16:47:20.711-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION
INFO
USER_SESSION [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="Joe.User@mycompan'
*escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION
[SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
[email protected]" APP="Ignio Uat OAG"
APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
apigniodashboard-uat.mycompany.com"
RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
REMOTE_IP="10.44.65.38"
USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36"]
SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
10.44.65.38#0122021-03-23T16:47:20.708-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
INFO
USER_AUTHZ [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="[email protected]" RESOURCE="/_dash-dependencies"
METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
APP="Ignio
Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
[email protected] RelayDomain=
apigniodashboard-uat.mycompany.com
[email protected]
SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac
OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36"] allow access to
resource#0122021-03-23T16:47:20.711-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION
INFO
USER_SESSION [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="Joe.User@mycompan'
*rawmsg*: '03-23T16:47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com
ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION
[SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="
[email protected]" APP="Ignio Uat OAG"
APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
apigniodashboard-uat.mycompany.com"
RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED"
REMOTE_IP="10.44.65.38"
USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36"]
SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP:
10.44.65.38#0122021-03-23T16:47:20.708-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY
INFO
USER_AUTHZ [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="[email protected]" RESOURCE="/_dash-dependencies"
METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0"
APP="Ignio
Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="
apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A -
SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username=
[email protected] RelayDomain=
apigniodashboard-uat.mycompany.com
[email protected]
SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000
maxActiveInterval=28800000 lastAccessedTime=1616536029221 "
REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac
OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192
Safari/537.36"] allow access to
resource#0122021-03-23T16:47:20.711-05:00
AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION
INFO
USER_SESSION [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0"
SUBJECT="Joe.User@mycompan'
By inspecting the rawmsg, I can see that field four (space-delimited)
indicates this is the ACCESS log. So if I were able to extract the log
identifier from the msg, I could then write all access logs to the same
daily file. There are other formats as well from the same device but the
idea is the same.
*Scott Slattery*
*Sr. Systems & Cloud Architect*
*Cloud, Compute, Information & Architecture Team*
motorolasolutions.com
*O: 602.529.8226*
*E*: [email protected]
On Tue, Mar 23, 2021 at 5:29 PM David Lang <[email protected]> wrote:
the source logfile name is not included in the payload by the syslog
spec.
It
may be in the case of your appliance, but we would need to see a sample
log to
understand ho to parse it.
based on your template, you are using app-name, which may be listed
separtely if
it's a RFC5424 format log, or may be part of the syslog tag if it's a
RFC3164
format log over the wire (neither format has a way to specify a source
log
file
by default)
you can look at
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e=
and see the *-cc
options that you could apply to the app-name to eliminate control
characters.
Again, we really need to see the original log message to understand
what's
what.
Please log it with the templateRSYSLOG_DebugFormat so we can see
exactly
what is
sent over the wire and how rsyslog has parsed it.
David Lang
On Tue, 23 Mar 2021, Scott Slattery via rsyslog
wrote:
Date: Tue, 23 Mar 2021 16:05:45 -0700
From: Scott Slattery via rsyslog <[email protected]>
To: John Chivian <[email protected]>
Cc: Scott Slattery <[email protected]>,
rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Altering forwarded logfile names
Thanks, John, let me try to clarify what I mean.
Normally when I forward from a remote server to the central log
server, I
can include a tag that can then be used to determine the file name I
want
on the central server. Since I have no real way to include this tag
from
the appliance, this is not an option.
I'm looking for a way of inspecting the incoming packets to
determining
the
source logfile name (which is included in the payload) and use that
filename on the target central server. Since there are multiple logs
being
sent (access, audit, monitor, etc.), I'd like to segregate these into
their
own files. I'm already using a template with the host information to
dynamically create the file names. I just don't know how I can go
beyond
this to also include the source logname.
Here's the template I'm using. It works for all other hosts where I
can
configure the tag but I get garbage names from the appliance. I had
hoped
that the appliance included some standard syslog tags but it doesn't
seem
so.
template(name="DynRemoteLogFile" type="string"
string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")
*Scott Slattery*
*Sr. Systems & Cloud Architect*
*Cloud, Compute, Information & Architecture Team*
motorolasolutions.com
*O: 602.529.8226*
*E*: [email protected]
On Tue, Mar 23, 2021 at 3:30 PM John Chivian <[email protected]>
wrote:
Your use of the term “file name” is confusing. When senders deliver
to
rsyslog over the network there is no exchange of files or filenames,
only
packets of information. Those packets are expected to be in a format
that
syslog understands such that useful information (header elements and
message body) may be parsed from them. If you as the rsyslog admin
choose
to use some of that header information to compose filenames for
output
files, then yes you are sort of at the mercy of the senders content
(especially if the sender doesn’t follow the syslog rules). However,
there
are functions in the advanced syntax that can be used to perform the
type
of character replacements you’re talking about.
It is common practice to use the syslog header/rsyslog property
element
called “hostname” for just such purposes. Is this what you’re
talking
about? You’d have to provide your configuration for real analysis,
at
least the part you perceive to be responsible for the problem.
Regards,
On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <
[email protected]> wrote:
I have a configured central log collector using rsyslog. A few of
the
devices forwarding their logs are appliances that have no
configuration
options other than the IP forwarding address and protocol. I cannot
control
what file names are being sent.
Unfortunately, they are sending unintelligible file names with
characters
that normally would be escaped. Is there any way I can control or
alter
the
incoming file name to normalize it to avoid these odd characters?
For example, could I establish a character map that maps the
unallowed
character to something acceptable?
thanks,
*Scott Slattery*
*Sr. Systems & Cloud Architect*
*Cloud, Compute, Information & Architecture Team*
motorolasolutions.com
*O: 602.529.822*
*E*: [email protected]
--
*For more information on how and why we collect your personal
information, please visit our Privacy Policy
<
https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
.*
_______________________________________________
rsyslog mailing list
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=
What's up with rsyslog? Follow
https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
--
*For more information on how and why we collect your personal
information, please visit our Privacy Policy
<
https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
.*
_______________________________________________
rsyslog mailing list
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=
What's up with rsyslog? Follow
https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
--
*For more information on how and why we collect your personal
information, please visit our Privacy Policy
<
https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
.*
_______________________________________________
rsyslog mailing list
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e=
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e=
What's up with rsyslog? Follow
https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e=
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
