Yes, all the logs are coming in this way. There are three separate logs being sent and each is incorrect in its own right but the same format behavior is consistent across each of the three logs. I will look further into the suggestions you have provided and have already reached out to the appliance support team and will raise the nonconformance with the RFC.
This does provide me some valuable guidance and I thank each of you for your feedback. *Scott Slattery* *Sr. Systems & Cloud Architect* *Cloud, Compute, Information & Architecture Team* motorolasolutions.com *O: 602.529.8226* *E*: [email protected] On Tue, Mar 23, 2021 at 11:22 PM <[email protected]> wrote: > There's something strange with this log entry. Notice that it looks as if > it were more than one log entries joined with LF character (escaped on > input as #012). And all "further" entries start with a - still > nonconformant to the RFC but "fuller" - timestamp containing year.It's the > leading event that seems to be trimmed in the front somehow. Do all events > arrive like this?Wysłano z telefonu Samsung<div> > </div><div> > </div><!-- originalMessage --><div>-------- Original message > --------</div><div>From: Scott Slattery via rsyslog < > [email protected]> </div><div>Date: 24/03/2021 01:52 > (GMT+01:00) </div><div>To: David Lang <[email protected]> </div><div>Cc: > Scott Slattery <[email protected]>, Scott Slattery via > rsyslog <[email protected]> </div><div>Subject: Re: [rsyslog] > Altering forwarded logfile names </div><div> > </div> > > Hi David, fortunately I had already done this. I'm including an actual log > entry but have anonymized the data to keep the actual user and email > address confidential: > > Debug line with all properties: > FROMHOST: 'ause1oagatst02.aws.mycompany.com' > , fromhost-ip: '10.41.102.143' > , HOSTNAME: 'ause1oagatst02.aws.mycompany.com' > , PRI: 13, > syslogtag '03-23T16:' > , programname: '03-23T16' > , APP-NAME: '03-23T16' > , PROCID: '-' > , MSGID: '-' > , TIMNESTAMP: 'Mar 23 21:47:13' > , STRUCTURED-DATA: '-' > , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY > ACCESS AUTHZ SESSION INFO USER_SESSION > [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT=" > [email protected]" APP="Ignio Uat OAG" > APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN=" > apigniodashboard-uat.mycompany.com" > RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38" > USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"] > SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP: > 10.44.65.38#0122021-03-23T16:47:20.708-05:00 > AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO > USER_AUTHZ [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" > SUBJECT="[email protected]" RESOURCE="/_dash-dependencies" > METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio > Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN=" > apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A - > SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username= > [email protected] RelayDomain=apigniodashboard-uat.mycompany.com > [email protected] > > SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport > RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X > 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 > Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000 > maxActiveInterval=28800000 lastAccessedTime=1616536029221 " > REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X > 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 > Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00 > AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO > USER_SESSION [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" > SUBJECT="Joe.User@mycompan' > *escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com > ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION > [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT=" > [email protected]" APP="Ignio Uat OAG" > APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN=" > apigniodashboard-uat.mycompany.com" > RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38" > USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"] > SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP: > 10.44.65.38#0122021-03-23T16:47:20.708-05:00 > AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO > USER_AUTHZ [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" > SUBJECT="[email protected]" RESOURCE="/_dash-dependencies" > METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio > Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN=" > apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A - > SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username= > [email protected] RelayDomain=apigniodashboard-uat.mycompany.com > [email protected] > > SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport > RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X > 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 > Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000 > maxActiveInterval=28800000 lastAccessedTime=1616536029221 " > REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X > 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 > Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00 > AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO > USER_SESSION [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" > SUBJECT="Joe.User@mycompan' > *rawmsg*: '03-23T16:47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com > ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION > [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT=" > [email protected]" APP="Ignio Uat OAG" > APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN=" > apigniodashboard-uat.mycompany.com" > RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38" > USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"] > SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP: > 10.44.65.38#0122021-03-23T16:47:20.708-05:00 > AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO > USER_AUTHZ [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" > SUBJECT="[email protected]" RESOURCE="/_dash-dependencies" > METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio > Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN=" > apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A - > SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username= > [email protected] RelayDomain=apigniodashboard-uat.mycompany.com > [email protected] > > SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport > RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X > 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 > Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000 > maxActiveInterval=28800000 lastAccessedTime=1616536029221 " > REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X > 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 > Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00 > AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO > USER_SESSION [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" > SUBJECT="Joe.User@mycompan' > > > By inspecting the rawmsg, I can see that field four (space-delimited) > indicates this is the ACCESS log. So if I were able to extract the log > identifier from the msg, I could then write all access logs to the same > daily file. There are other formats as well from the same device but the > idea is the same. > > *Scott Slattery* > > *Sr. Systems & Cloud Architect* > > *Cloud, Compute, Information & Architecture Team* > > motorolasolutions.com > > *O: 602.529.8226* > > *E*: [email protected] > > > > > On Tue, Mar 23, 2021 at 5:29 PM David Lang <[email protected]> wrote: > > > the source logfile name is not included in the payload by the syslog > spec. > > It > > may be in the case of your appliance, but we would need to see a sample > > log to > > understand ho to parse it. > > > > based on your template, you are using app-name, which may be listed > > separtely if > > it's a RFC5424 format log, or may be part of the syslog tag if it's a > > RFC3164 > > format log over the wire (neither format has a way to specify a source > log > > file > > by default) > > > > you can look at > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e= > > and see the *-cc > > options that you could apply to the app-name to eliminate control > > characters. > > > > Again, we really need to see the original log message to understand > what's > > what. > > Please log it with the templateRSYSLOG_DebugFormat so we can see exactly > > what is > > sent over the wire and how rsyslog has parsed it. > > > > David Lang > > > > On Tue, 23 Mar 2021, Scott Slattery via rsyslog > > wrote: > > > > > Date: Tue, 23 Mar 2021 16:05:45 -0700 > > > From: Scott Slattery via rsyslog <[email protected]> > > > To: John Chivian <[email protected]> > > > Cc: Scott Slattery <[email protected]>, > > > rsyslog-users <[email protected]> > > > Subject: Re: [rsyslog] Altering forwarded logfile names > > > > > > Thanks, John, let me try to clarify what I mean. > > > > > > Normally when I forward from a remote server to the central log > server, I > > > can include a tag that can then be used to determine the file name I > want > > > on the central server. Since I have no real way to include this tag > from > > > the appliance, this is not an option. > > > > > > I'm looking for a way of inspecting the incoming packets to determining > > the > > > source logfile name (which is included in the payload) and use that > > > filename on the target central server. Since there are multiple logs > > being > > > sent (access, audit, monitor, etc.), I'd like to segregate these into > > their > > > own files. I'm already using a template with the host information to > > > dynamically create the file names. I just don't know how I can go > beyond > > > this to also include the source logname. > > > > > > Here's the template I'm using. It works for all other hosts where I can > > > configure the tag but I get garbage names from the appliance. I had > hoped > > > that the appliance included some standard syslog tags but it doesn't > seem > > > so. > > > > > > template(name="DynRemoteLogFile" type="string" > > > > > > string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log") > > > > > > *Scott Slattery* > > > > > > *Sr. Systems & Cloud Architect* > > > > > > *Cloud, Compute, Information & Architecture Team* > > > > > > motorolasolutions.com > > > > > > *O: 602.529.8226* > > > > > > *E*: [email protected] > > > > > > > > > > > > > > > On Tue, Mar 23, 2021 at 3:30 PM John Chivian <[email protected]> > > wrote: > > > > > >> Your use of the term “file name” is confusing. When senders deliver > to > > >> rsyslog over the network there is no exchange of files or filenames, > > only > > >> packets of information. Those packets are expected to be in a format > > that > > >> syslog understands such that useful information (header elements and > > >> message body) may be parsed from them. If you as the rsyslog admin > > choose > > >> to use some of that header information to compose filenames for output > > >> files, then yes you are sort of at the mercy of the senders content > > >> (especially if the sender doesn’t follow the syslog rules). However, > > there > > >> are functions in the advanced syntax that can be used to perform the > > type > > >> of character replacements you’re talking about. > > >> > > >> It is common practice to use the syslog header/rsyslog property > element > > >> called “hostname” for just such purposes. Is this what you’re talking > > >> about? You’d have to provide your configuration for real analysis, at > > >> least the part you perceive to be responsible for the problem. > > >> > > >> Regards, > > >> > > >> > > >> > > >> > On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog < > > >> [email protected]> wrote: > > >> > > > >> > I have a configured central log collector using rsyslog. A few of > the > > >> > devices forwarding their logs are appliances that have no > > configuration > > >> > options other than the IP forwarding address and protocol. I cannot > > >> control > > >> > what file names are being sent. > > >> > > > >> > Unfortunately, they are sending unintelligible file names with > > characters > > >> > that normally would be escaped. Is there any way I can control or > > alter > > >> the > > >> > incoming file name to normalize it to avoid these odd characters? > > >> > > > >> > For example, could I establish a character map that maps the > unallowed > > >> > character to something acceptable? > > >> > > > >> > thanks, > > >> > > > >> > *Scott Slattery* > > >> > > > >> > *Sr. Systems & Cloud Architect* > > >> > > > >> > *Cloud, Compute, Information & Architecture Team* > > >> > > > >> > motorolasolutions.com > > >> > > > >> > *O: 602.529.822* > > >> > > > >> > *E*: [email protected] > > >> > > > >> > -- > > >> > > > >> > > > >> > *For more information on how and why we collect your personal > > >> > information, please visit our Privacy Policy > > >> > < > > >> > > > https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement > > >> >.* > > >> > _______________________________________________ > > >> > rsyslog mailing list > > >> > > > >> > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e= > > >> > > > >> > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e= > > >> > What's up with rsyslog? Follow > > >> > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e= > > >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > myriad > > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > >> DON'T LIKE THAT. > > >> > > >> > > > > > > -- > > > > > > > > > *For more information on how and why we collect your personal > > > information, please visit our Privacy Policy > > > < > > > https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement > > >.* > > > _______________________________________________ > > > rsyslog mailing list > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e= > > > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e= > > > What's up with rsyslog? Follow > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e= > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > -- > > > *For more information on how and why we collect your personal > information, please visit our Privacy Policy > < > https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement > >.* > _______________________________________________ > rsyslog mailing list > > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e= > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e= > What's up with rsyslog? Follow > https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e= > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. -- *For more information on how and why we collect your personal information, please visit our Privacy Policy <https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.* _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
