I would start off by filing a ticket with the appliance vendor complaining that they are not following the syslog RFCs (either the old or the new)
then what you will need to do is to have a conditional (probably based on fromhost-ip) to identify these logs and run them through a different parser (look at mmnormalize for a fast, flexible one, but you may be able to get away with the field() function if you only need to do one extraction)
and then to output the logs, you will need a new template that uses the variables that your parser pulls out of them since you can't use the default properties.
David LangOn Tue, 23 Mar 2021, Scott Slattery wrote:
Date: Tue, 23 Mar 2021 17:51:28 -0700 From: Scott Slattery <[email protected]> To: David Lang <[email protected]> Cc: Scott Slattery via rsyslog <[email protected]>, John Chivian <[email protected]> Subject: Re: [rsyslog] Altering forwarded logfile names Hi David, fortunately I had already done this. I'm including an actual log entry but have anonymized the data to keep the actual user and email address confidential: Debug line with all properties: FROMHOST: 'ause1oagatst02.aws.mycompany.com' , fromhost-ip: '10.41.102.143' , HOSTNAME: 'ause1oagatst02.aws.mycompany.com' , PRI: 13, syslogtag '03-23T16:' , programname: '03-23T16' , APP-NAME: '03-23T16' , PROCID: '-' , MSGID: '-' , TIMNESTAMP: 'Mar 23 21:47:13' , STRUCTURED-DATA: '-' , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT=" [email protected]" APP="Ignio Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"] SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP: 10.44.65.38#0122021-03-23T16:47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO USER_AUTHZ [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="[email protected]" RESOURCE="/_dash-dependencies" METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN=" apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A - SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username= [email protected] RelayDomain=apigniodashboard-uat.mycompany.com [email protected] SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000 maxActiveInterval=28800000 lastAccessedTime=1616536029221 " REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00 AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="Joe.User@mycompan' *escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT=" [email protected]" APP="Ignio Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"] SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP: 10.44.65.38#0122021-03-23T16:47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO USER_AUTHZ [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="[email protected]" RESOURCE="/_dash-dependencies" METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN=" apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A - SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username= [email protected] RelayDomain=apigniodashboard-uat.mycompany.com [email protected] SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000 maxActiveInterval=28800000 lastAccessedTime=1616536029221 " REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00 AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="Joe.User@mycompan' *rawmsg*: '03-23T16:47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT=" [email protected]" APP="Ignio Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"] SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP: 10.44.65.38#0122021-03-23T16:47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO USER_AUTHZ [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="[email protected]" RESOURCE="/_dash-dependencies" METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN=" apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A - SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username= [email protected] RelayDomain=apigniodashboard-uat.mycompany.com [email protected] SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000 maxActiveInterval=28800000 lastAccessedTime=1616536029221 " REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00 AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT="Joe.User@mycompan' By inspecting the rawmsg, I can see that field four (space-delimited) indicates this is the ACCESS log. So if I were able to extract the log identifier from the msg, I could then write all access logs to the same daily file. There are other formats as well from the same device but the idea is the same. *Scott Slattery* *Sr. Systems & Cloud Architect* *Cloud, Compute, Information & Architecture Team* motorolasolutions.com *O: 602.529.8226* *E*: [email protected] On Tue, Mar 23, 2021 at 5:29 PM David Lang <[email protected]> wrote:the source logfile name is not included in the payload by the syslog spec. It may be in the case of your appliance, but we would need to see a sample log to understand ho to parse it. based on your template, you are using app-name, which may be listed separtely if it's a RFC5424 format log, or may be part of the syslog tag if it's a RFC3164 format log over the wire (neither format has a way to specify a source log file by default) you can look at https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e= and see the *-cc options that you could apply to the app-name to eliminate control characters. Again, we really need to see the original log message to understand what's what. Please log it with the templateRSYSLOG_DebugFormat so we can see exactly what is sent over the wire and how rsyslog has parsed it. David Lang On Tue, 23 Mar 2021, Scott Slattery via rsyslog wrote:Date: Tue, 23 Mar 2021 16:05:45 -0700 From: Scott Slattery via rsyslog <[email protected]> To: John Chivian <[email protected]> Cc: Scott Slattery <[email protected]>, rsyslog-users <[email protected]> Subject: Re: [rsyslog] Altering forwarded logfile names Thanks, John, let me try to clarify what I mean. Normally when I forward from a remote server to the central log server, I can include a tag that can then be used to determine the file name I want on the central server. Since I have no real way to include this tag from the appliance, this is not an option. I'm looking for a way of inspecting the incoming packets to determiningthesource logfile name (which is included in the payload) and use that filename on the target central server. Since there are multiple logsbeingsent (access, audit, monitor, etc.), I'd like to segregate these intotheirown files. I'm already using a template with the host information to dynamically create the file names. I just don't know how I can go beyond this to also include the source logname. Here's the template I'm using. It works for all other hosts where I can configure the tag but I get garbage names from the appliance. I had hoped that the appliance included some standard syslog tags but it doesn't seem so. template(name="DynRemoteLogFile" type="string"string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log")*Scott Slattery* *Sr. Systems & Cloud Architect* *Cloud, Compute, Information & Architecture Team* motorolasolutions.com *O: 602.529.8226* *E*: [email protected] On Tue, Mar 23, 2021 at 3:30 PM John Chivian <[email protected]>wrote:Your use of the term “file name” is confusing. When senders deliver to rsyslog over the network there is no exchange of files or filenames,onlypackets of information. Those packets are expected to be in a formatthatsyslog understands such that useful information (header elements and message body) may be parsed from them. If you as the rsyslog adminchooseto use some of that header information to compose filenames for output files, then yes you are sort of at the mercy of the senders content (especially if the sender doesn’t follow the syslog rules). However,thereare functions in the advanced syntax that can be used to perform thetypeof character replacements you’re talking about. It is common practice to use the syslog header/rsyslog property element called “hostname” for just such purposes. Is this what you’re talking about? You’d have to provide your configuration for real analysis, at least the part you perceive to be responsible for the problem. Regards,On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog <[email protected]> wrote:I have a configured central log collector using rsyslog. A few of the devices forwarding their logs are appliances that have noconfigurationoptions other than the IP forwarding address and protocol. I cannotcontrolwhat file names are being sent. Unfortunately, they are sending unintelligible file names withcharactersthat normally would be escaped. Is there any way I can control oraltertheincoming file name to normalize it to avoid these odd characters? For example, could I establish a character map that maps the unallowed character to something acceptable? thanks, *Scott Slattery* *Sr. Systems & Cloud Architect* *Cloud, Compute, Information & Architecture Team* motorolasolutions.com *O: 602.529.822* *E*: [email protected] -- *For more information on how and why we collect your personal information, please visit our Privacy Policy <https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement.* _______________________________________________ rsyslog mailing listhttps://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e=https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e=What's up with rsyslog? Followhttps://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e=NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by amyriadof sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.-- *For more information on how and why we collect your personal information, please visit our Privacy Policy <https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement.* _______________________________________________ rsyslog mailing listhttps://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e=https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e=What's up with rsyslog? Followhttps://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e=NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriadof sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
