David is right on. The “2021-“ is missing from the “03-23T16:47:20.708-05:00”
> On Mar 23, 2021, at 21:14, David Lang <[email protected]> wrote: > > Ok, the problem here is that it's sending a completly invalid timestamp, > which is why it's showing up with the syslogtag/app-name as the beginning of > the timestamp. > > I would start off by filing a ticket with the appliance vendor complaining > that they are not following the syslog RFCs (either the old or the new) > > then what you will need to do is to have a conditional (probably based on > fromhost-ip) to identify these logs and run them through a different parser > (look at mmnormalize for a fast, flexible one, but you may be able to get > away with the field() function if you only need to do one extraction) > > and then to output the logs, you will need a new template that uses the > variables that your parser pulls out of them since you can't use the default > properties. > > David Lang > >> On Tue, 23 Mar 2021, Scott Slattery wrote: >> >> Date: Tue, 23 Mar 2021 17:51:28 -0700 >> From: Scott Slattery <[email protected]> >> To: David Lang <[email protected]> >> Cc: Scott Slattery via rsyslog <[email protected]>, >> John Chivian <[email protected]> >> Subject: Re: [rsyslog] Altering forwarded logfile names >> Hi David, fortunately I had already done this. I'm including an actual log >> entry but have anonymized the data to keep the actual user and email >> address confidential: >> >> Debug line with all properties: >> FROMHOST: 'ause1oagatst02.aws.mycompany.com' >> , fromhost-ip: '10.41.102.143' >> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com' >> , PRI: 13, >> syslogtag '03-23T16:' >> , programname: '03-23T16' >> , APP-NAME: '03-23T16' >> , PROCID: '-' >> , MSGID: '-' >> , TIMNESTAMP: 'Mar 23 21:47:13' >> , STRUCTURED-DATA: '-' >> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY >> ACCESS AUTHZ SESSION INFO USER_SESSION >> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT=" >> [email protected]" APP="Ignio Uat OAG" >> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com" >> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38" >> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) >> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"] >> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP: >> 10.44.65.38#0122021-03-23T16:47:20.708-05:00 >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO >> USER_AUTHZ [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" >> SUBJECT="[email protected]" RESOURCE="/_dash-dependencies" >> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio >> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN=" >> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A - >> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username= >> [email protected] RelayDomain=apigniodashboard-uat.mycompany.com >> [email protected] >> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport >> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 >> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000 >> maxActiveInterval=28800000 lastAccessedTime=1616536029221 " >> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 >> Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00 >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO >> USER_SESSION [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" >> SUBJECT="Joe.User@mycompan' >> *escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com >> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION >> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT=" >> [email protected]" APP="Ignio Uat OAG" >> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com" >> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38" >> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) >> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"] >> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP: >> 10.44.65.38#0122021-03-23T16:47:20.708-05:00 >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO >> USER_AUTHZ [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" >> SUBJECT="[email protected]" RESOURCE="/_dash-dependencies" >> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio >> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN=" >> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A - >> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username= >> [email protected] RelayDomain=apigniodashboard-uat.mycompany.com >> [email protected] >> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport >> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 >> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000 >> maxActiveInterval=28800000 lastAccessedTime=1616536029221 " >> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 >> Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00 >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO >> USER_SESSION [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" >> SUBJECT="Joe.User@mycompan' >> *rawmsg*: '03-23T16:47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com >> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION >> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" SUBJECT=" >> [email protected]" APP="Ignio Uat OAG" >> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN="apigniodashboard-uat.mycompany.com" >> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" REMOTE_IP="10.44.65.38" >> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) >> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"] >> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP: >> 10.44.65.38#0122021-03-23T16:47:20.708-05:00 >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ POLICY INFO >> USER_AUTHZ [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" >> SUBJECT="[email protected]" RESOURCE="/_dash-dependencies" >> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" APP="Ignio >> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN=" >> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A - >> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 oag_username= >> [email protected] RelayDomain=apigniodashboard-uat.mycompany.com >> [email protected] >> SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport >> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 >> Safari/537.36 creationTime=1616536016811 maxInactiveInterval=3600000 >> maxActiveInterval=28800000 lastAccessedTime=1616536029221 " >> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X >> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 >> Safari/537.36"] allow access to resource#0122021-03-23T16:47:20.711-05:00 >> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO >> USER_SESSION [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" >> SUBJECT="Joe.User@mycompan' >> >> >> By inspecting the rawmsg, I can see that field four (space-delimited) >> indicates this is the ACCESS log. So if I were able to extract the log >> identifier from the msg, I could then write all access logs to the same >> daily file. There are other formats as well from the same device but the >> idea is the same. >> >> *Scott Slattery* >> >> *Sr. Systems & Cloud Architect* >> >> *Cloud, Compute, Information & Architecture Team* >> >> motorolasolutions.com >> >> *O: 602.529.8226* >> >> *E*: [email protected] >> >> >> >> >>> On Tue, Mar 23, 2021 at 5:29 PM David Lang <[email protected]> wrote: >>> >>> the source logfile name is not included in the payload by the syslog spec. >>> It >>> may be in the case of your appliance, but we would need to see a sample >>> log to >>> understand ho to parse it. >>> >>> based on your template, you are using app-name, which may be listed >>> separtely if >>> it's a RFC5424 format log, or may be part of the syslog tag if it's a >>> RFC3164 >>> format log over the wire (neither format has a way to specify a source log >>> file >>> by default) >>> >>> you can look at >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e= >>> and see the *-cc >>> options that you could apply to the app-name to eliminate control >>> characters. >>> >>> Again, we really need to see the original log message to understand what's >>> what. >>> Please log it with the templateRSYSLOG_DebugFormat so we can see exactly >>> what is >>> sent over the wire and how rsyslog has parsed it. >>> >>> David Lang >>> >>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog >>> wrote: >>> >>>> Date: Tue, 23 Mar 2021 16:05:45 -0700 >>>> From: Scott Slattery via rsyslog <[email protected]> >>>> To: John Chivian <[email protected]> >>>> Cc: Scott Slattery <[email protected]>, >>>> rsyslog-users <[email protected]> >>>> Subject: Re: [rsyslog] Altering forwarded logfile names >>>> >>>> Thanks, John, let me try to clarify what I mean. >>>> >>>> Normally when I forward from a remote server to the central log server, I >>>> can include a tag that can then be used to determine the file name I want >>>> on the central server. Since I have no real way to include this tag from >>>> the appliance, this is not an option. >>>> >>>> I'm looking for a way of inspecting the incoming packets to determining >>> the >>>> source logfile name (which is included in the payload) and use that >>>> filename on the target central server. Since there are multiple logs >>> being >>>> sent (access, audit, monitor, etc.), I'd like to segregate these into >>> their >>>> own files. I'm already using a template with the host information to >>>> dynamically create the file names. I just don't know how I can go beyond >>>> this to also include the source logname. >>>> >>>> Here's the template I'm using. It works for all other hosts where I can >>>> configure the tag but I get garbage names from the appliance. I had hoped >>>> that the appliance included some standard syslog tags but it doesn't seem >>>> so. >>>> >>>> template(name="DynRemoteLogFile" type="string" >>>> >>> string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log") >>>> >>>> *Scott Slattery* >>>> >>>> *Sr. Systems & Cloud Architect* >>>> >>>> *Cloud, Compute, Information & Architecture Team* >>>> >>>> motorolasolutions.com >>>> >>>> *O: 602.529.8226* >>>> >>>> *E*: [email protected] >>>> >>>> >>>> >>>> >>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian <[email protected]> >>> wrote: >>>> >>>>> Your use of the term “file name” is confusing. When senders deliver to >>>>> rsyslog over the network there is no exchange of files or filenames, >>> only >>>>> packets of information. Those packets are expected to be in a format >>> that >>>>> syslog understands such that useful information (header elements and >>>>> message body) may be parsed from them. If you as the rsyslog admin >>> choose >>>>> to use some of that header information to compose filenames for output >>>>> files, then yes you are sort of at the mercy of the senders content >>>>> (especially if the sender doesn’t follow the syslog rules). However, >>> there >>>>> are functions in the advanced syntax that can be used to perform the >>> type >>>>> of character replacements you’re talking about. >>>>> >>>>> It is common practice to use the syslog header/rsyslog property element >>>>> called “hostname” for just such purposes. Is this what you’re talking >>>>> about? You’d have to provide your configuration for real analysis, at >>>>> least the part you perceive to be responsible for the problem. >>>>> >>>>> Regards, >>>>> >>>>> >>>>> >>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog < >>>>> [email protected]> wrote: >>>>>> >>>>>> I have a configured central log collector using rsyslog. A few of the >>>>>> devices forwarding their logs are appliances that have no >>> configuration >>>>>> options other than the IP forwarding address and protocol. I cannot >>>>> control >>>>>> what file names are being sent. >>>>>> >>>>>> Unfortunately, they are sending unintelligible file names with >>> characters >>>>>> that normally would be escaped. Is there any way I can control or >>> alter >>>>> the >>>>>> incoming file name to normalize it to avoid these odd characters? >>>>>> >>>>>> For example, could I establish a character map that maps the unallowed >>>>>> character to something acceptable? >>>>>> >>>>>> thanks, >>>>>> >>>>>> *Scott Slattery* >>>>>> >>>>>> *Sr. Systems & Cloud Architect* >>>>>> >>>>>> *Cloud, Compute, Information & Architecture Team* >>>>>> >>>>>> motorolasolutions.com >>>>>> >>>>>> *O: 602.529.822* >>>>>> >>>>>> *E*: [email protected] >>>>>> >>>>>> -- >>>>>> >>>>>> >>>>>> *For more information on how and why we collect your personal >>>>>> information, please visit our Privacy Policy >>>>>> < >>>>> >>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement >>>>>> .* >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> >>>>> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e= >>>>>> >>>>> >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e= >>>>>> What's up with rsyslog? Follow >>>>> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e= >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> myriad >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> DON'T LIKE THAT. >>>>> >>>>> >>>> >>>> -- >>>> >>>> >>>> *For more information on how and why we collect your personal >>>> information, please visit our Privacy Policy >>>> < >>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement >>>> .* >>>> _______________________________________________ >>>> rsyslog mailing list >>>> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e= >>>> >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e= >>>> What's up with rsyslog? Follow >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e= >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >> _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
