Good point David, I hadn't used the -X option but your point is valid so I'll restart my capture. Thanks again for your ongoing recommendations. ;)
*Scott Slattery* *Sr. Systems & Cloud Architect* *Cloud, Compute, Information & Architecture Team* motorolasolutions.com *O: 602.529.8226* *E*: [email protected] On Thu, Mar 25, 2021 at 1:15 PM David Lang <[email protected]> wrote: > you may want to capture with -X so that it decodes it into hex and you can > see > newlines vs linefeeds > > David Lang > > On Thu, 25 Mar 2021, Scott Slattery wrote: > > > Date: Thu, 25 Mar 2021 12:47:43 -0700 > > From: Scott Slattery <[email protected]> > > To: David Lang <[email protected]> > > Cc: Rainer Gerhards <[email protected]>, > > rsyslog-users <[email protected]>, [email protected] > > Subject: Re: [rsyslog] Altering forwarded logfile names > > > > Thanks David, I just grabbed my first capture and am looking at it. I'm > > using TCP vs UDP but your comment makes sense. I'll certainly share with > > you my observations once I have some data to refer to. This has been a > > learning experience. It's also the first time I've ever seen such an > issue > > with rsyslog so it's a bit of a puzzle. > > > > *Scott Slattery* > > > > *Sr. Systems & Cloud Architect* > > > > *Cloud, Compute, Information & Architecture Team* > > > > motorolasolutions.com > > > > *O: 602.529.8226* > > > > *E*: [email protected] > > > > > > > > > > On Thu, Mar 25, 2021 at 12:32 PM David Lang <[email protected]> wrote: > > > >> the rawmsg field in the debugformat output shows exactly what rsyslog is > >> seeing. > >> > >> the reason I asked you to check multiple entries is that if rsyslog does > >> not see > >> the separator (due to either multiple messages in one UDP packet, or > >> missing > >> newlines in a TCP stream) it will combine what are intended as multiple > >> messages > >> together into one message as far as rsyslog is concerned, and only break > >> it into > >> multiple messages when it hits the configured maxmessagesize size. That > >> breaks > >> things at arbitrary points in the message, so as you look at multiple > >> messages, > >> they will not all start the same way. > >> > >> a tcpdump -s0 -A -n (with the appropriate port/ip filter) will also show > >> exactly > >> what's happening over the wire. > >> > >> David Lang > >> > >> On Thu, 25 Mar 2021, Scott Slattery wrote: > >> > >>> Date: Thu, 25 Mar 2021 12:23:03 -0700 > >>> From: Scott Slattery <[email protected]> > >>> To: Rainer Gerhards <[email protected]> > >>> Cc: rsyslog-users <[email protected]>, David Lang < > [email protected] > >>> , > >>> [email protected] > >>> Subject: Re: [rsyslog] Altering forwarded logfile names > >>> > >>> Hey Guys, before I consume more of your time, I'm capturing the packets > >>> from the source device to determine if any of the payload is lost on > its > >>> way to the rsyslog collector. After I analyze further, I'll share my > >>> observations. I'm hoping nothing is getting dropped over the wire. > Thanks > >>> for your patience and help so far. I expect to have additional details > >>> shortly. > >>> > >>> *Scott Slattery* > >>> > >>> *Sr. Systems & Cloud Architect* > >>> > >>> *Cloud, Compute, Information & Architecture Team* > >>> > >>> motorolasolutions.com > >>> > >>> *O: 602.529.8226* > >>> > >>> *E*: [email protected] > >>> > >>> > >>> > >>> > >>> On Thu, Mar 25, 2021 at 12:51 AM Rainer Gerhards < > >> [email protected]> > >>> wrote: > >>> > >>>> Sorry if I didn't catch it, but did you post the configuration? If > >>>> not, it would probably be useful. > >>>> > >>>> Looking at the message properties, this looks like the message was > >>>> received via UDP but had a LF inside it which was intended as a frame > >>>> separator. If so, that is another bug in the vendor's implementation. > >>>> Framing is used for TCP, but not UDP. For UDP it is strictly one > >>>> message per package. > >>>> > >>>> HTH, > >>>> Rainer > >>>> > >>>> El mié, 24 mar 2021 a las 22:23, Scott Slattery via rsyslog > >>>> (<[email protected]>) escribió: > >>>>> > >>>>> From what I can discern since there are lots of transactions, there > >> seem > >>>> to > >>>>> be multiple payload formats. For all that I have so far analyzed, the > >>>> date > >>>>> truncation issue is the same regardless of the payload received. Like > >>>> you, > >>>>> I've never used mmnormalize but I will look at it as an option. I've > >>>>> reached out to the product support team to better understand if there > >> is > >>>> a > >>>>> appliance-side misconfiguration and something that can be corrected > >> there > >>>>> or if I need to make adjustments on the rsyslog side to clean up the > >>>>> records. > >>>>> > >>>>> What I meant is that there are multiple message formats (payload > data) > >>>> but > >>>>> there seems to be a consistent truncation of the date field just as > you > >>>> saw > >>>>> in the app-name tag field. > >>>>> > >>>>> I appreciate your ongoing feedback and help. This issue is new to me > >> and > >>>>> will need a creative solution if the vendor has no answers. > >>>>> > >>>>> Many thanks, > >>>>> > >>>>> > >>>>> *Scott Slattery* > >>>>> > >>>>> *Sr. Systems & Cloud Architect* > >>>>> > >>>>> *Cloud, Compute, Information & Architecture Team* > >>>>> > >>>>> motorolasolutions.com > >>>>> > >>>>> *O: 602.529.8226* > >>>>> > >>>>> *E*: [email protected] > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> On Wed, Mar 24, 2021 at 2:14 PM David Lang <[email protected]> wrote: > >>>>> > >>>>>> when you say 'each is incorrect but the same format' > >>>>>> > >>>>>> does that mean that every log has the year missing? or that every > log > >>>> is > >>>>>> combining the logs together? > >>>>>> > >>>>>> I'll note that it's possible to define a custom parser using the > >>>>>> mmnormalize > >>>>>> library and add it to the parsing stack. I helped define the need > for > >>>> such > >>>>>> a > >>>>>> capability, but haven't used it yet. > >>>>>> > >>>>>> David Lang > >>>>>> > >>>>>> On Wed, 24 Mar 2021, Scott Slattery wrote: > >>>>>> > >>>>>>> Date: Wed, 24 Mar 2021 08:58:07 -0700 > >>>>>>> From: Scott Slattery <[email protected]> > >>>>>>> To: [email protected] > >>>>>>> Cc: rsyslog-users <[email protected]>, David Lang < > >>>> [email protected] > >>>>>>> > >>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names > >>>>>>> > >>>>>>> Yes, all the logs are coming in this way. There are three separate > >>>> logs > >>>>>>> being sent and each is incorrect in its own right but the same > format > >>>>>>> behavior is consistent across each of the three logs. I will look > >>>> further > >>>>>>> into the suggestions you have provided and have already reached out > >>>> to > >>>>>> the > >>>>>>> appliance support team and will raise the nonconformance with the > >>>> RFC. > >>>>>>> > >>>>>>> This does provide me some valuable guidance and I thank each of you > >>>> for > >>>>>>> your feedback. > >>>>>>> > >>>>>>> *Scott Slattery* > >>>>>>> > >>>>>>> *Sr. Systems & Cloud Architect* > >>>>>>> > >>>>>>> *Cloud, Compute, Information & Architecture Team* > >>>>>>> > >>>>>>> motorolasolutions.com > >>>>>>> > >>>>>>> *O: 602.529.8226* > >>>>>>> > >>>>>>> *E*: [email protected] > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Tue, Mar 23, 2021 at 11:22 PM <[email protected]> > wrote: > >>>>>>> > >>>>>>>> There's something strange with this log entry. Notice that it > looks > >>>> as > >>>>>> if > >>>>>>>> it were more than one log entries joined with LF character > (escaped > >>>> on > >>>>>>>> input as #012). And all "further" entries start with a - still > >>>>>>>> nonconformant to the RFC but "fuller" - timestamp containing > >>>> year.It's > >>>>>> the > >>>>>>>> leading event that seems to be trimmed in the front somehow. Do > all > >>>>>> events > >>>>>>>> arrive like this?Wysłano z telefonu Samsung<div> > >>>>>>>> </div><div> > >>>>>>>> </div><!-- originalMessage --><div>-------- Original message > >>>>>>>> --------</div><div>From: Scott Slattery via rsyslog < > >>>>>>>> [email protected]> </div><div>Date: 24/03/2021 01:52 > >>>>>>>> (GMT+01:00) </div><div>To: David Lang <[email protected]> > >>>> </div><div>Cc: > >>>>>>>> Scott Slattery <[email protected]>, Scott > >>>> Slattery > >>>>>> via > >>>>>>>> rsyslog <[email protected]> </div><div>Subject: Re: > >>>> [rsyslog] > >>>>>>>> Altering forwarded logfile names </div><div> > >>>>>>>> </div> > >>>>>>>> > >>>>>>>> Hi David, fortunately I had already done this. I'm including an > >>>> actual > >>>>>> log > >>>>>>>> entry but have anonymized the data to keep the actual user and > email > >>>>>>>> address confidential: > >>>>>>>> > >>>>>>>> Debug line with all properties: > >>>>>>>> FROMHOST: 'ause1oagatst02.aws.mycompany.com' > >>>>>>>> , fromhost-ip: '10.41.102.143' > >>>>>>>> , HOSTNAME: 'ause1oagatst02.aws.mycompany.com' > >>>>>>>> , PRI: 13, > >>>>>>>> syslogtag '03-23T16:' > >>>>>>>> , programname: '03-23T16' > >>>>>>>> , APP-NAME: '03-23T16' > >>>>>>>> , PROCID: '-' > >>>>>>>> , MSGID: '-' > >>>>>>>> , TIMNESTAMP: 'Mar 23 21:47:13' > >>>>>>>> , STRUCTURED-DATA: '-' > >>>>>>>> , *msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com > >>>>>> ACCESS_GATEWAY > >>>>>>>> ACCESS AUTHZ SESSION INFO USER_SESSION > >>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" > SUBJECT=" > >>>>>>>> [email protected]" APP="Ignio Uat OAG" > >>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN=" > >>>>>>>> apigniodashboard-uat.mycompany.com" > >>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" > >>>>>> REMOTE_IP="10.44.65.38" > >>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) > >>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 > >>>>>> Safari/537.36"] > >>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP: > >>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00 > >>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ > POLICY > >>>>>> INFO > >>>>>>>> USER_AUTHZ > [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" > >>>>>>>> SUBJECT="[email protected]" > RESOURCE="/_dash-dependencies" > >>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" > >>>>>> APP="Ignio > >>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN=" > >>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A - > >>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 > oag_username= > >>>>>>>> [email protected] RelayDomain= > >>>>>> apigniodashboard-uat.mycompany.com > >>>>>>>> [email protected] > >>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport > >>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac > >>>> OS X > >>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) > Chrome/88.0.4324.192 > >>>>>>>> Safari/537.36 creationTime=1616536016811 > maxInactiveInterval=3600000 > >>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 " > >>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel > >>>> Mac > >>>>>> OS X > >>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) > Chrome/88.0.4324.192 > >>>>>>>> Safari/537.36"] allow access to > >>>>>> resource#0122021-03-23T16:47:20.711-05:00 > >>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ > >>>> SESSION > >>>>>> INFO > >>>>>>>> USER_SESSION > >>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" > >>>>>>>> SUBJECT="Joe.User@mycompan' > >>>>>>>> *escaped msg*: '47:20.708-05:00 AUSE1OAGATST02.aws.mycompany.com > >>>>>>>> ACCESS_GATEWAY ACCESS AUTHZ SESSION INFO USER_SESSION > >>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" > SUBJECT=" > >>>>>>>> [email protected]" APP="Ignio Uat OAG" > >>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN=" > >>>>>>>> apigniodashboard-uat.mycompany.com" > >>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" > >>>>>> REMOTE_IP="10.44.65.38" > >>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) > >>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 > >>>>>> Safari/537.36"] > >>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP: > >>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00 > >>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ > POLICY > >>>>>> INFO > >>>>>>>> USER_AUTHZ > [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" > >>>>>>>> SUBJECT="[email protected]" > RESOURCE="/_dash-dependencies" > >>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" > >>>>>> APP="Ignio > >>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN=" > >>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A - > >>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 > oag_username= > >>>>>>>> [email protected] RelayDomain= > >>>>>> apigniodashboard-uat.mycompany.com > >>>>>>>> [email protected] > >>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport > >>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac > >>>> OS X > >>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) > Chrome/88.0.4324.192 > >>>>>>>> Safari/537.36 creationTime=1616536016811 > maxInactiveInterval=3600000 > >>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 " > >>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel > >>>> Mac > >>>>>> OS X > >>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) > Chrome/88.0.4324.192 > >>>>>>>> Safari/537.36"] allow access to > >>>>>> resource#0122021-03-23T16:47:20.711-05:00 > >>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ > >>>> SESSION > >>>>>> INFO > >>>>>>>> USER_SESSION > >>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" > >>>>>>>> SUBJECT="Joe.User@mycompan' > >>>>>>>> *rawmsg*: '03-23T16:47:20.708-05:00 > >>>> AUSE1OAGATST02.aws.mycompany.com > >>>>>>>> ACCESS_GATEWAY *ACCESS* AUTHZ SESSION INFO USER_SESSION > >>>>>>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" > SUBJECT=" > >>>>>>>> [email protected]" APP="Ignio Uat OAG" > >>>>>>>> APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN=" > >>>>>>>> apigniodashboard-uat.mycompany.com" > >>>>>>>> RESULT="ALLOW" REASON="SESSION_INTEGRITY_VERIFIED" > >>>>>> REMOTE_IP="10.44.65.38" > >>>>>>>> USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) > >>>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 > >>>>>> Safari/537.36"] > >>>>>>>> SRF Request RemoteIP: 10.44.65.38 verified session RemoteIP: > >>>>>>>> 10.44.65.38#0122021-03-23T16:47:20.708-05:00 > >>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ > POLICY > >>>>>> INFO > >>>>>>>> USER_AUTHZ > [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" > >>>>>>>> SUBJECT="[email protected]" > RESOURCE="/_dash-dependencies" > >>>>>>>> METHOD="GET" POLICY="root" POLICY_TYPE="PROTECTED" DURATION="0" > >>>>>> APP="Ignio > >>>>>>>> Uat OAG" APP_TYPE="HEADERWEB2015_APP" APP_DOMAIN=" > >>>>>>>> apigniodashboard-uat.mycompany.com" RESULT="ALLOW" REASON="N/A - > >>>>>>>> SESSIONID=_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0 > oag_username= > >>>>>>>> [email protected] RelayDomain= > >>>>>> apigniodashboard-uat.mycompany.com > >>>>>>>> [email protected] > >>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > SourceAuthNType=urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport > >>>>>>>> RemoteIP=10.44.65.38 USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac > >>>> OS X > >>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) > Chrome/88.0.4324.192 > >>>>>>>> Safari/537.36 creationTime=1616536016811 > maxInactiveInterval=3600000 > >>>>>>>> maxActiveInterval=28800000 lastAccessedTime=1616536029221 " > >>>>>>>> REMOTE_IP="10.44.65.38" USER_AGENT="Mozilla/5.0 (Macintosh; Intel > >>>> Mac > >>>>>> OS X > >>>>>>>> 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) > Chrome/88.0.4324.192 > >>>>>>>> Safari/537.36"] allow access to > >>>>>> resource#0122021-03-23T16:47:20.711-05:00 > >>>>>>>> AUSE1OAGATST02.aws.mycompany.com ACCESS_GATEWAY ACCESS AUTHZ > >>>> SESSION > >>>>>> INFO > >>>>>>>> USER_SESSION > >>>> [SESSION_ID="_eddf580f5a5b9436ea27e5e6fa4515a2175ca69ac0" > >>>>>>>> SUBJECT="Joe.User@mycompan' > >>>>>>>> > >>>>>>>> > >>>>>>>> By inspecting the rawmsg, I can see that field four > >>>> (space-delimited) > >>>>>>>> indicates this is the ACCESS log. So if I were able to extract the > >>>> log > >>>>>>>> identifier from the msg, I could then write all access logs to the > >>>> same > >>>>>>>> daily file. There are other formats as well from the same device > >>>> but the > >>>>>>>> idea is the same. > >>>>>>>> > >>>>>>>> *Scott Slattery* > >>>>>>>> > >>>>>>>> *Sr. Systems & Cloud Architect* > >>>>>>>> > >>>>>>>> *Cloud, Compute, Information & Architecture Team* > >>>>>>>> > >>>>>>>> motorolasolutions.com > >>>>>>>> > >>>>>>>> *O: 602.529.8226* > >>>>>>>> > >>>>>>>> *E*: [email protected] > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On Tue, Mar 23, 2021 at 5:29 PM David Lang <[email protected]> wrote: > >>>>>>>> > >>>>>>>>> the source logfile name is not included in the payload by the > >>>> syslog > >>>>>>>> spec. > >>>>>>>>> It > >>>>>>>>> may be in the case of your appliance, but we would need to see a > >>>> sample > >>>>>>>>> log to > >>>>>>>>> understand ho to parse it. > >>>>>>>>> > >>>>>>>>> based on your template, you are using app-name, which may be > listed > >>>>>>>>> separtely if > >>>>>>>>> it's a RFC5424 format log, or may be part of the syslog tag if > >>>> it's a > >>>>>>>>> RFC3164 > >>>>>>>>> format log over the wire (neither format has a way to specify a > >>>> source > >>>>>>>> log > >>>>>>>>> file > >>>>>>>>> by default) > >>>>>>>>> > >>>>>>>>> you can look at > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Chojins_LinuxCNC-2DPolargraph&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=XRQ9OP8K-KeJO3-s6-unMBIRqEZONzs6npmrQYaXnds&e= > >>>>>>>>> and see the *-cc > >>>>>>>>> options that you could apply to the app-name to eliminate control > >>>>>>>>> characters. > >>>>>>>>> > >>>>>>>>> Again, we really need to see the original log message to > understand > >>>>>>>> what's > >>>>>>>>> what. > >>>>>>>>> Please log it with the templateRSYSLOG_DebugFormat so we can see > >>>>>> exactly > >>>>>>>>> what is > >>>>>>>>> sent over the wire and how rsyslog has parsed it. > >>>>>>>>> > >>>>>>>>> David Lang > >>>>>>>>> > >>>>>>>>> On Tue, 23 Mar 2021, Scott Slattery via rsyslog > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Date: Tue, 23 Mar 2021 16:05:45 -0700 > >>>>>>>>>> From: Scott Slattery via rsyslog <[email protected]> > >>>>>>>>>> To: John Chivian <[email protected]> > >>>>>>>>>> Cc: Scott Slattery <[email protected]>, > >>>>>>>>>> rsyslog-users <[email protected]> > >>>>>>>>>> Subject: Re: [rsyslog] Altering forwarded logfile names > >>>>>>>>>> > >>>>>>>>>> Thanks, John, let me try to clarify what I mean. > >>>>>>>>>> > >>>>>>>>>> Normally when I forward from a remote server to the central log > >>>>>>>> server, I > >>>>>>>>>> can include a tag that can then be used to determine the file > >>>> name I > >>>>>>>> want > >>>>>>>>>> on the central server. Since I have no real way to include this > >>>> tag > >>>>>>>> from > >>>>>>>>>> the appliance, this is not an option. > >>>>>>>>>> > >>>>>>>>>> I'm looking for a way of inspecting the incoming packets to > >>>>>> determining > >>>>>>>>> the > >>>>>>>>>> source logfile name (which is included in the payload) and use > >>>> that > >>>>>>>>>> filename on the target central server. Since there are multiple > >>>> logs > >>>>>>>>> being > >>>>>>>>>> sent (access, audit, monitor, etc.), I'd like to segregate these > >>>> into > >>>>>>>>> their > >>>>>>>>>> own files. I'm already using a template with the host > information > >>>> to > >>>>>>>>>> dynamically create the file names. I just don't know how I can > go > >>>>>>>> beyond > >>>>>>>>>> this to also include the source logname. > >>>>>>>>>> > >>>>>>>>>> Here's the template I'm using. It works for all other hosts > where > >>>> I > >>>>>> can > >>>>>>>>>> configure the tag but I get garbage names from the appliance. I > >>>> had > >>>>>>>> hoped > >>>>>>>>>> that the appliance included some standard syslog tags but it > >>>> doesn't > >>>>>>>> seem > >>>>>>>>>> so. > >>>>>>>>>> > >>>>>>>>>> template(name="DynRemoteLogFile" type="string" > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > string="/remote/%FROMHOST%-%FROMHOST-IP%/%$year%-%$month%-%$day%-%app-name%.log") > >>>>>>>>>> > >>>>>>>>>> *Scott Slattery* > >>>>>>>>>> > >>>>>>>>>> *Sr. Systems & Cloud Architect* > >>>>>>>>>> > >>>>>>>>>> *Cloud, Compute, Information & Architecture Team* > >>>>>>>>>> > >>>>>>>>>> motorolasolutions.com > >>>>>>>>>> > >>>>>>>>>> *O: 602.529.8226* > >>>>>>>>>> > >>>>>>>>>> *E*: [email protected] > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Tue, Mar 23, 2021 at 3:30 PM John Chivian < > >>>> [email protected]> > >>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Your use of the term “file name” is confusing. When senders > >>>> deliver > >>>>>>>> to > >>>>>>>>>>> rsyslog over the network there is no exchange of files or > >>>> filenames, > >>>>>>>>> only > >>>>>>>>>>> packets of information. Those packets are expected to be in a > >>>> format > >>>>>>>>> that > >>>>>>>>>>> syslog understands such that useful information (header > elements > >>>> and > >>>>>>>>>>> message body) may be parsed from them. If you as the rsyslog > >>>> admin > >>>>>>>>> choose > >>>>>>>>>>> to use some of that header information to compose filenames for > >>>>>> output > >>>>>>>>>>> files, then yes you are sort of at the mercy of the senders > >>>> content > >>>>>>>>>>> (especially if the sender doesn’t follow the syslog rules). > >>>> However, > >>>>>>>>> there > >>>>>>>>>>> are functions in the advanced syntax that can be used to > perform > >>>> the > >>>>>>>>> type > >>>>>>>>>>> of character replacements you’re talking about. > >>>>>>>>>>> > >>>>>>>>>>> It is common practice to use the syslog header/rsyslog property > >>>>>>>> element > >>>>>>>>>>> called “hostname” for just such purposes. Is this what you’re > >>>>>> talking > >>>>>>>>>>> about? You’d have to provide your configuration for real > >>>> analysis, > >>>>>> at > >>>>>>>>>>> least the part you perceive to be responsible for the problem. > >>>>>>>>>>> > >>>>>>>>>>> Regards, > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> On Mar 23, 2021, at 12:35, Scott Slattery via rsyslog < > >>>>>>>>>>> [email protected]> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> I have a configured central log collector using rsyslog. A few > >>>> of > >>>>>>>> the > >>>>>>>>>>>> devices forwarding their logs are appliances that have no > >>>>>>>>> configuration > >>>>>>>>>>>> options other than the IP forwarding address and protocol. I > >>>> cannot > >>>>>>>>>>> control > >>>>>>>>>>>> what file names are being sent. > >>>>>>>>>>>> > >>>>>>>>>>>> Unfortunately, they are sending unintelligible file names with > >>>>>>>>> characters > >>>>>>>>>>>> that normally would be escaped. Is there any way I can control > >>>> or > >>>>>>>>> alter > >>>>>>>>>>> the > >>>>>>>>>>>> incoming file name to normalize it to avoid these odd > >>>> characters? > >>>>>>>>>>>> > >>>>>>>>>>>> For example, could I establish a character map that maps the > >>>>>>>> unallowed > >>>>>>>>>>>> character to something acceptable? > >>>>>>>>>>>> > >>>>>>>>>>>> thanks, > >>>>>>>>>>>> > >>>>>>>>>>>> *Scott Slattery* > >>>>>>>>>>>> > >>>>>>>>>>>> *Sr. Systems & Cloud Architect* > >>>>>>>>>>>> > >>>>>>>>>>>> *Cloud, Compute, Information & Architecture Team* > >>>>>>>>>>>> > >>>>>>>>>>>> motorolasolutions.com > >>>>>>>>>>>> > >>>>>>>>>>>> *O: 602.529.822* > >>>>>>>>>>>> > >>>>>>>>>>>> *E*: [email protected] > >>>>>>>>>>>> > >>>>>>>>>>>> -- > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> *For more information on how and why we collect your personal > >>>>>>>>>>>> information, please visit our Privacy Policy > >>>>>>>>>>>> < > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement > >>>>>>>>>>>> .* > >>>>>>>>>>>> _______________________________________________ > >>>>>>>>>>>> rsyslog mailing list > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=O-radZKC6RhALSGrunmgfnDcUe0FBEzQXlwVMv4rwrk&e= > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=Ujl6rNYsQwlkacdBkNSQI3_ugt9iTahsA2ALpSb1zWA&e= > >>>>>>>>>>>> What's up with rsyslog? Follow > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=F25vuEW_UOr4xhEXRHv4FYzBC10xi8a7L7cY9KDJz-E&s=5gFALcKlKXLfCND69qR14lRU4iA42kMWjsC9PDoIb3Q&e= > >>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED > by > >>>> a > >>>>>>>>> myriad > >>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > >>>> if > >>>>>> you > >>>>>>>>>>> DON'T LIKE THAT. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> *For more information on how and why we collect your personal > >>>>>>>>>> information, please visit our Privacy Policy > >>>>>>>>>> < > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement > >>>>>>>>>> .* > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> rsyslog mailing list > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=4ENTgbqNRL4m9EpaPD487wHPCEOI1UMUrZ6zizJ25HE&e= > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=YboIrpBbwiXlhlR3JZnvNDi2QWxYQqNifb7d8JV6Xn0&e= > >>>>>>>>>> What's up with rsyslog? Follow > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIDaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=N7tTREp8_2WP8ZJPT8UpZsuFedG7Q4cil9vsQoiSiGM&s=gVD-Vwy9VAK7xAHPrmGhwhORXImwEoBcYZZVVG-KbZQ&e= > >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by > a > >>>>>>>> myriad > >>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if > >>>> you > >>>>>>>>> DON'T LIKE THAT. > >>>>>>>> > >>>>>>>> -- > >>>>>>>> > >>>>>>>> > >>>>>>>> *For more information on how and why we collect your personal > >>>>>>>> information, please visit our Privacy Policy > >>>>>>>> < > >>>>>>>> > >>>>>> > >>>> > >> > https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement > >>>>>>>>> .* > >>>>>>>> _______________________________________________ > >>>>>>>> rsyslog mailing list > >>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=XmKOVDZwhbJF7GnsDKu-uoz3xzXXdd_kesx3hMS_RFo&e= > >>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=cga3HVkV7UUOhHIbjCzyyabOX7hrcgro7F2gU2QB284&e= > >>>>>>>> What's up with rsyslog? Follow > >>>>>>>> > >>>>>> > >>>> > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIGaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=SloAZVc0I2seYTg6VejDjZaHBqa6pVo7oFvF4zTWPok&s=sH7WzGFu43DQHvNetrIn1fmGK-hbU7RsZv_t9hcmugw&e= > >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>> myriad > >>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > >>>> you > >>>>>>>> DON'T LIKE THAT. > >>>>>>> > >>>>>>> > >>>>> > >>>>> -- > >>>>> > >>>>> > >>>>> *For more information on how and why we collect your personal > >>>>> information, please visit our Privacy Policy > >>>>> < > >>>> > >> > https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement > >>>>> .* > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> > >>>> > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=7hc9b2gSfvsfQzXvsRVXqGeP4Rz_tJtrykpjqkGBIdk&e= > >>>>> > >>>> > >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=bDybhvts9KqI-WEfMHo7O40ulQK5M6Tg4tnKNRRIhE8&e= > >>>>> What's up with rsyslog? Follow > >>>> > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=vid2hj5QTU4sYumJIQ5fK64MsYLQqaSGhVmAC2kqwJQ&s=zPhpcUPlK_CWLiU75_WwjyQGz5mFTiXc1yyBIfppkrI&e= > >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> myriad > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>>> DON'T LIKE THAT. > >>>> > >>> > >>> > > > > -- *For more information on how and why we collect your personal information, please visit our Privacy Policy <https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.* _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
