Re: [rsyslog] sending VERSION in rsyslog message

2020-12-19 Thread David Lang via rsyslog
the timestamp in line 7 is not rfc5424 complint (among other things) David Lang On Sat, 19 Dec 2020, Prasad Koya via rsyslog wrote: Date: Sat, 19 Dec 2020 10:58:07 -0800 From: Prasad Koya via rsyslog To: rsyslog-users Cc: Prasad Koya Subject: [rsyslog] sending VERSION in rsyslog message Hi

Re: [rsyslog] TLS is killing me ;-)

2020-12-22 Thread David Lang via rsyslog
just as a FYI, 8.2010 includeed some pretty significant TLS improvements. I don't think they are related to what you are fighting, but I think you will want to upgrade (at least on the receiver) David Lang On Tue, 22 Dec 2020, Mariusz Kruk via rsyslog wrote: Date: Tue, 22 Dec 2020 12:45:03 +

Re: [rsyslog] TLS is killing me ;-)

2020-12-23 Thread David Lang via rsyslog
No, currently rsyslog only supports using a single cert across everything There are enhancement requests in to expand this, but I don't think any of them cover the problem of accepting multiple certs to facilitate a transition from one cert to another on the other end. David Lang On Wed, 23

Re: [rsyslog] rsyslog and config reload

2020-12-23 Thread David Lang via rsyslog
Currently rsyslog does not support reloading the config without a full stop and start. A large part of this is that the config loads code that changes the internals of rsyslog, and rsyslog has not gone to the effort to make it so that this code can be removed cleanly. There has been some dis

Re: [rsyslog] how to alter "msg" property with function replace()

2020-12-30 Thread David Lang via rsyslog
see https://www.rsyslog.com/doc/v8-stable/rainerscript/variable_property_types.html in the template, instead of msg you would use $.msg or $!msg (whichever you opt to use) and then you would do set $!msg = replace($msg, 'etc', ''); or set $.msg = replace($msg, 'etc', ''); let us know if y

Re: [rsyslog] rsyslog and config reload

2021-01-05 Thread David Lang via rsyslog
note that even if the UDP socket does not get closed, if you go too long without reading from it the OS buffer will fill up and messages will be lost. David Lang On Tue, 5 Jan 2021, Shirisha Dasari via rsyslog wrote: Date: Tue, 5 Jan 2021 04:22:22 -0700 (MST) From: Shirisha Dasari via rsyslog

Re: [rsyslog] Is there an easy way to send a msg to rsyslog via RELP as a Nagios check?

2021-01-11 Thread David Lang via rsyslog
One thing I like to do is to make sure all systems send a message on a frequent basis, then look for the message to not arrive. I like to do vmstat -t vmstat 60 or similar as it produces a very dense data set that's useful for after-the-fact investigation of problems David Lang

Re: [rsyslog] myhostname property

2021-01-12 Thread David Lang via rsyslog
On Tue, 12 Jan 2021, John Chivian via rsyslog wrote: I admit I am scratching my head on this one, any help is appreciated. This works in an output template… property(name=“$myhostname”) …but this does not work in a set statement… set $!thishost = $myhostname; try set $!thishost = $$my

Re: [rsyslog] Is there an easy way to send a msg to rsyslog via RELP as a Nagios check?

2021-01-13 Thread David Lang via rsyslog
meout never fires. If they stop arriving, the timeout fires and generates an alert. David Lang I think I've used the vmstat command before, but don't know a lot about it. I'll look more into it. Thanks for the suggestion. -Original Message- From: rsyslog On Beha

Re: [rsyslog] logging remote clients based on facility

2021-01-18 Thread David Lang via rsyslog
log with the template RSYSLOG_DebugFormat so you can see what's in each variable. I think that will answer your question (I don't think it's what you think it is) David Lang On Mon, 18 Jan 2021, Alex via rsyslog wrote: Date: Mon, 18 Jan 2021 21:52:28 -0500 From: Alex via rsyslog To: rsyslo

Re: [rsyslog] Processing one syslog message in multiple rulesets in parallel

2021-01-19 Thread David Lang via rsyslog
On Tue, 19 Jan 2021, Matthias Sitte via rsyslog wrote: This brings me to my questions: Is there a way to process a single syslog messages in multiple rulesets (action chains) in parallel without affecting each other? Is "call()" the right way to go, like use "call rs_siem; call rs_rtmon" insid

Re: [rsyslog] Processing one syslog message in multiple rulesets in parallel

2021-01-19 Thread David Lang via rsyslog
On Tue, 19 Jan 2021, John Chivian via rsyslog wrote: I have tended to put the queues on the enclosed actions rather than on the rulesets, especially in situations where multiple outbound actions within a ruleset might block independently of each other, but yes… the queue (somewhere) is requir

Re: [rsyslog] Processing one syslog message in multiple rulesets in parallel

2021-01-20 Thread David Lang via rsyslog
On Wed, 20 Jan 2021, Mariusz Kruk via rsyslog wrote: Question is - will the behaviour change if those get_source and output_action rulesets get their own queues? Yes, when you introduce a queue, the message gets duplicated into that queue and any changes to the original message won't apply to

Re: [rsyslog] sparseArray documentation

2021-02-02 Thread David Lang via rsyslog
it should be the largest index that is less than or equal to the key you are looking up. The use case in mind for this feature was the maxmind GeoIP database, which converts the IP addresses into 32bit integers and lists the first IP of each block so when you lookup an IP address, convert it

Re: [rsyslog] JSON extraction

2021-02-04 Thread David Lang via rsyslog
mmjsonparse is a pretty specialized use case, I would suggest that you look at mmnormalize, particularly the ability to parse variables. note that the quotes would need escaping with your example, what you posted isn't actually valid JSON. David Lang __

Re: [rsyslog] JSON extraction

2021-02-04 Thread David Lang via rsyslog
t":"venus", "msg": {"name":"bill", "address":"123 Anywhere", "City":"Cleveland", "State":"Ohio", "zip":"12345”}} …so that at least the structure is valid. If the intent is tru

Re: [rsyslog] rule assistance

2021-02-08 Thread David Lang via rsyslog
On Thu, 28 Jan 2021, Jason Prouty via rsyslog wrote: Not sure how to address this in a rule file the field for srcintf sometimes sent as srcintf=unknown-0 other times it is srcintf="rootprivate0" one has quotes the other does not. three options 1. make two rules, one for each variation 2.

Re: [rsyslog] Information about incoming logs.

2021-02-16 Thread David Lang via rsyslog
take a look at the impstats module, and the ability for the imtcp module to record per-sender stats if that isn't enough, you can create your own stats via the dyn_stats() functions David Lang On Tue, 16 Feb 2021, odrzen via rsyslog wrote: Date: Tue, 16 Feb 2021 23:04:40 + From: odrzen

Re: [rsyslog] rsyslog filters.

2021-02-17 Thread David Lang via rsyslog
change to the new if ... then syntax, it lets you do complex expressions. there is no reason to write any new filters using the old syntax. There was a time when the old syntax was faster, but that went away many years ago. David Lang On Wed, 17 Feb 2021, ashutosh via rsyslog wrote: Date: W

Re: [rsyslog] Information about incoming logs.

2021-02-18 Thread David Lang via rsyslog
I thought there was an option to have the input module keep per-sender stats, but I'm not finding it now. with dyn_stats() see https://www.rsyslog.com/doc/master/configuration/dyn_stats.html you create a stats type with the dyn_stats() call and you count with the dyn_inc() call stats inerv

Re: [rsyslog] Information about current syslog server

2021-02-18 Thread David Lang via rsyslog
see https://www.rsyslog.com/doc/v8-stable/configuration/modules/impstats.html also see https://www.rsyslog.com/doc/master/configuration/dyn_stats.html David Lang On Thu, 18 Feb 2021, odrzen via rsyslog wrote: Date: Thu, 18 Feb 2021 23:27:04 + From: odrzen via rsyslog Reply-To: odrzen ,

Re: [rsyslog] Weird issue not able to log to file or remote

2021-02-22 Thread David Lang via rsyslog
the 'typical' cause of this is iptables rules blocking access if you can't send logs to the target on port 1516, the queue will build enable impstats and see what the queue looks like (it helps to add name='foo' to the actions so the impstats output has the names rather than just action #) Da

Re: [rsyslog] Weird issue not able to log to file or remote

2021-02-22 Thread David Lang via rsyslog
the order of the actions so that the file write is first, then see what you get. But yes, enable impstats… it’s WELL worth the effort! Regards, On Feb 22, 2021, at 13:25, David Lang via rsyslog wrote: the 'typical' cause of this is iptables rules blocking access if you can't s

Re: [rsyslog] Dynamic subject text

2021-02-23 Thread David Lang via rsyslog
$.str = re_extract($msg, ..) is not valid syntax set $.str = re_extract($msg, ..); would be after you fix the syntax, if you are still having problems, after the set, log using the RSYSLOG_DebugFormt and that will write 10-line dump of just about everything that rsyslog knows about the log

Re: [rsyslog] Message in mailbody is not complete

2021-02-23 Thread David Lang via rsyslog
whenever you have a case where the content is not what you expect, log the message using the template RSYSLOG_DebugFormat so you can see how rsyslog understandsthe message. I'll bet that it's not getting parsed the way you are epecting. David Lang On Tue, 23 Feb 2021, HERCEK, Marián via rsys

Re: [rsyslog] forward raw syslog

2021-02-24 Thread David Lang via rsyslog
see https://www.rsyslog.com/doc/v8-stable/configuration/templates.html you can create a template to write the message in whatever format you need. an example is documented at https://stackoverflow.com/questions/37023745/rsyslog-config-to-store-only-raw-message-msg-to-a-file (in your case you ma

Re: [rsyslog] The right way to include more log files?

2021-02-24 Thread David Lang via rsyslog
you can tie a ruleset to a particular input, so all messages that arrive via that input only see the rules in that ruleset. to change the action 2 to something meaningful, add name='foo' in the action() statement and you will then get foo instead of 'action 2' in the pstats output pstats only

Re: [rsyslog] The right way to include more log files?

2021-02-24 Thread David Lang via rsyslog
On Wed, 24 Feb 2021, odrzen wrote: I am more wondering to understand what is the right way and why to use rulestes, actions or quests and especially in this case to send particular log files to a central rsyslog. There are a few reasons to use a ruleset 1. when you have an input (say a netw

Re: [rsyslog] The right way to include more log files?

2021-03-02 Thread David Lang via rsyslog
ruleseets only apply inside the instance or rsyslog that is running them. Once you send the logs to a new machine, you now have a separate problem. How will you identify the log you want to treat differently. If they are arriving via the same port as other logs, this can be difficult. I deal w

Re: [rsyslog] another n00b question about logging clients logs to their own directories based on hostnames

2021-03-03 Thread David Lang via rsyslog
log some of the messages with the template RSYSLOG_DebugFormat so we can see the log and how it's parsed, that should give us the info we need to figure out what's happening. David Lang On Wed, 3 Mar 2021, linksonice via rsyslog wrote: Date: Wed, 3 Mar 2021 11:21:38 -0700 (MST) From: linkso

Re: [rsyslog] another n00b question about logging clients logs to their own directories based on hostnames

2021-03-03 Thread David Lang via rsyslog
no, just something like /var/log/debugfile;RSYSLOG_DebugFormat (write a file and use the built in template) just adding a template to the file doesn't change any output. David Lang On Wed, 3 Mar 2021, linksonice via rsyslog wrote: Date: Wed, 3 Mar 2021 15:40:59 -0700 (MST) From: linksonice

Re: [rsyslog] The right way to include more log files?

2021-03-03 Thread David Lang via rsyslog
On Wed, 3 Mar 2021, odrzen wrote: Yes you understand my problem. And I understand that in the end, the problem is mainly in the central machine where it receives all these messages. From the remote machines, I have (probably) correctly defined the way they send their logs to the central machi

Re: [rsyslog] another n00b question about logging clients logs to their own directories based on hostnames

2021-03-04 Thread David Lang via rsyslog
your template is called REMOTEHOST so try logging with ?REMOTEHOST and see what happens (looking back, your prior message that lists the config line has a trailing /) the hostname of these two lines are different, so they should get written into two different directories (assuming permissio

Re: [rsyslog] another n00b question about logging clients logs to their own directories based on hostnames

2021-03-05 Thread David Lang via rsyslog
Cyril, there is no content in the messages you are sending. David Lang On Fri, 5 Mar 2021, Cyril Stoll via rsyslog wrote: Date: Fri, 5 Mar 2021 18:55:15 +0100 From: Cyril Stoll via rsyslog To: rsyslog@lists.adiscon.com Cc: cyril.st...@uzh.ch Subject: Re: [rsyslog] another n00b question about

Re: [rsyslog] another n00b question about logging clients logs to their own directories based on hostnames

2021-03-05 Thread David Lang via rsyslog
the exact same syntax that worked prior to v8 should still work. in the new action() format, you need to set the dynafile template to hve things like the date work (like they did with the prior one) what files are created when you use the remotehost template? David Lang On Fri, 5 Mar 2021, li

Re: [rsyslog] another n00b question about logging clients logs to their own directories based on hostnames

2021-03-08 Thread David Lang via rsyslog
are you sure you have permission to write there? and double check that there are no syntax errors (rsyslogd -N1) if you have include files, it may help to start rsyslog with -o /path/to/file to have rsyslog dump the config file as it understands it. you should be getting something in those di

Re: [rsyslog] Does imfile retry after getting permission denied?

2021-03-08 Thread David Lang via rsyslog
a HUP won't cause it to retry? David Lang On Tue, 9 Mar 2021, Rainer Gerhards via rsyslog wrote: Yes, the ignore is kind of permanent. However, even in inotify mode relatively recent (since 2 yrs ago?) versions of rsyslog also do some polling-like behaviour, so it should recover eventually. B

Re: [rsyslog] rsyslog .deb error

2021-03-10 Thread David Lang via rsyslog
what error do you get? Without information we can't know what's happening. does it start if you start it manually? David Lang On Wed, 10 Mar 2021, Milad Rezaei via rsyslog wrote: Hi I installed .deb package version 8.2010.0-1_amd64 on my Debian 10 and another ubuntu server and Rsyslog server

Re: [rsyslog] another n00b question about logging clients logs to their own directories based on hostnames

2021-03-10 Thread David Lang via rsyslog
there should be nothing special in the name (unless you managed to conflict with a built-in name), can you try the old name with this working format? David Lang On Wed, 10 Mar 2021, linksonice via rsyslog wrote: Date: Wed, 10 Mar 2021 11:14:25 -0700 (MST) From: linksonice via rsyslog To: rsy

Re: [rsyslog] another n00b question about logging clients logs to their own directories based on hostnames

2021-03-11 Thread David Lang via rsyslog
the problem is figuring out how to determine what is a windows sender and what isn't. What are you using to send the logs from the windows machines? The best option is probably to have that software tag the logs with something you can test for. David Lang On Thu, 11 Mar 2021, linksonice via

Re: [rsyslog] another n00b question about logging clients logs to their own directories based on hostnames

2021-03-11 Thread David Lang via rsyslog
everything that worked pre v6 should still be work. David Lang On Thu, 11 Mar 2021, linksonice via rsyslog wrote: Date: Thu, 11 Mar 2021 11:39:41 -0700 (MST) From: linksonice via rsyslog To: rsyslog@lists.adiscon.com Cc: linksonice Subject: Re: [rsyslog] another n00b question about logging c

Re: [rsyslog] Ruleset on IMTCP (or IMPTCP)

2021-03-16 Thread David Lang via rsyslog
yes, you cn create a ruleset for an input and then have it contain a rule for a specific host, or you can call a ruleset after an if statement I don't understand fully what you are trying to do that makes this hard. do you have an example of a config that you think should work that doesn't? Da

Re: [rsyslog] Ruleset on IMTCP (or IMPTCP)

2021-03-16 Thread David Lang via rsyslog
so you have it configured that anything that arrives over port 10514 is processed by the ruleset "DynRemoteServer" which consists of the configuration: action(type="omfile" dynaFile="RemoteServer") after which it is done processing logs. So if you want to have any if statements about the s

Re: [rsyslog] Logging issue using facility local3

2021-03-19 Thread David Lang via rsyslog
8.24 is about a 4 year old version (with some backported fixes by RedHat) that is no longer supported by the community. Please try a more up to date vesion That said, I have never seen the 19.* format used before, so I'm surprised that it works. I would suggest logging the problem messages wi

Re: [rsyslog] Altering forwarded logfile names

2021-03-23 Thread David Lang via rsyslog
can you show us a sample log? Rsyslog does have template options that will restrict non-printable characters (converting them to \xxx octal notation if you want) but we would need to see your config to have any way to figure out what's going on, and a sample of a bad log message (written with

Re: [rsyslog] Altering forwarded logfile names

2021-03-23 Thread David Lang via rsyslog
the source logfile name is not included in the payload by the syslog spec. It may be in the case of your appliance, but we would need to see a sample log to understand ho to parse it. based on your template, you are using app-name, which may be listed separtely if it's a RFC5424 format log, or

Re: [rsyslog] Altering forwarded logfile names

2021-03-23 Thread David Lang via rsyslog
Ok, the problem here is that it's sending a completly invalid timestamp, which is why it's showing up with the syslogtag/app-name as the beginning of the timestamp. I would start off by filing a ticket with the appliance vendor complaining that they are not following the syslog RFCs (either th

Re: [rsyslog] Altering forwarded logfile names

2021-03-24 Thread David Lang via rsyslog
if they are sending logs separated by lf instead of newline, then when they hit maxmessagesize bytes, it will truncate one log and start the next log in the middle of the message (which in this particular message may luck out to only truncate the year) look at the rawmsg field for several mess

Re: [rsyslog] Altering forwarded logfile names

2021-03-24 Thread David Lang via rsyslog
when you say 'each is incorrect but the same format' does that mean that every log has the year missing? or that every log is combining the logs together? I'll note that it's possible to define a custom parser using the mmnormalize library and add it to the parsing stack. I helped define the

Re: [rsyslog] Altering forwarded logfile names

2021-03-24 Thread David Lang via rsyslog
to clarify, I have a lot of mmnormalize experience, what I don't have is experience using the option to use mmnormalize to receive the message the other issue where the message separator seems to be linefeed insted of newline and is causing messages to run together. David Lang On Wed, 24 Mar

Re: [rsyslog] Altering forwarded logfile names

2021-03-25 Thread David Lang via rsyslog
the rawmsg field in the debugformat output shows exactly what rsyslog is seeing. the reason I asked you to check multiple entries is that if rsyslog does not see the separator (due to either multiple messages in one UDP packet, or missing newlines in a TCP stream) it will combine what are inten

Re: [rsyslog] Altering forwarded logfile names

2021-03-25 Thread David Lang via rsyslog
you may want to capture with -X so that it decodes it into hex and you can see newlines vs linefeeds David Lang On Thu, 25 Mar 2021, Scott Slattery wrote: Date: Thu, 25 Mar 2021 12:47:43 -0700 From: Scott Slattery To: David Lang Cc: Rainer Gerhards , rsyslog-users , mariusz.k...@safecom

Re: [rsyslog] Altering forwarded logfile names

2021-03-25 Thread David Lang via rsyslog
what version of rsyslog are you running. can you post your full config? if you are receiving via TCP and it's not splitting the logs based on newlines, something very odd is happening. David Lang On Thu, 25 Mar 2021, Scott Slattery wrote: Date: Thu, 25 Mar 2021 15:07:22 -0700 From: Scott S

Re: [rsyslog] Altering forwarded logfile names

2021-03-25 Thread David Lang via rsyslog
I don't think this is the problem, but 8.24 is 4+ years old now (releaseed Jan 2017, although the version maintained by RedHat and Amazon has some bugfixes backported), can you try with a current version (8.2102 or 8.2012) and see if you still have the problem. I don't remember of hearing about

Re: [rsyslog] Altering forwarded logfile names

2021-03-26 Thread David Lang via rsyslog
ahh, I missed that. A bit longer explination: before the date, the message should start out with '' or '1' depending on which RFC it's following (from the format of the date, this should have the 1 as required by RFC5424) without that it's triggering the non-standard extension that rsyslog sup

Re: [rsyslog] forward to 2nd server

2021-04-01 Thread David Lang via rsyslog
what you should be doing is look at the hostname in the message rather than the IP that the message appears to come from. As you have seen the IP is affected by relays, and it's also affected by NAT. omudpspoof can help, but it's a very substantial performance hit David Lang On Thu, 1 Apr 20

Re: [rsyslog] module error

2021-04-04 Thread David Lang via rsyslog
it's probably packaged as a separate package for your distro. David Lang On Sun, 4 Apr 2021, Milad Rezaei via rsyslog wrote: Date: Sun, 4 Apr 2021 14:29:44 +0430 From: Milad Rezaei via rsyslog To: rsyslog@lists.adiscon.com Cc: Milad Rezaei Subject: [rsyslog] module error Hi Dears I want to

Re: [rsyslog] rsyslog fails to collect FW traffic logs

2021-04-12 Thread David Lang via rsyslog
check your iptables rules David Lang On Mon, 12 Apr 2021, Erik.Moritz--- via rsyslog wrote: Date: Mon, 12 Apr 2021 11:47:16 + From: Erik.Moritz--- via rsyslog To: rsyslog@lists.adiscon.com Cc: erik.mor...@t-systems.com Subject: Re: [rsyslog] rsyslog fails to collect FW traffic logs Hello

Re: [rsyslog] messages are truncated after "-"

2021-04-25 Thread David Lang via rsyslog
please log a message with the template RSYSLOG_DebugFormat so we can see the raw message and how it's parsed. David Lang On Mon, 26 Apr 2021, ca--- via rsyslog wrote: Date: Mon, 26 Apr 2021 11:07:07 +0900 From: ca--- via rsyslog To: rsyslog-users Cc: c...@a-1.win Subject: [rsyslog] messages

Re: [rsyslog] messages are truncated after "-"

2021-04-26 Thread David Lang via rsyslog
you cut out the parts that we need to see to understand what's happening. we need the rawmsg field that you trimmed off. David Lang On Mon, 26 Apr 2021, c...@a-1.win wrote: Date: Mon, 26 Apr 2021 16:49:06 +0900 From: c...@a-1.win To: David Lang Cc: ca--- via rsyslog , c...@a-1.win Subject: R

Re: [rsyslog] Structured Data in Windows Event

2021-05-04 Thread David Lang via rsyslog
what software are you using to send the windows event data? can you show us an example of a log that's not working? (what the rawmsg looks like) David Lang On Tue, 4 May 2021, James Ward-Smith via rsyslog wrote: Hi, We are using a custom syslog header to parse Windows Events into syslog fo

Re: [rsyslog] Structured Data in Windows Event

2021-05-04 Thread David Lang via rsyslog
what is the config that sets the structured data? David Lang On Wed, 5 May 2021, James Ward-Smith wrote: Date: Wed, 5 May 2021 00:18:42 + From: James Ward-Smith To: David Lang Cc: James Ward-Smith via rsyslog Subject: Re: [rsyslog] Structured Data in Windows Event Hi, We are have got

Re: [rsyslog] Structured Data in Windows Event

2021-05-04 Thread David Lang via rsyslog
could you write the lot message on the windows machine with the template RSYSLOG_DebugFormat so that we can see what all the variables are and their contents? you can't set the default properties, you would need to set a variable like $!structured_data and use that in the template. but it's

Re: [rsyslog] Multiple mmnormalize calls on same message

2021-06-02 Thread David Lang via rsyslog
I have used multiple mmnormalize calls on the same message without a problem. That being said, if you can combine your rulesets, it's going to be more effiecient than having to make multiple calls. I believe the note in the docs is out of date. David Lang On Thu, 3 Jun 2021, rajeshksv via rs

Re: [rsyslog] impstats _sender_stat messages reported as string, not integer

2021-06-09 Thread David Lang via rsyslog
normally I am very strict on not breaking backwards compatibility, but I think this is a case where consumers are rare, and a large number of them aren't going to care if it's a sting or integer to start with, so I think it's better to fix it and include a warning in the release notes for the ne

Re: [rsyslog] rsyslog showing error "gnutls returned error on handshake: Could not negotiate a supported cipher suite. [v8.24.0 try http://www.rsyslog.com/e/2083 ]" for DHE-RSA cipher suites

2021-06-10 Thread David Lang via rsyslog
8.24 is a very old version at this point, and there has been a lot of work in this area in the last year. you are probably actually running the redhat fork of 8.24 with some backports added. please either contact RedHat for support of your forked version up update to a current (8.21xx) version

Re: [rsyslog] Odd parse_json Behavior With imfile Input

2021-06-17 Thread David Lang via rsyslog
whenever you have a parsing problem, log the messages with the template RSYSLOG_DebugFormat 90%+ of the time, we find that the contents of the variables are not what you are expecting, and the rest of the time, seeing the debug output lets us figure out what is going wrong with the parser Al

Re: [rsyslog] Odd parse_json Behavior With imfile Input

2021-06-17 Thread David Lang via rsyslog
what is msg and rawmsg in these two cases? what you are showing is the output, what we need to look at is the input to be parsed. David Lang On Thu, 17 Jun 2021, John Chivian wrote: Date: Thu, 17 Jun 2021 11:37:14 -0500 From: John Chivian To: David Lang Cc: John Chivian via rsyslog Subje

Re: [rsyslog] Problem with tool logger from sysklogd v2.2.3

2021-06-22 Thread David Lang via rsyslog
what version of rsyslog are you running? it seems odd that it's parsing the programname incorrectly (as '1') everything should just work with the new format. David Lang On Tue, 22 Jun 2021, Ansgar Puester via rsyslog wrote: Date: Tue, 22 Jun 2021 14:01:31 +0200 From: Ansgar Puester via rsysl

Re: [rsyslog] JSON extraction

2021-06-22 Thread David Lang via rsyslog
3 Anywhere", "City":"Cleveland", "State":"Ohio", "zip":"12345”}} …so that at least the structure is valid. If the intent is truly for the msg object to be a string, not a JSON object, then you must escape the quotes within it. I wo

Re: [rsyslog] Send all the logs under non standard location to remote server

2021-06-22 Thread David Lang via rsyslog
To: David Lang via rsyslog Cc: Ren You Subject: [rsyslog] Send all the logs under non standard location to remote server Hi, Can someone let me know how can I update /etc/rsyslog.conf file to send all the logs under /etc/apache2/logs/domlogs/* to remote server? The client is Oracle Linux

Re: [rsyslog] omkafka and custom data serialization

2021-06-24 Thread David Lang via rsyslog
rsyslog has a very powerful engine to let you format your data, but I don't know that it can do protobuf (it's text only), it would not be that hard to create a module that would format a message as protobuf (and Adiscon does professional services development at reasonable costs if you are inter

Re: [rsyslog] formatting the msg part to json

2021-07-02 Thread David Lang via rsyslog
please post a sample log and what you want the result to look like, that will make it much easier to figure out what you are meaning. David Lang On Fri, 2 Jul 2021, Asaf Segal via rsyslog wrote: Date: Fri, 2 Jul 2021 18:18:25 + From: Asaf Segal via rsyslog To: rsyslog@lists.adiscon.com C

Re: [rsyslog] Rsyslog insert DB problem

2021-07-02 Thread David Lang via rsyslog
we would need to see a sample message that did not get written. you may also want to look at the log messages of the database to see if it logged an error. one possible reason would be if the msg field included a quote character in it, but there are logs of other possible reasons. David Lang

Re: [rsyslog] Right place for STOP directive to prevent remote logs written to local messages

2021-07-06 Thread David Lang via rsyslog
paste the included file into your rsyslog.conf so you can see it the way rsyslog sees it (or write a combined file with the -o /path/to/combined/file command line option) then make two copies of the file. In the first one, remove all actions (including the stop), leaving only the config optio

Re: [rsyslog] RHEL / CentOS 8.1 omelasticsearch JSON issues

2021-07-08 Thread David Lang via rsyslog
full configuration please. David Lang On Fri, 9 Jul 2021, Daniel Pocock via rsyslog wrote: Date: Fri, 9 Jul 2021 01:48:07 +0200 From: Daniel Pocock via rsyslog To: rsyslog-users Cc: Daniel Pocock Subject: [rsyslog] RHEL / CentOS 8.1 omelasticsearch JSON issues I installed the omelasticsear

Re: [rsyslog] RHEL / CentOS 8.1 omelasticsearch JSON issues

2021-07-08 Thread David Lang via rsyslog
8.32 is quite a bit older than 8.1911 (2-3 years) Please log a message with the template RSYSLOG_DebugFormat on both so we can see what's different I also don't know what the default rsyslog.conf is on every system, so please include that as well. on 8.1911 you can start rsyslog with the c

Re: [rsyslog] RHEL / CentOS 8.1 omelasticsearch JSON issues

2021-07-08 Thread David Lang via rsyslog
ok, this shows the exact config that rsyslog is seeing, and it shows that the mmjsonparse call is not able to parse the message. I'm not exactly sure why. what is the other rsyslog instance you are using? you said 8.32, but here you mention sending from this config to the 'newer' rsyslog?? D

Re: [rsyslog] RHEL / CentOS 8.1 omelasticsearch JSON issues

2021-07-09 Thread David Lang via rsyslog
over the years the distros have put more effort into minimizing the optional dependencies that get pulled in, so more of the rsyslog modules have become separate packages when they pull in other things. it's nice when it's this easy to fix, but you also got a lesson in troubleshooting. Have

Re: [rsyslog] using Kibana / OpenSearch Dashboards to analyze logs during development

2021-07-09 Thread David Lang via rsyslog
multi-line logs are difficult to handle, it would be far easier on you if you can turn them into single-line logs as early in processing as possible. There is a lot of business analytics value in logs. the 'easy' way is to throw it into Splunk or ElasticSearch and depend on queries there, but t

Re: [rsyslog] using Kibana / OpenSearch Dashboards to analyze logs during development

2021-07-09 Thread David Lang via rsyslog
also, dig into mmnormalize (liblognorm), it's a very efficient parse engine for extracting values out of logs. The Dyn_stats() feature in rsyslog ends up being a rather powerful tool for summarizing things (SEC is more powerful, but you can do a lot with just dyn_stats() ) David Lang On Fri,

Re: [rsyslog] using Kibana / OpenSearch Dashboards to analyze logs during development

2021-07-09 Thread David Lang via rsyslog
here is a paper on the parsing performance https://www.fernuni-hagen.de/rechnerarchitektur/docs/rainer_gerhards.pdf On Fri, 9 Jul 2021, David Lang via rsyslog wrote: Date: Fri, 9 Jul 2021 15:40:19 -0700 (PDT) From: David Lang via rsyslog To: Jim Van Meggelen via rsyslog Cc: David Lang

Re: [rsyslog] CEE field values, ambiguities, sev, syslog!pri, etc

2021-07-10 Thread David Lang via rsyslog
Just a note that in practice, CEE is pretty much dead. Pretty much all that survived is the idea of using JSON to format the data and to use ! to be able to specify multi-level field names. Everything else in CEE should be treated as an idea that may or may not be useful rather than a RFC to be

Re: [rsyslog] CEE field values, ambiguities, sev, syslog!pri, etc

2021-07-10 Thread David Lang via rsyslog
On Sat, 10 Jul 2021, Daniel Pocock via rsyslog wrote: On 10/07/2021 22:38, David Lang wrote: Just a note that in practice, CEE is pretty much dead. Pretty much all that survived is the idea of using JSON to format the data and to use ! to be able to specify multi-level field names. Everything e

Re: [rsyslog] CEE field values, ambiguities, sev, syslog!pri, etc

2021-07-10 Thread David Lang via rsyslog
you may want to look and see if there are archives of the project lumberjack mailing list, that was a wider, slightly less academic, attempt to define a standard after the CEE effort floundered. David Lang On Sat, 10 Jul 2021, David Lang via rsyslog wrote: Date: Sat, 10 Jul 2021 14:22:14

Re: [rsyslog] Segfault after recursion too deeply

2021-07-16 Thread David Lang via rsyslog
a simple test to prevent calling the current ruleset seems like it ould be doable, but when you start talking chains of rulesets, tracking the full depth to prevent more indirect loops may be more difficult (or it may not be, Rainer would need to comment on this) I think it would be reasonable

Re: [rsyslog] Segfault after recursion too deeply

2021-07-19 Thread David Lang via rsyslog
On Sun, 18 Jul 2021, Rainer Gerhards via rsyslog wrote: I think that should be doable without too much trouble. I would appreciate a github issue tracker. Side-Note: I never intended to support general purpose loops, but of course there is always a way to do it ;-) that's why I suggested maki

Re: [rsyslog] Discard filters don't work

2021-07-28 Thread David Lang via rsyslog
you are probably discarding the message after it's been written out. but it's impossible to tell without seeing your full config and knowing hat file you are seeing the message in that you don't want there. if you start rsyslog ith the -o flag (-o /path/to/file) then the file will contain the

Re: [rsyslog] Discard filters don't work

2021-07-29 Thread David Lang via rsyslog
which point do you need me to elaborate? without the configs, I am only going to be able to guess. David Lang On Thu, 29 Jul 2021, Saint Michael wrote: Date: Thu, 29 Jul 2021 10:27:39 -0400 From: Saint Michael To: David Lang Cc: Saint Michael via rsyslog Subject: Re: [rsyslog] Discard fil

Re: [rsyslog] rsyslog imfile missing logs messages

2021-07-29 Thread David Lang via rsyslog
what is your rsyslog config, and how is the file being rotated (is it being truncted and filled again, or deleted and a new file created) do you have a cron job restarting rsyslog every 5 min or anything else like that? David Lang On Thu, 29 Jul 2021, Marcello Lorenzi via rsyslog wrote: Da

Re: [rsyslog] Discard filters don't work

2021-07-29 Thread David Lang via rsyslog
without seeing your configs (remember, I don't manage how RedHat sets their configs, so I don't know what their defaults are), I can only guess. But my guess is that rsyslog reads logs from journald David Lang On Thu, 29 Jul 2021, Saint Michael wrote: Date: Thu, 29 Jul 2021 15:45:37 -0400 Fr

Re: [rsyslog] Help with custom template and field for mysql output

2021-07-29 Thread David Lang via rsyslog
a sample of the log you are trying to parse would help a lot. David Lang On Thu, 29 Jul 2021, Bill Schoolfield via rsyslog wrote: Date: Thu, 29 Jul 2021 17:11:14 -0500 From: Bill Schoolfield via rsyslog To: rsyslog@lists.adiscon.com Cc: Bill Schoolfield Subject: [rsyslog] Help with custom te

Re: [rsyslog] syslogtag: only log daemon name, without the [pid]

2021-08-13 Thread David Lang via rsyslog
change syslogtag to programname (and you may need to add a : after the vriable) David Lang On Sat, 14 Aug 2021, Fourhundred Thecat via rsyslog wrote: Date: Sat, 14 Aug 2021 06:17:23 +0200 From: Fourhundred Thecat via rsyslog To: rsyslog@lists.adiscon.com Cc: Fourhundred Thecat <400the...@gmx.

Re: [rsyslog] rsyslogd HUP and logrotate

2021-08-27 Thread David Lang via rsyslog
signals (like HUP) are sent and processed asynchronously, so it can take a little bit of time to be handled. Your understanding is correct. what I normally do is not use logrotate, but do the rotation myself in a script run from cron that does the mv of the files, sends the hub, then sleeps for

Re: [rsyslog] rsyslogd HUP and logrotate

2021-08-27 Thread David Lang via rsyslog
On Fri, 27 Aug 2021, Jean-Baptiste Denis wrote: So, this is a kind of gray area unless rsyslog offers some kind of blocking mechanism that do the close/reopen and exits when its done. rsyslog could also handle the rotation, but I have only seen a size criteria using outchannel (https://www.rsy

Re: [rsyslog] imfile-state file not returning to zero after logrotate

2021-09-01 Thread David Lang via rsyslog
use postrotate to send a HUP to rsyslog so it closes the file and reopens it truncting a log file creates a race condition that will cause you to lose logs at some point. David Lang On Tue, 31 Aug 2021, John Chivian via rsyslog wrote: Date: Tue, 31 Aug 2021 21:59:58 -0500 From: John Chivian

Re: [rsyslog] imfile-state file not returning to zero after logrotate

2021-09-06 Thread David Lang via rsyslog
try adding nocopytruncate to your logrotate config, you may have some other config that is setting copytruncate as the default. David Lang On Tue, 7 Sep 2021, Andrew via rsyslog wrote: Date: Tue, 7 Sep 2021 07:30:27 +0930 From: Andrew via rsyslog To: Mariusz Kruk via rsyslog Cc: Andrew Sub

Re: [rsyslog] Struggling with the basics - trying to filter on text AND have logs go to /var/log/remote/yadayada

2021-09-08 Thread David Lang via rsyslog
look at the dynafile option for the action() statement (in the omfile page of the docs) David Lang On Thu, 9 Sep 2021, lists--- via rsyslog wrote: Date: Thu, 09 Sep 2021 06:53:42 +0100 From: lists--- via rsyslog To: rsyslog@lists.adiscon.com Cc: li...@kush-t.com Subject: [rsyslog] Struggling

Re: [rsyslog] Struggling with the basics - trying to filter on text AND have logs go to /var/log/remote/yadayada

2021-09-09 Thread David Lang via rsyslog
please post your full config, the example config does not have 101 lines, so it doesn't match the error you are posting. Also be aware that 8.24 is no about 5 years old and unsupported by the community, you are running something unique to redhat. that said, the imptcp module should be availab

Re: [rsyslog] Struggling with the basics - trying to filter on text AND have logs go to /var/log/remote/yadayada

2021-09-09 Thread David Lang via rsyslog
On Thu, 9 Sep 2021, Yuri Bushmelev via rsyslog wrote: Please consider to stop useing the $ThisConfigSyntaxStyle as "it will make your life miserable" (c) Reiner Gerhards .. There is nice new syntax made more than 10 years ago. please explain why template(name="TmplVPXMsg" type="string" string

  1   2   3   4   5   6   7   8   >