Re: [rsyslog] imjournal + json templates

2016-04-14 Thread David Lang
may be problems with variable names that start with _ I seem to remember seeing some patches around this and the imjournal module recently. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/pr

Re: [rsyslog] discarding msgs

2016-04-15 Thread David Lang
:-) Also, with rsyslog v7+ the config optimizer makes it so that if..then tests are just as fast as the :var, test ones. so just pick what makes the resulting config the easiest to understand David Lang On Fri, 15 Apr 2016, Stuart Kendrick wrote: Ahhh, OK. So this config line does suppres

Re: [rsyslog] pre-releease of rsyslog 08.18.0 available

2016-04-18 Thread David Lang
Just a note that the cron jobs to generate PPAs are failing due to the cleanup changes that require a PPA variable be defined in the config David Lang On Mon, 18 Apr 2016, Rainer Gerhards wrote: Hi all, I have just placed pre-release tarballs for tomorrow's upcoming rsyslog 8.18.0 re

Re: [rsyslog] Rsyslog exiting on signal 15

2016-04-18 Thread David Lang
it that can cause these sorts of corruptions (this is why libfastjson was forked from it) David Lang On Mon, 18 Apr 2016, Alec Swan wrote: Hi there, I've been seeing the following stack trace in /var/log/messages. It seems like something is trying to shut down rsyslog causing it to leav

Re: [rsyslog] Rsyslog exiting on signal 15

2016-04-18 Thread David Lang
of the OOM handler has changed over time, so it may vary depending on what you are running. David Lang On Tue, 19 Apr 2016, singh.janmejay wrote: Won't oom be a sigkill? You can determine which process issued term (syscall is kill). Match it with time of death to find the culprit. On Apr

[rsyslog] is there a way to convert a unix time value to a human readable string?

2016-04-19 Thread David Lang
I know that there is the ability to format a timestamp as a unix time value, but is there a way to take a variable that contains the unix time value and output it as a human readable string? i.e. if $!foo = "1454527741918" output "Feb 23 11:51:

Re: [rsyslog] segfault with rsyslog Version > 8.16.0

2016-04-20 Thread David Lang
tjson, and that fixes quite a number of known segfaulting issues. David Lang On Wed, 20 Apr 2016, Christopher Racky wrote: Date: Wed, 20 Apr 2016 14:18:25 +0200 From: Christopher Racky Reply-To: rsyslog-users To: rsyslog-users Subject: [rsyslog] segfault with rsyslog Version > 8.16.

Re: [rsyslog] segfault with rsyslog Version > 8.16.0

2016-04-20 Thread David Lang
for what little it's worth, I've been running the 8.18 git tree for a couple weeks and it's been better than the prior ones. But I am using the git version of libfastjson David Lang On Wed, 20 Apr 2016, Brian Knox wrote: Date: Wed, 20 Apr 2016 16:33:56 + From: Brian

Re: [rsyslog] rsyslog's batch api documentation inconsistency

2016-04-20 Thread David Lang
d approach, fix rsyslog to provide at-least-once guarantees or fix documentation so it mentions that rsyslog doesn't provide it? I think it should be at least once. It's a lot easier to weed out duplicates than to notice that something

Re: [rsyslog] retry if output module returns RS_RET_SUSPENDED

2016-04-20 Thread David Lang
Rainer, ping on this issue. David Lang On Tue, 29 Mar 2016, Kane Kim wrote: Date: Tue, 29 Mar 2016 10:46:49 -0700 From: Kane Kim Reply-To: rsyslog-users To: rsyslog-users Subject: Re: [rsyslog] retry if output module returns RS_RET_SUSPENDED Updated config to: module(load="omrelp&quo

Re: [rsyslog] rsyslog's batch api documentation inconsistency

2016-04-20 Thread David Lang
but you were busy at the time and were never able to follow up on the topic. David Lang On Wed, 20 Apr 2016, Rainer Gerhards wrote: Date: Wed, 20 Apr 2016 21:01:39 +0200 From: Rainer Gerhards Reply-To: rsyslog-users To: rsyslog-users Subject: Re: [rsyslog] rsyslog's batch api doc

Re: [rsyslog] retry if output module returns RS_RET_SUSPENDED

2016-04-20 Thread David Lang
rrors. David Lang On Wed, 20 Apr 2016, Rainer Gerhards wrote: I should also say that it had a release goal, so it is on my active to-do list. Sent from phone, thus brief. Am 20.04.2016 19:40 schrieb "Rainer Gerhards" : Yup... Look aty work Log - still no time... Was meant for last r

Re: [rsyslog] retry if output module returns RS_RET_SUSPENDED

2016-04-20 Thread David Lang
upt in it, not an entire batch. David Lang On Wed, 20 Apr 2016, Kane Kim wrote: What would happen if it will be retried forever at this point? Would it affect any other queues or only current action queue. If it will not affect anything else I'd really prefer it to block there until successfu

Re: [rsyslog] retry if output module returns RS_RET_SUSPENDED

2016-04-21 Thread David Lang
ld be satisfied with a simple retry/reconnect, but I think we should get things correct first, and then look to optimize it. David Lang On Wed, 20 Apr 2016, David Lang wrote: Date: Wed, 20 Apr 2016 23:44:51 -0700 (PDT) From: David Lang Reply-To: rsyslog-users To: rsyslog-users Subject: Re: [rsys

Re: [rsyslog] retry if output module returns RS_RET_SUSPENDED

2016-04-21 Thread David Lang
legit for the module to just say "temp problem" and tear down the connection and leave the retry up to the core logic. David Lang On Wed, Apr 20, 2016 at 11:44 PM, David Lang wrote: The original discussion when we created the batch mode was that there were two possible ways for

Re: [rsyslog] retry if output module returns RS_RET_SUSPENDED

2016-04-21 Thread David Lang
depending on how you look at it, it's serialized or parallel :-) in v7, the worker thread processed all messages for each action before moving to the next action. But that caused problems with things like global variables where you want to process all actions with one value (and one log mes

Re: [rsyslog] retry if output module returns RS_RET_SUSPENDED

2016-04-21 Thread David Lang
and 2, only for step 0 (start batch) and step 3 (mark batch as done either succeeded/failed) David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? F

Re: [rsyslog] retry if output module returns RS_RET_SUSPENDED

2016-04-21 Thread David Lang
x27;t work (global variables, counter modules, etc) David Lang On Thu, 21 Apr 2016, David Lang wrote: On Thu, 21 Apr 2016, Kane Kim wrote: Thanks for great explanation, David, that really helped me to understand this part. What I'm suggesting is essentially this (and please correct me

Re: [rsyslog] retry if output module returns RS_RET_SUSPENDED

2016-04-21 Thread David Lang
;ll write up a more formal conceptual document with diagrams so that we can have this all documented for the next time we have to look at it in the future :-) David Lang That's also one of the main things for me to take time (memory doesn't server well if you need to be very specifi

Re: [rsyslog] segfault with rsyslog Version > 8.16.0

2016-04-21 Thread David Lang
s from the adiscon repo (no stray packages with the same name/version from redhat repos) David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslo

Re: [rsyslog] retry if output module returns RS_RET_SUSPENDED

2016-04-21 Thread David Lang
On Thu, 21 Apr 2016, Rainer Gerhards wrote: 2016-04-21 9:57 GMT+02:00 David Lang : On Thu, 21 Apr 2016, Rainer Gerhards wrote: 2016-04-21 9:35 GMT+02:00 Kane Kim : Thanks for great explanation, David, that really helped me to understand this part. What I'm suggesting is essentially

Re: [rsyslog] segfault with rsyslog Version > 8.16.0

2016-04-21 Thread David Lang
On Thu, 21 Apr 2016, Rainer Gerhards wrote: 2016-04-21 10:13 GMT+02:00 David Lang : On Thu, 21 Apr 2016, Christopher Racky wrote: Hi, Thanks a lot so far. David you assumed right. I was not able to get a coredump as only one thread dies. I noticed, that on my RHEL 6.7 System I have json-c

Re: [rsyslog] rsyslog issue with new modsec_audit.log

2016-04-22 Thread David Lang
rsyslog should handle it just fine. Rsyslog detects that the inode of the filehas changed and starts from the beginning of the new file. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com

Re: [rsyslog] rsyslog issue with new modsec_audit.log

2016-04-22 Thread David Lang
written to it before rsyslog reads the file again, rsyslog could miss the fact that the file was truncated and miss data and/or spit out a partial log. David Lang On Fri, 22 Apr 2016, Ashish Barmase wrote: Date: Fri, 22 Apr 2016 17:08:18 + From: Ashish Barmase Reply-To: rsyslog-users

Re: [rsyslog] rsyslog issue with new modsec_audit.log

2016-04-25 Thread David Lang
t accepted, consider removing or replacing it are you mixing the legacy and new style syntax? can you show us your config? David Lang On Mon, Apr 25, 2016 at 12:00 PM, Muhammad Asif wrote: Dear Ashish, Thanks for reply. How you did this exactally. Any cron job or some thing else? Regards M.As

Re: [rsyslog] rsyslog issue with new modsec_audit.log

2016-04-25 Thread David Lang
to /var/spool/rsyslog? especially as user user/group syslog.syslog? as far as rotating the file go, you should not copy+delete the file, you should move the file, then HUP rsyslog and rsyslog will recreate the file as needed. David Lang On Mon, 25 Apr 2016, Muhammad Asif wrote: Date: Mon

Re: [rsyslog] Centralized Logging - AUDIT specifically

2016-04-28 Thread David Lang
rsyslog 5.8 is ancient (8.18 is current), there should be a 7.x version available through the redhat repos I have never used the aureport tool, but from the example on the page, you can give it the -f flag to specify a filename to look at. David Lang On Thu, 28 Apr 2016, Warron S French

Re: [rsyslog] TLS Anon Patch

2016-05-02 Thread David Lang
are you sure the Cisco is supporting TLS, not just doing plaintext over TCP? If the client has a cert to offer, we still want to gather the data from it, even if we don't require validation. David Lang On Mon, 2 May 2016, Wingard, Nathaniel wrote: Date: Mon, 2 May 2016 17:35:20 +

Re: [rsyslog] recreating a logging socket -- need to restart rsyslogd?

2016-05-02 Thread David Lang
of to make sure that rsyslog has the correct socket open after the HUP If this doesn't work, a full restart will be needed. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-service

Re: [rsyslog] Remote messages getting into local logs

2016-05-04 Thread David Lang
arrive via tcp or udp using the rules in the inbound ruleset (and using a separate queue) David Lang On Wed, 4 May 2016 16:38:34 +, Singh, Radesh wrote: Hello, Perhaps I'm overthinking this, but as I've confused myself pretty good, I'm reaching out to you guys. We noticed that

Re: [rsyslog] [Log delayed] Some old logs appears in the log file

2016-05-04 Thread David Lang
you show your full config? David Lang On Thu, 5 May 2016, Kuo Hugo wrote: Date: Thu, 5 May 2016 01:28:38 +0800 From: Kuo Hugo Reply-To: rsyslog-users To: rsyslog@lists.adiscon.com Subject: [rsyslog] [Log delayed] Some old logs appears in the log file Hi Folks, I observed a weird behaviour

Re: [rsyslog] Remote messages getting into local logs

2016-05-04 Thread David Lang
you need to use one syntax or the other $UDPServerRun 514 is the same as input(type="imudp" port="514") so remove the ServerRun lines and I think you have it. (although I think you should put the two input() statements on different lines) David Lang On Wed, 4 May 2016,

Re: [rsyslog] Remote messages getting into local logs

2016-05-04 Thread David Lang
Yes, you need to be using at least 7.x (current is 8.18) David Lang On Wed, 4 May 2016, Singh, Radesh wrote: Date: Wed, 4 May 2016 19:32:18 + From: "Singh, Radesh" Reply-To: rsyslog-users To: rsyslog-users Subject: Re: [rsyslog] Remote messages getting into local logs Rsysl

Re: [rsyslog] Remote messages getting into local logs

2016-05-04 Thread David Lang
y run into problems where the answers that we give are not going to work. David Lang On Wed, 4 May 2016, Singh, Radesh wrote: Date: Wed, 4 May 2016 19:42:46 + From: "Singh, Radesh" Reply-To: rsyslog-users To: rsyslog-users Subject: Re: [rsyslog] Remote messages getting into local lo

Re: [rsyslog] how to set property msgid value by rsyslog API ?

2016-05-05 Thread David Lang
the message that you have received and then create a custom template to output the message with the extracted value in the msgid field. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com

Re: [rsyslog] filter on rfc5424 field

2016-05-06 Thread David Lang
Even though the RFC allows it, it's hard to imagine a valid log without a timestamp or source. David Lang On Fri, 6 May 2016, Rainer Gerhards wrote: The best way to diagnose these kinds of problems is to add *.* /var/log/msgdebug;RSYSLOG_DebugFormat To the top of rsyslog.conf. The

Re: [rsyslog] how to set property msgid value by rsyslog API ?

2016-05-06 Thread David Lang
logs. David Lang On Fri, 6 May 2016, Zhou, Tao wrote: Thanks David for youy prompt reply. Did you mean rfc5424 ? if so, the message is deifned in BNF: SYSLOG-MSG = HEADER SP STRUCTURED-DATA [SP MSG] HEADER = PRI VERSION SP TIMESTAMP SP HOSTNAME SP APP-NAME SP PROCID SP MSGI

[rsyslog] performance while there is pending DA data

2016-05-11 Thread David Lang
in-memory queue)? David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list

Re: [rsyslog] rsyslog using domain instead of IP and hostname

2016-05-11 Thread David Lang
tead of 115-186-137-215.nayatel.pk. that depends on what data you have and where it came from. The default template uses $hostname in that field, but if the sender is violating the RFCs by not providing a hostname, rsyslog will make it's best guess by doing a reverse lookup ($fromhost) of th

Re: [rsyslog] retry if output module returns RS_RET_SUSPENDED

2016-05-11 Thread David Lang
I thought the last time the subject came up he was saying he needed help on this. I was actually going to ping you today to see if you had been able to get any time to look at it :-) David Lang On Wed, 11 May 2016, Kane Kim wrote: Date: Wed, 11 May 2016 10:46:52 -0700 From: Kane Kim

Re: [rsyslog] performance while there is pending DA data

2016-05-11 Thread David Lang
queue.dequeuebatchsize = "1000" queue.size="50" queue.filename="x_dc_queue" action.resumeRetryCount="-1" queue.maxfilesize="256m" queue.saveonshutdown="on") David Lang Rainer 2016-05-11 16:21 GMT+02:00 David Lang : I had a short o

Re: [rsyslog] retry if output module returns RS_RET_SUSPENDED

2016-05-12 Thread David Lang
ring module for json output. 2. sanitize mm module so that I can stick data into ES. David Lang On Wed, 11 May 2016, Kane Kim wrote: Sorry guys, I was busy as well for a couple of weeks, let's reiterate what we will have to do here. I guess as Rainer said first step should be to write a

Re: [rsyslog] Custom Log Not Forwarding Messages

2016-05-12 Thread David Lang
t it can make it so that you can't read the file. Just to be sure I understand your situation, normal logs from both files work, it's the imfile section that works on one and not on the other, correct? you can try starting rsyslog in debug mode (rsyslogd -dn) and see if the log giv

Re: [rsyslog] Custom Log Not Forwarding Messages

2016-05-12 Thread David Lang
to find the answers. David Lang On Thu, May 12, 2016 at 2:12 PM, David Lang wrote: On Thu, 12 May 2016, Thomas Lowry wrote: Hello, I am having an issue where I have two identical servers and a central log server. We will call the identical servers "server1" and "server2"

Re: [rsyslog] configuring failover to work with asynchronous queuing

2016-05-12 Thread David Lang
dFormat $ActionExecOnlyWhenPreviousIsSuspended off With the new format, queues and rulesets are really easy ruleset(name="foo" queue.type="linkedlist" ...) { *.info :omrelp:syslog1.example.com:1514;RSYSLOG_ForwardFormat $ActionExecOnlyWhenPreviousIsSuspen

Re: [rsyslog] configuring failover to work with asynchronous queuing

2016-05-12 Thread David Lang
That general approach is what I recommend in any case. David Lang On Thu, 12 May 2016, Joe Blow wrote: This probably won't answer your exact question but could you just use keepalived and create a VIP to send the syslog/relp to?  Would there be any downside to using a vrrp pool for

Re: [rsyslog] Custom Log Not Forwarding Messages

2016-05-12 Thread David Lang
ble to you through an update David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBL

Re: [rsyslog] Centos 6 rsyslog v5 Issue

2016-05-16 Thread David Lang
yslogd -N3" or similar 2. log with the format RSYSLOG_DebugFormat so that you can see what variables contain what values so that you can check that your filters match the variable contents. David Lang On Mon, 16 May 2016, J. Toby Knudsen wrote: Date: Mon, 16 May 2016 12:10:00 -0400 From:

Re: [rsyslog] feedback request: daily packages

2016-05-23 Thread David Lang
not most of the time. occasionally when I'm testing something (although not for the last few months as I've needed liblognorm2 that wasn't being shipped yet) David Lang On Mon, 23 May 2016, Rainer Gerhards wrote: Hi folks, I wonder if someone here from the list uses the da

Re: [rsyslog] Feedback requested: decisions for next release

2016-05-24 Thread David Lang
clude everything since at least one month). is it the libfastjson that's the question or the liblognorm v2? David Lang Hope that clarifies and thanks for the comments. I think I'll postpone if nobody says he would really like to have it with next weeks release (holding rsyslog for th

Re: [rsyslog] Setting TLS cipher suite / disabling RC4

2016-05-24 Thread David Lang
he options that we really should support and provide some sort of explination as to what they mean pretty easily. Unfortunatly this is why so many TLS related questions go unanswered for a while here on the list. David Lang ___ rsyslog ma

Re: [rsyslog] Help with rsyslog failover configuration

2016-05-24 Thread David Lang
e lisa presentation and the first article in my logging series: https://www.usenix.org/conference/lisa12/technical-sessions/presentation/lang_david https://www.usenix.org/publications/login/david-lang-series https://www.usenix.org/publications/login/april14/lang David Lang

Re: [rsyslog] Logs are delayed being pushed to Elasticsearch

2016-05-24 Thread David Lang
debug log locally (not necessarily the full messages, timestamp would be enough) you can then see if the local file in growing while things are not yet showing up in ES to see if the issue is on the sending side or on the receiving side. David Lang On Tue, 24 May 2016, Alec Swan wrote: Date

Re: [rsyslog] Setting TLS cipher suite / disabling RC4

2016-05-24 Thread David Lang
say, audits usually don't care about real risk, they have their checkboxes... David Lang On Tue, 24 May 2016, David Lang wrote: Date: Tue, 24 May 2016 16:16:22 -0700 (PDT) From: David Lang To: rsyslog-users Subject: Re: [rsyslog] Setting TLS cipher suite / disabling RC4 On Tue, 24 May 201

Re: [rsyslog] Fetch Logfiles reliably

2016-05-25 Thread David Lang
On Wed, 25 May 2016, Thomas Güttler wrote: Am 25.05.2016 um 08:26 schrieb Rainer Gerhards: 2016-05-25 8:22 GMT+02:00 Thomas Güttler : Am 24.05.2016 um 17:40 schrieb Rainer Gerhards: 2016-05-24 17:10 GMT+02:00 Thomas Güttler : Hi, I have some remote hosts which can't connect to our central

Re: [rsyslog] Fetch Logfiles reliably

2016-05-26 Thread David Lang
-site VPN. In either case, you will want to use RELP instead of just TCP http://blog.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.html http://blog.gerhards.net/2008/05/why-you-cant-build-reliable-tcp.html David Lang I will use RELP, thank you for this hint. Since it is based on tcp

Re: [rsyslog] Help with rsyslog failover configuration

2016-05-26 Thread David Lang
On Thu, 26 May 2016, Karri Vrkreddy wrote: Thanks David Lang for looking into this. Please correct me if I understood your suggestion incorrectly. I have changed earlier configuration to *** module(load="imudp") ruleset(name="central" queue.type="Link

Re: [rsyslog] Using OMPROG to output to a syslog stream?

2016-05-27 Thread David Lang
m on it's stdin. What your program does after that is strictly up to it. If you are wanting to just modify the message, you shouldn't use omprog, you should use mmexternal which lets you modify the log message in rsyslog with an external program and then

Re: [rsyslog] Having difficult getting MMEXternal work with rsyslog and an external perl program.

2016-05-30 Thread David Lang
ur perl program to write it's stdin to a file and make sure that what you get from rsyslog is what you think you are getting. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-s

[rsyslog] libfastjson build error

2016-06-01 Thread David Lang
Anyone have any suggestions (even quick-n-dirty ones) to let this work? David Lang /usr/src/libfastjson# make dist make dist-gzip am__post_remove_distdir='@:' make[1]: Entering directory `/usr/src/libfastjson' if test -d "libfastjson-0.99.3.master"; then find "lib

Re: [rsyslog] libfastjson build error

2016-06-01 Thread David Lang
On Wed, 1 Jun 2016, Thomas D. wrote: OK, workaround: Run "make" before "make dist". I had to do make clean; ./configure; make; make dist David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog ht

Re: [rsyslog] Logs are delayed being pushed to Elasticsearch

2016-06-02 Thread David Lang
the output for impstats to see how many messages are queued. David Lang Thanks, Alec On Sat, May 28, 2016 at 2:40 AM, Rainer Gerhards wrote: 2016-05-28 4:36 GMT+02:00 Alec Swan : I ran rsyslog in debug mode and now I am in the same state where a copy of the logs are still being writt

Re: [rsyslog] Logs are delayed being pushed to Elasticsearch

2016-06-02 Thread David Lang
og started back up with 925MB RAM usage. What parameter controls how much of queue data rsyslog will load up from disk on startup? It reads old messages in it as it has time, there is no specific parameter. I don't know what the chunk size is that it processes. David Lang Thanks, Alec

Re: [rsyslog] Can rsyslog send about 75 MB long file having one message?

2016-06-10 Thread David Lang
set. You can change this by editing the source, but above some point you will trip over other issues (assumptions about how large a message can be influencing buffer sizes, etc) you can try it, but *syslog is designed for messages of a few KB, so you are ~1000x the design size. David Lang

Re: [rsyslog] Can rsyslog process about 70 MB long one json message

2016-06-10 Thread David Lang
, but it's not the right tool to use with a screw. You may be able to get it to work, but it's still not the right tool. rsync or similar is the right answer when you have many files in the MB range to get to another system, not *syslo

[rsyslog] resume retries for queues on rulesets?

2016-06-13 Thread David Lang
using the new syntax, action.resumeRetryCount="-1" works on a queue for an action, but doesn't work for a queue on a ruleset, resumeRetryCount="-1" also doesn't work. What should work? David Lang ___ rsyslog mailing

Re: [rsyslog] message parsers with rulesets

2016-06-16 Thread David Lang
would make sense to try and globally define parsers in that case. David Lang regards Chris 2016-06-16 15:55 GMT+02:00 Rainer Gerhards : You need to specify them in the ruleset object, e.g. ruleset(name="remote" parser=["rsyslog.srcipinject","rsyslog.lastline&

Re: [rsyslog] Rsyslog stops reading logs after logrotate

2016-06-16 Thread David Lang
and then notice that the inode for the filename has changed (i.e. that the move happened), and close the old file and start reading the new one. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.c

Re: [rsyslog] Rsyslog stops reading logs after logrotate

2016-06-16 Thread David Lang
it doesn't require changing the source code of the logging library. Ok, just be aware that you will loose some logs as they rotate. what version of rsyslog are you running? current versions should not require a HUP. David Lang Thanks, Alec On Thu, Jun 16, 2016 at 11:36 AM, David Lang

Re: [rsyslog] Rsyslog stops reading logs after logrotate

2016-06-16 Thread David Lang
looking at: http://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html did you try the reopenOnTruncate option? David Lang On Thu, 16 Jun 2016, Alec Swan wrote: I am using rsyslog version 8.19. It did require restart (I haven't tried HUP yet) after logrotate, but I wond

Re: [rsyslog] Central logging solution

2016-06-20 Thread David Lang
https://www.usenix.org/publications/login/david-lang-series https://www.usenix.org/conference/lisa12/technical-sessions/presentation/lang_david you can use the (non-free) rsyslog windows agent, or nxlog to gather logs from windows and deliver them to central rsyslog servers. look through the

[rsyslog] mmnormalize rule database Re: mmgrok packages

2016-06-22 Thread David Lang
nt to have 50 different ways to handle the same Cisco message, but how do we pick which of the many different versions we are going to use? David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/pr

Re: [rsyslog] Central logging solution

2016-06-22 Thread David Lang
21 7e 154MB 25 8e 152MB 35 9e 150MB 31 David Lang I'd love to see the audispatch work you've done.  Also, in case you were wondering, I have a patent on my log-to-jpg-to-sql module, and the corresponding OCR extraction tool.  Cheers, JB   Original Message   From:warron.fre...@

Re: [rsyslog] Central logging solution

2016-06-22 Thread David Lang
hings haven't changed) At this point I've basically abandoned bzip2, I don't fine any cases where it's less cpu or more compression than is available via xz. About the only time I still use gzip is for the logs compressed by rsyslog. David Lang  When testing with firewall logs

Re: [rsyslog] Central logging solution

2016-06-22 Thread David Lang
On Wed, 22 Jun 2016, warron.french wrote: David Lang, you also have valid points obviously, but what I wonder is if the server crashes where is the data going to be then. At least one centralized on a NAS/SAN solution a massive hit to the server won't be as destruction to the data on th

Re: [rsyslog] Central logging solution

2016-06-22 Thread David Lang
On Wed, 22 Jun 2016, Joe Blow wrote: If you're trying for true HA/prod setup, I'd suggest looking at DRBD+pacemaker+corosync+crmsh+keepalived+rsyslog with local storage.  Why would use keepalived as well as pacemaker/corosync? It seems to me that you would use one or the other. I've given s

Re: [rsyslog] Central logging solution

2016-06-22 Thread David Lang
me. By then having one copied to the other based on which is the 'primary' I've got a rsyncable master 99%+ of the time, and the remainder of the time I have the individual files that I can inspect and merge manually if it absolutly matters (it seldom does so I just pick one a

Re: [rsyslog] Dealing with malformed messages

2016-06-23 Thread David Lang
build the list of culprits? or the best way to deal with them once you have the list? David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? F

Re: [rsyslog] mmnormalize rule database Re: mmgrok packages

2016-06-23 Thread David Lang
e the list. given the number of times this has come up, I'm sure there is some interest. Thanks for volunteering on this. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/

Re: [rsyslog] mmnormalize rule database Re: mmgrok packages

2016-06-23 Thread David Lang
the comfort to start using the parsing. Given that liblognorm is pretty insensitive to the number of rules, it may be that we can craft a combined rulebase that could ship by default with liblognorm as a starter for people. David Lang ___ rsyslog mai

Re: [rsyslog] Dealing with malformed messages

2016-06-24 Thread David Lang
ntain. David Lang Philippe -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog- boun...@lists.adiscon.com] On Behalf Of David Lang Sent: Thursday, June 23, 2016 7:08 PM To: rsyslog-users Subject: Re: [rsyslog] Dealing with malformed messages On Thu, 23 Jun

Re: [rsyslog] Invalid JSON from mmnormalize/liblognorm/omelasticsearch

2016-06-28 Thread David Lang
the feature, it can be available pretty quickly. David Lang On Tue, 28 Jun 2016, Alec Swan wrote: Hello, I am using rsyslog 8.19.0 with mmnorlamize and liblognorn to parse our log messages and convert them to JSON. After that I use omelaticsearch to send JSON to Elasticsearch 2.2.3. I noti

Re: [rsyslog] Invalid JSON from mmnormalize/liblognorm/omelasticsearch

2016-06-28 Thread David Lang
ding sure seems like it's complaining about '.' (which is ascii 46) do you have any way of finding out what the full string is that it's trying to process? have you asked elastic.io? (since the problem seems to be in their end), even if it's invalid json, they shou

Re: [rsyslog] Invalid JSON from mmnormalize/liblognorm/omelasticsearch

2016-06-29 Thread David Lang
file that has the Hindi characters near the beginning of it and try the file -bi again, see if it gives a more accurate answer. otherwise, you will have to track down what's writing the messages and try to set the character set there (or at least find out what character set it's using)

Re: [rsyslog] Invalid JSON from mmnormalize/liblognorm/omelasticsearch

2016-06-29 Thread David Lang
This is helping narrow things down. I would have rsyslog write to a file with the template that you use to send to elasticsearch. I would also use the liblognorm command-line tool to parse the file and output json. let's try to see where it breaks. David Lang On Wed, 29 Jun 2016,

Re: [rsyslog] rsyslog 8.4.2 and multiple actions per selector

2016-06-30 Thread David Lang
other than that, it will continue to work. the 'new way' of doing this would be: if ($programname == "named") and ($pri-text == "daemon.info") then { /var/log/syslog/named stop } use whichever way makes the most sense to you David Lang _

Re: [rsyslog] rsyslog 8.4.2 and multiple actions per selector

2016-06-30 Thread David Lang
On Thu, 30 Jun 2016, Marc Haber wrote: On Thu, Jun 30, 2016 at 12:59:25AM -0700, David Lang wrote: if ($programname == "named") and ($pri-text == "daemon.info") then { /var/log/syslog/named stop } Thanks, that was helpful. The docs need more examples. patches we

Re: [rsyslog] Problems CentOS 7 rsyslog 7.4.7

2016-06-30 Thread David Lang
are you getting any logs into rsyslog? do the mail.* logs show up in the systemd journal and just not in rsyslog? The first thing to do is to track down where things fail before starting to dig into the config files of any one thing. David Lang On Thu, 30 Jun 2016, Adrian P. van Bloois

Re: [rsyslog] rsyslog 8.4.2 and multiple actions per selector

2016-07-01 Thread David Lang
On Fri, 1 Jul 2016, Marc Haber wrote: On Thu, Jun 30, 2016 at 10:46:27AM -0700, David Lang wrote: On Thu, 30 Jun 2016, Marc Haber wrote: On Thu, Jun 30, 2016 at 12:59:25AM -0700, David Lang wrote: if ($programname == "named") and ($pri-text == "daemon.info") then { /

Re: [rsyslog] Strange behavior of rsyslog, once it is run as a service

2016-07-03 Thread David Lang
Linux logs for such failures. David Lang On Sun, 3 Jul 2016, Dr. Yury Zhauniarovich wrote: Date: Sun, 3 Jul 2016 16:48:49 + From: Dr. Yury Zhauniarovich Reply-To: rsyslog-users To: "rsyslog@lists.adiscon.com" Subject: [rsyslog] Strange behavior of rsyslog, once it is run as

Re: [rsyslog] Invalid JSON from mmnormalize/liblognorm/omelasticsearch

2016-07-06 Thread David Lang
first I've heard of this, we'll need to have Rainer comment on this. David Lang On Wed, 6 Jul 2016, Alec Swan wrote: Date: Wed, 6 Jul 2016 15:34:44 -0600 From: Alec Swan Reply-To: rsyslog-users To: rsyslog-users Subject: Re: [rsyslog] Invalid JSON from mmnormalize/

Re: [rsyslog] DA queue issues and 8.18 fixes

2016-07-07 Thread David Lang
themselves are corrupt. I am not aware of the exact details of the 8.18 improvements try the recover_qi.pl script (it's been posted before, I'm attempting to include it as an attachment here), see what it has to say about the queue files. David Lang#!/usr/bin/perl -w # recover rsyslog

Re: [rsyslog] Invalid JSON from mmnormalize/liblognorm/omelasticsearch

2016-07-07 Thread David Lang
As I understand Rainer's reply, if you compile from the current liblognorm master you will not have that limitation. This new branch will be released shortly (along with rsyslog 8.20) David Lang On Thu, 7 Jul 2016, Alec Swan wrote: Date: Thu, 7 Jul 2016 12:57:51 -0600 From: Alec Swan

Re: [rsyslog] Invalid JSON from mmnormalize/liblognorm/omelasticsearch

2016-07-07 Thread David Lang
Per Rainer, the 10k limit is in the normalizer tool, not in the library. In rsyslog there is a maxmessagesize parameter. That is the limiting factor even with the existing version. the new version of liblognorm/mmnormalize will work with existing version 1 rulesets. David Lang On Thu, 7

Re: [rsyslog] Invalid JSON from mmnormalize/liblognorm/omelasticsearch

2016-07-07 Thread David Lang
col = "ipv4-only" net.enableDNS = "off" parser.escapeControlCharactersOnReceive = "on" ) I don't know how ES could be complaining about a 50K message if it's defaulting to 8K David Lang On Thu, 7 Jul 2016, Alec Swan wrote: I remember seeing MaxMessageSize parameter, but

Re: [rsyslog] DA queue issues and 8.18 fixes

2016-07-08 Thread David Lang
(at which point rsyslog may try to flush the memory queues to disk if that's how you have it configured), and then if rsyslog doesn't shutdown fast enough (5 seconds or so), issues a kill -9. If there is too much data to flush to disk, this can cause the .qi file to not

Re: [rsyslog] %TIMESTAMP:::date-rfc3339% stopped working

2016-07-10 Thread David Lang
ot found online. The full config is at http://pastebin.com/ENkzmMT2 What have I done wrongly? what version of rsyslog are you running? David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/pr

Re: [rsyslog] libfastjson 0.99.3 released

2016-07-11 Thread David Lang
. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED

Re: [rsyslog] rsyslog 8.20.0 (v8-stable) released

2016-07-12 Thread David Lang
Also note as Rainer posted in a separate thread, binary packages of 8.20 are going to be delayed while work is done to figure out how to avoid the package problems of the last few days for future releases. David Lang On Tue, 12 Jul 2016, Florian Riedl wrote: Date: Tue, 12 Jul 2016 18:09:46

Re: [rsyslog] CentOS yum update failed

2016-07-12 Thread David Lang
s got fixed in < 10 hours, but also surprised if they didn't fix it within 18 hours or so, just due to time zone effects. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/profess

<    1   2   3   4   5   6   7   8   9   10   >