Re: [rsyslog] Rsyslog vs syslog-ng

[ 4]: > > I hope this discussion prompts some of the other high-volume rsyslog users on > this list to share some details also... > > Best regards, > > -- > Dave Caplinger | Chief Architect, Global Platform Engineering | NTT Security > Corporation > > > >

Re: [rsyslog] Rsyslog vs syslog-ng

I have used Rsyslog in the past to run a 66 node array (6 core VMs) to process 6M log-events / second (avg size 1.5 KB), which boils down to 69 Gbps of log traffic. This was being received over ptcp and delivered to the backend over omkafka. Rsyslog and HDFS (of Hadoop) were by-far the most

Re: [rsyslog] Transforming JSON Field Names in rawmsg

It seems like you can parse the message(https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmjsonparse.html) and set new fields (and unset the old fields) in rainerscript (https://www.rsyslog.com/doc/v8-stable/rainerscript/variable_property_types.html). On Wed, Oct 3, 2018 at 7:06 PM John

Re: [rsyslog] Imkafka/omkafka tuning

The omkafka action worker doesn't perform deliveries. Its responsibility is limited to enqueue and ack-processing. Actual deliveries are done by broker-bound threads and although the design delivers sufficient throughput with small kafka clusters, it chokes on lock contention as broker count goes

Re: [rsyslog] Problems with rsyslog Versions > 8.16 and Thread-Handling

Hi Kendall, I need a repro for this problem, it has remained open far too long for my comfort, but I simply haven't been able to get to the bottom of this over mail and issue-tracker. Does the image (rhel version) you use exist as an AMI on EC2? Or if you can spare some time, it'll help me

Re: [rsyslog] Duplicated logs on rolling file

Log4j has a way to rollover to new file (without truncating old file). I'd look at Rolling Appender if you want to use it with imfile. Or its simpler and more efficient to make log4j send logs over to Rsyslog (or any other syslog impl) directly. Here what a log4j properties file with syslog

Re: [rsyslog] conversion functions

as an optimization. On Nov 28, 2016 12:08 AM, "David Lang" <da...@lang.hm> wrote: > On Sun, 27 Nov 2016, singh.janmejay wrote: > > I was thinking of something like parse_int("0xa"), parse_int("012") and >> parse_int("10") all

Re: [rsyslog] conversion functions

-hierarchical and clubbing functions by their nature may be equally effective (but we'll have to maintain it purely by discipline, whereas namespacing will make it more natural). On Nov 27, 2016 5:27 PM, "Rainer Gerhards" <rgerha...@hq.adiscon.com> wrote: > 2016-11-27 12:29 GMT+0

Re: [rsyslog] conversion functions

o doing cint(val), clong(val), and trying > to get off the "too complex" conception, IMHO we should support that. > > Of course, internally we can use a single nicely abstracted function :-) > > Rainer > > 2016-11-27 8:45 GMT+01:00 David Lang <da...@lang.hm>:

Re: [rsyslog] conversion functions

I meant rainerscript functions. It'll boil down to equivalent functions in C too, but that is implementation detail. Did I misunderstand the context? On Nov 27, 2016 10:49 AM, "David Lang" <da...@lang.hm> wrote: > On Sun, 27 Nov 2016, singh.janmejay wrote: > > I feel i

Re: [rsyslog] conversion functions

I feel it's better to have parse_(expr) and cast_(expr) family of functions that return the appropriate value. Eg. If x is "12.3", parse_double(x) returns 12.3 (double precision), and cast_int(parse_double(x)) returns 12. Cast detects the source type and acts accordingly, parse expects

Re: [rsyslog] Disabling IMPTCP FlowControl

utility called tcpflood in rsyslog tests directory that can flood messages quite well. Ruleset behind imptcp can be configured to filter and discard those messages. On Nov 18, 2016 4:32 AM, "David Lang" <da...@lang.hm> wrote: > On Thu, 17 Nov 2016, matthew.gaetano wrote: > > singh.

Re: [rsyslog] Disabling IMPTCP FlowControl

A few questions: - how many threads is imptcp configured for? * how does cpu-util (%sys and %usr) change on receiver as it goes from healthy 30-40k ingestion to backlog building up? * what does the receiving throughput drop to (assuming it is 40k when all is well)? - what is omfwd

Re: [rsyslog] Disabling IMPTCP FlowControl

Yes, the snapshot from the time when it is under load will help. Has it just ramped-down on load or are you cycling connections too frequently? Or was sender restarted recently? On Fri, Nov 18, 2016 at 1:07 AM, matthew.gaetano wrote: > Sender: > [root@server1 ~]# netstat

Re: [rsyslog] Disabling IMPTCP FlowControl

What does 'netstat -antlp | grep -F ' say on both sender and receiver side? On Fri, Nov 18, 2016 at 12:43 AM, matthew.gaetano wrote: > My current average input on server1 is about 30-40k and we have seen spikes > of up to 100k; so i would say my target throughput would be

Re: [rsyslog] Disabling IMPTCP FlowControl

Hi Matthew, Do you have a target/planned throughput in mind(assuming other resources are sized correctly) and need help to achieve that, or do you just want to investigate what (in this case) lead to imptcp server not keeping up? On Thu, Nov 17, 2016 at 11:00 PM, matthew.gaetano

Re: [rsyslog] using lookup tables to change field names

I agree. On Thu, Nov 17, 2016 at 2:19 PM, Rainer Gerhards <rgerha...@hq.adiscon.com> wrote: > I'd tend to think that this would be valuable even if it is > performance-hungry... > > Just my 2cts, > Rainer > > 2016-11-17 9:48 GMT+01:00 singh.janmejay <singh.janme...@

Re: [rsyslog] using lookup tables to change field names

). Stream 1M such messages(say 20 unique messages repeated in arbitrary order) with some simple transformations in jq (say rename a key, concat a few values, sum a field etc). Helps? On Thu, Nov 17, 2016 at 1:22 PM, David Lang <da...@lang.hm> wrote: > On Thu, 17 Nov 2016, singh.janme

Re: [rsyslog] using lookup tables to change field names

This seems like exactly what 'jq' does, it is ability to modify json structures using a very flexible declarative language. Jq does advertise linking-as-a-library style integration. I was wondering how everyone feels about building 'mmjq'? On Nov 17, 2016 2:07 AM, "David Lang"

Re: [rsyslog] Segementation Fault when Rsyslog Starts

Pulled the attachments down, I don't have enough information to say this yet, but it does seem like memory corruption. Statement that scriptExec is trying to execute is at 0x30e1, if you still have the corefile, can you please check if that address is mapped RW? Another funny thing is, you seem

Re: [rsyslog] Segementation Fault when Rsyslog Starts

Did you forget to attach the gdb.txt file? On Wed, Nov 2, 2016 at 6:54 PM, shane.lawrence wrote: > Hi Singh, > > Thanks for your response. The gdb.txt file that I attached had the full > output of "thread apply all bt full", and I had the following debug

Re: [rsyslog] Lookup Table FH leak?

Sorry, I meant change log may have a ref(just have a look at master change log). On Oct 28, 2016 10:17 AM, "singh.janmejay" <singh.janme...@gmail.com> wrote: > FD leak on reload is fixed upstream already. Can't recall which release. > Release notes may have a ref. >

Re: [rsyslog] Lookup Table FH leak?

FD leak on reload is fixed upstream already. Can't recall which release. Release notes may have a ref. On Oct 28, 2016 6:15 AM, "David Lang" wrote: > I'm catching up on old mail, I think I saw fixes for the lookup table > functionality go through recently, can you check with the

Re: [rsyslog] I'm back

I was wondering about your silence on the mailing list. Welcome back. On Oct 28, 2016 12:03 AM, "David Lang" wrote: > I'm back on my feet, but don't have a new job yet, so I'm getting there. > > David Lang > > On Thu, 27 Oct 2016, Rainer Gerhards wrote: > > Hi David, >> >>

Re: [rsyslog] Fwd: Re: rsyslog kills entire system => force reboot

How long does it take to go thru one cycle of verifying the problem exists? I was wondering if bisecting would be viable? May not be required though, stats, entire config and all thread backtrace will likely give you/us enough clues. On Sep 16, 2016 12:30 PM, "Raffael Sahli"

Re: [rsyslog] Rsyslog exiting on signal 15

so it may vary depending on what you are running. > > David Lang > > On Tue, 19 Apr 2016, singh.janmejay wrote: > > Won't oom be a sigkill? >> >> You can determine which process issued term (syscall is kill). >> >> Match it with time of death to find the culpri

Re: [rsyslog] Rsyslog exiting on signal 15

Won't oom be a sigkill? You can determine which process issued term (syscall is kill). Match it with time of death to find the culprit. On Apr 19, 2016 12:32 AM, "Alec Swan" wrote: I checked dmesg output and didn't find anything related to rsyslog being killed even though I

Re: [rsyslog] another dyn_stats question

Here is the PR for unused-metric-ttl bug: https://github.com/rsyslog/rsyslog/pull/955 On Wed, Apr 6, 2016 at 8:35 AM, David Lang <da...@lang.hm> wrote: > looking pretty good. I've been running it for a couple days now. > > David Lang > > On Thu, 31 Mar 2016, singh.janmejay

Re: [rsyslog] Out of Office. (was: Out of Office. (was: Out of Office. (was: Out of Office. (was: Out of Office. (was: Out of Office. (was: Out of Office. (was: Out of Office. (was: Out of Office. (wa

It's an infinite loop(auto reply is replying to itself), can someone please kill that subscription? On Apr 13, 2016 12:11 AM, "Steve Clark" wrote: ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog

Re: [rsyslog] Rate limiting for imfile?

ote: > On 04/12/2016 11:31 AM, singh.janmejay wrote: > >> Im curious, why would you want to do that? >> > > > https://github.com/openshift/origin-aggregated-logging#throttling-logs-in-fluentd > > >> As a service that takes those logs and does something

Re: [rsyslog] Rate limiting for imfile?

Im curious, why would you want to do that? As a service that takes those logs and does something with it (store it, index it etc), you want to absorb acceptable bursts as soon as possible. If it is an unacceptably big burst, you are better off dropping messages and catching up with the latest set

Re: [rsyslog] dynastats problems with user defined variables?

definitely help me get quicker feedback and thus a quicker path > towards a final solution. > > The main thing of concern is this: > >https://github.com/rsyslog/libfastjson/issues/35 > > Rainer > > 2016-04-12 16:39 GMT+02:00 singh.janmejay <singh.janme...@gmail.com&

Re: [rsyslog] dynastats problems with user defined variables?

I cooked up this little systemtap script to track cause of off-cpu latency (with latency quantification). It identifies latencies by userspace-backtrace, kernel-backtrace and thread-id. It'll give us very useful data when you see low-TP with processing-backlog while CPU runs cold. Run it as $

Re: [rsyslog] dynastats problems with user defined variables?

regardless of edge and core relay counters being incremented. It was ~ 45k / sec (but again, neither the load profile, not the machine is not exactly like yours), so its not apples to apples. On Fri, Apr 8, 2016 at 12:15 PM, singh.janmejay <singh.janme...@gmail.com> wrote: > It doesn't treat

Re: [rsyslog] dynastats problems with user defined variables?

e digging (and some accidental stuff), is it possible that > it's running into grief if the contents of the variable are not a simple > word (spaces or other funny characters in the value)? > > David Lang > > On Wed, 23 Mar 2016, singh.janmejay wrote: > >> Date: Wed, 2

Re: [rsyslog] mmnormalize, mmjsonparse keyspace mapping

I have a convention of making loggers slap a special tag "obj@" where = a key based namespace. So obj@w3 would have website logs, obj@oms would have order-management-system logs etc. I extract the ns, and generate objects in the form: {host: ..., severity: , obj..x : ..., obj..y: ... } So

Re: [rsyslog] State of Lookup Tables Implementation

This page is the best starting point for all things configuration: http://www.rsyslog.com/doc/v8-stable/configuration/index.html On Tue, Apr 5, 2016 at 1:58 AM, singh.janmejay <singh.janme...@gmail.com> wrote: > http://www.rsyslog.com/doc/v8-stable/configuration/lookup_tables.html &g

Re: [rsyslog] State of Lookup Tables Implementation

http://www.rsyslog.com/doc/v8-stable/configuration/lookup_tables.html I think something went wrong with that link, we need to clean-up. On Tue, Apr 5, 2016 at 1:41 AM, Christian Ramseyer <r...@networkz.ch> wrote: > On 04/04/16 21:48, singh.janmejay wrote: >> The actual impleme

Re: [rsyslog] State of Lookup Tables Implementation

The actual implementation details deviate from proposal page at times. It's best to use documentation. At a high level, proposal describes it quite accurately though. On Apr 4, 2016 7:23 PM, "Christian Ramseyer" wrote: > Great thanks Rainer, I'll check it out. > > > On

Re: [rsyslog] another dyn_stats question

Will create a separate PR for unused-metric-ttl change. That is not included in the PR I posted above. On Fri, Apr 1, 2016 at 1:22 AM, singh.janmejay <singh.janme...@gmail.com> wrote: > Ready for merge now. > > On Thu, Mar 31, 2016 at 12:55 AM, singh.janmejay > <singh.janm

Re: [rsyslog] Loghost to loghost

We use SolrCloud for accessing last 0 to 4 hours of data(accessed using banana/silk). Older data is kept on hadoop, fronted by hive, accessed through either hue(for humans) or hive2 protocol, partitioned by date+hour. We ingress at an aggregate event rate of ~2M/sec or 7.2B/hr. Event sz ~1.5kb.

Re: [rsyslog] dynastats JSON output need work

Complete, ready for merge. On Thu, Mar 31, 2016 at 12:54 AM, singh.janmejay <singh.janme...@gmail.com> wrote: > PR created (not complete yet, not ready for merge): > https://github.com/rsyslog/rsyslog/pull/931 > > Have a quick look to get a feel for namespaced reporting of dyn

Re: [rsyslog] another dyn_stats question

Ready for merge now. On Thu, Mar 31, 2016 at 12:55 AM, singh.janmejay <singh.janme...@gmail.com> wrote: > Have opened a PR for this. Will fold it with other changes in the same > area. Here is the PR: https://github.com/rsyslog/rsyslog/pull/931 (not > ready for merge yet) > >

Re: [rsyslog] another dyn_stats question

Have opened a PR for this. Will fold it with other changes in the same area. Here is the PR: https://github.com/rsyslog/rsyslog/pull/931 (not ready for merge yet) On Wed, Mar 30, 2016 at 8:01 AM, singh.janmejay <singh.janme...@gmail.com> wrote: > Cool, I'll make dyn-stats bucket resett

Re: [rsyslog] dynastats JSON output need work

PR created (not complete yet, not ready for merge): https://github.com/rsyslog/rsyslog/pull/931 Have a quick look to get a feel for namespaced reporting of dyn-stats counters. On Fri, Mar 25, 2016 at 4:25 PM, singh.janmejay <singh.janme...@gmail.com> wrote: > Cool. Will fix b

Re: [rsyslog] another dyn_stats question

about backward compatibility of its behavior. On Mar 30, 2016 2:55 AM, "David Lang" <da...@lang.hm> wrote: > On Wed, 30 Mar 2016, singh.janmejay wrote: > > Metric owner is the subsystem (action is a perfect example). End-user is >> author of the config. >> &

Re: [rsyslog] another dyn_stats question

On Wed, 30 Mar 2016, singh.janmejay wrote: > > The problem is, pstats level control is an all or nothing kind of >> arrangement. >> >> Ideal setup would be to let metric-owner decide if metric should be >> resettable or not(which allows choosing if its a guage or not) an

Re: [rsyslog] another dyn_stats question

(counter) (or do they want it as a perpetual counter). In our case, the end-user control is very coarse-grained, where one can either get all counters as perpetual or all as resetting. On Wed, Mar 30, 2016 at 1:41 AM, David Lang <da...@lang.hm> wrote: > On Wed, 30 Mar 2016, singh.janme

Re: [rsyslog] another dyn_stats question

manipulate it with dyn_inc and dyn_dec. Resettable flag in dyn-stats bucket does the same thing as resettable-flag in statsobj static-counter. On Tue, Mar 29, 2016 at 3:13 AM, David Lang <da...@lang.hm> wrote: > On Tue, 29 Mar 2016, singh.janmejay wrote: > >> Well, from counter

Re: [rsyslog] enhancement idea, template optimizer/compiler

Makes sense. We probably don't even need to go as far as compiling. If we haven't tuned it in significant amount of time, we'll likely have big wins if we just invest in most significant PGOs. Compile to C will be hard, because it'll then require compiler and linker. Many environments will have

Re: [rsyslog] another dyn_stats question

in having two switches that get ANDed together for > resetting things? I could see two independent controls, or I could see the > pstats setting controlling everything, but I don't understand two getting > ANDed together. > > David Lang > > > On Tue, 29 Mar 2016, si

Re: [rsyslog] another dyn_stats question

n. > > David Lang > > On Mon, 28 Mar 2016, singh.janmejay wrote: > >> Date: Mon, 28 Mar 2016 23:59:46 +0530 >> >> From: singh.janmejay <singh.janme...@gmail.com> >> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com> >> To: rsyslog-users <rsys

Re: [rsyslog] another dyn_stats question

Sorry, I meant and'ed not or'ed. On Mon, Mar 28, 2016 at 11:02 PM, singh.janmejay <singh.janme...@gmail.com> wrote: > No, this reset mechanism is no different than the mechanism that > static-counters use. If you turn it off at impstats level, it won't > reset. > > Thi

Re: [rsyslog] another dyn_stats question

s(name="msgs_per_edge_relay" resettable="on" maxCardinality="3000" > unusedMetricLife="1200") > dyn_stats(name="msgs_per_core_relay" resettable="on" maxCardinality="3000" > unusedMetricLife="1200") > dyn_stats(name=&qu

Re: [rsyslog] another dyn_stats question

David, You have resetting turned-off in the config you shared with me. That would prevent resetting of counters. On Fri, Mar 25, 2016 at 8:21 PM, singh.janmejay <singh.janme...@gmail.com> wrote: > That is not supposed to happen. Although very unlikely, it may be an > environment spec

Re: [rsyslog] another dyn_stats question

able counters don't seem to be resetting for me. > > David Lang > > On Fri, 25 Mar 2016, singh.janmejay wrote: > > Well, the bug is metric-ttl ends up killing all metric instead of >> unused-metrics. This works for resettable counters, but for non-resettable >&g

Re: [rsyslog] dynastats JSON output need work

you do it! > > Rainer > > Sent from phone, thus brief. > Am 25.03.2016 06:21 schrieb "David Lang" <da...@lang.hm>: > > > As I understand Rainer, go . > > > > David Lang > > > > On Fri, 25 Mar 2016, singh.janmejay wrote: > > >

Re: [rsyslog] another dyn_stats question

t; David Lang > > On Fri, 25 Mar 2016, singh.janmejay wrote: > > This is a bug. I'll provide the fix on Monday(or in the worst case in a few >> days, in case I don't get a chance to work on this on Monday). >> >> Basically, I was supposed to have a shadow table to mark

Re: [rsyslog] dynastats JSON output need work

urge_triggered":0,"msgs_per_program.ops_overflow":0,"msgs_per_program.new_metric_add":0,"msgs_per_program.no_metric":0,"msgs_per_ p! > > rogram.metrics_purged":0,"msgs_per_program.ops_ignored":0,"msgs_per_program.purge_triggered":0,

Re: [rsyslog] another dyn_stats question

This is a bug. I'll provide the fix on Monday(or in the worst case in a few days, in case I don't get a chance to work on this on Monday). Basically, I was supposed to have a shadow table to mark used counters, and kill unused counters in next run of metric-ttl-cycle which wasn't present in first

Re: [rsyslog] dynastats JSON output need work

if they have no takers. On Wed, Mar 23, 2016 at 5:30 PM, Rainer Gerhards <rgerha...@hq.adiscon.com> wrote: > 2016-03-23 12:42 GMT+01:00 singh.janmejay <singh.janme...@gmail.com>: >> From variable-management pov, loop as a construct will be required as >> long we have a

Re: [rsyslog] dynastats JSON output need work

ist. Which in turn requires ways to work with that data-type. Doesn't it? On Wed, Mar 23, 2016 at 4:58 PM, singh.janmejay <singh.janme...@gmail.com> wrote: > In-code footprint of foreach enhancement (to handle object in addition > to array) will be fairly small (I am thinking

Re: [rsyslog] dynastats JSON output need work

h) is worse than first-class support (purely because of it using a round-about way of working). On Wed, Mar 23, 2016 at 3:01 PM, Rainer Gerhards <rgerha...@hq.adiscon.com> wrote: > 2016-03-22 20:34 GMT+01:00 singh.janmejay <singh.janme...@gmail.com>: >> What about wrapDynamicO

Re: [rsyslog] dynastats JSON output need work

, 2016 1:56 AM, "David Lang" <da...@lang.hm> wrote: > On Wed, 23 Mar 2016, singh.janmejay wrote: > > Makes sense. >> >> ..."counters":["a":80,"b":67] won't work, I think you meant >> ..."counters": [{"a" :

Re: [rsyslog] dynastats JSON output need work

Makes sense. ..."counters":["a":80,"b":67] won't work, I think you meant ..."counters": [{"a" : 80}, {"b": 67}]. So it boils down to foreach handling object or not. Thoughts? On Wed, Mar 23, 2016 at 1:18 AM, David Lang <da

Re: [rsyslog] dynastats problems with user defined variables?

abling the dyn_inc for edge relays. > > David Lang > > On Wed, 23 Mar 2016, singh.janmejay wrote: > >> Date: Wed, 23 Mar 2016 00:51:18 +0530 >> From: singh.janmejay <singh.janme...@gmail.com> >> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com> >> T

Re: [rsyslog] dynastats JSON output need work

d Lang <da...@lang.hm> wrote: > On Wed, 23 Mar 2016, singh.janmejay wrote: > >> Foreach can only work with arrays as of now. It can't work with >> objects (key-value pairs). So [{name: ... value: ...}, {..},...] is >> the only format that will work as of now. >>

Re: [rsyslog] dynastats problems with user defined variables?

It seems mostly idle(off cpu). Have you seen sched:off-cpu backtraces and backtrace-wise latency? On Wed, Mar 23, 2016 at 12:51 AM, singh.janmejay <singh.janme...@gmail.com> wrote: > You meant the first one was with "commented out" version right? (you > said uncommenting

Re: [rsyslog] dynastats problems with user defined variables?

0 0 > 100 0 0 > 0 0 62280 3203132584 2677066400 0 0 132 218 0 0 > 100 0 0 > 0 0 62280 3203912584 2677066800 0 8 203 347 0 0 > 100 0 0 > > > things just are not running with it uncommented. > > I'm going to send

Re: [rsyslog] dynastats JSON output need work

on-y. - enhancing foreach to work with {a: 10, b: 20...} On Wed, Mar 23, 2016 at 12:26 AM, David Lang <da...@lang.hm> wrote: > On Tue, 22 Mar 2016, singh.janmejay wrote: > >> How about this: we support a new flag in impstats which allows >> json-formatted stats-line to optionally

Re: [rsyslog] dynastats problems with user defined variables?

. On Wed, Mar 23, 2016 at 12:32 AM, David Lang <da...@lang.hm> wrote: > On Wed, 23 Mar 2016, singh.janmejay wrote: > >> On my cluster, each node (those 66 nodes) have 4 threads for ruleset >> (it does everything, lognorm based parsing, lookups using loopup >> ta

Re: [rsyslog] dynastats problems with user defined variables?

are working with? On Tue, Mar 22, 2016 at 8:38 PM, David Lang <da...@lang.hm> wrote: > On Tue, 22 Mar 2016, singh.janmejay wrote: > >> On Tue, Mar 22, 2016 at 5:01 AM, David Lang <da...@lang.hm> wrote: >>> >>> I added this to my configs >>> >>>

Re: [rsyslog] dynastats JSON output need work

How about this: we support a new flag in impstats which allows json-formatted stats-line to optionally use encapsulated/wrapped-layout? impstats(format="json" ...) generates {"name":"msg_per_host","origin":"dynstats.bucket","z-scribe1r-b":80,"SWEB10":67} however, impstats(format="json/w" ...)

Re: [rsyslog] dynastats problems with user defined variables?

Inline. On Tue, Mar 22, 2016 at 5:01 AM, David Lang wrote: > I added this to my configs > > # grep -B 1 -A 1 dyn_ /etc/rsyslog.conf > # custom stats to track in rsyslog > dyn_stats(name="msgs_per_host" resettable="on" maxCardinality="3000" > unusedMetricLife="1200") >

Re: [rsyslog] rsyslog 8.17.0 (v8-stable) released

This is a bug, here is the fix: https://github.com/rsyslog/rsyslog/pull/863 Workaround(in the meanwhile): Since you are not using dynstats, its safe to ignore that line (everything it reports is related to dynstats buckets). On Wed, Mar 9, 2016 at 9:34 PM, Andrew Davidoff

Re: [rsyslog] Identifying/quantifying log-loss or proving no-loss in a log-store

which is accounted for during measurement). On Sat, Feb 13, 2016 at 5:13 AM, David Lang <da...@lang.hm> wrote: > On Sat, 13 Feb 2016, singh.janmejay wrote: > >> The ideal solution would be one that identifies host, log-source and >> time of loss along with accurate

[rsyslog] Identifying/quantifying log-loss or proving no-loss in a log-store

Inviting ideas. Has anyone tried to quantify log-loss (#number of lines lost per day per sender etc) for a log-store? Let us consider the following setup: - An environment has several application nodes. Each app node hands-over its logs to local Rsyslog daemon(let us call it Ra,

Re: [rsyslog] Identifying/quantifying log-loss or proving no-loss in a log-store

The ideal solution would be one that identifies host, log-source and time of loss along with accurate number of messages lost. pstats makes sense, but correlating data from stats across large number of machines will be difficult (some machines may send stats slightly delayed which may skew

Re: [rsyslog] What would be the best place to document lookup-table and dyn-stats?

Done. PR: https://github.com/rsyslog/rsyslog-doc/pull/215 On Wed, Feb 10, 2016 at 2:31 PM, singh.janmejay <singh.janme...@gmail.com> wrote: > Makes sense. > > I'll link the PR here once done. > > -- > Regards, > Janmejay > > PS: Please blame the typos in this ma

Re: [rsyslog] What would be the best place to document lookup-table and dyn-stats?

with all those standard things. > > we do have a handful of other non-action/input/module statements. The > parser and global declarations are examples. the only thing that is unique > in table lookups is the ability to trigger the reload based on runtime > logic. > > Davi

Re: [rsyslog] What would be the best place to document lookup-table and dyn-stats?

() > clause as opposed to a different type of statement? the initial > lookup-table writeup was done a long time ago, before action() became a > significant part of the config languange. > > David Lang > > On Tue, 9 Feb 2016, singh.janmejay wrote: > > I guess everyone is O

[rsyslog] What would be the best place to document lookup-table and dyn-stats?

Does it go parallel to basic_structure or action etc? And where all should we link it from? -- Regards, Janmejay http://codehunk.wordpress.com ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog

Re: [rsyslog] What would be the best place to document lookup-table and dyn-stats?

Also, FYI, I am planning to copy over large portion of doc from lookup-table proposal. On Mon, Feb 8, 2016 at 2:59 PM, singh.janmejay <singh.janme...@gmail.com> wrote: > Does it go parallel to basic_structure or action etc? > > And where all should we link it from? > > --

Re: [rsyslog] What would be the best place to document lookup-table and dyn-stats?

I guess everyone is OK with putting it in as a function and linking out to details. -- Regards, Janmejay PS: Please blame the typos in this mail on my phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology. On Feb 8, 2016 7:50 PM, "singh.janmejay" &l

Re: [rsyslog] What would be the best place to document lookup-table and dyn-stats?

link to a page that talks about the declaration (and explains how it works etc). On Mon, Feb 8, 2016 at 5:15 PM, David Lang <da...@lang.hm> wrote: > On Mon, 8 Feb 2016, singh.janmejay wrote: > >> Does it go parallel to basic_structure or action etc? >> >> And wh

Re: [rsyslog] rsyslog next release and lookup-tables

Done. On Sun, Dec 20, 2015 at 8:52 PM, singh.janmejay <singh.janme...@gmail.com> wrote: > That is already the case. > > -- > Regards, > Janmejay > > PS: Please blame the typos in this mail on my phone's uncivilized soft > keyboard sporting it's not-so-smart-assi

Re: [rsyslog] rsyslog next release and lookup-tables

he table is loading, there will be queries against it > (from other worker threads is nothing else), and such queries should return > the data from the old version of the table. > > David Lang > > On Sat, 19 Dec 2015, singh.janmejay wrote: > > Date: Sat, 19 Dec 2015 20:43:41 +0530 >&g

Re: [rsyslog] rsyslog next release and lookup-tables

e sparse table type > is geoip lookup, using a table such as the one provided by MaxMind (some > for free, some at some costs), this can be a multi GB table that's being > loaded, so we don't want to halt log processing while the table is loading > > > On Fri, 18 Dec 2015, s

Re: [rsyslog] rsyslog next release and lookup-tables

uot;, "reload_failed") } On Thu, Nov 26, 2015 at 8:16 PM, singh.janmejay <singh.janme...@gmail.com> wrote: > Resurrecting this thread for a discussing a deviation from old > lookup_table proposal. > > The PR doesn't include load_lookup_table (a rainerscript statement > tha

Re: [rsyslog] rsyslog next release and lookup-tables

a remote destination. if (not reload_lookup_table("foo")) then { action(type="omfile"...) } Functionally, this is all that matters. Have a look at PR(comments) for details (backward compatibility issue etc). On Tue, Nov 3, 2015 at 7:37 PM, singh.janmejay <singh.janme...@gmail.com> wrote:

Re: [rsyslog] RFC: dynamic-stats support

both we'll be in a better position to decide which one should be kept. On Fri, Oct 9, 2015 at 3:20 AM, singh.janmejay <singh.janme...@gmail.com> wrote: > On Thu, Oct 8, 2015 at 11:07 PM, David Lang <da...@lang.hm> wrote: >> Atomic ops are actually rather expensive (almost

Re: [rsyslog] rsyslog next release and lookup-tables

argument that something >>> > non-existing can be a CVE. As such, I will create a new >>> > master-insecure branch, which will be a clone of the current master >>> > branch. Then I'll remove the lookup table code, so that the code base >>> > matches the documen

Re: [rsyslog] rsyslog next release and lookup-tables

Will cross-reference in the kill-feature PR. On Tue, Nov 3, 2015 at 7:37 PM, singh.janmejay <singh.janme...@gmail.com> wrote: > Sorry for the delay. Here is the PR: > https://github.com/rsyslog/rsyslog/pull/578 > > On Tue, Nov 3, 2015 at 6:02 PM, Rainer Gerhards > <

Re: [rsyslog] RFC: dynamic-stats support

-- Regards, Janmejay PS: Please blame the typos in this mail on my phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology. On Oct 7, 2015 11:25 PM, "David Lang" <da...@lang.hm> wrote: > > On Wed, 7 Oct 2015, singh.janmejay wrote: > >> -- &

Re: [rsyslog] RFC: dynamic-stats support

Yep, makes sense. I second your opinion, absolute consistency between metrics is not that valuable. On Thu, Oct 8, 2015 at 8:57 PM, Rainer Gerhards <rgerha...@hq.adiscon.com> wrote: > 2015-10-08 17:19 GMT+02:00 singh.janmejay <singh.janme...@gmail.com>: >> Did you mean i

Re: [rsyslog] RFC: dynamic-stats support

easy atomic ops you are thinking of, and > atomic ops can't operate on all sizes of data (for example, 32 bit systems > probably can't update a 64 bit value atomically) We should eventually move to thread-level accumulators for all metrics (static or dynamic). > > David Lang > > >

Re: [rsyslog] RFC: dynamic-stats support

-- Regards, Janmejay PS: Please blame the typos in this mail on my phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology. On Oct 7, 2015 3:26 AM, "David Lang" <da...@lang.hm> wrote: > > On Wed, 7 Oct 2015, singh.janmejay wrote: > >> -- &

Re: [rsyslog] RFC: dynamic-stats support

07 16:19 GMT+08:00 singh.janmejay <singh.janme...@gmail.com>: > >> -- >> Regards, >> Janmejay >> >> PS: Please blame the typos in this mail on my phone's uncivilized soft >> keyboard sporting it's not-so-smart-assist technology. >> >> On Oct 7, 201

Re: [rsyslog] Separation of actions based on log source - with good performance

Let us ship a full implementation in 8.14, let us not remove it. -- Regards, Janmejay PS: Please blame the typos in this mail on my phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology. On Oct 6, 2015 1:19 PM, "singh.janmejay" <singh.janme...@gmail.com&g

Re: [rsyslog] Separation of actions based on log source - with good performance

nuary 8.15 release iff it is ready by then. > > > @janmejay: please let me know if you would like to continue with your > > > work on lookup tables under this new situation. > > > > > > As soon as I have time, I'll check what else needs to be removed. Not

  1   2   3   >