Re: [rsyslog] Rsyslog vs syslog-ng

[singh.janmejay via rsyslog]

Relays (senders) in my case were Rsyslog too, ptcp stream-compressed
octate-counted pipes in and batched pipes out using omkafka (which was
using librdkafka with https://github.com/edenhill/librdkafka/pull/679
local patch, like I said earlier, upstream isn't always as nice as
here).

~ 600 relays per ingester-node (that is 600 connections) shooting into
each of the 66 nodes. We had alerts setup for rx buildup (on socket on
the receiver array), iirc at 3 MB, which never fired! Assuming equal
core utilization on rx and tx (remember tx was relatively more
expensive), we were doing all of this with 2 - 3 threads on rx side
(of 6 cores).

This is what Rsyslog is capable of!

On Thu, Feb 7, 2019 at 12:52 AM Dave Caplinger
 wrote:
>
> Hi Janmejay,
>
> Thanks for sharing your scenario as well.  Based on your log size data I 
> realized that I meant 24B logs/day rather than 24M!
>
> Here are some more details in case anyone else here is interested.  The data 
> I referred to was for a single day a few years ago, and the specifics based 
> on rsyslog-pstats data were:
>
> |  24,488,120,824 log lines
> |  12,649,411,012,274 bytes total
> |  15.1:1 compression ratio (6.6% of original)
> | 283,427 logs/sec avg
> | 516 bytes/log avg
>
> This data was received via ptcp by a pool of 12 central aggregators and came 
> from about 1000 remote collectors.  Since we control both the collectors and 
> the aggregators and they both run rsyslog, we could take advantage of the 
> stream compression support (which resulted in a significant reduction in 
> network bandwidth).
>
> There is a definite "long tail" to our log size distribution.  I don't have 
> the exact min/max numbers available any more for that day, but they can range 
> roughly between 50 and 8K bytes.  A size distribution similar to this sample 
> is pretty typical:
>
> |  # NumSamples = 53000; Min = 92.00; Max = 8206.00
> |  # Mean = 653.060623; Variance = 112272.110834; SD = 335.070307; Median 
> 521.50
> |  # each ∎ represents a count of 485
> | 92. -   903.4000 [ 36375]: 
> ∎∎∎
> |903.4000 -  1714.8000 [ 16362]: ∎
> |   1714.8000 -  2526.2000 [   219]:
> |   2526.2000 -  3337.6000 [19]:
> |   3337.6000 -  4149. [14]:
> |   4149. -  4960.4000 [ 4]:
> |   4960.4000 -  5771.8000 [ 1]:
> |   5771.8000 -  6583.2000 [ 0]:
> |   6583.2000 -  7394.6000 [ 2]:
> |   7394.6000 -  8206. [ 4]:
>
> I hope this discussion prompts some of the other high-volume rsyslog users on 
> this list to share some details also...
>
> Best regards,
>
> --
> Dave Caplinger | Chief Architect, Global Platform Engineering | NTT Security 
> Corporation
>
>
> > On Feb 6, 2019, at 3:03 AM, singh.janmejay  wrote:
> >
> > I have used Rsyslog in the past to run a 66 node array (6 core VMs) to
> > process 6M log-events / second (avg size 1.5 KB), which boils down to
> > 69 Gbps of log traffic. This was being received over ptcp and
> > delivered to the backend over omkafka.
> >
> > Rsyslog and HDFS (of Hadoop) were by-far the most trouble-free
> > components in this architecture. In addition to data-plane, Rsyslog
> > did accounting based on custom fields (such as cluster-id or
> > application-id), reported those stats, performed L7 service defence
> > (blocking abuse) and helped get data out for escallation to L3 defense
> > and much more.
> >
> > We saw everything break at this scale, including Rsyslog, but had to
> > make very few and relatively tiny changes to Rsyslog (all merged
> > upstream, btw) to get things going whereas most other components had
> > to be replaced or large parts of them had to be re-written.
> >
> > So when it comes to stability and performance, I totally swear by Rsyslog.
> >
> > When building such large systems one doesn't want to take over
> > maintainership of components used (which is sometimes the sorry state
> > things get into when upstream doesn't acknowledge the problems or
> > doesn't accept fixes for it). We were at 0 locally patched changes on
> > Rsyslog build, thanks to Rsyslog community which is super responsive
> > and helpful.
> >
> > If I ever build such a system again, I would choose Rsyslog for
> > log-mux with my eyes closed.
> >
> > On Wed, Feb 6, 2019 at 5:30 AM Dave Caplinger via rsyslog
> >  wrote:
> >>
> >> Hi Vishal,
> >>
> >> It's been many years since we switched from (open source) syslog-ng to 
> >> rsyslog.  We did it because we w

Re: [rsyslog] Rsyslog vs syslog-ng

[singh.janmejay via rsyslog]

I have used Rsyslog in the past to run a 66 node array (6 core VMs) to
process 6M log-events / second (avg size 1.5 KB), which boils down to
69 Gbps of log traffic. This was being received over ptcp and
delivered to the backend over omkafka.

Rsyslog and HDFS (of Hadoop) were by-far the most trouble-free
components in this architecture. In addition to data-plane, Rsyslog
did accounting based on custom fields (such as cluster-id or
application-id), reported those stats, performed L7 service defence
(blocking abuse) and helped get data out for escallation to L3 defense
and much more.

We saw everything break at this scale, including Rsyslog, but had to
make very few and relatively tiny changes to Rsyslog (all merged
upstream, btw) to get things going whereas most other components had
to be replaced or large parts of them had to be re-written.

So when it comes to stability and performance, I totally swear by Rsyslog.

When building such large systems one doesn't want to take over
maintainership of components used (which is sometimes the sorry state
things get into when upstream doesn't acknowledge the problems or
doesn't accept fixes for it). We were at 0 locally patched changes on
Rsyslog build, thanks to Rsyslog community which is super responsive
and helpful.

If I ever build such a system again, I would choose Rsyslog for
log-mux with my eyes closed.

On Wed, Feb 6, 2019 at 5:30 AM Dave Caplinger via rsyslog
 wrote:
>
> Hi Vishal,
>
> It's been many years since we switched from (open source) syslog-ng to 
> rsyslog.  We did it because we were struggling with configuration complexity 
> and performance issues, and also because Balabit supported certain features 
> (such as local disk buffering) only in the commercial-only version, but their 
> pricing model was not justifiable for us at the time.  I'm sure that 
> syslog-ng has changed quite a bit in that time so a comparison today may not 
> turn out the same way, but we've been very happy with rsyslog.  If syslog 
> collection is important to you or your organization, my opinion is that 
> rsyslog is the best choice.  The prompt adoption of new technologies over the 
> years such as json, liblognorm, stream-compression, elasticsearch, kafka, and 
> many others demonstrate that this project continues to have an active user 
> and development community.
>
> As for performance: we are certainly not the largest-volume users of rsyslog, 
> but as one anecdotal example I happen to have handy, we've used rsyslog to 
> collect over 24 million logs (around 12.5 TB) per day.  You will need to 
> learn about action queues, buffer sizing, pstats, and tuning, but rsyslog can 
> handle it.
>
> Additionally, we've been happy to help financially support the rsyslog 
> project by maintaining an Adiscon support contract, as several others on this 
> mailing-list do as well (just to clarify that unwillingness to pay was not 
> the reason we decided not to go with syslog-ng pro).
>
> Best regards,
>
> --
> Dave Caplinger | Chief Architect, Global Platform Engineering | NTT Security 
> Corporation
>
> > On Feb 4, 2019, at 12:45 AM, vishal via rsyslog  
> > wrote:
> >
> > Hi,
> > I am evaluating rsyslog and syslogng for our project.
> > Though aware of some of the differences and pros and cons, but still would 
> > like to know the differences which users have faced and evaluated in terms 
> > of ease of use, robustness, handling huge volumes of logs and deployment 
> > scenarios (single host to multi host cluster) , and if there are any other 
> > important areas to be considered.
> >
> > The general deployment would be,
> >
> > Log sources -> rsyslog/syslogng -> elasticsearch
> >
> >
> > Thanks.
> >
> >
> > ___
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> > LIKE THAT.
>
>
> Confidentiality Notice: The content of this communication, along with any 
> attachments, is covered by federal and state law governing electronic 
> communications and may contain confidential and legally privileged 
> information. If the reader of this message is not the intended recipient, you 
> are hereby notified that any dissemination, distribution, use or copying of 
> the information contained herein is strictly prohibited. If you have received 
> this communication in error, please immediately contact us by telephone at 
> 402.361.3000 or e-mail security-ameri...@nttsecurity.com.
>
> Copyright 2000-2018 NTT Security (US) Inc., a wholly-owned subsidiary of NTT 
> Group. All rights reserved. NTT Security is a trademark of NTT Security GMBH.
>
> ___
> rsyslog mailing list
> 

Re: [rsyslog] Transforming JSON Field Names in rawmsg

[singh.janmejay via rsyslog]

It seems like you can parse the
message(https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmjsonparse.html)
and set new fields (and unset the old fields) in rainerscript
(https://www.rsyslog.com/doc/v8-stable/rainerscript/variable_property_types.html).
On Wed, Oct 3, 2018 at 7:06 PM John Chivian  wrote:
>
> Thank you David.  You're comment has led me to this...
>
>ruleset(name="fix_names") {
>  action(
>type="mmexternal"
>name="normalize-names"
>binary="/etc/rsyslog.d/transforms/prefix_names.sh"
>interface.input="rawmsg"
>output="/logspool/prefix_results.txt"
>  )
>  action(
>type="omfile"
>name="mmexternal-debug"
>template="rawmsg"
>File="/logspool/mext.out"
>  )
> }
>
> I know the external script is functioning because the prefix_results.txt
> file (from the first action) shows the correct results, but when I then
> immediately write out the rawmsg (in the second action) I get the
> original, unmodified value.
>
> I must be missing something fundamental, and will revisit the
> documentation, but I'd be grateful for any words of wisdom or guidance.
>
> Thanks, John
>
> On 10/2/18 3:11 PM, David Lang wrote:
> > On Tue, 2 Oct 2018, John Chivian wrote:
> >
> >> Hello Group:
> >>
> >>I am trying to determine the best way to transform the field names
> >> of a simple JSON object that is rawmsg.  The objects are fluid having
> >> both numeric and string content, but are always in the form...
> >>
> >> { "aStr": "aString","bStr": "bString","cNum": 0,"dStr":
> >> "cString" }
> >>
> >>I need to add a prefix "ny_" to the field names such that the
> >> result would be...
> >>
> >> { "ny_aStr": "aString","ny_bStr": "bString","ny_cNum":
> >> 0,"ny_dStr": "cString" }
> >>
> >>I have a sed script with extractions that can do this
> >> transformation...
> >>
> >> 's/\([^"]\+\)"\([^"]\+\)":/\1"ny_\2":/g'
> >>
> >>...but I don't know if it's possible to integrate that into a
> >> template with the property replacer, or if there's a better cleaner
> >> way to do it.
> >>
> >>Any and all recommendations are greatly appreciated.
> >
> > I think you would have to resort to code outside of rsyslog (either a
> > custom mm module or mmexternal to call a script of your devising) to
> > change the field contents like that for an arbitrary and changing list
> > of fields.
> >
> > David Lang
> > ___
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> > if you DON'T LIKE THAT.
>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Imkafka/omkafka tuning

[singh.janmejay via rsyslog]

The omkafka action worker doesn't perform deliveries. Its
responsibility is limited to enqueue and ack-processing. Actual
deliveries are done by broker-bound threads and although the design
delivers sufficient throughput with small kafka clusters, it chokes on
lock contention as broker count goes up.

I created it with this patch
(https://github.com/edenhill/librdkafka/pull/679) to fix this problem
(IIRC we had a successful 300 broker benchmark aggregating to 6M
events per second), this allows controlling #threads for deliveries
independent of #brokers reducing severity of contention (have a look
at screenshots in the PR). The maintainer of librdkafka had a
different design in mind, I haven't followed up on the change after
fixing it for myself.

On Wed, Jan 3, 2018 at 10:25 PM, Andrew Griffin via rsyslog
 wrote:
> I use omkafka heavily - the confParam setting is what you’re looking for, 
> basically all of the standard Kafka producer / consumer related parameters 
> are available there.  (see  
> https://kafka.apache.org/documentation/#producerconfigs 
> ).  What you set and 
> how you set it is heavily dependent on your use case - number of producers, 
> number of consumers, number / size of topics, throughput expectations, etc.  
> But in general for omkafka (producer) you’ll get the most results tuning 
> buffer.memory and batch.size.  Those will help you adjust the size and 
> frequency of producing messages.
>
> To add threads out to Kafka?  I’m not positive but I’d assume adding multiple 
> rulesets may accomplish that goal - say, multiple listening ports, each with 
> it’s own dedicated ruleset.  That would add additional rsyslog queues, and 
> thus multiple output queues. Though in our setup we have 1 instance of 
> rsyslog running per Kafka broker, and have never run in to a case where 
> either rsyslog or Kafka gets bottlenecked
>
> Andrew
>
>> On Dec 23, 2017, at 8:56 PM, deoren 
>>  wrote:
>>
>>
>>
>> On 12/22/2017 9:52 AM, Luigi Tagliamonte via rsyslog wrote:
>>> Hi there!
>>> What are the tunable parameters for this module, like:
>>> - an option to increase the number of threads for kafka processing
>>> - number of messages to process per req.
>>> - etc..
>>> Regards
>>> L.
>>
>>
>> Module docs:
>>
>> * http://www.rsyslog.com/doc/v8-stable/configuration/modules/omkafka.html
>> * http://www.rsyslog.com/doc/v8-stable/configuration/modules/imkafka.html
>>
>> Disclaimer: I know little (if anything) about Apache Kafka, but I found
>> the following mentioned in the module docs:
>>
>> ###
>> confParam [parameter]
>> Type: Array
>>
>> Default: none
>>
>> Permits to specify Kafka options. Rather than offering a myriad of
>> config settings to match the Kafka parameters, we provide this setting
>> here as a vehicle to set any Kafka parameter. This has the big advantage
>> that Kafka parameters that come up in new releases can immediately be used.
>>
>> Note that we use librdkafka for the Kafka connection, so the parameters
>> are actually those that librdkafka supports. As of our understanding,
>> this is a superset of the native Kafka parameters.
>> ###
>>
>>
>> That bit is the same for both modules. The omkafka module has this in
>> addition to the above:
>>
>>
>> ###
>> topicConfParam [parameter]
>> Type: Array
>>
>> Default: none
>>
>> In essence the same as confParam, but for the Kafka topic.
>> ###
>>
>>
>> I assume that means that you can use the confParam parameter for the
>> rsyslog imkafka/omkafka modules to pass through native Kafka
>> parameters/settings.
>>
>> Hope that helps. If not, I hope that someone with more knowledge of this
>> module can chime in and assist.
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
>> LIKE THAT.
>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog

Re: [rsyslog] Problems with rsyslog Versions > 8.16 and Thread-Handling

[singh.janmejay via rsyslog]

Hi Kendall,

I need a repro for this problem, it has remained open far too long for my
comfort, but I simply haven't been able to get to the bottom of this over
mail and issue-tracker.

Does the image (rhel version) you use exist as an AMI on EC2? Or if you can
spare some time, it'll help me immensely if you can pass me a virtual box
image.

Or if possible, can you move a box reproducing this to DMZ, strip the
config down to the least necessary, remove all sensitive material out of it
and let me debug it in place?

I need this issue to reproduce consistently while I debug it. Any other
ideas on how we may do this?

On 04-Jun-2017 7:26 AM, "Kendall Green via rsyslog" <
rsyslog@lists.adiscon.com> wrote:

> Updated following details on Issue:
> https://github.com/rsyslog/rsyslog/issues/1071
>
> I've recently setup lookup tables to control which ruleset to call with
> $fromhost-ip. The call appears to work properly, however there's a segfault
> in /var/log/messages on service rsyslog start, and the reload lookup tables
> does not function on HUP nor with message trigger. The segfault only
> happened after config of lookup tables, with and without the load and
> reload as part of ruleset. The ruleset does call to another ruleset by set
> $.var and if $.var == $fromhost-ip, using similar syntax as example configs
> from lookup in rsyslog docs:
>
> lookup_table(name="test_lu" file="/etc/rsyslog.d/lu/test_lu.json"
> reloadOnHUP="off")
>
> if ($rawmsg contains "test_lu")
> then {
>reload_lookup_table("test_lu, "unk")
>stop
> }
>
> set $.test = lookup("test_lu", $fromhost-ip);
>
> Tried other config tests, with same result, the match lookup table value
> works to call other ruleset, but reload lookup table does not take effect,
> as reload only happens when restarting rsyslog, and not with message
> trigger or HUP. All tests with lookup table loaded has segfault in
> /var/log/message on start. I had commented out the lookup() and
> reload_lookup_table() and started with just loading the lookup_table in
> config and still got segfault. Plus, tried with the reloadOnHUP on or off,
> but segfault either way. The segfault does not appear after removing /
> commenting out the lookup table from config, and these configs have not
> produced segfaults in the past.
>
> kernel: rsyslogd[5434]: segfault at 7faab145f9d0 ip 7faab44e5213 sp
> 7fff94cf9940 error 4 in libpthread-2.12.so[7faab44dd000+17000]
>
> Confirmed issue on rhel-6.9 with rsyslog v8.24, v8.26.x, and v8.27.0.
> Verified libjson-c is not installed, and the issue was not resolved with
> upgrade of libfastjson4-0.99.4-1.el6 to latest libfastjson4-0.99.5-1.el6.
> Is there another version of libfastjson should use?
>
> On Thu, Jan 12, 2017 at 6:48 AM, David Lang  wrote:
>
> > is the version you are using linked to json-c or libfastjson?
> >
> > we know there were thread-safe problems as a result of json-c
> >
> > David Lang
> >
> > On Thu, 12 Jan 2017, Christopher Racky via rsyslog wrote:
> >
> > Date: Thu, 12 Jan 2017 12:43:31 +0100
> >> From: Christopher Racky via rsyslog 
> >> To: rsyslog-users 
> >> Cc: Christopher Racky 
> >> Subject: [rsyslog] Problems with rsyslog Versions > 8.16 and
> >> Thread-Handling
> >>
> >>
> >> Hello,
> >>
> >> I still have big issues with different Servers running RHEL 6.8 (incl.
> >> latest updates) and rsyslog > 8.16.
> >>
> >> While 8.16 works fine, all following versions, including 8.24 which
> >> was released a few days ago leads to problems that results in
> >> core-dumps:
> >> rsyslogd[1408]: segfault at 7f9f2910b9d0 ip 7f9f2e18e213 sp
> >> 7ffd3d9fe080 error 4 in libpthread-2.12.so[7f9f2e186000+17000]
> >> rsyslogd[4201]: segfault at 7f991c0789d0 ip 7f99210fb213 sp
> >> 7ffec5155240 error 4 in libpthread-2.12.so[7f99210f3000+17000]
> >> rsyslogd[2783]: segfault at 7f5f695399d0 ip 7f5f6e5bd213 sp
> >> 7ffe4b4da5c0 error 4 in libpthread-2.12.so[7f5f6e5b5000+17000]
> >> rsyslogd[6816]: segfault at 7f519ad269d0 ip 7f519fdaa213 sp
> >> 7ffd22bf2470 error 4 in libpthread-2.12.so[7f519fda2000+17000]
> >>
> >>
> >> My hope was that this topic was solved by
> >> https://github.com/rsyslog/rsyslog/pull/1274
> >>
> >> But it was not.
> >> As problems seems not (directly) related to lookup-table, I guess it
> >> also has a problem with thread handling.
> >> https://github.com/rsyslog/rsyslog/issues/1071
> >>
> >> Do you have any further idea?
> >>
> >> With single-thread mode problem does not appear and also in lab, I was
> >> not able to reproduce it.
> >> But on several production servers the issue occures quite often.
> >>
> >>
> >> regards
> >> Chris
> >> ___
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow 

Re: [rsyslog] Duplicated logs on rolling file

[singh.janmejay via rsyslog]

Log4j has a way to rollover to new file (without truncating old file).
I'd look at Rolling Appender if you want to use it with imfile.

Or its simpler and more efficient to make log4j send logs over to
Rsyslog (or any other syslog impl) directly. Here what a log4j
properties file with syslog appender may look like:

log4j.rootLogger=debug, syslog
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.EnhancedPatternLayout
log4j.appender.stdout.layout.ConversionPattern=%5p [%t] (%F:%L) - %m%n
%throwable{1000}

log4j.appender.syslog=org.apache.log4j.net.SyslogAppender
log4j.appender.syslog.syslogHost=127.0.0.1:11514
log4j.appender.syslog.layout=org.apache.log4j.EnhancedPatternLayout
log4j.appender.syslog.layout.conversionPattern=log@1,myapp,myteam
%d{ISO8601} %-5p [%t] %c{2} %x - %m%n %throwable{1000}
log4j.appender.syslog.Facility=LOCAL0



On Fri, Jun 2, 2017 at 4:23 PM, Rainer Gerhards
 wrote:
> Hola Lucas,
>
>
> 2017-06-02 11:39 GMT+02:00 Lucas Ventura Carro via rsyslog
> :
>> Hallo,
>>
>> I'm developing a webapp Java, which logs to a rolling file using log4j-2.
>> The logged file(s) are later forwarded with rsyslog to 2 actions: 'omfwd'
>> to a Splunk server and 'omprog'.
>>
>> I've been executing the high load tests on the webapp, and once they finish
>> I've realized that on both destinations are more logs than in files. And
>> these duplicated logs are the first(s) line(s) of the rolled files.
>>
>> When trying to activate rsyslog with debug[1], doesn't send all the logs
>> from the original files then.
>>
>> I've published a simpler example[2] which reproduces almost all the times
>> the issue. Is a Java application which can be run with gradle inside a
>> docker container.
>>
>> There is also a Splunk server runnable with docker, in the same repo.
>>
>> Can be a rsyslog rolling detection issue?
>
> It sounds like the log4j config overwrites the same inode. If so, this
> is the root cause of the problem. Overwriting the same inode is highly
> racy. On a busy system, you will almost always loose some logs. It can
> also happen that the logic that *tries* to detect overwrites
> misdetects them and duplicates potentially large amounts of data. We
> have seen this in the past and for that reason imfile did not support
> overwrites at all for quite some while.
>
> Based on user requests, we *experiementally* added the option
> "reopenOnTruncate" to support this somewhat again. However, as we know
> this is racy and highly problematic, we do not even try to debug such
> situations - we've done this often enough in the past and the plain
> truth is that problems are inevitable in that mode.
>
> The real cure is to ensure that a new inode is created when the log
> file is rolled over.
>
> We should probably add an even bolder warning to the doc that highly
> discourages to use "reopenOnTruncate".
>
> Sorry I have no better answer, but that's simply how it is...
> Rainer
>>
>> Thanks,
>>
>>   [1]: http://www.rsyslog.com/doc/v8-stable/troubleshooting/debug.html
>>   [2]: https://github.com/lucasvc/rsyslog-tests/tree/master/log4j2-rollover
>> --
>> Lucas
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
>> LIKE THAT.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] conversion functions

[singh.janmejay]

Intermediate form was necessary for the convert(3) form right?

The small, narrow scope functions don't need an intermediate form.

If we come across a usecase to translate say decimal represented integer to
octal, we create a function called base(decimal-value, 8). This doesn't
require one to convert to intermediate representation before conversion
back to desired format.

Alternatively we can choose a intermediate representation based
implementation but then optimizers will have to be a lot more complex and
have capability to analyze AST and generate optimal code eliminating the IR
form. In this approach IR will not exist at runtime, so all we need is a
AST level marker that points the optimizer to the right place.

In convert(3) form IR will be completely internal, but then the abstraction
will be slightly more complex for a new user. Implementation may again
choose to eliminate IR as an  optimization.

On Nov 28, 2016 12:08 AM, "David Lang" <da...@lang.hm> wrote:

> On Sun, 27 Nov 2016, singh.janmejay wrote:
>
> I was thinking of something like parse_int("0xa"), parse_int("012") and
>> parse_int("10") all returning 10.
>>
>> I didn't really have representing numbers as strings in mind, but it is
>> certainly needed.
>>
>
> what was the intermediate form that you were expecting to use?
>
> David Lang
>
> I can't really think of a very elegant signature for that.
>>
>> So something like str_hex(10) would return "0xa" and user can use
>> substring
>> function or field function to trim out the prefix.
>>
>> str(10) would return "10", str(15.2) returns "15.2" and so on?
>>
>> So signature is str_(expr) and str is basically an alias to str_dec.
>> Here str_dec(expr) will accept both floating-point or integer. Others will
>> expect integer.
>>
>> On Nov 27, 2016 2:58 PM, "Rainer Gerhards" <rgerha...@hq.adiscon.com>
>> wrote:
>>
>> I also think we should use special functions for each type of conversion.
>>>
>>> Reasoning: I actually think a single function is a very nice
>>> abstraction. rsyslog does a lot of nice and very good abstractions.
>>> Looking at the feedback and the overall conception "rsyslog is too
>>> complex to use easily", I conclude that most people have problems with
>>> abstractions. They are used to doing cint(val), clong(val), and trying
>>> to get off the "too complex" conception, IMHO we should support that.
>>>
>>> Of course, internally we can use a single nicely abstracted function :-)
>>>
>>> Rainer
>>>
>>> 2016-11-27 8:45 GMT+01:00 David Lang <da...@lang.hm>:
>>>
>>>> On Sun, 27 Nov 2016, singh.janmejay wrote:
>>>>
>>>> I meant rainerscript functions. It'll boil down to equivalent functions
>>>>>
>>>> in
>>>
>>>> C too, but that is implementation detail.
>>>>>
>>>>> Did I misunderstand the context?
>>>>>
>>>>
>>>>
>>>> the context I was thinking of was rsyslog.conf. I just wasn't thinking
>>>> of
>>>> how your suggestion would work.
>>>>
>>>> so you are suggesting that we have X2num and then num2x functions, so if
>>>>
>>> you
>>>
>>>> needed to convert hex to octal you would do
>>>>
>>>> set $.var1 = hex2num("$.var");
>>>> set $.var2 = num2oct("$.var1);
>>>>
>>>> vs
>>>>
>>>> set $.var2 = convert("$.var","hex","oct");
>>>>
>>>> am I understanding you correctly?
>>>>
>>>>
>>>> David Lang
>>>>
>>>> On Nov 27, 2016 10:49 AM, "David Lang" <da...@lang.hm> wrote:
>>>>>
>>>>> On Sun, 27 Nov 2016, singh.janmejay wrote:
>>>>>>
>>>>>> I feel it's better to have parse_(expr) and cast_(expr)
>>>>>> family
>>>>>>
>>>>>>>
>>>>>>> of functions that return the appropriate value.
>>>>>>>
>>>>>>> Eg. If x is "12.3", parse_double(x) returns 12.3 (double precision),
>>>>>>>
>>>>>> and
>>>
>>>> cast_int(parse_double(x)) returns 12.
>>>>>>>
>>>>>>> Cast detects the source type and acts accordingly, parse expects
>>>>>>> 

Re: [rsyslog] conversion functions

[singh.janmejay]

The direction I was aiming for was simple functions that are compose-able
with easy-to-guess names.

Also, may be at some point it makes sense to namespace functions. Eg. I can
imagine parse being a namespace, allowing for shorter function names under
that namespace. We can namespace by prefixing _ to function names, but
a more syntactically specific namespacing mechanism like . or : may
be worth thinking about.

The benefit is, someone looking at documentation can find all related
functions in one section or page.

Even if we choose to not namespace, making function documentation
2-level-hierarchical and clubbing functions by their nature may be equally
effective (but we'll have to maintain it purely by discipline, whereas
namespacing will make it more natural).

On Nov 27, 2016 5:27 PM, "Rainer Gerhards" <rgerha...@hq.adiscon.com> wrote:

> 2016-11-27 12:29 GMT+01:00 singh.janmejay <singh.janme...@gmail.com>:
> > I was thinking of something like parse_int("0xa"), parse_int("012") and
> > parse_int("10") all returning 10.
> >
> > I didn't really have representing numbers as strings in mind, but it is
> > certainly needed.
>
> I did not mean this, it also already happens when required and we have
> also cstr(), which does not give you full control over the format.
>
> I meant what you said with e.g.pare_int(), which we also already have as
> cnum():
>
> http://www.rsyslog.com/doc/v8-stable/rainerscript/functions.html
>
> My point is that while I like a single abstracted function, it is less
> along the typical user expectations than cnum("1"), cint("1"), etc...
>
> So I think we should identify which conversions we actually need and
> craft individual functions for them - for sake of user experience.
>
> Rainer
> >
> > I can't really think of a very elegant signature for that.
> >
> > So something like str_hex(10) would return "0xa" and user can use
> substring
> > function or field function to trim out the prefix.
> >
> > str(10) would return "10", str(15.2) returns "15.2" and so on?
> >
> > So signature is str_(expr) and str is basically an alias to
> str_dec.
> > Here str_dec(expr) will accept both floating-point or integer. Others
> will
> > expect integer.
> >
> > On Nov 27, 2016 2:58 PM, "Rainer Gerhards" <rgerha...@hq.adiscon.com>
> wrote:
> >
> >> I also think we should use special functions for each type of
> conversion.
> >>
> >> Reasoning: I actually think a single function is a very nice
> >> abstraction. rsyslog does a lot of nice and very good abstractions.
> >> Looking at the feedback and the overall conception "rsyslog is too
> >> complex to use easily", I conclude that most people have problems with
> >> abstractions. They are used to doing cint(val), clong(val), and trying
> >> to get off the "too complex" conception, IMHO we should support that.
> >>
> >> Of course, internally we can use a single nicely abstracted function :-)
> >>
> >> Rainer
> >>
> >> 2016-11-27 8:45 GMT+01:00 David Lang <da...@lang.hm>:
> >> > On Sun, 27 Nov 2016, singh.janmejay wrote:
> >> >
> >> >> I meant rainerscript functions. It'll boil down to equivalent
> functions
> >> in
> >> >> C too, but that is implementation detail.
> >> >>
> >> >> Did I misunderstand the context?
> >> >
> >> >
> >> > the context I was thinking of was rsyslog.conf. I just wasn't
> thinking of
> >> > how your suggestion would work.
> >> >
> >> > so you are suggesting that we have X2num and then num2x functions, so
> if
> >> you
> >> > needed to convert hex to octal you would do
> >> >
> >> > set $.var1 = hex2num("$.var");
> >> > set $.var2 = num2oct("$.var1);
> >> >
> >> > vs
> >> >
> >> > set $.var2 = convert("$.var","hex","oct");
> >> >
> >> > am I understanding you correctly?
> >> >
> >> >
> >> > David Lang
> >> >
> >> >> On Nov 27, 2016 10:49 AM, "David Lang" <da...@lang.hm> wrote:
> >> >>
> >> >>> On Sun, 27 Nov 2016, singh.janmejay wrote:
> >> >>>
> >> >>> I feel it's better to have parse_(expr) and cast_(expr)
> >> >>> family
> >> >>>>
> >> >>>

Re: [rsyslog] conversion functions

[singh.janmejay]

I was thinking of something like parse_int("0xa"), parse_int("012") and
parse_int("10") all returning 10.

I didn't really have representing numbers as strings in mind, but it is
certainly needed.

I can't really think of a very elegant signature for that.

So something like str_hex(10) would return "0xa" and user can use substring
function or field function to trim out the prefix.

str(10) would return "10", str(15.2) returns "15.2" and so on?

So signature is str_(expr) and str is basically an alias to str_dec.
Here str_dec(expr) will accept both floating-point or integer. Others will
expect integer.

On Nov 27, 2016 2:58 PM, "Rainer Gerhards" <rgerha...@hq.adiscon.com> wrote:

> I also think we should use special functions for each type of conversion.
>
> Reasoning: I actually think a single function is a very nice
> abstraction. rsyslog does a lot of nice and very good abstractions.
> Looking at the feedback and the overall conception "rsyslog is too
> complex to use easily", I conclude that most people have problems with
> abstractions. They are used to doing cint(val), clong(val), and trying
> to get off the "too complex" conception, IMHO we should support that.
>
> Of course, internally we can use a single nicely abstracted function :-)
>
> Rainer
>
> 2016-11-27 8:45 GMT+01:00 David Lang <da...@lang.hm>:
> > On Sun, 27 Nov 2016, singh.janmejay wrote:
> >
> >> I meant rainerscript functions. It'll boil down to equivalent functions
> in
> >> C too, but that is implementation detail.
> >>
> >> Did I misunderstand the context?
> >
> >
> > the context I was thinking of was rsyslog.conf. I just wasn't thinking of
> > how your suggestion would work.
> >
> > so you are suggesting that we have X2num and then num2x functions, so if
> you
> > needed to convert hex to octal you would do
> >
> > set $.var1 = hex2num("$.var");
> > set $.var2 = num2oct("$.var1);
> >
> > vs
> >
> > set $.var2 = convert("$.var","hex","oct");
> >
> > am I understanding you correctly?
> >
> >
> > David Lang
> >
> >> On Nov 27, 2016 10:49 AM, "David Lang" <da...@lang.hm> wrote:
> >>
> >>> On Sun, 27 Nov 2016, singh.janmejay wrote:
> >>>
> >>> I feel it's better to have parse_(expr) and cast_(expr)
> >>> family
> >>>>
> >>>> of functions that return the appropriate value.
> >>>>
> >>>> Eg. If x is "12.3", parse_double(x) returns 12.3 (double precision),
> and
> >>>> cast_int(parse_double(x)) returns 12.
> >>>>
> >>>> Cast detects the source type and acts accordingly, parse expects
> >>>> expression
> >>>> to produce a string.
> >>>>
> >>>
> >>> are you meaning to have these in the rsyslog.conf file or as functions
> in
> >>> the C source?
> >>>
> >>> David Lang
> >>>
> >>> On Nov 27, 2016 4:14 AM, "David Lang" <da...@lang.hm> wrote:
> >>>>
> >>>>
> >>>> We currently have issues open for a few different conversion functions
> >>>>>
> >>>>> (some date related, ip2num)
> >>>>>
> >>>>> should we start down the path of having a bunch of different
> functions
> >>>>> for
> >>>>> all the permutations of source and destination formats? or should we
> >>>>> have a
> >>>>> convert() function that lets you specify the source and destination
> >>>>> formats
> >>>>>
> >>>>> There are a bunch of 'types' that can all be represented internally
> >>>>> with
> >>>>> a
> >>>>> fairly simple binary representation
> >>>>>
> >>>>> decimal
> >>>>> hex (including variations like we commonly see for mac addresses and
> >>>>> ipv6)
> >>>>> octal
> >>>>> ipv4
> >>>>> dates
> >>>>>
> >>>>> these may be extended slightly from being integers to having a
> >>>>> fixed-point
> >>>>> fractional component (not floating point, which can't properly
> >>>>> represent
> >>>>> 0.1)
> >>>>>
> >>>>> so should 

Re: [rsyslog] conversion functions

[singh.janmejay]

I meant rainerscript functions. It'll boil down to equivalent functions in
C too, but that is implementation detail.

Did I misunderstand the context?

On Nov 27, 2016 10:49 AM, "David Lang" <da...@lang.hm> wrote:

> On Sun, 27 Nov 2016, singh.janmejay wrote:
>
> I feel it's better to have parse_(expr) and cast_(expr) family
>> of functions that return the appropriate value.
>>
>> Eg. If x is "12.3", parse_double(x) returns 12.3 (double precision), and
>> cast_int(parse_double(x)) returns 12.
>>
>> Cast detects the source type and acts accordingly, parse expects
>> expression
>> to produce a string.
>>
>
> are you meaning to have these in the rsyslog.conf file or as functions in
> the C source?
>
> David Lang
>
> On Nov 27, 2016 4:14 AM, "David Lang" <da...@lang.hm> wrote:
>>
>> We currently have issues open for a few different conversion functions
>>> (some date related, ip2num)
>>>
>>> should we start down the path of having a bunch of different functions
>>> for
>>> all the permutations of source and destination formats? or should we
>>> have a
>>> convert() function that lets you specify the source and destination
>>> formats
>>>
>>> There are a bunch of 'types' that can all be represented internally with
>>> a
>>> fairly simple binary representation
>>>
>>> decimal
>>> hex (including variations like we commonly see for mac addresses and
>>> ipv6)
>>> octal
>>> ipv4
>>> dates
>>>
>>> these may be extended slightly from being integers to having a
>>> fixed-point
>>> fractional component (not floating point, which can't properly represent
>>> 0.1)
>>>
>>> so should we create a function
>>>
>>> convert(var,sformat,dformatt[,extradata])
>>>
>>> where the function converts the variable contents from sformat to the
>>> internal binary representation and then from that to the dformat
>>> (allowing
>>> the optional extradata parameter to specify tweaks to the dformat)
>>>
>>> thoughts.
>>>
>>> David Lang
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] conversion functions

[singh.janmejay]

I feel it's better to have parse_(expr) and cast_(expr) family
of functions that return the appropriate value.

Eg. If x is "12.3", parse_double(x) returns 12.3 (double precision), and
cast_int(parse_double(x)) returns 12.

Cast detects the source type and acts accordingly, parse expects expression
to produce a string.

On Nov 27, 2016 4:14 AM, "David Lang"  wrote:

> We currently have issues open for a few different conversion functions
> (some date related, ip2num)
>
> should we start down the path of having a bunch of different functions for
> all the permutations of source and destination formats? or should we have a
> convert() function that lets you specify the source and destination formats
>
> There are a bunch of 'types' that can all be represented internally with a
> fairly simple binary representation
>
> decimal
> hex (including variations like we commonly see for mac addresses and ipv6)
> octal
> ipv4
> dates
>
> these may be extended slightly from being integers to having a fixed-point
> fractional component (not floating point, which can't properly represent
> 0.1)
>
> so should we create a function
>
> convert(var,sformat,dformatt[,extradata])
>
> where the function converts the variable contents from sformat to the
> internal binary representation and then from that to the dformat (allowing
> the optional extradata parameter to specify tweaks to the dformat)
>
> thoughts.
>
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Disabling IMPTCP FlowControl

[singh.janmejay]

Sys% 55 and usr% 18 is very surprising.

The other question I forgot to ask was around octate-counted vs
octate-stuffed framing. But had that been a problem, it should have
reflected as those percentages being the other way (usr% should have been
significantly higher than sys%).

If this is a new enough linux image, can you get debug symbols for Linux,
libc and rsyslog and share output of "perf record -g -a -F 97" when it's
healthy and when it's not?

You'll need to launch "perf report" to view the data. If possible please
share screenshots with all rows collapsed, and another one with the top few
(upto >2%, say) expanded.

Also, what was the load-avg at the time of degraded vs healthy receiver.

Please do capture the netstat grep output if it happens again.

Btw, it may be possible to reproduce it on-demand if its purely a function
of load. There is a utility called tcpflood in rsyslog tests directory that
can flood messages quite well. Ruleset behind imptcp can be configured to
filter and discard those messages.

On Nov 18, 2016 4:32 AM, "David Lang" <da...@lang.hm> wrote:

> On Thu, 17 Nov 2016, matthew.gaetano wrote:
>
> singh.janmejay wrote
>>
>>> how many threads is imptcp configured for?
>>>
>>
>> On the reciever imptcp threads is set to 7, so 8 total (equal to the
>> number
>> of cores on the server).
>>
>
> how did you arrive at this number? Rsyslog can behave very badly with too
> many threads (the threads end up competing for locks on the main queue)
>
> what is your batch size set to?
>
> You should only set the number of threads above 1 if you see that the
> worker threads are hitting max CPU, and try increasing batch size first
> before adding additional threads.
>
> you can see the per-thread usage on TOP by hitting 'H'
>
>
>> singh.janmejay wrote
>>
>>> how does cpu-util (%sys and %usr) change on receiver as it goes from
>>> healthy 30-40k ingestion to backlog building up?
>>>
>>
>> During normal operations user time is roughly 30% and system time is
>> roughly
>> 15%. During the incident user time had highs of 80-82% and lows of 10-15%.
>> System time had highs of 70-80% and lows of 10-15%. In an hour period
>> during
>> the issue avg system time was 55% and avg user time was 18%. Under normal
>> conditions the avg system time for an hour period was 6% and user time was
>> 21% (not percentages are much small in the early morning when the input
>> halfed - the numbers are representative of the time the incident occurred
>> and under normal 30-40k conditions)
>>
>>
>> singh.janmejay wrote
>>
>>> what does the receiving throughput drop to (assuming it is 40k when all
>>> is
>>> well)?
>>>
>>
>> the throughput will bounce between highs of 20k-100k and lows of a 30-15k
>>
>>
>> singh.janmejay wrote
>>
>>> what is omfwd queue.workerthreads (assuming minimum messages is not very
>>> high, so all of them will be used as backlog builds up)?
>>>
>>
>> Our omfwd action on the sender side has a dedicated queue. The queue has
>> been set to max of 8 worker threads and a min msg size of 1. The
>> action
>> its self has nothing set on it. I do not currently track the number of
>> used
>> threads in our monitor tools ... its on my to-do-list.
>>
>
> again, too many threads can cause horrible problems when there is
> contention.
>
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Disabling IMPTCP FlowControl

[singh.janmejay]

A few questions:
- how many threads is imptcp configured for?
  * how does cpu-util (%sys and %usr) change on receiver as it goes
from healthy 30-40k ingestion to backlog building up?
  * what does the receiving throughput drop to (assuming it is 40k
when all is well)?
- what is omfwd queue.workerthreads (assuming minimum messages is not
very high, so all of them will be used as backlog builds up)?
- have you set processOnPoller off on receiver side?

On Fri, Nov 18, 2016 at 2:05 AM, matthew.gaetano  wrote:
> Rainer Gerhards wrote
>> I just checked the code, there currently is no such feature. I surgery to
>> open a feature enhancement tracker (issue) at GitHub, it looks ready to
>> do.
>>
>> Rainer
>
> Submitted feature request: https://github.com/rsyslog/rsyslog/issues/1266
>
>
>
> -
> ~Regards
>
> Matthew Gaetano
> --
> View this message in context: 
> http://rsyslog-users.1305293.n2.nabble.com/Disabling-IMPTCP-FlowControl-tp7591485p7591493.html
> Sent from the rsyslog-users mailing list archive at Nabble.com.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Disabling IMPTCP FlowControl

[singh.janmejay]

Yes, the snapshot from the time when it is under load will help.

Has it just ramped-down on load or are you cycling connections too
frequently? Or was sender restarted recently?

On Fri, Nov 18, 2016 at 1:07 AM, matthew.gaetano  wrote:
> Sender:
> [root@server1 ~]# netstat -antlp | grep -F 2
> tcp0  0 X.X.X.X:40368 X.X.X.X:2 TIME_WAIT   -
> tcp0   3720 X.X.X.X:40500 X.X.X.X:2 ESTABLISHED
> 8383/rsyslogd
> tcp0  0 X.X.X.X:40354 X.X.X.X:2 TIME_WAIT   -
> tcp0  0 X.X.X.X:40406 X.X.X.X:2 TIME_WAIT   -
> tcp0  0 X.X.X.X:40360 X.X.X.X:2 TIME_WAIT   -
> tcp0  0 X.X.X.X:40392 X.X.X.X:2 TIME_WAIT   -
> tcp0  0 X.X.X.X:40384 X.X.X.X:2 TIME_WAIT   -
> tcp0  0 X.X.X.X:40420 X.X.X.X:2 TIME_WAIT   -
> tcp0  0 X.X.X.X:40376 X.X.X.X:2 TIME_WAIT   -
> tcp0  0 X.X.X.X:40322 X.X.X.X:2 TIME_WAIT   -
>
>
> Reciever:
> [root@server2 ~]# netstat -antlp | grep -F 2
> tcp0  0 0.0.0.0:2   0.0.0.0:*   LISTEN
> 1808/rsyslogd
> tcp0  0 X.X.X.X:2 X.X.X.X:33820 ESTABLISHED
> 1808/rsyslogd
> tcp6   0  0 :::2:::*LISTEN
> 1808/rsyslogd
>
>
> Note: the issue is not currently on going. A restart of server2 fixes the
> problem. Its intermittent, we ran for three days without problems. I have
> made a note internally to get the nestat info again when it happens next.
>
>
>
> -
> ~Regards
>
> Matthew Gaetano
> --
> View this message in context: 
> http://rsyslog-users.1305293.n2.nabble.com/Disabling-IMPTCP-FlowControl-tp7591485p7591489.html
> Sent from the rsyslog-users mailing list archive at Nabble.com.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Disabling IMPTCP FlowControl

[singh.janmejay]

What does 'netstat -antlp | grep -F ' say on both
sender and receiver side?

On Fri, Nov 18, 2016 at 12:43 AM, matthew.gaetano
 wrote:
> My current average input on server1 is about 30-40k and we have seen spikes
> of up to 100k; so i would say my target throughput would be around 100k.
>
> server1 is our main collector were server2 is just another rsyslog server
> where we are doing development on parsing and ingesting to elasticsearch.
> The goal is to keep server1 from queuing unless the connection to server2 is
> down or the server2 main ruleset queue becomes full.
>
> The problem at the moment is that server1 queues on the imptcp destination
> when server2 is neither down nor is is queue full. We are trying to identify
> why. At present we suspect its something to do with the communication
> between the two rsyslog servers.
>
>
>
>
>
> -
> ~Regards
>
> Matthew Gaetano
> --
> View this message in context: 
> http://rsyslog-users.1305293.n2.nabble.com/Disabling-IMPTCP-FlowControl-tp7591485p7591487.html
> Sent from the rsyslog-users mailing list archive at Nabble.com.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Disabling IMPTCP FlowControl

[singh.janmejay]

Hi Matthew,

Do you have a target/planned throughput in mind(assuming other
resources are sized correctly) and need help to achieve that, or do
you just want to investigate what (in this case) lead to imptcp server
not keeping up?

On Thu, Nov 17, 2016 at 11:00 PM, matthew.gaetano
 wrote:
> Hello,
>
> I currently have an rsyslog server forwarding events via the "omfwd" module
> to an alternate rsyslog server which is listening using the "imptcp" module.
> The intent is to use the compression feature, though at present it is
> disabled for troubleshooting.
>
> These spikes come to the second rsyslog server but shortly there after we
> can see that the input on the impctp module fluctuation from 10k to 1million
> over the course of a minute or so. This continues until the primary syslog
> server start queuing to disk. The second rsyslog server, despite having its
> own queue, never fills or writes to disk.
>
> This suggests that the issue lies in the way the too systems are
> communicating within the tcp connection settings. Reading the documentation
> the imtcp module has something called FlowControl on by default. This is not
> the same for imptcp, though it does have "RateLimit.Interval" on. By default
> it is set to 0 disabling rate limiting.
>
> Is the parameter "RateLimit.Interval" in imptcp the equivlent to
> "FlowControl" in imtcp? if not is there a way to disable FlowControl in
> imptcp, or is there something else that could be causing back pressure on
> the first rsyslog server?
>
> Thanks
>
>
>
>
>
> -
> ~Regards
>
> Matthew Gaetano
> --
> View this message in context: 
> http://rsyslog-users.1305293.n2.nabble.com/Disabling-IMPTCP-FlowControl-tp7591485.html
> Sent from the rsyslog-users mailing list archive at Nabble.com.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] using lookup tables to change field names

[singh.janmejay]

I agree.

On Thu, Nov 17, 2016 at 2:19 PM, Rainer Gerhards
<rgerha...@hq.adiscon.com> wrote:
> I'd tend to think that this would be valuable even if it is
> performance-hungry...
>
> Just my 2cts,
> Rainer
>
> 2016-11-17 9:48 GMT+01:00 singh.janmejay <singh.janme...@gmail.com>:
>> Not sure, I can figure-out a lower-bound. Here is the setup I have in mind.
>>
>> Fabricate a mix of 1 - 2 kb payloads (say 10 primitive values and 10
>> complex data-types as values, 20 keys at root level, may be 40 keys
>> overall). This is a fairly large message (perhaps 99th quantile for
>> most deployments).
>>
>> Stream 1M such messages(say 20 unique messages repeated in arbitrary
>> order) with some simple transformations in jq (say rename a key,
>> concat a few values, sum a field etc).
>>
>> Helps?
>>
>> On Thu, Nov 17, 2016 at 1:22 PM, David Lang <da...@lang.hm> wrote:
>>> On Thu, 17 Nov 2016, singh.janmejay wrote:
>>>
>>>> This seems like exactly what 'jq' does, it is ability to modify json
>>>> structures using a very flexible declarative language.
>>>>
>>>> Jq does advertise linking-as-a-library style integration. I was wondering
>>>> how everyone feels about building 'mmjq'?
>>>
>>>
>>> what's it's performance like?
>>>
>>> David Lang
>>>
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>>> LIKE THAT.
>>
>>
>>
>> --
>> Regards,
>> Janmejay
>> http://codehunk.wordpress.com
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
>> LIKE THAT.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] using lookup tables to change field names

[singh.janmejay]

Not sure, I can figure-out a lower-bound. Here is the setup I have in mind.

Fabricate a mix of 1 - 2 kb payloads (say 10 primitive values and 10
complex data-types as values, 20 keys at root level, may be 40 keys
overall). This is a fairly large message (perhaps 99th quantile for
most deployments).

Stream 1M such messages(say 20 unique messages repeated in arbitrary
order) with some simple transformations in jq (say rename a key,
concat a few values, sum a field etc).

Helps?

On Thu, Nov 17, 2016 at 1:22 PM, David Lang <da...@lang.hm> wrote:
> On Thu, 17 Nov 2016, singh.janmejay wrote:
>
>> This seems like exactly what 'jq' does, it is ability to modify json
>> structures using a very flexible declarative language.
>>
>> Jq does advertise linking-as-a-library style integration. I was wondering
>> how everyone feels about building 'mmjq'?
>
>
> what's it's performance like?
>
> David Lang
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] using lookup tables to change field names

[singh.janmejay]

This seems like exactly what 'jq' does, it is ability to modify json
structures using a very flexible declarative language.

Jq does advertise linking-as-a-library style integration. I was wondering
how everyone feels about building 'mmjq'?

On Nov 17, 2016 2:07 AM, "David Lang"  wrote:

> I don't think there is a good way to do what you are wanting to do.
>
> The only things I can think of that may work are:
>
> create a new string and use mmnormalize to parse it
>
> use mmexternal to parse things and return a json string of the 'fixed'
> values.
>
> David Lang
>
> On Wed, 16 Nov 2016, matthew.gaetano wrote:
>
> Hello,
>>
>> I am a bit stuck and wondering if anyone can suggest something to help,
>> though its entirely possible that what I am attempting to do can not be
>> done
>> ... currently.
>>
>> I am currently using liblognorm to parse a multitude of logs and with a
>> standard set of field names. Though when it comes to using field types
>> like
>> iptables and cef its not possible to dictate what the outcome of the field
>> name is. This is understandable because the whole point of it is to grab
>> the
>> key-value pairs that would a) not be known and/or b) not be in a specific
>> order. Thought the challenge still remains would one conform to a set of
>> field names.
>>
>> With the release of lookup tables i decided to to build a string table
>> with
>> the key representing the "bad" field name, and the value of said key
>> representing the "good" field name. From there a foreach loop could be
>> used
>> to go through and swap out the key names accordingly; example:
>>
>> foreach ($.var in $!var) do {
>>set $.newkey = lookup("map", $.var!key);
>>if ($.newkey != "") then {
>>set $.var!key = $.newkey;
>> }
>> }
>>
>> This works mostly however there are two problems i cant seem to overcome.
>> The first is that this only works for the first layer. So if the variable
>> has multiple levels they will not get evaluated. The second is that there
>> does not seem to be a way to for me to push my adjusted keys and values
>> into
>> a new or existing variable as set and unset do not permit any expression
>> use.
>>
>> Any suggestions? or am i just insane?
>>
>> Thank you
>>
>> ~Regards
>>
>> Matthew Gaetano
>>
>>
>>
>>
>>
>>
>> -
>> ~Regards
>>
>> Matthew Gaetano
>> --
>> View this message in context: http://rsyslog-users.1305293.n
>> 2.nabble.com/using-lookup-tables-to-change-field-names-tp7591472.html
>> Sent from the rsyslog-users mailing list archive at Nabble.com.
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Segementation Fault when Rsyslog Starts

[singh.janmejay]

Pulled the attachments down, I don't have enough information to say
this yet, but it does seem like memory corruption.

Statement that scriptExec is trying to execute is at 0x30e1, if you
still have the corefile, can you please check if that address is
mapped RW?

Another funny thing is, you seem to have 2021 threads (almost all
timed-waiting inside asyncWriterThread function). I haven't seem strm
code, so not sure why it'd spawn this large number of threads.
Something for us to check.

If this is indeed memory corruption running with valgrind may help
validate (if we are lucky, it may even help find the starting point of
corruption).

On Wed, Nov 2, 2016 at 10:56 PM, shane.lawrence
 wrote:
> I uploaded it to nabble and there is a link in my first message. I'm sure
> some people on the mailing list have small mailboxes, so I didn't want to
> spam everyone with a 3MB attachment.  These are the URLs from my original
> post:
> http://rsyslog-users.1305293.n2.nabble.com/file/n7591426/rsyslog.conf
> http://rsyslog-users.1305293.n2.nabble.com/file/n7591426/gdb.txt
>
> If they're still not working for you, just let me know and I'll find another
> way to send them.
>
> Thanks.
>
>
>
> --
> View this message in context: 
> http://rsyslog-users.1305293.n2.nabble.com/Segementation-Fault-when-Rsyslog-Starts-tp7591426p7591437.html
> Sent from the rsyslog-users mailing list archive at Nabble.com.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Segementation Fault when Rsyslog Starts

[singh.janmejay]

Did you forget to attach the gdb.txt file?

On Wed, Nov 2, 2016 at 6:54 PM, shane.lawrence
 wrote:
> Hi Singh,
>
> Thanks for your response. The gdb.txt file that I attached had the full
> output of "thread apply all bt full", and I had the following debug packages
> installed:
> - gcc-base-debuginfo.x86_64
> - gcc-debuginfo.x86_64
> - glibc-debuginfo.x86_64
> - glibc-debuginfo-common.x86_64
> - libestr-debuginfo.x86_64
> - libnet-debuginfo.x86_64
> - rsyslog-debuginfo.x86_64
> - uuid-debuginfo.x86_64
>
> There's still a fair bit that shows as "optimized out", but I think that's
> because of the compiler options used for the v8-stable package.
>
> If there's anything I can do to provide more/better info, please let me
> know.
>
> Thanks!
>
>
>
> --
> View this message in context: 
> http://rsyslog-users.1305293.n2.nabble.com/Segementation-Fault-when-Rsyslog-Starts-tp7591426p7591433.html
> Sent from the rsyslog-users mailing list archive at Nabble.com.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Lookup Table FH leak?

[singh.janmejay]

Sorry, I meant change log may have a ref(just have a look at master change
log).

On Oct 28, 2016 10:17 AM, "singh.janmejay" <singh.janme...@gmail.com> wrote:

> FD leak on reload is fixed upstream already. Can't recall which release.
> Release notes may have a ref.
>
> On Oct 28, 2016 6:15 AM, "David Lang" <da...@lang.hm> wrote:
>
>> I'm catching up on old mail, I think I saw fixes for the lookup table
>> functionality go through recently, can you check with the latest version?
>>
>> David Lang
>>
>> On Mon, 8 Aug 2016, Christian Ramseyer wrote:
>>
>> Date: Mon, 8 Aug 2016 14:31:06 +0200
>>> From: Christian Ramseyer <r...@networkz.ch>
>>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>> Subject: [rsyslog] Lookup Table FH leak?
>>>
>>> Hi
>>>
>>> I've been testing the lookup table feature in 8.18.0 for a while now and
>>> it works great, good stuff!
>>>
>>> However there seems to be an issue that reloading the table leaks a
>>> filehandle:
>>>
>>> # config:
>>>
>>> lookup_table(name="mylookup" file="/etc/lookup.json" reloadOnHUP="on")
>>>
>>> # freshly started rsyslog, 1 FH as expected
>>>
>>> #  lsof -p $(pgrep -f rsyslog/sbin/rsyslogd) | fgrep .json
>>> rsyslogd 14135 root 3r REG  254,1 8467305  537540076 /etc/lookup.json
>>>
>>> # HUP a few times
>>>
>>> #  kill -1  $(pgrep -f rsyslog/sbin/rsyslogd)
>>> #  kill -1  $(pgrep -f rsyslog/sbin/rsyslogd)
>>> #  kill -1  $(pgrep -f rsyslog/sbin/rsyslogd)
>>>
>>> # results in 1 FH per HUP
>>>
>>> #  lsof -p $(pgrep -f rsyslog/sbin/rsyslogd) | fgrep .json
>>> rsyslogd 14135 root  3r REG  254,1 8467305  537540076 /etc/lookup.json
>>> rsyslogd 14135 root 11r REG  254,1 8467305  537540076 /etc/lookup.json
>>> rsyslogd 14135 root 12r REG  254,1 8467305  537540076 /etc/lookup.json
>>> rsyslogd 14135 root 13r REG  254,1 8467305  537540076 /etc/lookup.json
>>>
>>>
>>> Is this a known issue? I've looked at the open issues and commits in
>>> 8.19.0/8.20.0 and couldn't find any mention of it so I didn't try a
>>> newer release for now. But I'm of course willing to upgrade if you guys
>>> think this is addressed already.
>>>
>>> Build Info:
>>>
>>> rsyslogd 8.18.0, compiled with:
>>> PLATFORM:   x86_64-pc-linux-gnu
>>> PLATFORM (lsb_release -d):
>>> FEATURE_REGEXP: Yes
>>> GSSAPI Kerberos 5 support:  No
>>> FEATURE_DEBUG (debug build, slow code): No
>>> 32bit Atomic operations supported:  Yes
>>> 64bit Atomic operations supported:  Yes
>>> memory allocator:   system default
>>> Runtime Instrumentation (slow code):No
>>> uuid support:   Yes
>>> Number of Bits in RainerScript integers: 64
>>>
>>> Thanks
>>> Christian
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Lookup Table FH leak?

[singh.janmejay]

FD leak on reload is fixed upstream already. Can't recall which release.
Release notes may have a ref.

On Oct 28, 2016 6:15 AM, "David Lang"  wrote:

> I'm catching up on old mail, I think I saw fixes for the lookup table
> functionality go through recently, can you check with the latest version?
>
> David Lang
>
> On Mon, 8 Aug 2016, Christian Ramseyer wrote:
>
> Date: Mon, 8 Aug 2016 14:31:06 +0200
>> From: Christian Ramseyer 
>> Reply-To: rsyslog-users 
>> To: rsyslog-users 
>> Subject: [rsyslog] Lookup Table FH leak?
>>
>> Hi
>>
>> I've been testing the lookup table feature in 8.18.0 for a while now and
>> it works great, good stuff!
>>
>> However there seems to be an issue that reloading the table leaks a
>> filehandle:
>>
>> # config:
>>
>> lookup_table(name="mylookup" file="/etc/lookup.json" reloadOnHUP="on")
>>
>> # freshly started rsyslog, 1 FH as expected
>>
>> #  lsof -p $(pgrep -f rsyslog/sbin/rsyslogd) | fgrep .json
>> rsyslogd 14135 root 3r REG  254,1 8467305  537540076 /etc/lookup.json
>>
>> # HUP a few times
>>
>> #  kill -1  $(pgrep -f rsyslog/sbin/rsyslogd)
>> #  kill -1  $(pgrep -f rsyslog/sbin/rsyslogd)
>> #  kill -1  $(pgrep -f rsyslog/sbin/rsyslogd)
>>
>> # results in 1 FH per HUP
>>
>> #  lsof -p $(pgrep -f rsyslog/sbin/rsyslogd) | fgrep .json
>> rsyslogd 14135 root  3r REG  254,1 8467305  537540076 /etc/lookup.json
>> rsyslogd 14135 root 11r REG  254,1 8467305  537540076 /etc/lookup.json
>> rsyslogd 14135 root 12r REG  254,1 8467305  537540076 /etc/lookup.json
>> rsyslogd 14135 root 13r REG  254,1 8467305  537540076 /etc/lookup.json
>>
>>
>> Is this a known issue? I've looked at the open issues and commits in
>> 8.19.0/8.20.0 and couldn't find any mention of it so I didn't try a
>> newer release for now. But I'm of course willing to upgrade if you guys
>> think this is addressed already.
>>
>> Build Info:
>>
>> rsyslogd 8.18.0, compiled with:
>> PLATFORM:   x86_64-pc-linux-gnu
>> PLATFORM (lsb_release -d):
>> FEATURE_REGEXP: Yes
>> GSSAPI Kerberos 5 support:  No
>> FEATURE_DEBUG (debug build, slow code): No
>> 32bit Atomic operations supported:  Yes
>> 64bit Atomic operations supported:  Yes
>> memory allocator:   system default
>> Runtime Instrumentation (slow code):No
>> uuid support:   Yes
>> Number of Bits in RainerScript integers: 64
>>
>> Thanks
>> Christian
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] I'm back

[singh.janmejay]

I was wondering about your silence on the mailing list.

Welcome back.

On Oct 28, 2016 12:03 AM, "David Lang"  wrote:

> I'm back on my feet, but don't have a new job yet, so I'm getting there.
>
> David Lang
>
> On Thu, 27 Oct 2016, Rainer Gerhards wrote:
>
> Hi David,
>>
>> welcome back, you have been missed :-)
>>
>> I hope your are doing well again and everyting has worked out to your
>> favor!
>>
>> Rainer
>>
>> 2016-10-27 11:10 GMT+02:00 David Lang :
>>
>>> I left my job and broke my ankle the next day and just dropped out of
>>> everything for a while.
>>>
>>> The volume of traffic related to rsyslog has been quite significant,
>>> which
>>> is a good thing, but kept being a "amd I really ready to dive back into
>>> that" barrier :-)
>>>
>>> It's great to see so many people talking on the list and new active
>>> contributers.
>>>
>>> so I'm catching up on things from the last couple of months.
>>>
>>> David Lang
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of
>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>>> LIKE THAT.
>>>
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Fwd: Re: rsyslog kills entire system => force reboot

[singh.janmejay]

How long does it take to go thru one cycle of verifying the problem exists?
I was wondering if bisecting would be viable?

May not be required though, stats, entire config and all thread backtrace
will likely give you/us enough clues.

On Sep 16, 2016 12:30 PM, "Raffael Sahli"  wrote:

> yep, I can confirm that the problem is gone.
> Downgrade back to 8.20 solved the problem.
>
> Anybody with the same problem?
>
>
>  Forwarded Message 
> Subject: Re: [rsyslog] rsyslog kills entire system => force reboot
> Date: Mon, 12 Sep 2016 11:03:58 +0200
> From: Raffael Sahli 
> To: rsyslog@lists.adiscon.com
>
> fyi since the downgrade to 8.20 (from 8.21), we didn't notice any problems.
>
>
>
> On 09.09.2016 15:48, Raffael Sahli wrote:
>
>> On 09.09.2016 15:09, David Lang wrote:
>>  > On Fri, 9 Sep 2016, Raffael Sahli wrote:
>>
>>  >>
>>  >> Actually I tried $ActionResumeRetryCount with a value 10, @see 2nd
>>  >> configuration. But faced the same problem.
>>  >>
>>  >>
>>  >> Strange thing is, I deployed new rsyslog configs without the remote
>>  >> forwarding, but this morning one server was unresponsive again, same
>>  >> problem.
>>  >>
>>  >> Does anybody know, can this also happen without remote forwarding?
>>  >
>>  > where are your local logs being written? is there any chance that it's
>>  > running out of space or otherwise falling behind (think of a slow NFS
>>  > server)
>>  >
>>  > remember that even with retries = 10 rsyslog won't stop completely, but
>>  > it will slow things down drastically so that it appears to be dead.
>>
>> No, just the local filesystem.
>> And the fs and disk i/o is fine.
>>
>>
>>  >
>>  >> Maybe this more a general syslog problem, as far as I know the RFC,
>>  >> since syslog should never loose any messages by default.
>>  >> I just like to know what rsyslog config I should use with remote
>>  >> forwarding, but without any timeout for syslog services if syslog is
>>  >> somehow unresponsive.
>>  >
>>  > per the syslog spec it should block forever if it can't deliver the
>>  > message.
>>
>> Yeah thats the point, I don't get that
>>
>>  >
>>  > But to really see what's going on, configure impstats and have it write
>>  > to a local file, that will let you see what's going on when it appears
>>  > to stalls.
>>
>> Mhm will try it out, or/and try downgrade to an earlier version since I
>> did not have such problems before.
>>
>>
>>
>>
>>
>
> --
> Raffael Sahli
>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Rsyslog exiting on signal 15

[singh.janmejay]

I'm saying it should be kill based on the intuition that it can't be
handled. So it is the quickest way to free memory. Not sure though, need to
check code.
On Apr 19, 2016 1:23 AM, "David Lang" <da...@lang.hm> wrote:

> I'm not sure. I know I have seen some cases where rsyslog has exited
> saying it was signal 15 when it was really SIGHUP that it received when
> things are corrupted.
>
> I know that it's preferred to let a program exit with SIGTERM rather than
> SIGKILL if at all possible, and the behavior of the OOM handler has changed
> over time, so it may vary depending on what you are running.
>
> David Lang
>
> On Tue, 19 Apr 2016, singh.janmejay wrote:
>
> Won't oom be a sigkill?
>>
>> You can determine which process issued term (syscall is kill).
>>
>> Match it with time of death to find the culprit.
>> On Apr 19, 2016 12:32 AM, "Alec Swan" <alecs...@gmail.com> wrote:
>>
>> I checked dmesg output and didn't find anything related to rsyslog being
>> killed even though I found that other programs were killed at completely
>> different times due to OOM.
>>
>> I will be looking forward to 8.18 release to fix these.
>>
>> Thanks,
>>
>> Alec
>>
>> On Mon, Apr 18, 2016 at 11:50 AM, David Lang <da...@lang.hm> wrote:
>>
>> signal 15 is SIGTERM, which tells rsyslog to shutdown gracefully. Check to
>>> see if you are hitting OOM or something like that (check dmesg)
>>>
>>> I have seen this happen when things get corrupted and a log rotation
>>> happens. 8.14 was still using json-c which we found has some nasty race
>>> conditions in it that can cause these sorts of corruptions (this is why
>>> libfastjson was forked from it)
>>>
>>> David Lang
>>>
>>>
>>> On Mon, 18 Apr 2016, Alec Swan wrote:
>>>
>>> Hi there,
>>>
>>>>
>>>> I've been seeing the following stack trace in /var/log/messages. It
>>>> seems
>>>> like something is trying to shut down rsyslog causing it to leave its
>>>> queues in a bad state. The log line preceding rsyslog termination with
>>>> signal 15 happens 14 minutes prior and doesn't give me any information
>>>> on
>>>> who tried to terminate rsyslog. dmesg doesn't show rsyslog being killed
>>>> either. Any ideas how to track down what kills rsyslog?
>>>>
>>>> Apr 17 15:57:37 m0058601 dhclient[771]: bound to ...
>>>> Apr 17 16:11:58 m0058601 rsyslogd: [origin software="rsyslogd"
>>>> swVersion="8.14.0" x-pid="2408" x-info="http://www.rsyslog.com;]
>>>> exiting
>>>> on
>>>> signal 15.
>>>> Apr 17 16:12:05 m0058601 rsyslogd: [origin software="rsyslogd"
>>>> swVersion="8.14.0" x-pid="10955" x-info="http://www.rsyslog.com;] start
>>>> Apr 17 16:12:05 m0058601 rsyslogd-2040: fatal error on disk queue
>>>> 'omelasticsearch-event-indexer-http-request.log queue[DA]', emergency
>>>> switch to direct mode [v8.14.0 try http://www.rsyslog.com/e/2040 ]
>>>> Apr 17 16:12:05 m0058601 rsyslogd-2040: fatal error on disk queue
>>>> 'omelasticsearch-syslog queue[DA]', emergency switch to direct mode
>>>> [v8.14.0 try http://www.rsyslog.com/e/2040 ]
>>>> Apr 17 16:12:04 m0058601 rsyslogd-2221: module 'imuxsock' already in
>>>> this
>>>> config, cannot be added  [v8.14.0 try http://www.rsyslog.com/e/2221 ]
>>>> Apr 17 16:12:04 m0058601 rsyslogd-2221: module 'imklog' already in this
>>>> config, cannot be added  [v8.14.0 try http://www.rsyslog.com/e/2221 ]
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Alec
>>>> ___
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T LIKE THAT.
>>>>
>>>> ___
>>>>
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>

Re: [rsyslog] Rsyslog exiting on signal 15

[singh.janmejay]

Won't oom be a sigkill?

You can determine which process issued term (syscall is kill).

Match it with time of death to find the culprit.
On Apr 19, 2016 12:32 AM, "Alec Swan"  wrote:

I checked dmesg output and didn't find anything related to rsyslog being
killed even though I found that other programs were killed at completely
different times due to OOM.

I will be looking forward to 8.18 release to fix these.

Thanks,

Alec

On Mon, Apr 18, 2016 at 11:50 AM, David Lang  wrote:

> signal 15 is SIGTERM, which tells rsyslog to shutdown gracefully. Check to
> see if you are hitting OOM or something like that (check dmesg)
>
> I have seen this happen when things get corrupted and a log rotation
> happens. 8.14 was still using json-c which we found has some nasty race
> conditions in it that can cause these sorts of corruptions (this is why
> libfastjson was forked from it)
>
> David Lang
>
>
> On Mon, 18 Apr 2016, Alec Swan wrote:
>
> Hi there,
>>
>> I've been seeing the following stack trace in /var/log/messages. It seems
>> like something is trying to shut down rsyslog causing it to leave its
>> queues in a bad state. The log line preceding rsyslog termination with
>> signal 15 happens 14 minutes prior and doesn't give me any information on
>> who tried to terminate rsyslog. dmesg doesn't show rsyslog being killed
>> either. Any ideas how to track down what kills rsyslog?
>>
>> Apr 17 15:57:37 m0058601 dhclient[771]: bound to ...
>> Apr 17 16:11:58 m0058601 rsyslogd: [origin software="rsyslogd"
>> swVersion="8.14.0" x-pid="2408" x-info="http://www.rsyslog.com;] exiting
>> on
>> signal 15.
>> Apr 17 16:12:05 m0058601 rsyslogd: [origin software="rsyslogd"
>> swVersion="8.14.0" x-pid="10955" x-info="http://www.rsyslog.com;] start
>> Apr 17 16:12:05 m0058601 rsyslogd-2040: fatal error on disk queue
>> 'omelasticsearch-event-indexer-http-request.log queue[DA]', emergency
>> switch to direct mode [v8.14.0 try http://www.rsyslog.com/e/2040 ]
>> Apr 17 16:12:05 m0058601 rsyslogd-2040: fatal error on disk queue
>> 'omelasticsearch-syslog queue[DA]', emergency switch to direct mode
>> [v8.14.0 try http://www.rsyslog.com/e/2040 ]
>> Apr 17 16:12:04 m0058601 rsyslogd-2221: module 'imuxsock' already in this
>> config, cannot be added  [v8.14.0 try http://www.rsyslog.com/e/2221 ]
>> Apr 17 16:12:04 m0058601 rsyslogd-2221: module 'imklog' already in this
>> config, cannot be added  [v8.14.0 try http://www.rsyslog.com/e/2221 ]
>>
>>
>> Thanks,
>>
>> Alec
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] another dyn_stats question

[singh.janmejay]

Here is the PR for unused-metric-ttl bug:
https://github.com/rsyslog/rsyslog/pull/955

On Wed, Apr 6, 2016 at 8:35 AM, David Lang <da...@lang.hm> wrote:
> looking pretty good. I've been running it for a couple days now.
>
> David Lang
>
> On Thu, 31 Mar 2016, singh.janmejay wrote:
>
>> Date: Thu, 31 Mar 2016 00:55:44 +0530
>>
>> From: singh.janmejay <singh.janme...@gmail.com>
>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] another dyn_stats question
>>
>> Have opened a PR for this. Will fold it with other changes in the same
>> area. Here is the PR: https://github.com/rsyslog/rsyslog/pull/931 (not
>> ready for merge yet)
>>
>> On Wed, Mar 30, 2016 at 8:01 AM, singh.janmejay
>> <singh.janme...@gmail.com> wrote:
>>>
>>> Cool, I'll make dyn-stats bucket resettable flag the only thing that
>>> controls resetting of its accumulators. It will or won't reset regardless
>>> of
>>> impstats reset flag value.
>>>
>>> This is again changing behavior from previous release but is a fairly
>>> minor
>>> thing, so we'll just document it and not worry about backward
>>> compatibility
>>> of its behavior.
>>>
>>> On Mar 30, 2016 2:55 AM, "David Lang" <da...@lang.hm> wrote:
>>>>
>>>>
>>>> On Wed, 30 Mar 2016, singh.janmejay wrote:
>>>>
>>>>> Metric owner is the subsystem (action is a perfect example). End-user
>>>>> is
>>>>> author of the config.
>>>>
>>>>
>>>>
>>>> Except that dyn_stats doen't exist except in the config, so it's the
>>>> "End-user" who sets everything about them.
>>>>
>>>>> In this case ideal situation would be to let action decide which of its
>>>>> metrics are resettable, and let end-user decide which of the resettable
>>>>> metric accumulators(subset of all metrics) do they actually want to
>>>>> reset.
>>>>
>>>>
>>>>
>>>> I disagree, the module author should not be deciding if something is
>>>> resettable or not.
>>>>
>>>> The module author is either reporting the current state of something
>>>> (like
>>>> queue length) where there is no resetting, or they are reporting the
>>>> state
>>>> of a counter (number of messages processed), and it should be up to the
>>>> config author if they want to reset this every reporting period or not.
>>>>
>>>>> In case of dyn-stats, end-user is metric-owner.
>>>>>
>>>>> So we should probably disassociate from impstats reset flag and let
>>>>> bucket-level resettable flag be the only way to control it.
>>>>>
>>>>> That is what you expected the behavior would be, right? So it's more
>>>>> intuitive too.
>>>>>
>>>>> Should we do this?
>>>>
>>>>
>>>>
>>>> I think so.
>>>>
>>>>> For static counters though, the problem still remains. If you want to
>>>>> reset
>>>>> omkafka resettable counters, but don't want resetting for imudp
>>>>> resettable
>>>>> counters, as of now we have no way to configure it like that.
>>>>
>>>>
>>>>
>>>> Well, I see that as a legacy interface, so I don't think we need to
>>>> worry
>>>> about changing that.
>>>>
>>>> Monitoring systems are almost always able to handle counters and convert
>>>> them to rate-of-change.
>>>>
>>>> resetting a counter is always racy with updates.
>>>>
>>>> If a monitoring system is looking at the stats, you almost always want
>>>> them to be non-reset counters. It's only if a person is looking at them
>>>> that
>>>> you want to show the delta since the last report in the output.
>>>>
>>>> David Lang
>>>>
>>>>> On Mar 30, 2016 2:06 AM, "David Lang" <da...@lang.hm> wrote:
>>>>>
>>>>>> On Wed, 30 Mar 2016, singh.janmejay wrote:
>>>>>>
>>>>>> The problem is, pstats level control is an all or nothing kind of
>>>>>>>
>>>>>>>
>>>>>>> arrangement.
>&g

Re: [rsyslog] Out of Office. (was: Out of Office. (was: Out of Office. (was: Out of Office. (was: Out of Office. (was: Out of Office. (was: Out of Office. (was: Out of Office. (was: Out of Office. (wa

[singh.janmejay]

It's an infinite loop(auto reply is replying to itself), can someone please
kill that subscription?
On Apr 13, 2016 12:11 AM, "Steve Clark"  wrote:


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Rate limiting for imfile?

[singh.janmejay]

If you really want to do it, it's possible to use omuxsock and imuxsock
pair to pass the logs thru before dispatching it over.  If it is replacing
fluentd, with this loop it will still be a huge performance win.
On Apr 12, 2016 11:24 PM, "Rich Megginson" <rmegg...@redhat.com> wrote:

> On 04/12/2016 11:31 AM, singh.janmejay wrote:
>
>> Im curious, why would you want to do that?
>>
>
>
> https://github.com/openshift/origin-aggregated-logging#throttling-logs-in-fluentd
>
>
>> As a service that takes those logs and does something with it (store
>> it, index it etc), you want to absorb acceptable bursts as soon as
>> possible.
>>
>
> Unless you don't.
>
> If it is an unacceptably big burst, you are better off
>> dropping messages and catching up with the latest set of messages
>> (because when someone debugs a "current" problem, the latest logs are
>> the most useful, and in case of a very large burst current messages
>> end up waiting to ingest way to long).
>>
>
> Ok, but that's not the way the solution works now, that I'm proposing to
> use rsyslog for instead.
>
>
>> In our case, we allow bursts upto x% (using pre-planned burst buffer)
>> and discard messages when that burst-limit is breached. But we do it
>> at the service's ingestion-frontend(an array of Rsyslog receivers,
>> slurping logs in over tcp), so it acts as a gate-keeper to the
>> service. This allows service users to burst at individual nodes level,
>> as long as their cluster overall stays within planned-capacity +
>> burst-buffer.
>>
>> On Tue, Apr 12, 2016 at 10:26 PM, David Lang <da...@lang.hm> wrote:
>>
>>> On Tue, 12 Apr 2016, Rich Megginson wrote:
>>>
>>> I see there is rate limiting for certain types of input
>>>> http://blog.gerhards.net/2012/10/rate-limiting-in-rsyslog-732.htm
>>>> <http://blog.gerhards.net/2012/10/rate-limiting-in-rsyslog-732.html>
>>>> Is it possible to do rate limiting on imfile?
>>>> I tried this with rsyslog v 8.12:
>>>>
>>>> input(name="imfile" file="/var/log/mybadapp.log"
>>>> ratelimit.burst="100")
>>>> rsyslogd -N 1 says "parameter ratelimit.burst not known"
>>>>
>>>
>>> yep, that parameter doesn't exist for imfile, look at
>>> http://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html
>>> for
>>> your options.
>>>
>>> it does have the batch size, but there is nothing that limit the rate
>>> that
>>> it processes batches.
>>>
>>> David Lang
>>>
>>>
>>> I'm trying to do something like I can do with fluentd - in a file input I
>>>> can specify read_lines_limit
>>>>
>>>> I suppose I could do something with the input queue, but I want to do
>>>> rate
>>>> limiting before the log data even gets to the queue - I want to read
>>>> slowly
>>>> from the file.
>>>>
>>>> I could also do some extra-process tricks, like putting some sort of
>>>> named
>>>> pipe script intermediary between the file and rsyslog, and have rsyslog
>>>> read
>>>> from the pipe instead, and have the script do the rate limiting, but
>>>> that's
>>>> an "inelegant" solution.
>>>>
>>>> ___
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T
>>>> LIKE THAT.
>>>>
>>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of
>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>>> LIKE THAT.
>>>
>>
>>
>>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Rate limiting for imfile?

[singh.janmejay]

Im curious, why would you want to do that?

As a service that takes those logs and does something with it (store
it, index it etc), you want to absorb acceptable bursts as soon as
possible. If it is an unacceptably big burst, you are better off
dropping messages and catching up with the latest set of messages
(because when someone debugs a "current" problem, the latest logs are
the most useful, and in case of a very large burst current messages
end up waiting to ingest way to long).

In our case, we allow bursts upto x% (using pre-planned burst buffer)
and discard messages when that burst-limit is breached. But we do it
at the service's ingestion-frontend(an array of Rsyslog receivers,
slurping logs in over tcp), so it acts as a gate-keeper to the
service. This allows service users to burst at individual nodes level,
as long as their cluster overall stays within planned-capacity +
burst-buffer.

On Tue, Apr 12, 2016 at 10:26 PM, David Lang  wrote:
> On Tue, 12 Apr 2016, Rich Megginson wrote:
>
>>
>> I see there is rate limiting for certain types of input
>> http://blog.gerhards.net/2012/10/rate-limiting-in-rsyslog-732.htm
>> 
>> Is it possible to do rate limiting on imfile?
>> I tried this with rsyslog v 8.12:
>>
>>input(name="imfile" file="/var/log/mybadapp.log" ratelimit.burst="100")
>> rsyslogd -N 1 says "parameter ratelimit.burst not known"
>
>
> yep, that parameter doesn't exist for imfile, look at
> http://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html for
> your options.
>
> it does have the batch size, but there is nothing that limit the rate that
> it processes batches.
>
> David Lang
>
>
>> I'm trying to do something like I can do with fluentd - in a file input I
>> can specify read_lines_limit
>>
>> I suppose I could do something with the input queue, but I want to do rate
>> limiting before the log data even gets to the queue - I want to read slowly
>> from the file.
>>
>> I could also do some extra-process tricks, like putting some sort of named
>> pipe script intermediary between the file and rsyslog, and have rsyslog read
>> from the pipe instead, and have the script do the rate limiting, but that's
>> an "inelegant" solution.
>>
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
>>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] dynastats problems with user defined variables?

[singh.janmejay]

I'll try to help. But can't make promises on how long it'll take me to
get to it (I already slip in time-estimates, you can see some of that
in play in this thread). Often things go haywire at the wrong time and
suck away a lot of bandwidth.

I think extent of de-risking we can achieve through a good
micro-benchmark in this case will be sufficient. But once all of these
changes are in place, we can build a synthetic benchmark which is easy
to run (like one shell-script to run the whole thing and collect
results). Then we can distribute it to users to run in their own
environments (so we'll have results across different architectures,
virtualization stacks, OSes, allocators, compilers, flavors of distro
etc).

On Tue, Apr 12, 2016 at 8:24 PM, Rainer Gerhards
<rgerha...@hq.adiscon.com> wrote:
> Hi Janmejay,
>
> thanks for being so persistent with this. I am currently working on
> libfastjson and hope to have a much different version available
> shortly. In theory, it could give quite a performance boost, but this
> is not totally clear, a lot is really experimental. Would you be
> willing to give the new code a try when it is ready? That would
> definitely help me get quicker feedback and thus a quicker path
> towards a final solution.
>
> The main thing of concern is this:
>
>https://github.com/rsyslog/libfastjson/issues/35
>
> Rainer
>
> 2016-04-12 16:39 GMT+02:00 singh.janmejay <singh.janme...@gmail.com>:
>> I cooked up this little systemtap script to track cause of off-cpu
>> latency (with latency quantification). It identifies latencies by
>> userspace-backtrace, kernel-backtrace and thread-id.
>>
>> It'll give us very useful data when you see low-TP with
>> processing-backlog while CPU runs cold.
>>
>> Run it as $ sudo stap -v 

Re: [rsyslog] dynastats problems with user defined variables?

[singh.janmejay]

I cooked up this little systemtap script to track cause of off-cpu
latency (with latency quantification). It identifies latencies by
userspace-backtrace, kernel-backtrace and thread-id.

It'll give us very useful data when you see low-TP with
processing-backlog while CPU runs cold.

Run it as $ sudo stap -v 

Re: [rsyslog] dynastats problems with user defined variables?

[singh.janmejay]

David,

Did a few runs with those lines present / commented out with
rsyslog-master. I see no difference in either on-cpu or off-cpu
performance. The left-hand-side is with edge-relay and core-relay
lines being active, and the right-side has them commented out. Here is
the cpu profile:
https://drive.google.com/open?id=0B_XhUZLNFT4dWDFOZlB6VW1hc0E
The hypercall on top is cpu-idle call. This was on a c1.4large box on ec2.

Both were on the same version of Rsyslog (built off master). I'll do
another round of tests with the version I run on production (which is
patched fork of 8.12, afair) to identify any large change in cpu
profile.

This is a much more simplified version of config though. I extracted
valid (non garbled) json messages from debug log(got 9089 lines) and
pumped them in 100 times using tcpflood.

tcpflood call was timed, it took between 19.7 - 20.5 seconds for each
run regardless of edge and core relay counters being incremented. It
was ~ 45k / sec (but again, neither the load profile, not the machine
is not exactly like yours), so its not apples to apples.


On Fri, Apr 8, 2016 at 12:15 PM, singh.janmejay
<singh.janme...@gmail.com> wrote:
> It doesn't treat them any different. It takes the key and puts it in a
> hash-table and set an accumulator as its value. Then it'll look it up
> every time you ask it to increment the counter followed by actual
> increment. That is about it.
>
> The only way I can foresee the contents affecting its behavior is the
> hash-fn takes more CPU time when given a large string. But that should
> be on-cpu and not off-cpu time.
>
> Today im working on a getting an Rsyslog build running with config
> into which I inject messages from your debug-log (i'll extract
> messages and tcpflood them in) and try to understand why it behaves
> different between those lines commented out or not.
>
> If I fail to reproduce it in this simple setup, we'll have to track
> backtraces that are taking Rsyslog off-cpu in your environment.
>
> On Fri, Mar 25, 2016 at 12:57 AM, David Lang <da...@lang.hm> wrote:
>> doing a little more digging (and some accidental stuff), is it possible that
>> it's running into grief if the contents of the variable are not a simple
>> word (spaces or other funny characters in the value)?
>>
>> David Lang
>>
>> On Wed, 23 Mar 2016, singh.janmejay wrote:
>>
>>> Date: Wed, 23 Mar 2016 01:07:49 +0530
>>>
>>> From: singh.janmejay <singh.janme...@gmail.com>
>>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>> Subject: Re: [rsyslog] dynastats problems with user defined variables?
>>>
>>> Yep, it looked like that. Its interesting though, its 100% off CPU
>>> (very unique).
>>>
>>> On Wed, Mar 23, 2016 at 1:02 AM, David Lang <da...@lang.hm> wrote:
>>>>
>>>> Yes, the first one is with the user variables commented out, exactly what
>>>> I
>>>> pasted in from the grep.
>>>>
>>>> the second is enabling the dyn_inc for edge relays.
>>>>
>>>> David Lang
>>>>
>>>> On Wed, 23 Mar 2016, singh.janmejay wrote:
>>>>
>>>>> Date: Wed, 23 Mar 2016 00:51:18 +0530
>>>>> From: singh.janmejay <singh.janme...@gmail.com>
>>>>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>>> Subject: Re: [rsyslog] dynastats problems with user defined variables?
>>>>>
>>>>> You meant the first one was with "commented out" version right? (you
>>>>> said uncommenting edge_relay, so double-checking).
>>>>>
>>>>> On Wed, Mar 23, 2016 at 12:38 AM, David Lang <da...@lang.hm> wrote:
>>>>>>
>>>>>>
>>>>>> here is vmstat 1 running
>>>>>>
>>>>>> # vmstat 1
>>>>>> procs ---memory-- ---swap-- -io -system--
>>>>>> --cpu-
>>>>>>  r  b   swpd   free   buff  cache   si   sobibo   in   cs us sy
>>>>>> id
>>>>>> wa st
>>>>>>  2  0  62280 2363804584 272803760121   59601 46
>>>>>> 3
>>>>>> 51  0 0
>>>>>>  1  0  62280 2349476584 2728061200 0 87042  979  891  4
>>>>>> 0
>>>>>> 95  0 0
>>>>>>  1  0  62280 2330488584 2728090800 0 0  388  258  4
>>>>>>

Re: [rsyslog] dynastats problems with user defined variables?

[singh.janmejay]

It doesn't treat them any different. It takes the key and puts it in a
hash-table and set an accumulator as its value. Then it'll look it up
every time you ask it to increment the counter followed by actual
increment. That is about it.

The only way I can foresee the contents affecting its behavior is the
hash-fn takes more CPU time when given a large string. But that should
be on-cpu and not off-cpu time.

Today im working on a getting an Rsyslog build running with config
into which I inject messages from your debug-log (i'll extract
messages and tcpflood them in) and try to understand why it behaves
different between those lines commented out or not.

If I fail to reproduce it in this simple setup, we'll have to track
backtraces that are taking Rsyslog off-cpu in your environment.

On Fri, Mar 25, 2016 at 12:57 AM, David Lang <da...@lang.hm> wrote:
> doing a little more digging (and some accidental stuff), is it possible that
> it's running into grief if the contents of the variable are not a simple
> word (spaces or other funny characters in the value)?
>
> David Lang
>
> On Wed, 23 Mar 2016, singh.janmejay wrote:
>
>> Date: Wed, 23 Mar 2016 01:07:49 +0530
>>
>> From: singh.janmejay <singh.janme...@gmail.com>
>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] dynastats problems with user defined variables?
>>
>> Yep, it looked like that. Its interesting though, its 100% off CPU
>> (very unique).
>>
>> On Wed, Mar 23, 2016 at 1:02 AM, David Lang <da...@lang.hm> wrote:
>>>
>>> Yes, the first one is with the user variables commented out, exactly what
>>> I
>>> pasted in from the grep.
>>>
>>> the second is enabling the dyn_inc for edge relays.
>>>
>>> David Lang
>>>
>>> On Wed, 23 Mar 2016, singh.janmejay wrote:
>>>
>>>> Date: Wed, 23 Mar 2016 00:51:18 +0530
>>>> From: singh.janmejay <singh.janme...@gmail.com>
>>>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Subject: Re: [rsyslog] dynastats problems with user defined variables?
>>>>
>>>> You meant the first one was with "commented out" version right? (you
>>>> said uncommenting edge_relay, so double-checking).
>>>>
>>>> On Wed, Mar 23, 2016 at 12:38 AM, David Lang <da...@lang.hm> wrote:
>>>>>
>>>>>
>>>>> here is vmstat 1 running
>>>>>
>>>>> # vmstat 1
>>>>> procs ---memory-- ---swap-- -io -system--
>>>>> --cpu-
>>>>>  r  b   swpd   free   buff  cache   si   sobibo   in   cs us sy
>>>>> id
>>>>> wa st
>>>>>  2  0  62280 2363804584 272803760121   59601 46
>>>>> 3
>>>>> 51  0 0
>>>>>  1  0  62280 2349476584 2728061200 0 87042  979  891  4
>>>>> 0
>>>>> 95  0 0
>>>>>  1  0  62280 2330488584 2728090800 0 0  388  258  4
>>>>> 0
>>>>> 96  0 0
>>>>>  1  0  62280 2314744584 2728114800 0 0  396  255  4
>>>>> 0
>>>>> 96  0 0
>>>>>  1  0  62280 2297736584 2728138800 0 0  376  245  4
>>>>> 0
>>>>> 96  0 0
>>>>>  2  0  62280 2546260584 2724527200 0 4  639  568  6
>>>>> 1
>>>>> 94  0 0
>>>>>  2  0  62280 2464300584 2724592800 0  1716  936 1146  8
>>>>> 0
>>>>> 91  0 0
>>>>>  2  0  62280 2394452584 2724644400 0 8  687  333  8
>>>>> 0
>>>>> 92  0 0
>>>>>
>>>>>
>>>>> here is vmstat 1 uncommenting edge_relay
>>>>>
>>>>> # vmstat 1
>>>>> procs ---memory-- ---swap-- -io -system--
>>>>> --cpu-
>>>>>  r  b   swpd   free   buff  cache   si   sobibo   in   cs us sy
>>>>> id
>>>>> wa st
>>>>>  1  0  62280 3198672584 267706240121   59601 46
>>>>> 3
>>>>> 51  0 0
>>>>>  0  0  62280 3198904584 2677066400 0 0  151  247  0
>>>>> 0
>>>>> 100 0  0
>>>>>  

Re: [rsyslog] mmnormalize, mmjsonparse keyspace mapping

[singh.janmejay]

I have a convention of making loggers slap a special tag "obj@"
where  = a key based namespace.
So obj@w3 would have website logs, obj@oms would have
order-management-system logs etc.
I extract the ns, and generate objects in the form: {host: ...,
severity: , obj..x : ..., obj..y: ... }

So  allows users to not conflict across un-expected contexts.
Turns out, when users want to retrieve logs, in almost all scenarios
they want logs from a specific . Infact fields that have similar
names, have very different semantics across contexts, eg. status field
for web-server access logs is http-status, whereas in a stage-machine
log event is current-state, so pushing them in the same field makes
little sense.

On Thu, Apr 7, 2016 at 11:26 PM, Matt Ford  wrote:
> Hi, yes I've gotten fairly deep into ES.  We would like to be able to
> search over and perhaps even calculate based on fields with the same
> name and different types.  What's more we don't know when developers
> will add services and new json logs.  Without the rsyslog change - as
> far as I can see the only way to do this to have per app indexes (as
> per the tip at the very bottom of this page
> https://www.elastic.co/guide/en/elasticsearch/guide/current/mapping.html).
>
> This is all cool stuff!  Please keep the ideas coming :-)
>
> One thing that makes me nervous is the overhead of doing the
> transformation via normalize on rsyslog.  Some the applications
> generate a lot of logs per second.
>
> On 7 April 2016 at 18:38, Dave Caplinger  
> wrote:
>> On Apr 7, 2016, at 12:04 PM, Matt Ford  wrote:
>>>
>>> Thanks for the help thus far I'm able to parse arbitrary json logs and
>>> get them into kafka very nicely.
>>> However, due to the many different systems in use there is key
>>> namespace clashes in the final destination (Elasticsearch)
>>>
>>> I have some JSON logs like this from one app
>>>
>>> { "login": 234343,... }
>>>
>>> and some JSON logs like this from another app
>>>
>>> { "login": "matt",... }
>>>
>>> Is it possible to parse and change the key space to look like this
>>>
>>> { "app1_login": 234343, "app1_XX:": }
>>> { "app2_login": "matt", "app2_XX:":...}
>>
>> I'm not sure how deep into ElasticSearch you've gotten, but it sounds like 
>> maybe you're seeing the result of automatic type mapping where the first 
>> field called "login" happens to be interpreted as a number, and later on a 
>> string value shows up and fails to be indexed because ElasticSearch now 
>> expects only numeric values.  You can solve this at ElasticSearch directly 
>> by having an explicit mapping (for example, "login" is a string), which in 
>> this case would force the numeric login value to be inserted as a string 
>> instead.
>>
>> (See: 
>> https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html 
>> )
>>
>> So you don't have to change the upstream JSON sources if you don't want to 
>> (though you certainly could do that instead).
>>
>> - Dave Caplinger
>>
>>
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
>> LIKE THAT.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] State of Lookup Tables Implementation

[singh.janmejay]

This page is the best starting point for all things configuration:
http://www.rsyslog.com/doc/v8-stable/configuration/index.html

On Tue, Apr 5, 2016 at 1:58 AM, singh.janmejay <singh.janme...@gmail.com> wrote:
> http://www.rsyslog.com/doc/v8-stable/configuration/lookup_tables.html
>
> I think something went wrong with that link, we need to clean-up.
>
> On Tue, Apr 5, 2016 at 1:41 AM, Christian Ramseyer <r...@networkz.ch> wrote:
>> On 04/04/16 21:48, singh.janmejay wrote:
>>> The actual implementation details deviate from proposal page at times. It's
>>> best to use documentation.
>>
>> Sorry could you post a link to said documentation? At a first glance I
>> can only find
>> http://www.rsyslog.com/doc/v8-stable/rainerscript/lookup_tables.html but
>> it's just a single sentence with a link back to the same page.
>>
>> Thanks
>> Christian
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
>> LIKE THAT.
>
>
>
> --
> Regards,
> Janmejay
> http://codehunk.wordpress.com



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] State of Lookup Tables Implementation

[singh.janmejay]

http://www.rsyslog.com/doc/v8-stable/configuration/lookup_tables.html

I think something went wrong with that link, we need to clean-up.

On Tue, Apr 5, 2016 at 1:41 AM, Christian Ramseyer <r...@networkz.ch> wrote:
> On 04/04/16 21:48, singh.janmejay wrote:
>> The actual implementation details deviate from proposal page at times. It's
>> best to use documentation.
>
> Sorry could you post a link to said documentation? At a first glance I
> can only find
> http://www.rsyslog.com/doc/v8-stable/rainerscript/lookup_tables.html but
> it's just a single sentence with a link back to the same page.
>
> Thanks
> Christian
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] State of Lookup Tables Implementation

[singh.janmejay]

The actual implementation details deviate from proposal page at times. It's
best to use documentation.

At a high level, proposal describes it quite accurately though.
On Apr 4, 2016 7:23 PM, "Christian Ramseyer"  wrote:

> Great thanks Rainer, I'll check it out.
>
>
> On 04/04/16 15:38, Rainer Gerhards wrote:
> > It is part of 8.17, only flagged as experimental. So you should be good
> in
> > using it.
> >
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] another dyn_stats question

[singh.janmejay]

Will create a separate PR for unused-metric-ttl change. That is not
included in the PR I posted above.

On Fri, Apr 1, 2016 at 1:22 AM, singh.janmejay <singh.janme...@gmail.com> wrote:
> Ready for merge now.
>
> On Thu, Mar 31, 2016 at 12:55 AM, singh.janmejay
> <singh.janme...@gmail.com> wrote:
>> Have opened a PR for this. Will fold it with other changes in the same
>> area. Here is the PR: https://github.com/rsyslog/rsyslog/pull/931 (not
>> ready for merge yet)
>>
>> On Wed, Mar 30, 2016 at 8:01 AM, singh.janmejay
>> <singh.janme...@gmail.com> wrote:
>>> Cool, I'll make dyn-stats bucket resettable flag the only thing that
>>> controls resetting of its accumulators. It will or won't reset regardless of
>>> impstats reset flag value.
>>>
>>> This is again changing behavior from previous release but is a fairly minor
>>> thing, so we'll just document it and not worry about backward compatibility
>>> of its behavior.
>>>
>>> On Mar 30, 2016 2:55 AM, "David Lang" <da...@lang.hm> wrote:
>>>>
>>>> On Wed, 30 Mar 2016, singh.janmejay wrote:
>>>>
>>>>> Metric owner is the subsystem (action is a perfect example). End-user is
>>>>> author of the config.
>>>>
>>>>
>>>> Except that dyn_stats doen't exist except in the config, so it's the
>>>> "End-user" who sets everything about them.
>>>>
>>>>> In this case ideal situation would be to let action decide which of its
>>>>> metrics are resettable, and let end-user decide which of the resettable
>>>>> metric accumulators(subset of all metrics) do they actually want to
>>>>> reset.
>>>>
>>>>
>>>> I disagree, the module author should not be deciding if something is
>>>> resettable or not.
>>>>
>>>> The module author is either reporting the current state of something (like
>>>> queue length) where there is no resetting, or they are reporting the state
>>>> of a counter (number of messages processed), and it should be up to the
>>>> config author if they want to reset this every reporting period or not.
>>>>
>>>>> In case of dyn-stats, end-user is metric-owner.
>>>>>
>>>>> So we should probably disassociate from impstats reset flag and let
>>>>> bucket-level resettable flag be the only way to control it.
>>>>>
>>>>> That is what you expected the behavior would be, right? So it's more
>>>>> intuitive too.
>>>>>
>>>>> Should we do this?
>>>>
>>>>
>>>> I think so.
>>>>
>>>>> For static counters though, the problem still remains. If you want to
>>>>> reset
>>>>> omkafka resettable counters, but don't want resetting for imudp
>>>>> resettable
>>>>> counters, as of now we have no way to configure it like that.
>>>>
>>>>
>>>> Well, I see that as a legacy interface, so I don't think we need to worry
>>>> about changing that.
>>>>
>>>> Monitoring systems are almost always able to handle counters and convert
>>>> them to rate-of-change.
>>>>
>>>> resetting a counter is always racy with updates.
>>>>
>>>> If a monitoring system is looking at the stats, you almost always want
>>>> them to be non-reset counters. It's only if a person is looking at them 
>>>> that
>>>> you want to show the delta since the last report in the output.
>>>>
>>>> David Lang
>>>>
>>>>> On Mar 30, 2016 2:06 AM, "David Lang" <da...@lang.hm> wrote:
>>>>>
>>>>>> On Wed, 30 Mar 2016, singh.janmejay wrote:
>>>>>>
>>>>>> The problem is, pstats level control is an all or nothing kind of
>>>>>>>
>>>>>>> arrangement.
>>>>>>>
>>>>>>> Ideal setup would be to let metric-owner decide if metric should be
>>>>>>> resettable or not(which allows choosing if its a guage or not) and
>>>>>>> then let end-user choose if they really want to reset each resettable
>>>>>>> metric(counter) (or do they want it as a perpetual counter).
>>>>>>>
>>>>>>
>>>>>> there is only one entity in control of the rsyslog.conf 

Re: [rsyslog] Loghost to loghost

[singh.janmejay]

We use SolrCloud for accessing last 0 to 4 hours of data(accessed using
banana/silk). Older data is kept on hadoop, fronted by hive, accessed
through either hue(for humans) or hive2 protocol, partitioned by date+hour.

We ingress at an aggregate event rate of ~2M/sec or 7.2B/hr. Event sz
~1.5kb.
On Apr 1, 2016 5:07 AM, "David Lang"  wrote:

> On Thu, 31 Mar 2016, Singh, Radesh wrote:
>
> That seems very slick.
>>
>> In this config:
>> client -> (logshost_dc1 <=> loghost_dc2) -> analysis
>> With the rule:
>> if $fromhost-ip != [] then @@remote-VIP
>>
>> I'm assuming that remote-VIP will round robin between loghost_dc1 and
>> loghost_dc2, and then whichever loghost receives the message will just
>> forward the message to its peer.
>> And then they both have a rule to forward to whatever you are using for
>> analysis.
>>
>
> actually each loghost is actually a failover pair with a vip shared
> between them, clients in each datacenter talk to the VIP in that
> datacenter, and then the loghosts make sure the log gets to both
> datacenters, and then each datacenter forwards to local analysis tools (so
> that everything 'just works' if one datacenter goes away, the remaining one
> has all the logs, although not in the same order due to speed-of-light lag)
>
>
> Out of curiousity, can you share what you use for analysis?
>>
>> We have an application that can dump 50GB+ of logs per server in a single
>> day.
>> Rsyslog handles logging just fine, but trying to find 'a message' is
>> particularly painful.
>>
>
> I roll my logs every minute, and I have rules that pull apart my ~200
> million logs/day into all sorts of different categories. This by itself
> makes it possible to find things with grep. This is somewhere around 100GB
> of logs/day (I'd have to go back and check, I do serious compression, but
> also have multiple copies of logs, so it's not a trivial question to answer)
>
> But what you really want (and I'm in the process of setting up) is to have
> a copy of the data go to Splunk or ElasticSearch. Properly provisioned they
> do wonders for finding things. At my last job, we were able to search for
> an IP in the last three years of logs and get an answer back in <3 min. We
> used Splunk with some serious hardware (>1TB ram, 320 1TB drives, 20
> machines, 160 cores) but this was at 100K logs/sec
>
> I've written articles in ;login magazine on this topic and the basic design
> https://www.usenix.org/publications/login/david-lang-series
> https://www.usenix.org/publications/login/april14/lang
>
> https://www.usenix.org/conference/lisa12/technical-sessions/presentation/lang_david
>
>
> https://www.usenix.org/publications/login/april-2013-volume-38-number-2/wireless-means-radio
>
> https://www.usenix.org/conference/lisa12/technical-sessions/presentation/lang_david_wirelesshttps://www.usenix.org/conference/lisa12/technical-sessions/presentation/lang_david_wireless
>
> David Lang
>
>
>
>
>
>
>
> R. Singh
>> Sr. Systems Administrator
>> Middleware/PTC Support
>> 904-633-5745
>>
>> RC Offering: SC07507098
>>
>>
>> H0\/\/ T0/\/\0RR0\/\/ /\/\0\/35
>>
>> "Give instruction to a wise man, and he will be yet wiser : teach a just
>> man, and he will increase in learning." - Proverbs 9:9
>>
>>
>> -Original Message-
>> From: rsyslog-boun...@lists.adiscon.com [mailto:
>> rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang
>> Sent: Thursday, March 31, 2016 2:48 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] Loghost to loghost
>>
>> On Thu, 31 Mar 2016, Singh, Radesh wrote:
>>
>> Thank you for that tip.
>>> Are you just saying that we need to ensure to make sure that we don't
>>> sent our loghosts to point at each other?
>>> In other words...
>>>
>>> Client->loghost<=>loghost...
>>>
>>
>> or server1 -> server2 -> server3 -> server1
>>
>> I actually do client -> (logshost_dc1 <=> loghost_dc2) -> analysis
>>
>> on each loghost (failover pair in each datacenter), I have a filter
>>
>> if $fromhost-ip != [] then @@remote-VIP
>>
>> (simplified to not include queues, etc)
>>
>> This makes it so that if the log doesn't come from the remote dc, I send
>> it there, but that loops can't happen.
>>
>> David Lang
>>
>>
>>>
>>> R. Singh
>>> Sr. Systems Administrator
>>> Middleware/PTC Support
>>> 904-633-5745
>>>
>>> RC Offering: SC07507098
>>>
>>>
>>> H0\/\/ T0/\/\0RR0\/\/ /\/\0\/35
>>>
>>> "Give instruction to a wise man, and he will be yet wiser : teach a
>>> just man, and he will increase in learning." - Proverbs 9:9
>>>
>>>
>>> -Original Message-
>>> From: rsyslog-boun...@lists.adiscon.com
>>> [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang
>>> Sent: Thursday, March 31, 2016 2:11 PM
>>> To: rsyslog-users
>>> Subject: Re: [rsyslog] Loghost to loghost
>>>
>>> On Thu, 31 Mar 2016, Rainer Gerhards wrote:
>>>
>>> 2016-03-31 18:29 GMT+02:00 Singh, Radesh :

> Hello,
>
> I've been looking online, and perhaps I'm just not putting in the
> 

Re: [rsyslog] dynastats JSON output need work

[singh.janmejay]

Complete, ready for merge.

On Thu, Mar 31, 2016 at 12:54 AM, singh.janmejay
<singh.janme...@gmail.com> wrote:
> PR created (not complete yet, not ready for merge):
> https://github.com/rsyslog/rsyslog/pull/931
>
> Have a quick look to get a feel for namespaced reporting of dyn-stats 
> counters.
>
> On Fri, Mar 25, 2016 at 4:25 PM, singh.janmejay
> <singh.janme...@gmail.com> wrote:
>> Cool. Will fix both stats representation and foreach in one PR.
>> On Mar 25, 2016 3:10 PM, "Rainer Gerhards" <rgerha...@hq.adiscon.com> wrote:
>>
>>> ACK! Please go, let's set aside my probably too strong concerns. I am
>>> perfectly happy when you say it's a small change and glad you do it!
>>>
>>> Rainer
>>>
>>> Sent from phone, thus brief.
>>> Am 25.03.2016 06:21 schrieb "David Lang" <da...@lang.hm>:
>>>
>>> > As I understand Rainer, go .
>>> >
>>> > David Lang
>>> >
>>> > On Fri, 25 Mar 2016, singh.janmejay wrote:
>>> >
>>> > Makes sense. So final call on foreach? Go or no-go?
>>> >> On Mar 25, 2016 2:38 AM, "David Lang" <da...@lang.hm> wrote:
>>> >>
>>> >> Just a note, the dynastats output line needs this as well as the bucket
>>> >>> line.
>>> >>>
>>> >>> This isn't remote-input, but it needs the same ability to iterate over
>>> >>> things.
>>> >>>
>>> >>> example:
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> {"name":"global","origin":"dynstats","message_framing.ops_overflow":0,"message_framing.new_metric_add":5,"message_framing.no_metric":0,"message_framing.metrics_purged":0,"message_framing.ops_ignored":0,"message_framing.purge_triggered":0,"msgs_per_host.ops_overflow":0,"msgs_per_host.new_metric_add":0,"msgs_per_host.no_metric":0,"msgs_per_host.metrics_purged":0,"msgs_per_host.ops_ignored":0,"msgs_per_host.purge_triggered":0,"msgs_per_edge_relay.ops_overflow":0,"msgs_per_edge_relay.new_metric_add":0,"msgs_per_edge_relay.no_metric":0,"msgs_per_edge_relay.metrics_purged":0,"msgs_per_edge_relay.ops_ignored":0,"msgs_per_edge_relay.purge_triggered":0,"msgs_per_core_relay.ops_overflow":0,"msgs_per_core_relay.new_metric_add":0,"msgs_per_core_relay.no_metric":0,"msgs_per_core_relay.metrics_purged":0,"msgs_per_core_relay.ops_ignored":0,"msgs_per_core_relay.purge_triggered":0,"msgs_per_program.ops_overflow":0,"msgs_per_program.new_metric_add":0,"msgs_per_program.no_metric":0,"msgs_pe
>>>  !
>>> >>>
>>> >> r_
>>> >
>>> >> p!
>>> >>
>>> >>>
>>> >>>
>>> >>>
>>> rogram.metrics_purged":0,"msgs_per_program.ops_ignored":0,"msgs_per_program.purge_triggered":0,"msgs_per_tag.ops_overflow":0,"msgs_per_tag.new_metric_add":0,"msgs_per_tag.no_metric":0,"msgs_per_tag.metrics_purged":0,"msgs_per_tag.ops_ignored":0,"msgs_per_tag.purge_triggered":0}
>>> >>>
>>> >>> David Lang
>>> >>>
>>> >>> On Wed, 23 Mar 2016, singh.janmejay wrote:
>>> >>>
>>> >>> Date: Wed, 23 Mar 2016 09:10:47 +0530
>>> >>>
>>> >>>> From: singh.janmejay <singh.janme...@gmail.com>
>>> >>>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>>> >>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>> >>>> Subject: Re: [rsyslog] dynastats JSON output need work
>>> >>>>
>>> >>>> Agree, if most of us like it, I can implement foreach for objects. As
>>> of
>>> >>>> now I use omprog to call a script which uses jq and awk to make it
>>> >>>> opentsdb
>>> >>>> and redis friendly. I could have used omredis if foreach worked for
>>> >>>> objects.
>>> >>>>
>>> >>>> Let us conclude.
>>> >>>>
>>> >>>> @Rainer any thoughts on foreach for object?
>>> >>>> On Mar 23, 2016 1:56 AM, "David La

Re: [rsyslog] another dyn_stats question

[singh.janmejay]

Ready for merge now.

On Thu, Mar 31, 2016 at 12:55 AM, singh.janmejay
<singh.janme...@gmail.com> wrote:
> Have opened a PR for this. Will fold it with other changes in the same
> area. Here is the PR: https://github.com/rsyslog/rsyslog/pull/931 (not
> ready for merge yet)
>
> On Wed, Mar 30, 2016 at 8:01 AM, singh.janmejay
> <singh.janme...@gmail.com> wrote:
>> Cool, I'll make dyn-stats bucket resettable flag the only thing that
>> controls resetting of its accumulators. It will or won't reset regardless of
>> impstats reset flag value.
>>
>> This is again changing behavior from previous release but is a fairly minor
>> thing, so we'll just document it and not worry about backward compatibility
>> of its behavior.
>>
>> On Mar 30, 2016 2:55 AM, "David Lang" <da...@lang.hm> wrote:
>>>
>>> On Wed, 30 Mar 2016, singh.janmejay wrote:
>>>
>>>> Metric owner is the subsystem (action is a perfect example). End-user is
>>>> author of the config.
>>>
>>>
>>> Except that dyn_stats doen't exist except in the config, so it's the
>>> "End-user" who sets everything about them.
>>>
>>>> In this case ideal situation would be to let action decide which of its
>>>> metrics are resettable, and let end-user decide which of the resettable
>>>> metric accumulators(subset of all metrics) do they actually want to
>>>> reset.
>>>
>>>
>>> I disagree, the module author should not be deciding if something is
>>> resettable or not.
>>>
>>> The module author is either reporting the current state of something (like
>>> queue length) where there is no resetting, or they are reporting the state
>>> of a counter (number of messages processed), and it should be up to the
>>> config author if they want to reset this every reporting period or not.
>>>
>>>> In case of dyn-stats, end-user is metric-owner.
>>>>
>>>> So we should probably disassociate from impstats reset flag and let
>>>> bucket-level resettable flag be the only way to control it.
>>>>
>>>> That is what you expected the behavior would be, right? So it's more
>>>> intuitive too.
>>>>
>>>> Should we do this?
>>>
>>>
>>> I think so.
>>>
>>>> For static counters though, the problem still remains. If you want to
>>>> reset
>>>> omkafka resettable counters, but don't want resetting for imudp
>>>> resettable
>>>> counters, as of now we have no way to configure it like that.
>>>
>>>
>>> Well, I see that as a legacy interface, so I don't think we need to worry
>>> about changing that.
>>>
>>> Monitoring systems are almost always able to handle counters and convert
>>> them to rate-of-change.
>>>
>>> resetting a counter is always racy with updates.
>>>
>>> If a monitoring system is looking at the stats, you almost always want
>>> them to be non-reset counters. It's only if a person is looking at them that
>>> you want to show the delta since the last report in the output.
>>>
>>> David Lang
>>>
>>>> On Mar 30, 2016 2:06 AM, "David Lang" <da...@lang.hm> wrote:
>>>>
>>>>> On Wed, 30 Mar 2016, singh.janmejay wrote:
>>>>>
>>>>> The problem is, pstats level control is an all or nothing kind of
>>>>>>
>>>>>> arrangement.
>>>>>>
>>>>>> Ideal setup would be to let metric-owner decide if metric should be
>>>>>> resettable or not(which allows choosing if its a guage or not) and
>>>>>> then let end-user choose if they really want to reset each resettable
>>>>>> metric(counter) (or do they want it as a perpetual counter).
>>>>>>
>>>>>
>>>>> there is only one entity in control of the rsyslog.conf file, how do you
>>>>> get to 'metric-owner' vs 'end-user'?
>>>>>
>>>>> David Lang
>>>>>
>>>>> In our case, the end-user control is very coarse-grained, where one
>>>>>>
>>>>>> can either get all counters as perpetual or all as resetting.
>>>>>>
>>>>>> On Wed, Mar 30, 2016 at 1:41 AM, David Lang <da...@lang.hm> wrote:
>>>>>>
>>>>>>> On Wed, 30 Mar 2016, singh.janmejay wrote:
>>

Re: [rsyslog] another dyn_stats question

[singh.janmejay]

Have opened a PR for this. Will fold it with other changes in the same
area. Here is the PR: https://github.com/rsyslog/rsyslog/pull/931 (not
ready for merge yet)

On Wed, Mar 30, 2016 at 8:01 AM, singh.janmejay
<singh.janme...@gmail.com> wrote:
> Cool, I'll make dyn-stats bucket resettable flag the only thing that
> controls resetting of its accumulators. It will or won't reset regardless of
> impstats reset flag value.
>
> This is again changing behavior from previous release but is a fairly minor
> thing, so we'll just document it and not worry about backward compatibility
> of its behavior.
>
> On Mar 30, 2016 2:55 AM, "David Lang" <da...@lang.hm> wrote:
>>
>> On Wed, 30 Mar 2016, singh.janmejay wrote:
>>
>>> Metric owner is the subsystem (action is a perfect example). End-user is
>>> author of the config.
>>
>>
>> Except that dyn_stats doen't exist except in the config, so it's the
>> "End-user" who sets everything about them.
>>
>>> In this case ideal situation would be to let action decide which of its
>>> metrics are resettable, and let end-user decide which of the resettable
>>> metric accumulators(subset of all metrics) do they actually want to
>>> reset.
>>
>>
>> I disagree, the module author should not be deciding if something is
>> resettable or not.
>>
>> The module author is either reporting the current state of something (like
>> queue length) where there is no resetting, or they are reporting the state
>> of a counter (number of messages processed), and it should be up to the
>> config author if they want to reset this every reporting period or not.
>>
>>> In case of dyn-stats, end-user is metric-owner.
>>>
>>> So we should probably disassociate from impstats reset flag and let
>>> bucket-level resettable flag be the only way to control it.
>>>
>>> That is what you expected the behavior would be, right? So it's more
>>> intuitive too.
>>>
>>> Should we do this?
>>
>>
>> I think so.
>>
>>> For static counters though, the problem still remains. If you want to
>>> reset
>>> omkafka resettable counters, but don't want resetting for imudp
>>> resettable
>>> counters, as of now we have no way to configure it like that.
>>
>>
>> Well, I see that as a legacy interface, so I don't think we need to worry
>> about changing that.
>>
>> Monitoring systems are almost always able to handle counters and convert
>> them to rate-of-change.
>>
>> resetting a counter is always racy with updates.
>>
>> If a monitoring system is looking at the stats, you almost always want
>> them to be non-reset counters. It's only if a person is looking at them that
>> you want to show the delta since the last report in the output.
>>
>> David Lang
>>
>>> On Mar 30, 2016 2:06 AM, "David Lang" <da...@lang.hm> wrote:
>>>
>>>> On Wed, 30 Mar 2016, singh.janmejay wrote:
>>>>
>>>> The problem is, pstats level control is an all or nothing kind of
>>>>>
>>>>> arrangement.
>>>>>
>>>>> Ideal setup would be to let metric-owner decide if metric should be
>>>>> resettable or not(which allows choosing if its a guage or not) and
>>>>> then let end-user choose if they really want to reset each resettable
>>>>> metric(counter) (or do they want it as a perpetual counter).
>>>>>
>>>>
>>>> there is only one entity in control of the rsyslog.conf file, how do you
>>>> get to 'metric-owner' vs 'end-user'?
>>>>
>>>> David Lang
>>>>
>>>> In our case, the end-user control is very coarse-grained, where one
>>>>>
>>>>> can either get all counters as perpetual or all as resetting.
>>>>>
>>>>> On Wed, Mar 30, 2016 at 1:41 AM, David Lang <da...@lang.hm> wrote:
>>>>>
>>>>>> On Wed, 30 Mar 2016, singh.janmejay wrote:
>>>>>>
>>>>>> Consider a queue-size, its not really a counter, its a guage. It will
>>>>>>>
>>>>>>> provide point-in-time value of size of the queue. Resetting such a
>>>>>>> value will make no sense. Some static metrics we report are guages
>>>>>>> (so
>>>>>>> subsystem supporting the metric must identify it to be resettable or
>>>>>>> not).
>&g

Re: [rsyslog] dynastats JSON output need work

[singh.janmejay]

PR created (not complete yet, not ready for merge):
https://github.com/rsyslog/rsyslog/pull/931

Have a quick look to get a feel for namespaced reporting of dyn-stats counters.

On Fri, Mar 25, 2016 at 4:25 PM, singh.janmejay
<singh.janme...@gmail.com> wrote:
> Cool. Will fix both stats representation and foreach in one PR.
> On Mar 25, 2016 3:10 PM, "Rainer Gerhards" <rgerha...@hq.adiscon.com> wrote:
>
>> ACK! Please go, let's set aside my probably too strong concerns. I am
>> perfectly happy when you say it's a small change and glad you do it!
>>
>> Rainer
>>
>> Sent from phone, thus brief.
>> Am 25.03.2016 06:21 schrieb "David Lang" <da...@lang.hm>:
>>
>> > As I understand Rainer, go .
>> >
>> > David Lang
>> >
>> > On Fri, 25 Mar 2016, singh.janmejay wrote:
>> >
>> > Makes sense. So final call on foreach? Go or no-go?
>> >> On Mar 25, 2016 2:38 AM, "David Lang" <da...@lang.hm> wrote:
>> >>
>> >> Just a note, the dynastats output line needs this as well as the bucket
>> >>> line.
>> >>>
>> >>> This isn't remote-input, but it needs the same ability to iterate over
>> >>> things.
>> >>>
>> >>> example:
>> >>>
>> >>>
>> >>>
>> >>>
>> {"name":"global","origin":"dynstats","message_framing.ops_overflow":0,"message_framing.new_metric_add":5,"message_framing.no_metric":0,"message_framing.metrics_purged":0,"message_framing.ops_ignored":0,"message_framing.purge_triggered":0,"msgs_per_host.ops_overflow":0,"msgs_per_host.new_metric_add":0,"msgs_per_host.no_metric":0,"msgs_per_host.metrics_purged":0,"msgs_per_host.ops_ignored":0,"msgs_per_host.purge_triggered":0,"msgs_per_edge_relay.ops_overflow":0,"msgs_per_edge_relay.new_metric_add":0,"msgs_per_edge_relay.no_metric":0,"msgs_per_edge_relay.metrics_purged":0,"msgs_per_edge_relay.ops_ignored":0,"msgs_per_edge_relay.purge_triggered":0,"msgs_per_core_relay.ops_overflow":0,"msgs_per_core_relay.new_metric_add":0,"msgs_per_core_relay.no_metric":0,"msgs_per_core_relay.metrics_purged":0,"msgs_per_core_relay.ops_ignored":0,"msgs_per_core_relay.purge_triggered":0,"msgs_per_program.ops_overflow":0,"msgs_per_program.new_metric_add":0,"msgs_per_program.no_metric":0,"msgs_pe
>>  !
>> >>>
>> >> r_
>> >
>> >> p!
>> >>
>> >>>
>> >>>
>> >>>
>> rogram.metrics_purged":0,"msgs_per_program.ops_ignored":0,"msgs_per_program.purge_triggered":0,"msgs_per_tag.ops_overflow":0,"msgs_per_tag.new_metric_add":0,"msgs_per_tag.no_metric":0,"msgs_per_tag.metrics_purged":0,"msgs_per_tag.ops_ignored":0,"msgs_per_tag.purge_triggered":0}
>> >>>
>> >>> David Lang
>> >>>
>> >>> On Wed, 23 Mar 2016, singh.janmejay wrote:
>> >>>
>> >>> Date: Wed, 23 Mar 2016 09:10:47 +0530
>> >>>
>> >>>> From: singh.janmejay <singh.janme...@gmail.com>
>> >>>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>> >>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> >>>> Subject: Re: [rsyslog] dynastats JSON output need work
>> >>>>
>> >>>> Agree, if most of us like it, I can implement foreach for objects. As
>> of
>> >>>> now I use omprog to call a script which uses jq and awk to make it
>> >>>> opentsdb
>> >>>> and redis friendly. I could have used omredis if foreach worked for
>> >>>> objects.
>> >>>>
>> >>>> Let us conclude.
>> >>>>
>> >>>> @Rainer any thoughts on foreach for object?
>> >>>> On Mar 23, 2016 1:56 AM, "David Lang" <da...@lang.hm> wrote:
>> >>>>
>> >>>> On Wed, 23 Mar 2016, singh.janmejay wrote:
>> >>>>
>> >>>>>
>> >>>>> Makes sense.
>> >>>>>
>> >>>>>
>> >>>>>> ..."counters":["a":80,"b":67] won't work, I think you meant
>> >>>

Re: [rsyslog] another dyn_stats question

[singh.janmejay]

Cool, I'll make dyn-stats bucket resettable flag the only thing that
controls resetting of its accumulators. It will or won't reset regardless
of impstats reset flag value.

This is again changing behavior from previous release but is a fairly minor
thing, so we'll just document it and not worry about backward compatibility
of its behavior.
On Mar 30, 2016 2:55 AM, "David Lang" <da...@lang.hm> wrote:

> On Wed, 30 Mar 2016, singh.janmejay wrote:
>
> Metric owner is the subsystem (action is a perfect example). End-user is
>> author of the config.
>>
>
> Except that dyn_stats doen't exist except in the config, so it's the
> "End-user" who sets everything about them.
>
> In this case ideal situation would be to let action decide which of its
>> metrics are resettable, and let end-user decide which of the resettable
>> metric accumulators(subset of all metrics) do they actually want to reset.
>>
>
> I disagree, the module author should not be deciding if something is
> resettable or not.
>
> The module author is either reporting the current state of something (like
> queue length) where there is no resetting, or they are reporting the state
> of a counter (number of messages processed), and it should be up to the
> config author if they want to reset this every reporting period or not.
>
> In case of dyn-stats, end-user is metric-owner.
>>
>> So we should probably disassociate from impstats reset flag and let
>> bucket-level resettable flag be the only way to control it.
>>
>> That is what you expected the behavior would be, right? So it's more
>> intuitive too.
>>
>> Should we do this?
>>
>
> I think so.
>
> For static counters though, the problem still remains. If you want to reset
>> omkafka resettable counters, but don't want resetting for imudp resettable
>> counters, as of now we have no way to configure it like that.
>>
>
> Well, I see that as a legacy interface, so I don't think we need to worry
> about changing that.
>
> Monitoring systems are almost always able to handle counters and convert
> them to rate-of-change.
>
> resetting a counter is always racy with updates.
>
> If a monitoring system is looking at the stats, you almost always want
> them to be non-reset counters. It's only if a person is looking at them
> that you want to show the delta since the last report in the output.
>
> David Lang
>
> On Mar 30, 2016 2:06 AM, "David Lang" <da...@lang.hm> wrote:
>>
>> On Wed, 30 Mar 2016, singh.janmejay wrote:
>>>
>>> The problem is, pstats level control is an all or nothing kind of
>>>
>>>> arrangement.
>>>>
>>>> Ideal setup would be to let metric-owner decide if metric should be
>>>> resettable or not(which allows choosing if its a guage or not) and
>>>> then let end-user choose if they really want to reset each resettable
>>>> metric(counter) (or do they want it as a perpetual counter).
>>>>
>>>>
>>> there is only one entity in control of the rsyslog.conf file, how do you
>>> get to 'metric-owner' vs 'end-user'?
>>>
>>> David Lang
>>>
>>> In our case, the end-user control is very coarse-grained, where one
>>>
>>>> can either get all counters as perpetual or all as resetting.
>>>>
>>>> On Wed, Mar 30, 2016 at 1:41 AM, David Lang <da...@lang.hm> wrote:
>>>>
>>>> On Wed, 30 Mar 2016, singh.janmejay wrote:
>>>>>
>>>>> Consider a queue-size, its not really a counter, its a guage. It will
>>>>>
>>>>>> provide point-in-time value of size of the queue. Resetting such a
>>>>>> value will make no sense. Some static metrics we report are guages (so
>>>>>> subsystem supporting the metric must identify it to be resettable or
>>>>>> not).
>>>>>>
>>>>>>
>>>>>
>>>>> but that won't work well with the current policy.
>>>>>
>>>>> you can't have anything as a gauge unless you eliminate the possiblity
>>>>> for
>>>>> any of the pstats data to be gauges.
>>>>>
>>>>> If pstats doesn't reset values, then dyn_stats can't implement gauges.
>>>>> If
>>>>> pststs does reset values, you have turned a lot of things that probably
>>>>> shouldn't reset into gauges.
>>>>>
>>>>> it would make far more sense to define this purely on a per-stats
>>>>> basis.

Re: [rsyslog] another dyn_stats question

[singh.janmejay]

Metric owner is the subsystem (action is a perfect example). End-user is
author of the config.

In this case ideal situation would be to let action decide which of its
metrics are resettable, and let end-user decide which of the resettable
metric accumulators(subset of all metrics) do they actually want to reset.

In case of dyn-stats, end-user is metric-owner.

So we should probably disassociate from impstats reset flag and let
bucket-level resettable flag be the only way to control it.

That is what you expected the behavior would be, right? So it's more
intuitive too.

Should we do this?

For static counters though, the problem still remains. If you want to reset
omkafka resettable counters, but don't want resetting for imudp resettable
counters, as of now we have no way to configure it like that.
On Mar 30, 2016 2:06 AM, "David Lang" <da...@lang.hm> wrote:

> On Wed, 30 Mar 2016, singh.janmejay wrote:
>
> The problem is, pstats level control is an all or nothing kind of
>> arrangement.
>>
>> Ideal setup would be to let metric-owner decide if metric should be
>> resettable or not(which allows choosing if its a guage or not) and
>> then let end-user choose if they really want to reset each resettable
>> metric(counter) (or do they want it as a perpetual counter).
>>
>
> there is only one entity in control of the rsyslog.conf file, how do you
> get to 'metric-owner' vs 'end-user'?
>
> David Lang
>
> In our case, the end-user control is very coarse-grained, where one
>> can either get all counters as perpetual or all as resetting.
>>
>> On Wed, Mar 30, 2016 at 1:41 AM, David Lang <da...@lang.hm> wrote:
>>
>>> On Wed, 30 Mar 2016, singh.janmejay wrote:
>>>
>>> Consider a queue-size, its not really a counter, its a guage. It will
>>>> provide point-in-time value of size of the queue. Resetting such a
>>>> value will make no sense. Some static metrics we report are guages (so
>>>> subsystem supporting the metric must identify it to be resettable or
>>>> not).
>>>>
>>>
>>>
>>> but that won't work well with the current policy.
>>>
>>> you can't have anything as a gauge unless you eliminate the possiblity
>>> for
>>> any of the pstats data to be gauges.
>>>
>>> If pstats doesn't reset values, then dyn_stats can't implement gauges. If
>>> pststs does reset values, you have turned a lot of things that probably
>>> shouldn't reset into gauges.
>>>
>>> it would make far more sense to define this purely on a per-stats basis.
>>>
>>> pstats has some items that are always gauges (size/maxsize/etc) and other
>>> items that are allowed to be treated as gauges or as counters.
>>>
>>> In my case, I don't want to reset those things (submitted/enqueued) that
>>> can
>>> be counters, but that prevents me from having dyn_stats items that are
>>> gauges.
>>>
>>> David Lang
>>>
>>>
>>> Yes, restart is a problem. But restarts are really in-frequent enough
>>>> to not matter. When they do happen, dx/dt chart shows a big dip, but
>>>> people still do track perpetual counters.
>>>>
>>>> Actually its only dyn_inc (we don't have a dyn_dec). But as we go
>>>> forward, we may expose capability to support guages and may manipulate
>>>> it with dyn_inc and dyn_dec.
>>>>
>>>> Resettable flag in dyn-stats bucket does the same thing as
>>>> resettable-flag in statsobj static-counter.
>>>>
>>>> On Tue, Mar 29, 2016 at 3:13 AM, David Lang <da...@lang.hm> wrote:
>>>>
>>>>>
>>>>> On Tue, 29 Mar 2016, singh.janmejay wrote:
>>>>>
>>>>> Well, from counter pov, some counters by nature need to be resettable,
>>>>>> some are actually guages (so resetting makes little sense) and some
>>>>>> are perpetual (so again resetting is not the right thing to do).
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> can you explain the different categories better? I don't see how any
>>>>> stats
>>>>> can be 'by nature' one form or another, since it's possible to convert
>>>>> the
>>>>> data from one form to another.
>>>>>
>>>>> The idea of perpetual counters doesn't work if the contents of the
>>>>> counters
>>>>> are lost at restart.
>>>>>
>>>>> It could be that I didn't read the do

Re: [rsyslog] another dyn_stats question

[singh.janmejay]

The problem is, pstats level control is an all or nothing kind of arrangement.

Ideal setup would be to let metric-owner decide if metric should be
resettable or not(which allows choosing if its a guage or not) and
then let end-user choose if they really want to reset each resettable
metric(counter) (or do they want it as a perpetual counter).

In our case, the end-user control is very coarse-grained, where one
can either get all counters as perpetual or all as resetting.

On Wed, Mar 30, 2016 at 1:41 AM, David Lang <da...@lang.hm> wrote:
> On Wed, 30 Mar 2016, singh.janmejay wrote:
>
>> Consider a queue-size, its not really a counter, its a guage. It will
>> provide point-in-time value of size of the queue. Resetting such a
>> value will make no sense. Some static metrics we report are guages (so
>> subsystem supporting the metric must identify it to be resettable or
>> not).
>
>
> but that won't work well with the current policy.
>
> you can't have anything as a gauge unless you eliminate the possiblity for
> any of the pstats data to be gauges.
>
> If pstats doesn't reset values, then dyn_stats can't implement gauges. If
> pststs does reset values, you have turned a lot of things that probably
> shouldn't reset into gauges.
>
> it would make far more sense to define this purely on a per-stats basis.
>
> pstats has some items that are always gauges (size/maxsize/etc) and other
> items that are allowed to be treated as gauges or as counters.
>
> In my case, I don't want to reset those things (submitted/enqueued) that can
> be counters, but that prevents me from having dyn_stats items that are
> gauges.
>
> David Lang
>
>
>> Yes, restart is a problem. But restarts are really in-frequent enough
>> to not matter. When they do happen, dx/dt chart shows a big dip, but
>> people still do track perpetual counters.
>>
>> Actually its only dyn_inc (we don't have a dyn_dec). But as we go
>> forward, we may expose capability to support guages and may manipulate
>> it with dyn_inc and dyn_dec.
>>
>> Resettable flag in dyn-stats bucket does the same thing as
>> resettable-flag in statsobj static-counter.
>>
>> On Tue, Mar 29, 2016 at 3:13 AM, David Lang <da...@lang.hm> wrote:
>>>
>>> On Tue, 29 Mar 2016, singh.janmejay wrote:
>>>
>>>> Well, from counter pov, some counters by nature need to be resettable,
>>>> some are actually guages (so resetting makes little sense) and some
>>>> are perpetual (so again resetting is not the right thing to do).
>>>
>>>
>>>
>>> can you explain the different categories better? I don't see how any
>>> stats
>>> can be 'by nature' one form or another, since it's possible to convert
>>> the
>>> data from one form to another.
>>>
>>> The idea of perpetual counters doesn't work if the contents of the
>>> counters
>>> are lost at restart.
>>>
>>> It could be that I didn't read the documentation well enough and missed
>>> references to ways to manipulate them besides dyn_inc and dyn_dec
>>>
>>> David Lang
>>>
>>>
>>>> A global control needs to be at impstats level because some users may
>>>> want to access data in rate-of-change (dx/dt) form rather than
>>>> absolute value form (it can be derived by adding up the numbers and
>>>> studying the slope, but its prone to error when some messages are
>>>> dropped etc). So user needs to have a say in how counters should be
>>>> reported, should they be reset or not.
>>>>
>>>> Same goes for dyn-stats. User may want a perpetual counter for one
>>>> dyn-stats bucket (regardless of other buckets being configured to
>>>> reset), so its important to allow a particular bucket to be configured
>>>> as non-resetting.
>>>>
>>>> However, if user chooses to view all the counters in cumulative form,
>>>> the same config that works for static-counters should work for
>>>> dyn-stats counters too (so impstats flag is relevant as well).
>>>>
>>>> And'ing is necessary because it doesn't make sense to reset a
>>>> by-nature non-resettable counter.
>>>>
>>>>
>>>> On Tue, Mar 29, 2016 at 12:25 AM, David Lang <da...@lang.hm> wrote:
>>>>>
>>>>>
>>>>> What is the thinking in having two switches that get ANDed together for
>>>>> resetting things? I could see two independent controls, or I could see
>>>>> the
>>>

Re: [rsyslog] another dyn_stats question

[singh.janmejay]

Consider a queue-size, its not really a counter, its a guage. It will
provide point-in-time value of size of the queue. Resetting such a
value will make no sense. Some static metrics we report are guages (so
subsystem supporting the metric must identify it to be resettable or
not).

Yes, restart is a problem. But restarts are really in-frequent enough
to not matter. When they do happen, dx/dt chart shows a big dip, but
people still do track perpetual counters.

Actually its only dyn_inc (we don't have a dyn_dec). But as we go
forward, we may expose capability to support guages and may manipulate
it with dyn_inc and dyn_dec.

Resettable flag in dyn-stats bucket does the same thing as
resettable-flag in statsobj static-counter.

On Tue, Mar 29, 2016 at 3:13 AM, David Lang <da...@lang.hm> wrote:
> On Tue, 29 Mar 2016, singh.janmejay wrote:
>
>> Well, from counter pov, some counters by nature need to be resettable,
>> some are actually guages (so resetting makes little sense) and some
>> are perpetual (so again resetting is not the right thing to do).
>
>
> can you explain the different categories better? I don't see how any stats
> can be 'by nature' one form or another, since it's possible to convert the
> data from one form to another.
>
> The idea of perpetual counters doesn't work if the contents of the counters
> are lost at restart.
>
> It could be that I didn't read the documentation well enough and missed
> references to ways to manipulate them besides dyn_inc and dyn_dec
>
> David Lang
>
>
>> A global control needs to be at impstats level because some users may
>> want to access data in rate-of-change (dx/dt) form rather than
>> absolute value form (it can be derived by adding up the numbers and
>> studying the slope, but its prone to error when some messages are
>> dropped etc). So user needs to have a say in how counters should be
>> reported, should they be reset or not.
>>
>> Same goes for dyn-stats. User may want a perpetual counter for one
>> dyn-stats bucket (regardless of other buckets being configured to
>> reset), so its important to allow a particular bucket to be configured
>> as non-resetting.
>>
>> However, if user chooses to view all the counters in cumulative form,
>> the same config that works for static-counters should work for
>> dyn-stats counters too (so impstats flag is relevant as well).
>>
>> And'ing is necessary because it doesn't make sense to reset a
>> by-nature non-resettable counter.
>>
>>
>> On Tue, Mar 29, 2016 at 12:25 AM, David Lang <da...@lang.hm> wrote:
>>>
>>> What is the thinking in having two switches that get ANDed together for
>>> resetting things? I could see two independent controls, or I could see
>>> the
>>> pstats setting controlling everything, but I don't understand two getting
>>> ANDed together.
>>>
>>> David Lang
>>>
>>>
>>> On Tue, 29 Mar 2016, singh.janmejay wrote:
>>>
>>>> Will add this to the documentation when I touch it for
>>>> output-restructuring.
>>>>
>>>> On Tue, Mar 29, 2016 at 12:11 AM, David Lang <da...@lang.hm> wrote:
>>>>>
>>>>>
>>>>> ahh, that's not at all clear from the documentation.
>>>>>
>>>>> I turned off resetting the counters at the pstats level because I was
>>>>> getting soem errors/coredumps from them (quite a few months ago, so it
>>>>> may
>>>>> be solved now) I'm currently getting >100 lines of pstats output now,
>>>>> before
>>>>> I add dyn_stats to things.
>>>>>
>>>>> I'll wait for the restructuring of the dyn_stats output and the new
>>>>> foreach
>>>>> and then try again.
>>>>>
>>>>> David Lang
>>>>>
>>>>> On Mon, 28 Mar 2016, singh.janmejay wrote:
>>>>>
>>>>>> Date: Mon, 28 Mar 2016 23:59:46 +0530
>>>>>>
>>>>>> From: singh.janmejay <singh.janme...@gmail.com>
>>>>>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>>>> Subject: Re: [rsyslog] another dyn_stats question
>>>>>>
>>>>>> Sorry, I meant and'ed not or'ed.
>>>>>>
>>>>>> On Mon, Mar 28, 2016 at 11:02 PM, singh.janmejay
>>>>>> <singh.janme...@gmail.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>

Re: [rsyslog] enhancement idea, template optimizer/compiler

[singh.janmejay]

Makes sense.

We probably don't even need to go as far as compiling. If we haven't tuned
it in significant amount of time, we'll likely have big wins if we just
invest in most significant PGOs.

Compile to C will be hard, because it'll then require compiler and linker.
Many environments will have policies around not having compiler or linker
installed on production env.
On Mar 29, 2016 12:36 AM, "Peter Portante" 
wrote:

> On Mon, Mar 28, 2016 at 2:15 PM, David Lang  wrote:
>
> > Templates are static constructs, but when they are used, they are
> > essentially interpreted each time they are accessed.
> >
> > A few years ago we introduced the idea that templates could be hard-coded
> > in C (the built-in templates were done this way, and string modules were
> > introduced to allow for further cusomizations). This was enough of an
> > improvement to result in a 10%+ improvement in total throughput.
> >
> > I'm wondering if it would make sense to try and 'compile' a template down
> > to C (or something extremely efficient to process) at startup time rather
> > than going to hand-coded string modules.
> >
> > thoughts?
>
>
> FWIW, I'd be in favor of doing that.  Great idea.
>
> -peter
>
>
> >
> >
> > David Lang
> > ___
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
> >
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] another dyn_stats question

[singh.janmejay]

Well, from counter pov, some counters by nature need to be resettable,
some are actually guages (so resetting makes little sense) and some
are perpetual (so again resetting is not the right thing to do).

A global control needs to be at impstats level because some users may
want to access data in rate-of-change (dx/dt) form rather than
absolute value form (it can be derived by adding up the numbers and
studying the slope, but its prone to error when some messages are
dropped etc). So user needs to have a say in how counters should be
reported, should they be reset or not.

Same goes for dyn-stats. User may want a perpetual counter for one
dyn-stats bucket (regardless of other buckets being configured to
reset), so its important to allow a particular bucket to be configured
as non-resetting.

However, if user chooses to view all the counters in cumulative form,
the same config that works for static-counters should work for
dyn-stats counters too (so impstats flag is relevant as well).

And'ing is necessary because it doesn't make sense to reset a
by-nature non-resettable counter.


On Tue, Mar 29, 2016 at 12:25 AM, David Lang <da...@lang.hm> wrote:
> What is the thinking in having two switches that get ANDed together for
> resetting things? I could see two independent controls, or I could see the
> pstats setting controlling everything, but I don't understand two getting
> ANDed together.
>
> David Lang
>
>
> On Tue, 29 Mar 2016, singh.janmejay wrote:
>
>> Will add this to the documentation when I touch it for
>> output-restructuring.
>>
>> On Tue, Mar 29, 2016 at 12:11 AM, David Lang <da...@lang.hm> wrote:
>>>
>>> ahh, that's not at all clear from the documentation.
>>>
>>> I turned off resetting the counters at the pstats level because I was
>>> getting soem errors/coredumps from them (quite a few months ago, so it
>>> may
>>> be solved now) I'm currently getting >100 lines of pstats output now,
>>> before
>>> I add dyn_stats to things.
>>>
>>> I'll wait for the restructuring of the dyn_stats output and the new
>>> foreach
>>> and then try again.
>>>
>>> David Lang
>>>
>>> On Mon, 28 Mar 2016, singh.janmejay wrote:
>>>
>>>> Date: Mon, 28 Mar 2016 23:59:46 +0530
>>>>
>>>> From: singh.janmejay <singh.janme...@gmail.com>
>>>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Subject: Re: [rsyslog] another dyn_stats question
>>>>
>>>> Sorry, I meant and'ed not or'ed.
>>>>
>>>> On Mon, Mar 28, 2016 at 11:02 PM, singh.janmejay
>>>> <singh.janme...@gmail.com> wrote:
>>>>>
>>>>>
>>>>> No, this reset mechanism is no different than the mechanism that
>>>>> static-counters use. If you turn it off at impstats level, it won't
>>>>> reset.
>>>>>
>>>>> This flag basically sets CTR_FLAG_RESETTABLE (statsobj.h).
>>>>>
>>>>> Its or'ed with stats-reporter's choice of resetting or not in this
>>>>> way: (bResetCtrs && (pCtr->flags & CTR_FLAG_RESETTABLE)), where
>>>>> bResetCtrs comes from that boolean param in impstats.
>>>>>
>>>>> On Mon, Mar 28, 2016 at 9:58 PM, David Lang <da...@lang.hm> wrote:
>>>>>>
>>>>>>
>>>>>> the first one should not be resetting, but the rest should be,
>>>>>> correct?
>>>>>>
>>>>>> David Lang
>>>>>>
>>>>>>
>>>>>> dyn_stats(name="message_framing" resettable="off"
>>>>>> maxCardinality="3000"
>>>>>> unusedMetricLife="1200")
>>>>>> dyn_stats(name="msgs_per_host" resettable="on" maxCardinality="3000"
>>>>>> unusedMetricLife="1200")
>>>>>> dyn_stats(name="msgs_per_edge_relay" resettable="on"
>>>>>> maxCardinality="3000"
>>>>>> unusedMetricLife="1200")
>>>>>> dyn_stats(name="msgs_per_core_relay" resettable="on"
>>>>>> maxCardinality="3000"
>>>>>> unusedMetricLife="1200")
>>>>>> dyn_stats(name="msgs_per_program" resettable="on"
>>>>>> maxCardinality="3000"
>>>>>> unused

Re: [rsyslog] another dyn_stats question

[singh.janmejay]

Will add this to the documentation when I touch it for output-restructuring.

On Tue, Mar 29, 2016 at 12:11 AM, David Lang <da...@lang.hm> wrote:
> ahh, that's not at all clear from the documentation.
>
> I turned off resetting the counters at the pstats level because I was
> getting soem errors/coredumps from them (quite a few months ago, so it may
> be solved now) I'm currently getting >100 lines of pstats output now, before
> I add dyn_stats to things.
>
> I'll wait for the restructuring of the dyn_stats output and the new foreach
> and then try again.
>
> David Lang
>
> On Mon, 28 Mar 2016, singh.janmejay wrote:
>
>> Date: Mon, 28 Mar 2016 23:59:46 +0530
>>
>> From: singh.janmejay <singh.janme...@gmail.com>
>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] another dyn_stats question
>>
>> Sorry, I meant and'ed not or'ed.
>>
>> On Mon, Mar 28, 2016 at 11:02 PM, singh.janmejay
>> <singh.janme...@gmail.com> wrote:
>>>
>>> No, this reset mechanism is no different than the mechanism that
>>> static-counters use. If you turn it off at impstats level, it won't
>>> reset.
>>>
>>> This flag basically sets CTR_FLAG_RESETTABLE (statsobj.h).
>>>
>>> Its or'ed with stats-reporter's choice of resetting or not in this
>>> way: (bResetCtrs && (pCtr->flags & CTR_FLAG_RESETTABLE)), where
>>> bResetCtrs comes from that boolean param in impstats.
>>>
>>> On Mon, Mar 28, 2016 at 9:58 PM, David Lang <da...@lang.hm> wrote:
>>>>
>>>> the first one should not be resetting, but the rest should be, correct?
>>>>
>>>> David Lang
>>>>
>>>>
>>>> dyn_stats(name="message_framing" resettable="off" maxCardinality="3000"
>>>> unusedMetricLife="1200")
>>>> dyn_stats(name="msgs_per_host" resettable="on" maxCardinality="3000"
>>>> unusedMetricLife="1200")
>>>> dyn_stats(name="msgs_per_edge_relay" resettable="on"
>>>> maxCardinality="3000"
>>>> unusedMetricLife="1200")
>>>> dyn_stats(name="msgs_per_core_relay" resettable="on"
>>>> maxCardinality="3000"
>>>> unusedMetricLife="1200")
>>>> dyn_stats(name="msgs_per_program" resettable="on" maxCardinality="3000"
>>>> unusedMetricLife="1200")
>>>> dyn_stats(name="msgs_per_tag" resettable="on" maxCardinality="3000"
>>>> unusedMetricLife="1200")
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, 28 Mar 2016, singh.janmejay wrote:
>>>>
>>>>> Date: Mon, 28 Mar 2016 14:37:12 +0530
>>>>> From: singh.janmejay <singh.janme...@gmail.com>
>>>>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>>> Subject: Re: [rsyslog] another dyn_stats question
>>>>>
>>>>>
>>>>> David,
>>>>>
>>>>> You have resetting turned-off in the config you shared with me. That
>>>>> would prevent resetting of counters.
>>>>>
>>>>> On Fri, Mar 25, 2016 at 8:21 PM, singh.janmejay
>>>>> <singh.janme...@gmail.com> wrote:
>>>>>>
>>>>>>
>>>>>> That is not supposed to happen. Although very unlikely, it may be an
>>>>>> environment specific bug. Do all dyn-stats tests pass in your local
>>>>>> env?
>>>>>>
>>>>>> Also, are you using something other than impstats to report it?
>>>>>>
>>>>>> On Mar 25, 2016 7:24 PM, "David Lang" <da...@lang.hm> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> resettable counters don't seem to be resetting for me.
>>>>>>>
>>>>>>> David Lang
>>>>>>>
>>>>>>> On Fri, 25 Mar 2016, singh.janmejay wrote:
>>>>>>>
>>>>>>>> Well, the bug is metric-ttl ends up killing all metric instead of
>>>>>>>> unused-metrics. This works for resettable counters, but for
>>&g

Re: [rsyslog] another dyn_stats question

[singh.janmejay]

Sorry, I meant and'ed not or'ed.

On Mon, Mar 28, 2016 at 11:02 PM, singh.janmejay
<singh.janme...@gmail.com> wrote:
> No, this reset mechanism is no different than the mechanism that
> static-counters use. If you turn it off at impstats level, it won't
> reset.
>
> This flag basically sets CTR_FLAG_RESETTABLE (statsobj.h).
>
> Its or'ed with stats-reporter's choice of resetting or not in this
> way: (bResetCtrs && (pCtr->flags & CTR_FLAG_RESETTABLE)), where
> bResetCtrs comes from that boolean param in impstats.
>
> On Mon, Mar 28, 2016 at 9:58 PM, David Lang <da...@lang.hm> wrote:
>> the first one should not be resetting, but the rest should be, correct?
>>
>> David Lang
>>
>>
>> dyn_stats(name="message_framing" resettable="off" maxCardinality="3000"
>> unusedMetricLife="1200")
>> dyn_stats(name="msgs_per_host" resettable="on" maxCardinality="3000"
>> unusedMetricLife="1200")
>> dyn_stats(name="msgs_per_edge_relay" resettable="on" maxCardinality="3000"
>> unusedMetricLife="1200")
>> dyn_stats(name="msgs_per_core_relay" resettable="on" maxCardinality="3000"
>> unusedMetricLife="1200")
>> dyn_stats(name="msgs_per_program" resettable="on" maxCardinality="3000"
>> unusedMetricLife="1200")
>> dyn_stats(name="msgs_per_tag" resettable="on" maxCardinality="3000"
>> unusedMetricLife="1200")
>>
>>
>>
>>
>> On Mon, 28 Mar 2016, singh.janmejay wrote:
>>
>>> Date: Mon, 28 Mar 2016 14:37:12 +0530
>>> From: singh.janmejay <singh.janme...@gmail.com>
>>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>> Subject: Re: [rsyslog] another dyn_stats question
>>>
>>>
>>> David,
>>>
>>> You have resetting turned-off in the config you shared with me. That
>>> would prevent resetting of counters.
>>>
>>> On Fri, Mar 25, 2016 at 8:21 PM, singh.janmejay
>>> <singh.janme...@gmail.com> wrote:
>>>>
>>>> That is not supposed to happen. Although very unlikely, it may be an
>>>> environment specific bug. Do all dyn-stats tests pass in your local env?
>>>>
>>>> Also, are you using something other than impstats to report it?
>>>>
>>>> On Mar 25, 2016 7:24 PM, "David Lang" <da...@lang.hm> wrote:
>>>>>
>>>>>
>>>>> resettable counters don't seem to be resetting for me.
>>>>>
>>>>> David Lang
>>>>>
>>>>> On Fri, 25 Mar 2016, singh.janmejay wrote:
>>>>>
>>>>>> Well, the bug is metric-ttl ends up killing all metric instead of
>>>>>> unused-metrics. This works for resettable counters, but for
>>>>>> non-resettable
>>>>>> counters it kills the accumulation prematurely.
>>>>>>
>>>>>> The fix will identify which counters are being used and will not kill
>>>>>> them.
>>>>>> So it won't discard accumulated value as long as it is being
>>>>>> incremented
>>>>>> at
>>>>>> least once within the TTL time.
>>>>>> On Mar 25, 2016 10:53 AM, "David Lang" <da...@lang.hm> wrote:
>>>>>>
>>>>>>> What I'm seeing is that unless the reset time matches the pstats
>>>>>>> interval,
>>>>>>> the data output doesn't reset each pstats report.
>>>>>>>
>>>>>>> David Lang
>>>>>>>
>>>>>>> On Fri, 25 Mar 2016, singh.janmejay wrote:
>>>>>>>
>>>>>>> This is a bug. I'll provide the fix on Monday(or in the worst case in
>>>>>>> a
>>>>>>> few
>>>>>>>>
>>>>>>>>
>>>>>>>> days, in case I don't get a chance to work on this on Monday).
>>>>>>>>
>>>>>>>> Basically, I was supposed to have a shadow table to mark used
>>>>>>>> counters,
>>>>>>>> and
>>>>>>>> kill unused counters in next run of metric-ttl-cycle which wasn't
>>>>>>>> present
>>>>&

Re: [rsyslog] another dyn_stats question

[singh.janmejay]

No, this reset mechanism is no different than the mechanism that
static-counters use. If you turn it off at impstats level, it won't
reset.

This flag basically sets CTR_FLAG_RESETTABLE (statsobj.h).

Its or'ed with stats-reporter's choice of resetting or not in this
way: (bResetCtrs && (pCtr->flags & CTR_FLAG_RESETTABLE)), where
bResetCtrs comes from that boolean param in impstats.

On Mon, Mar 28, 2016 at 9:58 PM, David Lang <da...@lang.hm> wrote:
> the first one should not be resetting, but the rest should be, correct?
>
> David Lang
>
>
> dyn_stats(name="message_framing" resettable="off" maxCardinality="3000"
> unusedMetricLife="1200")
> dyn_stats(name="msgs_per_host" resettable="on" maxCardinality="3000"
> unusedMetricLife="1200")
> dyn_stats(name="msgs_per_edge_relay" resettable="on" maxCardinality="3000"
> unusedMetricLife="1200")
> dyn_stats(name="msgs_per_core_relay" resettable="on" maxCardinality="3000"
> unusedMetricLife="1200")
> dyn_stats(name="msgs_per_program" resettable="on" maxCardinality="3000"
> unusedMetricLife="1200")
> dyn_stats(name="msgs_per_tag" resettable="on" maxCardinality="3000"
> unusedMetricLife="1200")
>
>
>
>
> On Mon, 28 Mar 2016, singh.janmejay wrote:
>
>> Date: Mon, 28 Mar 2016 14:37:12 +0530
>> From: singh.janmejay <singh.janme...@gmail.com>
>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] another dyn_stats question
>>
>>
>> David,
>>
>> You have resetting turned-off in the config you shared with me. That
>> would prevent resetting of counters.
>>
>> On Fri, Mar 25, 2016 at 8:21 PM, singh.janmejay
>> <singh.janme...@gmail.com> wrote:
>>>
>>> That is not supposed to happen. Although very unlikely, it may be an
>>> environment specific bug. Do all dyn-stats tests pass in your local env?
>>>
>>> Also, are you using something other than impstats to report it?
>>>
>>> On Mar 25, 2016 7:24 PM, "David Lang" <da...@lang.hm> wrote:
>>>>
>>>>
>>>> resettable counters don't seem to be resetting for me.
>>>>
>>>> David Lang
>>>>
>>>> On Fri, 25 Mar 2016, singh.janmejay wrote:
>>>>
>>>>> Well, the bug is metric-ttl ends up killing all metric instead of
>>>>> unused-metrics. This works for resettable counters, but for
>>>>> non-resettable
>>>>> counters it kills the accumulation prematurely.
>>>>>
>>>>> The fix will identify which counters are being used and will not kill
>>>>> them.
>>>>> So it won't discard accumulated value as long as it is being
>>>>> incremented
>>>>> at
>>>>> least once within the TTL time.
>>>>> On Mar 25, 2016 10:53 AM, "David Lang" <da...@lang.hm> wrote:
>>>>>
>>>>>> What I'm seeing is that unless the reset time matches the pstats
>>>>>> interval,
>>>>>> the data output doesn't reset each pstats report.
>>>>>>
>>>>>> David Lang
>>>>>>
>>>>>> On Fri, 25 Mar 2016, singh.janmejay wrote:
>>>>>>
>>>>>> This is a bug. I'll provide the fix on Monday(or in the worst case in
>>>>>> a
>>>>>> few
>>>>>>>
>>>>>>>
>>>>>>> days, in case I don't get a chance to work on this on Monday).
>>>>>>>
>>>>>>> Basically, I was supposed to have a shadow table to mark used
>>>>>>> counters,
>>>>>>> and
>>>>>>> kill unused counters in next run of metric-ttl-cycle which wasn't
>>>>>>> present
>>>>>>> in first cut because I started with resetting counter test cases
>>>>>>> first,
>>>>>>> then somehow this slipped my mind.
>>>>>>>
>>>>>>
>>>>>> On Mar 25, 2016 4:12 AM, "David Lang" <da...@lang.hm> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> the unusedMetricLife parameter when setting up a stat say

Re: [rsyslog] another dyn_stats question

[singh.janmejay]

David,

You have resetting turned-off in the config you shared with me. That
would prevent resetting of counters.

On Fri, Mar 25, 2016 at 8:21 PM, singh.janmejay
<singh.janme...@gmail.com> wrote:
> That is not supposed to happen. Although very unlikely, it may be an
> environment specific bug. Do all dyn-stats tests pass in your local env?
>
> Also, are you using something other than impstats to report it?
>
> On Mar 25, 2016 7:24 PM, "David Lang" <da...@lang.hm> wrote:
>>
>> resettable counters don't seem to be resetting for me.
>>
>> David Lang
>>
>> On Fri, 25 Mar 2016, singh.janmejay wrote:
>>
>>> Well, the bug is metric-ttl ends up killing all metric instead of
>>> unused-metrics. This works for resettable counters, but for
>>> non-resettable
>>> counters it kills the accumulation prematurely.
>>>
>>> The fix will identify which counters are being used and will not kill
>>> them.
>>> So it won't discard accumulated value as long as it is being incremented
>>> at
>>> least once within the TTL time.
>>> On Mar 25, 2016 10:53 AM, "David Lang" <da...@lang.hm> wrote:
>>>
>>>> What I'm seeing is that unless the reset time matches the pstats
>>>> interval,
>>>> the data output doesn't reset each pstats report.
>>>>
>>>> David Lang
>>>>
>>>> On Fri, 25 Mar 2016, singh.janmejay wrote:
>>>>
>>>> This is a bug. I'll provide the fix on Monday(or in the worst case in a
>>>> few
>>>>>
>>>>> days, in case I don't get a chance to work on this on Monday).
>>>>>
>>>>> Basically, I was supposed to have a shadow table to mark used counters,
>>>>> and
>>>>> kill unused counters in next run of metric-ttl-cycle which wasn't
>>>>> present
>>>>> in first cut because I started with resetting counter test cases first,
>>>>> then somehow this slipped my mind.
>>>>>
>>>>
>>>> On Mar 25, 2016 4:12 AM, "David Lang" <da...@lang.hm> wrote:
>>>>>
>>>>>
>>>>> the unusedMetricLife parameter when setting up a stat says that it's to
>>>>>>
>>>>>> purge an unused stat.
>>>>>>
>>>>>> But I've got an example where I create about 5 stats in the bucket and
>>>>>> what I think I'm seeing is that the stats are cumulative, not reset
>>>>>> each
>>>>>> time they are dumped out, but they get reset every unusedMetricLife
>>>>>> seconds.
>>>>>>
>>>>>> Is this my imagination? or is the behavior not matching the
>>>>>> documentation?
>>>>>>
>>>>>> David Lang
>>>>>> ___
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com/professional-services/
>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>> myriad
>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>>> DON'T LIKE THAT.
>>>>>>
>>>>>> ___
>>>>>
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>> myriad
>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T LIKE THAT.
>>>>>
>>>>> ___
>>>>
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T LIKE THAT.
>>>>
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>>> LIKE THAT.
>>>
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] another dyn_stats question

[singh.janmejay]

That is not supposed to happen. Although very unlikely, it may be an
environment specific bug. Do all dyn-stats tests pass in your local env?

Also, are you using something other than impstats to report it?
On Mar 25, 2016 7:24 PM, "David Lang" <da...@lang.hm> wrote:

> resettable counters don't seem to be resetting for me.
>
> David Lang
>
> On Fri, 25 Mar 2016, singh.janmejay wrote:
>
> Well, the bug is metric-ttl ends up killing all metric instead of
>> unused-metrics. This works for resettable counters, but for non-resettable
>> counters it kills the accumulation prematurely.
>>
>> The fix will identify which counters are being used and will not kill
>> them.
>> So it won't discard accumulated value as long as it is being incremented
>> at
>> least once within the TTL time.
>> On Mar 25, 2016 10:53 AM, "David Lang" <da...@lang.hm> wrote:
>>
>> What I'm seeing is that unless the reset time matches the pstats interval,
>>> the data output doesn't reset each pstats report.
>>>
>>> David Lang
>>>
>>> On Fri, 25 Mar 2016, singh.janmejay wrote:
>>>
>>> This is a bug. I'll provide the fix on Monday(or in the worst case in a
>>> few
>>>
>>>> days, in case I don't get a chance to work on this on Monday).
>>>>
>>>> Basically, I was supposed to have a shadow table to mark used counters,
>>>> and
>>>> kill unused counters in next run of metric-ttl-cycle which wasn't
>>>> present
>>>> in first cut because I started with resetting counter test cases first,
>>>> then somehow this slipped my mind.
>>>>
>>>>
>>> On Mar 25, 2016 4:12 AM, "David Lang" <da...@lang.hm> wrote:
>>>
>>>>
>>>> the unusedMetricLife parameter when setting up a stat says that it's to
>>>>
>>>>> purge an unused stat.
>>>>>
>>>>> But I've got an example where I create about 5 stats in the bucket and
>>>>> what I think I'm seeing is that the stats are cumulative, not reset
>>>>> each
>>>>> time they are dumped out, but they get reset every unusedMetricLife
>>>>> seconds.
>>>>>
>>>>> Is this my imagination? or is the behavior not matching the
>>>>> documentation?
>>>>>
>>>>> David Lang
>>>>> ___
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>> myriad
>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T LIKE THAT.
>>>>>
>>>>> ___
>>>>>
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T LIKE THAT.
>>>>
>>>> ___
>>>>
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] dynastats JSON output need work

[singh.janmejay]

Cool. Will fix both stats representation and foreach in one PR.
On Mar 25, 2016 3:10 PM, "Rainer Gerhards" <rgerha...@hq.adiscon.com> wrote:

> ACK! Please go, let's set aside my probably too strong concerns. I am
> perfectly happy when you say it's a small change and glad you do it!
>
> Rainer
>
> Sent from phone, thus brief.
> Am 25.03.2016 06:21 schrieb "David Lang" <da...@lang.hm>:
>
> > As I understand Rainer, go .
> >
> > David Lang
> >
> > On Fri, 25 Mar 2016, singh.janmejay wrote:
> >
> > Makes sense. So final call on foreach? Go or no-go?
> >> On Mar 25, 2016 2:38 AM, "David Lang" <da...@lang.hm> wrote:
> >>
> >> Just a note, the dynastats output line needs this as well as the bucket
> >>> line.
> >>>
> >>> This isn't remote-input, but it needs the same ability to iterate over
> >>> things.
> >>>
> >>> example:
> >>>
> >>>
> >>>
> >>>
> {"name":"global","origin":"dynstats","message_framing.ops_overflow":0,"message_framing.new_metric_add":5,"message_framing.no_metric":0,"message_framing.metrics_purged":0,"message_framing.ops_ignored":0,"message_framing.purge_triggered":0,"msgs_per_host.ops_overflow":0,"msgs_per_host.new_metric_add":0,"msgs_per_host.no_metric":0,"msgs_per_host.metrics_purged":0,"msgs_per_host.ops_ignored":0,"msgs_per_host.purge_triggered":0,"msgs_per_edge_relay.ops_overflow":0,"msgs_per_edge_relay.new_metric_add":0,"msgs_per_edge_relay.no_metric":0,"msgs_per_edge_relay.metrics_purged":0,"msgs_per_edge_relay.ops_ignored":0,"msgs_per_edge_relay.purge_triggered":0,"msgs_per_core_relay.ops_overflow":0,"msgs_per_core_relay.new_metric_add":0,"msgs_per_core_relay.no_metric":0,"msgs_per_core_relay.metrics_purged":0,"msgs_per_core_relay.ops_ignored":0,"msgs_per_core_relay.purge_triggered":0,"msgs_per_program.ops_overflow":0,"msgs_per_program.new_metric_add":0,"msgs_per_program.no_metric":0,"msgs_pe
>  !
> >>>
> >> r_
> >
> >> p!
> >>
> >>>
> >>>
> >>>
> rogram.metrics_purged":0,"msgs_per_program.ops_ignored":0,"msgs_per_program.purge_triggered":0,"msgs_per_tag.ops_overflow":0,"msgs_per_tag.new_metric_add":0,"msgs_per_tag.no_metric":0,"msgs_per_tag.metrics_purged":0,"msgs_per_tag.ops_ignored":0,"msgs_per_tag.purge_triggered":0}
> >>>
> >>> David Lang
> >>>
> >>> On Wed, 23 Mar 2016, singh.janmejay wrote:
> >>>
> >>> Date: Wed, 23 Mar 2016 09:10:47 +0530
> >>>
> >>>> From: singh.janmejay <singh.janme...@gmail.com>
> >>>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
> >>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
> >>>> Subject: Re: [rsyslog] dynastats JSON output need work
> >>>>
> >>>> Agree, if most of us like it, I can implement foreach for objects. As
> of
> >>>> now I use omprog to call a script which uses jq and awk to make it
> >>>> opentsdb
> >>>> and redis friendly. I could have used omredis if foreach worked for
> >>>> objects.
> >>>>
> >>>> Let us conclude.
> >>>>
> >>>> @Rainer any thoughts on foreach for object?
> >>>> On Mar 23, 2016 1:56 AM, "David Lang" <da...@lang.hm> wrote:
> >>>>
> >>>> On Wed, 23 Mar 2016, singh.janmejay wrote:
> >>>>
> >>>>>
> >>>>> Makes sense.
> >>>>>
> >>>>>
> >>>>>> ..."counters":["a":80,"b":67] won't work, I think you meant
> >>>>>> ..."counters": [{"a" : 80}, {"b": 67}].
> >>>>>>
> >>>>>>
> >>>>>> ahh, showing my lack of knowledge of JSON :-) talking things
> through:
> >>>>>
> >>>>> so our choices are:
> >>>>>
> >>>>> "counters": [{"a":80},{"b":67}]
> >>>>> vs
> >>>>> "counters": {"a":80,"b":67}
> >>>&g

Re: [rsyslog] another dyn_stats question

[singh.janmejay]

Well, the bug is metric-ttl ends up killing all metric instead of
unused-metrics. This works for resettable counters, but for non-resettable
counters it kills the accumulation prematurely.

The fix will identify which counters are being used and will not kill them.
So it won't discard accumulated value as long as it is being incremented at
least once within the TTL time.
On Mar 25, 2016 10:53 AM, "David Lang" <da...@lang.hm> wrote:

> What I'm seeing is that unless the reset time matches the pstats interval,
> the data output doesn't reset each pstats report.
>
> David Lang
>
> On Fri, 25 Mar 2016, singh.janmejay wrote:
>
> This is a bug. I'll provide the fix on Monday(or in the worst case in a few
>> days, in case I don't get a chance to work on this on Monday).
>>
>> Basically, I was supposed to have a shadow table to mark used counters,
>> and
>> kill unused counters in next run of metric-ttl-cycle which wasn't present
>> in first cut because I started with resetting counter test cases first,
>> then somehow this slipped my mind.
>>
>
> On Mar 25, 2016 4:12 AM, "David Lang" <da...@lang.hm> wrote:
>>
>> the unusedMetricLife parameter when setting up a stat says that it's to
>>> purge an unused stat.
>>>
>>> But I've got an example where I create about 5 stats in the bucket and
>>> what I think I'm seeing is that the stats are cumulative, not reset each
>>> time they are dumped out, but they get reset every unusedMetricLife
>>> seconds.
>>>
>>> Is this my imagination? or is the behavior not matching the
>>> documentation?
>>>
>>> David Lang
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] dynastats JSON output need work

[singh.janmejay]

Makes sense. So final call on foreach? Go or no-go?
On Mar 25, 2016 2:38 AM, "David Lang" <da...@lang.hm> wrote:

> Just a note, the dynastats output line needs this as well as the bucket
> line.
>
> This isn't remote-input, but it needs the same ability to iterate over
> things.
>
> example:
>
>
> {"name":"global","origin":"dynstats","message_framing.ops_overflow":0,"message_framing.new_metric_add":5,"message_framing.no_metric":0,"message_framing.metrics_purged":0,"message_framing.ops_ignored":0,"message_framing.purge_triggered":0,"msgs_per_host.ops_overflow":0,"msgs_per_host.new_metric_add":0,"msgs_per_host.no_metric":0,"msgs_per_host.metrics_purged":0,"msgs_per_host.ops_ignored":0,"msgs_per_host.purge_triggered":0,"msgs_per_edge_relay.ops_overflow":0,"msgs_per_edge_relay.new_metric_add":0,"msgs_per_edge_relay.no_metric":0,"msgs_per_edge_relay.metrics_purged":0,"msgs_per_edge_relay.ops_ignored":0,"msgs_per_edge_relay.purge_triggered":0,"msgs_per_core_relay.ops_overflow":0,"msgs_per_core_relay.new_metric_add":0,"msgs_per_core_relay.no_metric":0,"msgs_per_core_relay.metrics_purged":0,"msgs_per_core_relay.ops_ignored":0,"msgs_per_core_relay.purge_triggered":0,"msgs_per_program.ops_overflow":0,"msgs_per_program.new_metric_add":0,"msgs_per_program.no_metric":0,"msgs_per_
 p!
>
> rogram.metrics_purged":0,"msgs_per_program.ops_ignored":0,"msgs_per_program.purge_triggered":0,"msgs_per_tag.ops_overflow":0,"msgs_per_tag.new_metric_add":0,"msgs_per_tag.no_metric":0,"msgs_per_tag.metrics_purged":0,"msgs_per_tag.ops_ignored":0,"msgs_per_tag.purge_triggered":0}
>
> David Lang
>
> On Wed, 23 Mar 2016, singh.janmejay wrote:
>
> Date: Wed, 23 Mar 2016 09:10:47 +0530
>> From: singh.janmejay <singh.janme...@gmail.com>
>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] dynastats JSON output need work
>>
>> Agree, if most of us like it, I can implement foreach for objects. As of
>> now I use omprog to call a script which uses jq and awk to make it
>> opentsdb
>> and redis friendly. I could have used omredis if foreach worked for
>> objects.
>>
>> Let us conclude.
>>
>> @Rainer any thoughts on foreach for object?
>> On Mar 23, 2016 1:56 AM, "David Lang" <da...@lang.hm> wrote:
>>
>> On Wed, 23 Mar 2016, singh.janmejay wrote:
>>>
>>> Makes sense.
>>>
>>>>
>>>> ..."counters":["a":80,"b":67] won't work, I think you meant
>>>> ..."counters": [{"a" : 80}, {"b": 67}].
>>>>
>>>>
>>> ahh, showing my lack of knowledge of JSON :-) talking things through:
>>>
>>> so our choices are:
>>>
>>> "counters": [{"a":80},{"b":67}]
>>> vs
>>> "counters": {"a":80,"b":67}
>>>
>>> running these through jq gives
>>>
>>> {
>>>   "counters": [
>>> {
>>>   "a": 80
>>> },
>>> {
>>>   "b": 67
>>> }
>>>   ]
>>> }
>>>
>>> vs
>>>
>>> {
>>>   "counters": {
>>> "a": 80,
>>> "b": 67
>>>   }
>>> }
>>>
>>> or to reference a
>>> .counters[0].a
>>> vs
>>> .counters.a
>>>
>>> The latter seems to clearly be the cleaner and more logical one to me.
>>>
>>> Does this seem sane?
>>>
>>> But to use it, we need a variation of foreach that gives the object name
>>> as well as the object contents.
>>>
>>> David Lang
>>>
>>> So it boils down to foreach handling object or not.
>>>
>>>>
>>>> Thoughts?
>>>>
>>>> On Wed, Mar 23, 2016 at 1:18 AM, David Lang <da...@lang.hm> wrote:
>>>>
>>>> On Wed, 23 Mar 2016, singh.janmejay wrote:
>>>>>
>>>>> What about wrapDynamicObjects="on|off"? That is required regardless,
>>>>>
>>>>>> right? (if we want to preserve bac

Re: [rsyslog] another dyn_stats question

[singh.janmejay]

This is a bug. I'll provide the fix on Monday(or in the worst case in a few
days, in case I don't get a chance to work on this on Monday).

Basically, I was supposed to have a shadow table to mark used counters, and
kill unused counters in next run of metric-ttl-cycle which wasn't present
in first cut because I started with resetting counter test cases first,
then somehow this slipped my mind.
On Mar 25, 2016 4:12 AM, "David Lang"  wrote:

> the unusedMetricLife parameter when setting up a stat says that it's to
> purge an unused stat.
>
> But I've got an example where I create about 5 stats in the bucket and
> what I think I'm seeing is that the stats are cumulative, not reset each
> time they are dumped out, but they get reset every unusedMetricLife seconds.
>
> Is this my imagination? or is the behavior not matching the documentation?
>
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] dynastats JSON output need work

[singh.janmejay]

To me this is under the same umbrella of features (loop over
collection-style data). That data comes in a few flavors, and we are
doing it in small increments instead of doing it in one-shot and in a
way it does look like feature-creep. But I think some variant (if not
this statement in current form) will be required.

About supporting it in the long term, it makes sense to kill features
when users no longer defend it (by defending I mean someone should
come forward to maintain it, fix it, re-design it etc). Basically, its
a move towards encouraging the cycle of trying ideas and killing them
if they have no takers.


On Wed, Mar 23, 2016 at 5:30 PM, Rainer Gerhards
<rgerha...@hq.adiscon.com> wrote:
> 2016-03-23 12:42 GMT+01:00 singh.janmejay <singh.janme...@gmail.com>:
>> From variable-management pov, loop as a construct will be required as
>> long we have array, associative-array etc.
>>
>> But logs do have multi-valued fields, they have key-value pairs etc right?
>>
>> So regardless of the change in variable-system,
>> array/associative-array data-type will continue to exist. Which in
>> turn requires ways to work with that data-type.
>>
>> Doesn't it?
>
> yeah, but... I have a strong feeling of feature creep here.
> Originally, the config should not support any looping at all. Then we
> had arrays and said "well, it would be nice if we could process them".
> Now we have other types of objects and again "it would be nice if we
> could process them". Next we probably say "well, a for(i = 0 ; i< 10;
> i++) loop would be nice"...
>
> You see what I mean.
>
> Anyhow, maybe we need to go towards full-blown scripting capabilities.
> Not sure the engine is really up to that, but... Given the fact that I
> didn't strongly oppose the current loops system, I think you have a
> good point if the extension can easily be done without adding extra
> complexity. Still I am a bit nervous about the long-term implications,
> but let's see how things evolve.
>
> Rainer
>>
>> On Wed, Mar 23, 2016 at 4:58 PM, singh.janmejay
>> <singh.janme...@gmail.com> wrote:
>>> In-code footprint of foreach enhancement (to handle object in addition
>>> to array) will be fairly small (I am thinking under 40 lines?).
>>> Majority of addition will be in form of tests (unless we choose to
>>> change the rainerscript signature of that statement).
>>>
>>> Alternatively, from data-traversal functional-completeness pov, the
>>> same thing can be achieved without touching foreach impl, in a
>>> slightly high-overhead way.
>>>
>>> We can create a new rainerscript fn make_list() =>
>>> [kv-tuples...] (or mklist).
>>>
>>> Eg.
>>> {"foo" : "bar", "baz" : "quux"} becomes [{"key": "foo", "value":
>>> "bar"}, {"key": "baz", "value": "quux"}]
>>>
>>> Clearly it will have inferior performance, but it'll make foreach work
>>> for all object use-cases.
>>>
>>> I really feel in terms of complexity though, this (transform approach)
>>> is worse than first-class support (purely because of it using a
>>> round-about way of working).
>>>
>>>
>>> On Wed, Mar 23, 2016 at 3:01 PM, Rainer Gerhards
>>> <rgerha...@hq.adiscon.com> wrote:
>>>> 2016-03-22 20:34 GMT+01:00 singh.janmejay <singh.janme...@gmail.com>:
>>>>> What about wrapDynamicObjects="on|off"? That is required regardless,
>>>>> right? (if we want to preserve backward compatibility). Unless we
>>>>> choose to change it anyway (which im fine with).
>>>>>
>>>>> @Rainer: what do you think about foreach handling object?
>>>>
>>>> so far I could only manage to have a glimpse at the conversation. I
>>>> admit that I am really concerned about all the extra complexity we
>>>> introduce. And do so for what I think is a border-use case at best.
>>>> There is a lot of work that should really be done in regard to
>>>> variables and performance and I am unsure if this is the right time to
>>>> do large extensions...
>>>>
>>>> I was happy with the dynstats as it looked like a very contained
>>>> solution that did not affect much else. But now it looks we need to
>>>> touch a big deal of code ... code that's not really ready for that...
>>>>
>>>> Maybe I get a different view when I find time to read all this more 
>>>&g

Re: [rsyslog] dynastats JSON output need work

[singh.janmejay]

>From variable-management pov, loop as a construct will be required as
long we have array, associative-array etc.

But logs do have multi-valued fields, they have key-value pairs etc right?

So regardless of the change in variable-system,
array/associative-array data-type will continue to exist. Which in
turn requires ways to work with that data-type.

Doesn't it?

On Wed, Mar 23, 2016 at 4:58 PM, singh.janmejay
<singh.janme...@gmail.com> wrote:
> In-code footprint of foreach enhancement (to handle object in addition
> to array) will be fairly small (I am thinking under 40 lines?).
> Majority of addition will be in form of tests (unless we choose to
> change the rainerscript signature of that statement).
>
> Alternatively, from data-traversal functional-completeness pov, the
> same thing can be achieved without touching foreach impl, in a
> slightly high-overhead way.
>
> We can create a new rainerscript fn make_list() =>
> [kv-tuples...] (or mklist).
>
> Eg.
> {"foo" : "bar", "baz" : "quux"} becomes [{"key": "foo", "value":
> "bar"}, {"key": "baz", "value": "quux"}]
>
> Clearly it will have inferior performance, but it'll make foreach work
> for all object use-cases.
>
> I really feel in terms of complexity though, this (transform approach)
> is worse than first-class support (purely because of it using a
> round-about way of working).
>
>
> On Wed, Mar 23, 2016 at 3:01 PM, Rainer Gerhards
> <rgerha...@hq.adiscon.com> wrote:
>> 2016-03-22 20:34 GMT+01:00 singh.janmejay <singh.janme...@gmail.com>:
>>> What about wrapDynamicObjects="on|off"? That is required regardless,
>>> right? (if we want to preserve backward compatibility). Unless we
>>> choose to change it anyway (which im fine with).
>>>
>>> @Rainer: what do you think about foreach handling object?
>>
>> so far I could only manage to have a glimpse at the conversation. I
>> admit that I am really concerned about all the extra complexity we
>> introduce. And do so for what I think is a border-use case at best.
>> There is a lot of work that should really be done in regard to
>> variables and performance and I am unsure if this is the right time to
>> do large extensions...
>>
>> I was happy with the dynstats as it looked like a very contained
>> solution that did not affect much else. But now it looks we need to
>> touch a big deal of code ... code that's not really ready for that...
>>
>> Maybe I get a different view when I find time to read all this more 
>> in-depth
>>
>> Rainer
>>>
>>> On Wed, Mar 23, 2016 at 1:01 AM, David Lang <da...@lang.hm> wrote:
>>>> On Wed, 23 Mar 2016, singh.janmejay wrote:
>>>>
>>>>> Foreach can only work with arrays as of now. It can't work with
>>>>> objects (key-value pairs). So [{name: ... value: ...}, {..},...] is
>>>>> the only format that will work as of now.
>>>>>
>>>>> We can enhance foreach to work with objects.
>>>>>
>>>>> I can make a flag available at dyn-stats bucket level, which can
>>>>> control serialization format, but that would really be a hack.
>>>>>
>>>>> From single-responsibility pattern pov, impstats should be the only
>>>>> component that decides how to layout data for user to see.
>>>>>
>>>>> How about this:
>>>>>
>>>>> impstats(format="json" wrapDynamicObjects="on"...)?
>>>>>
>>>>> It defaults to off, which keeps backward compatibility.
>>>>>
>>>>> So what do you guys think about:
>>>>> - wrapDynamicObjects="on|off"
>>>>> - generating [{name: a, value: 10}, {...} ...] vs. {a: 10, ...}
>>>>> (foreach will handle the former out of the box, but later is concise,
>>>>> readable and light-weight in addition to being more json-y.
>>>>> - enhancing foreach to work with {a: 10, b: 20...}
>>>>
>>>>
>>>> If we can enhance foreach to work with the concise format, I would rather
>>>> wait for it instead of introducing the wrapping version.
>>>>
>>>> I'm thinking that foreach walks through arrays, rather than mixing 
>>>> concepts,
>>>> a foreachobject that gives us a name and contents for a {} list of objects
>>>> may be the right thing to do?
>>>>
>>>> foreach ju

Re: [rsyslog] dynastats JSON output need work

[singh.janmejay]

In-code footprint of foreach enhancement (to handle object in addition
to array) will be fairly small (I am thinking under 40 lines?).
Majority of addition will be in form of tests (unless we choose to
change the rainerscript signature of that statement).

Alternatively, from data-traversal functional-completeness pov, the
same thing can be achieved without touching foreach impl, in a
slightly high-overhead way.

We can create a new rainerscript fn make_list() =>
[kv-tuples...] (or mklist).

Eg.
{"foo" : "bar", "baz" : "quux"} becomes [{"key": "foo", "value":
"bar"}, {"key": "baz", "value": "quux"}]

Clearly it will have inferior performance, but it'll make foreach work
for all object use-cases.

I really feel in terms of complexity though, this (transform approach)
is worse than first-class support (purely because of it using a
round-about way of working).


On Wed, Mar 23, 2016 at 3:01 PM, Rainer Gerhards
<rgerha...@hq.adiscon.com> wrote:
> 2016-03-22 20:34 GMT+01:00 singh.janmejay <singh.janme...@gmail.com>:
>> What about wrapDynamicObjects="on|off"? That is required regardless,
>> right? (if we want to preserve backward compatibility). Unless we
>> choose to change it anyway (which im fine with).
>>
>> @Rainer: what do you think about foreach handling object?
>
> so far I could only manage to have a glimpse at the conversation. I
> admit that I am really concerned about all the extra complexity we
> introduce. And do so for what I think is a border-use case at best.
> There is a lot of work that should really be done in regard to
> variables and performance and I am unsure if this is the right time to
> do large extensions...
>
> I was happy with the dynstats as it looked like a very contained
> solution that did not affect much else. But now it looks we need to
> touch a big deal of code ... code that's not really ready for that...
>
> Maybe I get a different view when I find time to read all this more 
> in-depth
>
> Rainer
>>
>> On Wed, Mar 23, 2016 at 1:01 AM, David Lang <da...@lang.hm> wrote:
>>> On Wed, 23 Mar 2016, singh.janmejay wrote:
>>>
>>>> Foreach can only work with arrays as of now. It can't work with
>>>> objects (key-value pairs). So [{name: ... value: ...}, {..},...] is
>>>> the only format that will work as of now.
>>>>
>>>> We can enhance foreach to work with objects.
>>>>
>>>> I can make a flag available at dyn-stats bucket level, which can
>>>> control serialization format, but that would really be a hack.
>>>>
>>>> From single-responsibility pattern pov, impstats should be the only
>>>> component that decides how to layout data for user to see.
>>>>
>>>> How about this:
>>>>
>>>> impstats(format="json" wrapDynamicObjects="on"...)?
>>>>
>>>> It defaults to off, which keeps backward compatibility.
>>>>
>>>> So what do you guys think about:
>>>> - wrapDynamicObjects="on|off"
>>>> - generating [{name: a, value: 10}, {...} ...] vs. {a: 10, ...}
>>>> (foreach will handle the former out of the box, but later is concise,
>>>> readable and light-weight in addition to being more json-y.
>>>> - enhancing foreach to work with {a: 10, b: 20...}
>>>
>>>
>>> If we can enhance foreach to work with the concise format, I would rather
>>> wait for it instead of introducing the wrapping version.
>>>
>>> I'm thinking that foreach walks through arrays, rather than mixing concepts,
>>> a foreachobject that gives us a name and contents for a {} list of objects
>>> may be the right thing to do?
>>>
>>> foreach just returns a single object while foreachobject needs to return the
>>> object and name.
>>>
>>> although, if we ever get the ability to address arrays directly, being able
>>> to look at the array position would be the equivalent of the name.
>>>
>>> David Lang
>>>
>>>
>>>
>>>> On Wed, Mar 23, 2016 at 12:26 AM, David Lang <da...@lang.hm> wrote:
>>>>>
>>>>> On Tue, 22 Mar 2016, singh.janmejay wrote:
>>>>>
>>>>>> How about this: we support a new flag in impstats which allows
>>>>>> json-formatted stats-line to optionally use
>>>>>> encapsulated/wrapped-layout?
>>>>>>
>>>>>> impstats(format="json&quo

Re: [rsyslog] dynastats JSON output need work

[singh.janmejay]

Agree, if most of us like it, I can implement foreach for objects. As of
now I use omprog to call a script which uses jq and awk to make it opentsdb
and redis friendly. I could have used omredis if foreach worked for objects.

Let us conclude.

@Rainer any thoughts on foreach for object?
On Mar 23, 2016 1:56 AM, "David Lang" <da...@lang.hm> wrote:

> On Wed, 23 Mar 2016, singh.janmejay wrote:
>
> Makes sense.
>>
>> ..."counters":["a":80,"b":67] won't work, I think you meant
>> ..."counters": [{"a" : 80}, {"b": 67}].
>>
>
> ahh, showing my lack of knowledge of JSON :-) talking things through:
>
> so our choices are:
>
> "counters": [{"a":80},{"b":67}]
> vs
> "counters": {"a":80,"b":67}
>
> running these through jq gives
>
> {
>   "counters": [
> {
>   "a": 80
> },
> {
>   "b": 67
> }
>   ]
> }
>
> vs
>
> {
>   "counters": {
> "a": 80,
> "b": 67
>   }
> }
>
> or to reference a
> .counters[0].a
> vs
> .counters.a
>
> The latter seems to clearly be the cleaner and more logical one to me.
>
> Does this seem sane?
>
> But to use it, we need a variation of foreach that gives the object name
> as well as the object contents.
>
> David Lang
>
> So it boils down to foreach handling object or not.
>>
>> Thoughts?
>>
>> On Wed, Mar 23, 2016 at 1:18 AM, David Lang <da...@lang.hm> wrote:
>>
>>> On Wed, 23 Mar 2016, singh.janmejay wrote:
>>>
>>> What about wrapDynamicObjects="on|off"? That is required regardless,
>>>> right? (if we want to preserve backward compatibility). Unless we
>>>> choose to change it anyway (which im fine with).
>>>>
>>>
>>>
>>> I think the fact that data from the log could overwrite name or origin
>>> is a
>>> fatal flaw and the existing format should not be allowed. So I think we
>>> should break backwards compatibility here (if it was more established,
>>> I'd
>>> be much more reluctant to do so, but at this point, I think it's only
>>> early
>>> adopters who are using it)
>>>
>>> So if we are always going to push things down one level, the only
>>> question
>>> is the format
>>>
>>> {"name":"h","origin":"dynstats.bucket"},"counters":{"a":80,"b":67}}
>>>
>>> vs
>>>
>>> {"name":"h","origin":"dynstats.bucket"},"counters":["a":80,"b":67]}
>>>
>>> or the horrible
>>>
>>>
>>> {"name":"h","origin":"dynstats.bucket"},"counters":[{"name":"a","value":80},{"name":"b","value":67}]}
>>>
>>> I really want to avoid this third one. I'd say it's probably better to
>>> have
>>> to use an external script to process things than make the third one a
>>> standard :-)
>>>
>>> David Lang
>>>
>>>
>>> @Rainer: what do you think about foreach handling object?
>>>>
>>>> On Wed, Mar 23, 2016 at 1:01 AM, David Lang <da...@lang.hm> wrote:
>>>>
>>>>>
>>>>> On Wed, 23 Mar 2016, singh.janmejay wrote:
>>>>>
>>>>> Foreach can only work with arrays as of now. It can't work with
>>>>>> objects (key-value pairs). So [{name: ... value: ...}, {..},...] is
>>>>>> the only format that will work as of now.
>>>>>>
>>>>>> We can enhance foreach to work with objects.
>>>>>>
>>>>>> I can make a flag available at dyn-stats bucket level, which can
>>>>>> control serialization format, but that would really be a hack.
>>>>>>
>>>>>> From single-responsibility pattern pov, impstats should be the only
>>>>>> component that decides how to layout data for user to see.
>>>>>>
>>>>>> How about this:
>>>>>>
>>>>>> impstats(format="json" wrapDynamicObjects="on"...)?
>>>>>>
>>>>>> It defaults to off, which keeps backward compatibility.
>>>>>>
>>>>>> So wh

Re: [rsyslog] dynastats JSON output need work

[singh.janmejay]

Makes sense.

..."counters":["a":80,"b":67] won't work, I think you meant
..."counters": [{"a" : 80}, {"b": 67}].

So it boils down to foreach handling object or not.

Thoughts?

On Wed, Mar 23, 2016 at 1:18 AM, David Lang <da...@lang.hm> wrote:
> On Wed, 23 Mar 2016, singh.janmejay wrote:
>
>> What about wrapDynamicObjects="on|off"? That is required regardless,
>> right? (if we want to preserve backward compatibility). Unless we
>> choose to change it anyway (which im fine with).
>
>
> I think the fact that data from the log could overwrite name or origin is a
> fatal flaw and the existing format should not be allowed. So I think we
> should break backwards compatibility here (if it was more established, I'd
> be much more reluctant to do so, but at this point, I think it's only early
> adopters who are using it)
>
> So if we are always going to push things down one level, the only question
> is the format
>
> {"name":"h","origin":"dynstats.bucket"},"counters":{"a":80,"b":67}}
>
> vs
>
> {"name":"h","origin":"dynstats.bucket"},"counters":["a":80,"b":67]}
>
> or the horrible
>
> {"name":"h","origin":"dynstats.bucket"},"counters":[{"name":"a","value":80},{"name":"b","value":67}]}
>
> I really want to avoid this third one. I'd say it's probably better to have
> to use an external script to process things than make the third one a
> standard :-)
>
> David Lang
>
>
>> @Rainer: what do you think about foreach handling object?
>>
>> On Wed, Mar 23, 2016 at 1:01 AM, David Lang <da...@lang.hm> wrote:
>>>
>>> On Wed, 23 Mar 2016, singh.janmejay wrote:
>>>
>>>> Foreach can only work with arrays as of now. It can't work with
>>>> objects (key-value pairs). So [{name: ... value: ...}, {..},...] is
>>>> the only format that will work as of now.
>>>>
>>>> We can enhance foreach to work with objects.
>>>>
>>>> I can make a flag available at dyn-stats bucket level, which can
>>>> control serialization format, but that would really be a hack.
>>>>
>>>> From single-responsibility pattern pov, impstats should be the only
>>>> component that decides how to layout data for user to see.
>>>>
>>>> How about this:
>>>>
>>>> impstats(format="json" wrapDynamicObjects="on"...)?
>>>>
>>>> It defaults to off, which keeps backward compatibility.
>>>>
>>>> So what do you guys think about:
>>>> - wrapDynamicObjects="on|off"
>>>> - generating [{name: a, value: 10}, {...} ...] vs. {a: 10, ...}
>>>> (foreach will handle the former out of the box, but later is concise,
>>>> readable and light-weight in addition to being more json-y.
>>>> - enhancing foreach to work with {a: 10, b: 20...}
>>>
>>>
>>>
>>> If we can enhance foreach to work with the concise format, I would rather
>>> wait for it instead of introducing the wrapping version.
>>>
>>> I'm thinking that foreach walks through arrays, rather than mixing
>>> concepts,
>>> a foreachobject that gives us a name and contents for a {} list of
>>> objects
>>> may be the right thing to do?
>>>
>>> foreach just returns a single object while foreachobject needs to return
>>> the
>>> object and name.
>>>
>>> although, if we ever get the ability to address arrays directly, being
>>> able
>>> to look at the array position would be the equivalent of the name.
>>>
>>> David Lang
>>>
>>>
>>>
>>>> On Wed, Mar 23, 2016 at 12:26 AM, David Lang <da...@lang.hm> wrote:
>>>>>
>>>>>
>>>>> On Tue, 22 Mar 2016, singh.janmejay wrote:
>>>>>
>>>>>> How about this: we support a new flag in impstats which allows
>>>>>> json-formatted stats-line to optionally use
>>>>>> encapsulated/wrapped-layout?
>>>>>>
>>>>>> impstats(format="json" ...) generates
>>>>>>
>>>>>>
>>>>>>
>>>>>> {"name":"msg_per_host","origin":"dynstats.bucket&quo

Re: [rsyslog] dynastats problems with user defined variables?

[singh.janmejay]

Yep, it looked like that. Its interesting though, its 100% off CPU
(very unique).

On Wed, Mar 23, 2016 at 1:02 AM, David Lang <da...@lang.hm> wrote:
> Yes, the first one is with the user variables commented out, exactly what I
> pasted in from the grep.
>
> the second is enabling the dyn_inc for edge relays.
>
> David Lang
>
> On Wed, 23 Mar 2016, singh.janmejay wrote:
>
>> Date: Wed, 23 Mar 2016 00:51:18 +0530
>> From: singh.janmejay <singh.janme...@gmail.com>
>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] dynastats problems with user defined variables?
>>
>> You meant the first one was with "commented out" version right? (you
>> said uncommenting edge_relay, so double-checking).
>>
>> On Wed, Mar 23, 2016 at 12:38 AM, David Lang <da...@lang.hm> wrote:
>>>
>>> here is vmstat 1 running
>>>
>>> # vmstat 1
>>> procs ---memory-- ---swap-- -io -system--
>>> --cpu-
>>>  r  b   swpd   free   buff  cache   si   sobibo   in   cs us sy
>>> id
>>> wa st
>>>  2  0  62280 2363804584 272803760121   59601 46
>>> 3
>>> 51  0 0
>>>  1  0  62280 2349476584 2728061200 0 87042  979  891  4
>>> 0
>>> 95  0 0
>>>  1  0  62280 2330488584 2728090800 0 0  388  258  4
>>> 0
>>> 96  0 0
>>>  1  0  62280 2314744584 2728114800 0 0  396  255  4
>>> 0
>>> 96  0 0
>>>  1  0  62280 2297736584 2728138800 0 0  376  245  4
>>> 0
>>> 96  0 0
>>>  2  0  62280 2546260584 2724527200 0 4  639  568  6
>>> 1
>>> 94  0 0
>>>  2  0  62280 2464300584 2724592800 0  1716  936 1146  8
>>> 0
>>> 91  0 0
>>>  2  0  62280 2394452584 2724644400 0 8  687  333  8
>>> 0
>>> 92  0 0
>>>
>>>
>>> here is vmstat 1 uncommenting edge_relay
>>>
>>> # vmstat 1
>>> procs ---memory-- ---swap-- -io -system--
>>> --cpu-
>>>  r  b   swpd   free   buff  cache   si   sobibo   in   cs us sy
>>> id
>>> wa st
>>>  1  0  62280 3198672584 267706240121   59601 46
>>> 3
>>> 51  0 0
>>>  0  0  62280 3198904584 2677066400 0 0  151  247  0
>>> 0
>>> 100 0  0
>>>  0  0  62280 3200332584 2677066400 0 0  128  233  0
>>> 0
>>> 100 0  0
>>>  0  0  62280 3202244584 2677066400 0 4  172  315  0
>>> 0
>>> 100 0  0
>>>  0  0  62280 3203132584 2677066400 0 0  132  218  0
>>> 0
>>> 100 0  0
>>>  0  0  62280 3203912584 2677066800 0 8  203  347  0
>>> 0
>>> 100 0  0
>>>
>>>
>>> things just are not running with it uncommented.
>>>
>>> I'm going to send you my config directly, it is a very heavy config.
>>>
>>> David Lang
>>>
>>>
>>>
>>> On Wed, 23 Mar 2016, singh.janmejay wrote:
>>>
>>>> On my cluster, each node (those 66 nodes) have 4 threads for ruleset
>>>> (it does everything, lognorm based parsing, lookups using loopup
>>>> table, re_extract, string concat, several if-else based conditions and
>>>> queues up for egress using omkafka and I generally see 3 being
>>>> utilized). This setup runs at ~37k msg/s (2.5M / 66 = 37k).
>>>>
>>>> 300k/Min is still 5k/s which should be doable on a single-thd (unless
>>>> your ruleset is much heavier than mine). Im thinking for a
>>>> equivalently complex ruleset 37k / 4  = 9k should be achievable.
>>>>
>>>> I'll try to reproduce this. Can you share out the exact commit-id you
>>>> are working with?
>>>>
>>>> On Tue, Mar 22, 2016 at 8:38 PM, David Lang <da...@lang.hm> wrote:
>>>>>
>>>>>
>>>>> On Tue, 22 Mar 2016, singh.janmejay wrote:
>>>>>
>>>>>> On Tue, Mar 22, 2016 at 5:01 AM, David Lang <da...@lang.hm> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I added this to my configs
>>>>>>>

Re: [rsyslog] dynastats JSON output need work

[singh.janmejay]

What about wrapDynamicObjects="on|off"? That is required regardless,
right? (if we want to preserve backward compatibility). Unless we
choose to change it anyway (which im fine with).

@Rainer: what do you think about foreach handling object?

On Wed, Mar 23, 2016 at 1:01 AM, David Lang <da...@lang.hm> wrote:
> On Wed, 23 Mar 2016, singh.janmejay wrote:
>
>> Foreach can only work with arrays as of now. It can't work with
>> objects (key-value pairs). So [{name: ... value: ...}, {..},...] is
>> the only format that will work as of now.
>>
>> We can enhance foreach to work with objects.
>>
>> I can make a flag available at dyn-stats bucket level, which can
>> control serialization format, but that would really be a hack.
>>
>> From single-responsibility pattern pov, impstats should be the only
>> component that decides how to layout data for user to see.
>>
>> How about this:
>>
>> impstats(format="json" wrapDynamicObjects="on"...)?
>>
>> It defaults to off, which keeps backward compatibility.
>>
>> So what do you guys think about:
>> - wrapDynamicObjects="on|off"
>> - generating [{name: a, value: 10}, {...} ...] vs. {a: 10, ...}
>> (foreach will handle the former out of the box, but later is concise,
>> readable and light-weight in addition to being more json-y.
>> - enhancing foreach to work with {a: 10, b: 20...}
>
>
> If we can enhance foreach to work with the concise format, I would rather
> wait for it instead of introducing the wrapping version.
>
> I'm thinking that foreach walks through arrays, rather than mixing concepts,
> a foreachobject that gives us a name and contents for a {} list of objects
> may be the right thing to do?
>
> foreach just returns a single object while foreachobject needs to return the
> object and name.
>
> although, if we ever get the ability to address arrays directly, being able
> to look at the array position would be the equivalent of the name.
>
> David Lang
>
>
>
>> On Wed, Mar 23, 2016 at 12:26 AM, David Lang <da...@lang.hm> wrote:
>>>
>>> On Tue, 22 Mar 2016, singh.janmejay wrote:
>>>
>>>> How about this: we support a new flag in impstats which allows
>>>> json-formatted stats-line to optionally use
>>>> encapsulated/wrapped-layout?
>>>>
>>>> impstats(format="json" ...) generates
>>>>
>>>>
>>>> {"name":"msg_per_host","origin":"dynstats.bucket","z-scribe1r-b":80,"SWEB10":67}
>>>>
>>>> however,
>>>>
>>>> impstats(format="json/w" ...) generates {"header":
>>>> {"name":"msg_per_host","origin":"dynstats.bucket"}, "counters" :
>>>> {"z-scribe1r-b":80,"SWEB10":67}}
>>>>
>>>> This is relevant, the serilization format we use right now mixes
>>>> pre-defined keys with counter-names and it can affect regular static
>>>> counters too.
>>>
>>>
>>>
>>> with the existing pstats output, there is no ability for user-defined
>>> data
>>> to become a tag name, so there is no potential for ambiguity. but with
>>> dynastats, this is not a possibility, and the format we use should
>>> prevent
>>> problems.
>>>
>>> Also, just from a conceptual point of view, why should the bucket
>>> contents
>>> be at the same level as the bucket name?
>>>
>>> other than backwards compatibility, what advantage is there of the
>>> current
>>> version in JSON? the documentation uses the plain text equivalant, which
>>> is
>>> perfectly legitimate because there is an order to the line, and after you
>>> get past the name and origin, everything else on the line is name-value
>>> pairs of counters, again, no ambiguity.
>>>
>>> But with JSON, I don't believe that you can depend on tools maintaining
>>> (or
>>> even identifying) the order of the elements, and if you have multiple
>>> elements with the same name, it's implementation dependent as to which
>>> one
>>> will be seen.
>>>
>>> So purely from a correctness and defensive programming point of view, I
>>> think the current JSON serialization should be changed, with the old
>>> format
>>> no longer being an option.
>>>
>>>
>>>
>>>
>>> As to the details of the ne

Re: [rsyslog] dynastats problems with user defined variables?

[singh.janmejay]

It seems mostly idle(off cpu). Have you seen sched:off-cpu backtraces
and backtrace-wise latency?

On Wed, Mar 23, 2016 at 12:51 AM, singh.janmejay
<singh.janme...@gmail.com> wrote:
> You meant the first one was with "commented out" version right? (you
> said uncommenting edge_relay, so double-checking).
>
> On Wed, Mar 23, 2016 at 12:38 AM, David Lang <da...@lang.hm> wrote:
>> here is vmstat 1 running
>>
>> # vmstat 1
>> procs ---memory-- ---swap-- -io -system--
>> --cpu-
>>  r  b   swpd   free   buff  cache   si   sobibo   in   cs us sy id
>> wa st
>>  2  0  62280 2363804584 272803760121   59601 46  3
>> 51  0 0
>>  1  0  62280 2349476584 2728061200 0 87042  979  891  4  0
>> 95  0 0
>>  1  0  62280 2330488584 2728090800 0 0  388  258  4  0
>> 96  0 0
>>  1  0  62280 2314744584 2728114800 0 0  396  255  4  0
>> 96  0 0
>>  1  0  62280 2297736584 2728138800 0 0  376  245  4  0
>> 96  0 0
>>  2  0  62280 2546260584 2724527200 0 4  639  568  6  1
>> 94  0 0
>>  2  0  62280 2464300584 2724592800 0  1716  936 1146  8  0
>> 91  0 0
>>  2  0  62280 2394452584 2724644400 0 8  687  333  8  0
>> 92  0 0
>>
>>
>> here is vmstat 1 uncommenting edge_relay
>>
>> # vmstat 1
>> procs ---memory-- ---swap-- -io -system--
>> --cpu-
>>  r  b   swpd   free   buff  cache   si   sobibo   in   cs us sy id
>> wa st
>>  1  0  62280 3198672584 267706240121   59601 46  3
>> 51  0 0
>>  0  0  62280 3198904584 2677066400 0 0  151  247  0  0
>> 100 0  0
>>  0  0  62280 3200332584 2677066400 0 0  128  233  0  0
>> 100 0  0
>>  0  0  62280 3202244584 2677066400 0 4  172  315  0  0
>> 100 0  0
>>  0  0  62280 3203132584 2677066400 0 0  132  218  0  0
>> 100 0  0
>>  0  0  62280 3203912584 2677066800 0 8  203  347  0  0
>> 100 0  0
>>
>>
>> things just are not running with it uncommented.
>>
>> I'm going to send you my config directly, it is a very heavy config.
>>
>> David Lang
>>
>>
>>
>> On Wed, 23 Mar 2016, singh.janmejay wrote:
>>
>>> On my cluster, each node (those 66 nodes) have 4 threads for ruleset
>>> (it does everything, lognorm based parsing, lookups using loopup
>>> table, re_extract, string concat, several if-else based conditions and
>>> queues up for egress using omkafka and I generally see 3 being
>>> utilized). This setup runs at ~37k msg/s (2.5M / 66 = 37k).
>>>
>>> 300k/Min is still 5k/s which should be doable on a single-thd (unless
>>> your ruleset is much heavier than mine). Im thinking for a
>>> equivalently complex ruleset 37k / 4  = 9k should be achievable.
>>>
>>> I'll try to reproduce this. Can you share out the exact commit-id you
>>> are working with?
>>>
>>> On Tue, Mar 22, 2016 at 8:38 PM, David Lang <da...@lang.hm> wrote:
>>>>
>>>> On Tue, 22 Mar 2016, singh.janmejay wrote:
>>>>
>>>>> On Tue, Mar 22, 2016 at 5:01 AM, David Lang <da...@lang.hm> wrote:
>>>>>>
>>>>>>
>>>>>> I added this to my configs
>>>>>>
>>>>>> # grep -B 1 -A 1 dyn_ /etc/rsyslog.conf
>>>>>> # custom stats to track in rsyslog
>>>>>> dyn_stats(name="msgs_per_host" resettable="on" maxCardinality="3000"
>>>>>> unusedMetricLife="1200")
>>>>>> dyn_stats(name="msgs_per_edge_relay" resettable="on"
>>>>>> maxCardinality="3000"
>>>>>> unusedMetricLife="1200")
>>>>>> dyn_stats(name="msgs_per_core_relay" resettable="on"
>>>>>> maxCardinality="3000"
>>>>>> unusedMetricLife="1200")
>>>>>> dyn_stats(name="msgs_per_program" resettable="on" maxCardinality="3000"
>>>>>> unusedMetricLife="1200")
>>>>>> dyn_stats(name="msgs_per_tag" resettable="on" maxCardinality="3000"
>>>>>> unusedMetricLife="1200")
>>>>>>
>>>>

Re: [rsyslog] dynastats problems with user defined variables?

[singh.janmejay]

You meant the first one was with "commented out" version right? (you
said uncommenting edge_relay, so double-checking).

On Wed, Mar 23, 2016 at 12:38 AM, David Lang <da...@lang.hm> wrote:
> here is vmstat 1 running
>
> # vmstat 1
> procs ---memory-- ---swap-- -io -system--
> --cpu-
>  r  b   swpd   free   buff  cache   si   sobibo   in   cs us sy id
> wa st
>  2  0  62280 2363804584 272803760121   59601 46  3
> 51  0 0
>  1  0  62280 2349476584 2728061200 0 87042  979  891  4  0
> 95  0 0
>  1  0  62280 2330488584 2728090800 0 0  388  258  4  0
> 96  0 0
>  1  0  62280 2314744584 2728114800 0 0  396  255  4  0
> 96  0 0
>  1  0  62280 2297736584 2728138800 0 0  376  245  4  0
> 96  0 0
>  2  0  62280 2546260584 2724527200 0 4  639  568  6  1
> 94  0 0
>  2  0  62280 2464300584 2724592800 0  1716  936 1146  8  0
> 91  0 0
>  2  0  62280 2394452584 2724644400 0 8  687  333  8  0
> 92  0 0
>
>
> here is vmstat 1 uncommenting edge_relay
>
> # vmstat 1
> procs ---memory-- ---swap-- -io -system--
> --cpu-
>  r  b   swpd   free   buff  cache   si   sobibo   in   cs us sy id
> wa st
>  1  0  62280 3198672584 267706240121   59601 46  3
> 51  0 0
>  0  0  62280 3198904584 2677066400 0 0  151  247  0  0
> 100 0  0
>  0  0  62280 3200332584 2677066400 0 0  128  233  0  0
> 100 0  0
>  0  0  62280 3202244584 2677066400 0 4  172  315  0  0
> 100 0  0
>  0  0  62280 3203132584 2677066400 0 0  132  218  0  0
> 100 0  0
>  0  0  62280 3203912584 2677066800 0 8  203  347  0  0
> 100 0  0
>
>
> things just are not running with it uncommented.
>
> I'm going to send you my config directly, it is a very heavy config.
>
> David Lang
>
>
>
> On Wed, 23 Mar 2016, singh.janmejay wrote:
>
>> On my cluster, each node (those 66 nodes) have 4 threads for ruleset
>> (it does everything, lognorm based parsing, lookups using loopup
>> table, re_extract, string concat, several if-else based conditions and
>> queues up for egress using omkafka and I generally see 3 being
>> utilized). This setup runs at ~37k msg/s (2.5M / 66 = 37k).
>>
>> 300k/Min is still 5k/s which should be doable on a single-thd (unless
>> your ruleset is much heavier than mine). Im thinking for a
>> equivalently complex ruleset 37k / 4  = 9k should be achievable.
>>
>> I'll try to reproduce this. Can you share out the exact commit-id you
>> are working with?
>>
>> On Tue, Mar 22, 2016 at 8:38 PM, David Lang <da...@lang.hm> wrote:
>>>
>>> On Tue, 22 Mar 2016, singh.janmejay wrote:
>>>
>>>> On Tue, Mar 22, 2016 at 5:01 AM, David Lang <da...@lang.hm> wrote:
>>>>>
>>>>>
>>>>> I added this to my configs
>>>>>
>>>>> # grep -B 1 -A 1 dyn_ /etc/rsyslog.conf
>>>>> # custom stats to track in rsyslog
>>>>> dyn_stats(name="msgs_per_host" resettable="on" maxCardinality="3000"
>>>>> unusedMetricLife="1200")
>>>>> dyn_stats(name="msgs_per_edge_relay" resettable="on"
>>>>> maxCardinality="3000"
>>>>> unusedMetricLife="1200")
>>>>> dyn_stats(name="msgs_per_core_relay" resettable="on"
>>>>> maxCardinality="3000"
>>>>> unusedMetricLife="1200")
>>>>> dyn_stats(name="msgs_per_program" resettable="on" maxCardinality="3000"
>>>>> unusedMetricLife="1200")
>>>>> dyn_stats(name="msgs_per_tag" resettable="on" maxCardinality="3000"
>>>>> unusedMetricLife="1200")
>>>>>
>>>>> --
>>>>>   /var/log/sources-messages;sources
>>>>>   set $.inc = dyn_inc("msgs_per_host", $hostname);
>>>>>   set $.inc = dyn_inc("msgs_per_program", $programname);
>>>>> #  if $!trusted!edge!relay != "" then {
>>>>> #set $.inc = dyn_inc("msgs_per_edge_relay", $!trusted!edge!relay);
>>>>> #  }
>>>>> #  if $!trusted!core!relay != "" then {
>>>>> #set $.inc = dyn_inc("msgs_per_edge_relay&qu

Re: [rsyslog] dynastats JSON output need work

[singh.janmejay]

Foreach can only work with arrays as of now. It can't work with
objects (key-value pairs). So [{name: ... value: ...}, {..},...] is
the only format that will work as of now.

We can enhance foreach to work with objects.

I can make a flag available at dyn-stats bucket level, which can
control serialization format, but that would really be a hack.

>From single-responsibility pattern pov, impstats should be the only
component that decides how to layout data for user to see.

How about this:

impstats(format="json" wrapDynamicObjects="on"...)?

It defaults to off, which keeps backward compatibility.

So what do you guys think about:
- wrapDynamicObjects="on|off"
- generating [{name: a, value: 10}, {...} ...] vs. {a: 10, ...}
(foreach will handle the former out of the box, but later is concise,
readable and light-weight in addition to being more json-y.
- enhancing foreach to work with {a: 10, b: 20...}


On Wed, Mar 23, 2016 at 12:26 AM, David Lang <da...@lang.hm> wrote:
> On Tue, 22 Mar 2016, singh.janmejay wrote:
>
>> How about this: we support a new flag in impstats which allows
>> json-formatted stats-line to optionally use
>> encapsulated/wrapped-layout?
>>
>> impstats(format="json" ...) generates
>>
>> {"name":"msg_per_host","origin":"dynstats.bucket","z-scribe1r-b":80,"SWEB10":67}
>>
>> however,
>>
>> impstats(format="json/w" ...) generates {"header":
>> {"name":"msg_per_host","origin":"dynstats.bucket"}, "counters" :
>> {"z-scribe1r-b":80,"SWEB10":67}}
>>
>> This is relevant, the serilization format we use right now mixes
>> pre-defined keys with counter-names and it can affect regular static
>> counters too.
>
>
> with the existing pstats output, there is no ability for user-defined data
> to become a tag name, so there is no potential for ambiguity. but with
> dynastats, this is not a possibility, and the format we use should prevent
> problems.
>
> Also, just from a conceptual point of view, why should the bucket contents
> be at the same level as the bucket name?
>
> other than backwards compatibility, what advantage is there of the current
> version in JSON? the documentation uses the plain text equivalant, which is
> perfectly legitimate because there is an order to the line, and after you
> get past the name and origin, everything else on the line is name-value
> pairs of counters, again, no ambiguity.
>
> But with JSON, I don't believe that you can depend on tools maintaining (or
> even identifying) the order of the elements, and if you have multiple
> elements with the same name, it's implementation dependent as to which one
> will be seen.
>
> So purely from a correctness and defensive programming point of view, I
> think the current JSON serialization should be changed, with the old format
> no longer being an option.
>
>
>
>
> As to the details of the new format
>
> What I'm wanting to do with the counters is something like
>
> if $!origin == "dynstats.bucket" then {
>   foreach $.tag $!counters {
> /var/log/stats;format
>   }
> }
>
> to output one line per counter.
>
> I'm very flexible in how to do this, but I would much rather be able to do
> this inside rsyslog than have to serialize things to an external script,
> have it parse the json and process it.
>
> my initial thinking was just do
>
> counters: [ "z-scribe1r-b":80,"SWEB10":67 ]
>
> but as I'm typing this, I realize that doesn't work as I don't have a way to
> break $.tag down to reference the name and the value.
>
> I'd hate to have to do something like
>
> counters: [{"name":"z-scribe1r-b","value":80 },{"name":"SWEB10","value":67}]
>
> this mirrors the misuse of XML that gives it such a horrible reputation. But
> unless we introduce some new function to rsyslog to break things down, I
> don't see a better way. If we do need to do something like this, I sure
> would not want to make it the default JSON, which would result in two
> different formats. I hate the idea of starting to have different formats
> because of subtypes of data (what is someone wants the cee version of this
> for example, you start to have orthoginal format decisions)
>
>
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] dynastats problems with user defined variables?

[singh.janmejay]

Awesome.

Here is what I plan to do:
- Prepare load-profile to pump 300k/min 1.5kB messages
- Run with the commit-id you are on
- Run with commit-id my internal cluster is on
- Compare TP, jitter and CPU profile

I may not get a chance to work on it this week though, can only pick
it up next week.

On Wed, Mar 23, 2016 at 12:32 AM, David Lang <da...@lang.hm> wrote:
> On Wed, 23 Mar 2016, singh.janmejay wrote:
>
>> On my cluster, each node (those 66 nodes) have 4 threads for ruleset
>> (it does everything, lognorm based parsing, lookups using loopup
>> table, re_extract, string concat, several if-else based conditions and
>> queues up for egress using omkafka and I generally see 3 being
>> utilized). This setup runs at ~37k msg/s (2.5M / 66 = 37k).
>>
>> 300k/Min is still 5k/s which should be doable on a single-thd (unless
>> your ruleset is much heavier than mine). Im thinking for a
>> equivalently complex ruleset 37k / 4  = 9k should be achievable.
>>
>> I'll try to reproduce this. Can you share out the exact commit-id you
>> are working with?
>
>
> # cat build.config.201603171640
> ./rsyslog/ branch: * master
> ./rsyslog/ version: v8.17.0-50-gbbc6812
> ./liblognorm/ branch: * master
> ./liblognorm/ version: v1.1.2-270-gf48f3ae
> ./rsyslog-doc/ branch: * master
> ./rsyslog-doc/ version: v8.3.5-345-gdda3f6f
> ./liblogging/ branch: * master
> ./liblogging/ version: v1.0.5-8-gc957d39
> ./librelp/ branch: * master
> ./librelp/ version: v1.2.9-1-g414ee34
> ./libestr/ branch: * master
> ./libestr/ version: v0.1.10-3-gb60d2ce
> ./json-c/ branch: * libfastjson
> ./json-c/ version: 80c1f69
> ./rsyslog-pkg-ubuntu-upstream/ branch: * master
> ./rsyslog-pkg-ubuntu-upstream/ version: f460f89
> ./rsyslog-pkg-ubuntu.old/ branch: * master
> ./rsyslog-pkg-ubuntu.old/ version: f460f89
>
> I'll be updating to the following almost immediatly
>
> # cat build.config.201603211911
> ./rsyslog/ branch: * master
> ./rsyslog/ version: v8.17.0-53-g2578a73
> ./liblognorm/ branch: * master
> ./liblognorm/ version: v1.1.2-270-gf48f3ae
> ./rsyslog-doc/ branch: * master
> ./rsyslog-doc/ version: v8.3.5-345-gdda3f6f
> ./liblogging/ branch: * master
> ./liblogging/ version: v1.0.5-8-gc957d39
> ./librelp/ branch: * master
> ./librelp/ version: v1.2.9-1-g414ee34
> ./libestr/ branch: * master
> ./libestr/ version: v0.1.10-3-gb60d2ce
> ./json-c/ branch: * libfastjson
> ./json-c/ version: 80c1f69
>
> this is the output of
>
> /usr/bin/git describe origin --always
>
> which says what the most recent tagged release, how many commit since that
> tag, and the hash of the commit.
>
> David Lang
>
>
>> On Tue, Mar 22, 2016 at 8:38 PM, David Lang <da...@lang.hm> wrote:
>>>
>>> On Tue, 22 Mar 2016, singh.janmejay wrote:
>>>
>>>> On Tue, Mar 22, 2016 at 5:01 AM, David Lang <da...@lang.hm> wrote:
>>>>>
>>>>>
>>>>> I added this to my configs
>>>>>
>>>>> # grep -B 1 -A 1 dyn_ /etc/rsyslog.conf
>>>>> # custom stats to track in rsyslog
>>>>> dyn_stats(name="msgs_per_host" resettable="on" maxCardinality="3000"
>>>>> unusedMetricLife="1200")
>>>>> dyn_stats(name="msgs_per_edge_relay" resettable="on"
>>>>> maxCardinality="3000"
>>>>> unusedMetricLife="1200")
>>>>> dyn_stats(name="msgs_per_core_relay" resettable="on"
>>>>> maxCardinality="3000"
>>>>> unusedMetricLife="1200")
>>>>> dyn_stats(name="msgs_per_program" resettable="on" maxCardinality="3000"
>>>>> unusedMetricLife="1200")
>>>>> dyn_stats(name="msgs_per_tag" resettable="on" maxCardinality="3000"
>>>>> unusedMetricLife="1200")
>>>>>
>>>>> --
>>>>>   /var/log/sources-messages;sources
>>>>>   set $.inc = dyn_inc("msgs_per_host", $hostname);
>>>>>   set $.inc = dyn_inc("msgs_per_program", $programname);
>>>>> #  if $!trusted!edge!relay != "" then {
>>>>> #set $.inc = dyn_inc("msgs_per_edge_relay", $!trusted!edge!relay);
>>>>> #  }
>>>>> #  if $!trusted!core!relay != "" then {
>>>>> #set $.inc = dyn_inc("msgs_per_edge_relay", $!trusted!core!relay);
>>>>> #  }
>>>>> --
>>>>> }
>>>>>  

Re: [rsyslog] dynastats problems with user defined variables?

[singh.janmejay]

On my cluster, each node (those 66 nodes) have 4 threads for ruleset
(it does everything, lognorm based parsing, lookups using loopup
table, re_extract, string concat, several if-else based conditions and
queues up for egress using omkafka and I generally see 3 being
utilized). This setup runs at ~37k msg/s (2.5M / 66 = 37k).

300k/Min is still 5k/s which should be doable on a single-thd (unless
your ruleset is much heavier than mine). Im thinking for a
equivalently complex ruleset 37k / 4  = 9k should be achievable.

I'll try to reproduce this. Can you share out the exact commit-id you
are working with?

On Tue, Mar 22, 2016 at 8:38 PM, David Lang <da...@lang.hm> wrote:
> On Tue, 22 Mar 2016, singh.janmejay wrote:
>
>> On Tue, Mar 22, 2016 at 5:01 AM, David Lang <da...@lang.hm> wrote:
>>>
>>> I added this to my configs
>>>
>>> # grep -B 1 -A 1 dyn_ /etc/rsyslog.conf
>>> # custom stats to track in rsyslog
>>> dyn_stats(name="msgs_per_host" resettable="on" maxCardinality="3000"
>>> unusedMetricLife="1200")
>>> dyn_stats(name="msgs_per_edge_relay" resettable="on"
>>> maxCardinality="3000"
>>> unusedMetricLife="1200")
>>> dyn_stats(name="msgs_per_core_relay" resettable="on"
>>> maxCardinality="3000"
>>> unusedMetricLife="1200")
>>> dyn_stats(name="msgs_per_program" resettable="on" maxCardinality="3000"
>>> unusedMetricLife="1200")
>>> dyn_stats(name="msgs_per_tag" resettable="on" maxCardinality="3000"
>>> unusedMetricLife="1200")
>>>
>>> --
>>>   /var/log/sources-messages;sources
>>>   set $.inc = dyn_inc("msgs_per_host", $hostname);
>>>   set $.inc = dyn_inc("msgs_per_program", $programname);
>>> #  if $!trusted!edge!relay != "" then {
>>> #set $.inc = dyn_inc("msgs_per_edge_relay", $!trusted!edge!relay);
>>> #  }
>>> #  if $!trusted!core!relay != "" then {
>>> #set $.inc = dyn_inc("msgs_per_edge_relay", $!trusted!core!relay);
>>> #  }
>>> --
>>> }
>>> #set $.inc = dyn_inc("msgs_per_tag", $.custommessage);
>>> /var/log/messages-tags;manual
>>>
>>> if I uncomment any of the lines that refer to $! or $. variables, rsyslog
>>> basically stops processing messages (a few messages seem to be processed
>>> in
>>> spurts, bit a lot of processing never happens)
>>>
>>> any chance that this has problems with locking around user defined
>>> variables?
>>
>>
>> Not sure, I run it in production (66 6-cpu(HT) haswell VMs handling an
>> aggregate of ~2.5 M-msg / sec (~1.5kB each)) and haven't seen such
>> jitters, but then I don't use 3 level deep keys either and in my case,
>> the feature is backported to 8.12 branch), so still uses json-c (not
>> libfastjson). I do use it with user-defined variables too, just not
>> nested as deep.
>>
>> We don't treat variables differently after dereferencing them in the
>> top-most layer handling rainerscript fn-calls, so $hostname and
>> $!x!y!z, as far as any rainerscript fn invocation is considered, are
>> the same (performance-wise) except for the top-level dereference
>> (outside of fn-impl-body).
>>
>> Some more info may come handy in reproducing it:
>> - What is the message throughput (in msg/s) and what is the avg msg size?
>
>
> with these commented out as above, ~50k messages/min normal, ~200K
> message/min if I'm replaying queued logs (box saturated, see the other
> thread where 8.17 final seems to have slowed down from ~300K messages/min
> that I was getting with a 8.16+ git)
>
>> - Do you see messages being written to sources-messages file, while
>> they don't get written to messages-tags file?
>
>
> Yes, but not all messages. I have an alarm that goes off if sources-messages
> isn't written to in two 35 second checks (70 seconds total), that alarm
> never fired. But there would be a burst of messages show up in
> sources-messages and then 20 or so seconds before another burst would show
> up.
>
>> - Do you see a significant jump in voluntary context switches(say per
>> 10 seconds or even per second) when you uncomment those lines? How big
>> is the jump?
>> - How does the observation change if you replace it with this kind of
>> variable access pattern:
>>
>> set $.x = $!trusted!edge!relay;
>> if $.x !=

Re: [rsyslog] dynastats JSON output need work

[singh.janmejay]

How about this: we support a new flag in impstats which allows
json-formatted stats-line to optionally use
encapsulated/wrapped-layout?

impstats(format="json" ...) generates
{"name":"msg_per_host","origin":"dynstats.bucket","z-scribe1r-b":80,"SWEB10":67}

however,

impstats(format="json/w" ...) generates {"header":
{"name":"msg_per_host","origin":"dynstats.bucket"}, "counters" :
{"z-scribe1r-b":80,"SWEB10":67}}

This is relevant, the serilization format we use right now mixes
pre-defined keys with counter-names and it can affect regular static
counters too.

Thoughts?

On Tue, Mar 22, 2016 at 8:46 PM, David Lang  wrote:
> On Tue, 22 Mar 2016, Rainer Gerhards wrote:
>
>> 2016-03-22 15:57 GMT+01:00 David Lang :
>>>
>>> On Tue, 22 Mar 2016, Rainer Gerhards wrote:
>>>
 2016-03-21 22:27 GMT+01:00 David Lang :
>
>
> The json produced by dynastats needs some work, currently it looks
> like:
>
>
>
> {"name":"msg_per_host","origin":"dynstats.bucket","z-scribe1r-b":80,"SWEB10":67}
>
> This is not something that can readily be handled, and since the
> hostname
> is
> at the same level as name or origin, it's just begging for abuse.
>
> It would be far better to put these into an array that could then be
> handled
> via foreach
>
> something like:
>
>
> {"name":"msg_per_host","origin":"dynstats.bucket",items:["z-scribe1r-b":80,"SWEB10":67]}
>
> Yes, this has hit a stable release, but the docs still haven't made it
> to
> the website (which is still showing 8.16) and so I don't think anyone
> is
> depending on this syntax yet, so I think we can get away with changing
> it.



 Two things: first of all, the doc tarball went out and probably is
 included in some distros. Also, this *is* the released version. So I
 think it would be a bad thing to change this without mentioning it. If
>>>
>>>
>>>
>>> I agree it should be mentioned, probably in the release announcement
>>>
 there is agreement to change, I agree it's not a big deal at this
 stage. While seldom, we have made some breaking changes of such a
 small magnitude in the past as well.
>>>
>>>
>>>
>>> So do people agree or disagree with my suggested change?
>>
>>
>> I am neutral, as I don't see any use case for me and my customers for
>> this functionality. I also don't think it breaks any rsyslog concept
>> (that's why I merged in the frist place).
>
>
> I've been delivering logs to sec and having it write per-minute summary data
> out, I'm running into the case where the sec process is maxing out (hitting
> 100% cpu and single-threaded), so I'm experimenting with doing this in
> rsyslog and having rsyslog spit out the data to a monitoring system.
>
> I'm also going to be looking at the functionality that I think I saw go in
> recently to trigger if a source hasn't reported in too long a timeframe.
> That's also something I'm starting to have as a bottleneck. I expect that
> when I try to do that I'll be trying to do more than rsyslog is setup to do,
> but we'll see.
>
> David Lang
>
>
>> Conceptionally, I am leaning towards your argument, but I don't want
>> to put any effort into a patch or strong discussion TBH.
>>
>> Rainer
>>>
>>>
 Secondly, the website update failed due to a merge conflict inside a
 script that wasn't properly detected. This is fixed now. So the
 correct version will hopefully show up soon on the site (but IMHO
 without affecting anything else, as said in the first paragraph).
>>>
>>>
>>>
>>> Glad to hear this.
>>>
>>>
>>> David Lang
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of
>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>>> LIKE THAT.
>>
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
>>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com

Re: [rsyslog] dynastats problems with user defined variables?

[singh.janmejay]

Inline.

On Tue, Mar 22, 2016 at 5:01 AM, David Lang  wrote:
> I added this to my configs
>
> # grep -B 1 -A 1 dyn_ /etc/rsyslog.conf
> # custom stats to track in rsyslog
> dyn_stats(name="msgs_per_host" resettable="on" maxCardinality="3000"
> unusedMetricLife="1200")
> dyn_stats(name="msgs_per_edge_relay" resettable="on" maxCardinality="3000"
> unusedMetricLife="1200")
> dyn_stats(name="msgs_per_core_relay" resettable="on" maxCardinality="3000"
> unusedMetricLife="1200")
> dyn_stats(name="msgs_per_program" resettable="on" maxCardinality="3000"
> unusedMetricLife="1200")
> dyn_stats(name="msgs_per_tag" resettable="on" maxCardinality="3000"
> unusedMetricLife="1200")
>
> --
>   /var/log/sources-messages;sources
>   set $.inc = dyn_inc("msgs_per_host", $hostname);
>   set $.inc = dyn_inc("msgs_per_program", $programname);
> #  if $!trusted!edge!relay != "" then {
> #set $.inc = dyn_inc("msgs_per_edge_relay", $!trusted!edge!relay);
> #  }
> #  if $!trusted!core!relay != "" then {
> #set $.inc = dyn_inc("msgs_per_edge_relay", $!trusted!core!relay);
> #  }
> --
> }
> #set $.inc = dyn_inc("msgs_per_tag", $.custommessage);
> /var/log/messages-tags;manual
>
> if I uncomment any of the lines that refer to $! or $. variables, rsyslog
> basically stops processing messages (a few messages seem to be processed in
> spurts, bit a lot of processing never happens)
>
> any chance that this has problems with locking around user defined
> variables?

Not sure, I run it in production (66 6-cpu(HT) haswell VMs handling an
aggregate of ~2.5 M-msg / sec (~1.5kB each)) and haven't seen such
jitters, but then I don't use 3 level deep keys either and in my case,
the feature is backported to 8.12 branch), so still uses json-c (not
libfastjson). I do use it with user-defined variables too, just not
nested as deep.

We don't treat variables differently after dereferencing them in the
top-most layer handling rainerscript fn-calls, so $hostname and
$!x!y!z, as far as any rainerscript fn invocation is considered, are
the same (performance-wise) except for the top-level dereference
(outside of fn-impl-body).

Some more info may come handy in reproducing it:
- What is the message throughput (in msg/s) and what is the avg msg size?
- Do you see messages being written to sources-messages file, while
they don't get written to messages-tags file?
- Do you see a significant jump in voluntary context switches(say per
10 seconds or even per second) when you uncomment those lines? How big
is the jump?
- How does the observation change if you replace it with this kind of
variable access pattern:

set $.x = $!trusted!edge!relay;
if $.x != "" then {
  set $.inc = dyn_inc("msgs_per_edge_relay", $.x);
}

- Is this stock 8.17 build?
- Is this x86-64 Linux? What distro (and version)?

>
> Also, the documentation doesn't say why we are setting $.inc to the result
> of the command, what does $.inc contain after the command is run? from the
> documentation, it looks like it contains an error code (at least the way
> it's used is similar to error code returns in other languages), but I don't
> see what the errors are in the doc page.

Yes, it is the error-code. It has value 0 when successful, an non-zero
when it fails to increment (it may fail due to maxCardinality being
hit or due to a contended for lock etc (it doesn't wait for the lock,
it uses try-lock and doesn't increment the counter if it can't get a
shared-lock over the bucket).

These are rsyslog error-codes. It does an error pass-thru.

I'll add this to documentation (didn't know this was missed).

>
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog 8.17.0 (v8-stable) released

[singh.janmejay]

This is a bug, here is the fix: https://github.com/rsyslog/rsyslog/pull/863

Workaround(in the meanwhile): Since you are not using dynstats, its
safe to ignore that line (everything it reports is related to dynstats
buckets).

On Wed, Mar 9, 2016 at 9:34 PM, Andrew Davidoff  wrote:
> On Tue, Mar 8, 2016 at 9:54 AM, Florian Riedl  wrote:
>> Hi all,
>>
>> We have released rsyslog 8.17.0.
>
> I just started testing this release and with the same config I was
> using for 8.13 (which may be the issue) I'm seeing the following
> incomplete JSON being dropped in my rsyslog stats log for "global".
> I'm including the lines before and after for context.
>
> 2016-03-09T15:49:29.427521+00:00 01.syslog.dev.lax1 rsyslogd-pstats:
> {"name":"imudp(w0)","origin":"imudp","called.recvmmsg":4,"called.recvmsg":0,"msgs.received":2}
> 2016-03-09T15:50:30.150578+00:00 01.syslog.dev.lax1 rsyslogd-pstats:
> {"name":"global","origin":"dynstats",
> 2016-03-09T15:50:30.150732+00:00 01.syslog.dev.lax1 rsyslogd-pstats:
> {"name":"action
> 0","origin":"core.action","processed":0,"failed":0,"suspended":0,"suspended.duration":0,"resumed":0}
>
> I'm not sure if this indicates a configuration issue or a bug, but
> wanted to pass it along.
>
> My pstats config looks like this (looking back over the docs, I am not
> sure anymore why I broke out the file handler into a ruleset instead
> of using log.file):
>
> module(
> load="impstats"
> interval="60"
> format="json"
> ruleset="pstats")
>
> ruleset(
> name="pstats"
> queue.type="FixedArray") {
>
> action(
> type="omfile"
> file="/var/log/rsyslog.stats.log")
> }
>
> I also tried this format (no ruleset) with the same results:
>
> module(
> load="impstats"
> interval="60"
> format="json"
> log.file="/var/log/rsyslog.stats.log")
>
>
>
> Thanks.
> Andy
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Identifying/quantifying log-loss or proving no-loss in a log-store

[singh.janmejay]

@Thomas: This is not about testing and quantifying loss during a test.
Its about quantifying it during normal operation. I see it as a choice
between:
A. deploy the strongest protocol at every system-boundary and test
each one rigorously and each change rigorously to identify or bound
loss in test conditions, and expect nothing unexpected to show up in
production
B. do the former and measure loss in production to identify that
something unexpected happened
C. deploy efficient protocols at all system-boundaries and measure
loss (as long as loss stays within an acceptable level, deployment
benefits from all the efficiency gains)

I am talking in the context of C.

If/when loss is above acceptable level, one needs to debug and fix the
problem. Both B and C provide the data required to identify
situation(s) when such debugging needs to happen.

The approach of stamping on one end an measuring on the other treats
all intermediate hops as a blackbox. For instance, it can be used to
quantify losses in face of frequent machine failures or down-time free
maintenance etc.

@David: As of now, I am thinking of end-of-the-day style measurement
(basically report number of messages lost at a good-enough
granularity, say host x severity).

I am thinking of this as something independent of frequency of outages
and unrelated to maintenance windows. Im thinking of it as a report
that captures extent of loss, where one can pull down several months
of this data and verify loss was never beyond a acceptable level,
compare it across days when load profile was very different (the day
when too many circuit-breakers engaged etc).

I haven't thought through this, but reset may not be required.
Basically let the counter count-up and wrap-around (as long as
wrap-around is well defined behavior which is accounted for during
measurement).


On Sat, Feb 13, 2016 at 5:13 AM, David Lang <da...@lang.hm> wrote:
> On Sat, 13 Feb 2016, singh.janmejay wrote:
>
>> The ideal solution would be one that identifies host, log-source and
>> time of loss along with accurate number of messages lost.
>>
>> pstats makes sense, but correlating data from stats across large
>> number of machines will be difficult (some machines may send stats
>> slightly delayed which may skew aggregation etc).
>
>
> if you don't reset the counters, they keep increasing, so over time the
> error due to the slew becomes a very minor componnent.
>
>
>> One approach I can think of: slap a stream-identifier and
>> sequence-number on each received message, then find gaps in sequence
>> number for a session-id on the other side (as a query over log-store
>> etc).
>
>
> I'll point out that generating/checking a monotonic sequence number destroys
> parallelism, and so it can seriously hurt performance.
>
> Are you trying to detect problems 'on the fly' as they happen? or at the end
> of the hour/day saying 'hey, there was a problem at some point'
>
> how frequent do you think problems are? I would suggest that you run some
> stress tests on your equipment/network and push things until you do have
> problems, so you can track when they happen. I expect that you will find
> that they don't start happening until you have much higher loads than you
> expect (at least after a bit of tuning), and this can make it so that the
> most invastive solutions aren't needed.
>
> David Lang
>
>
>> Large issues such as producer suddenly going silent can be detected
>> using macro mechanisms (like pstats).
>>
>> On Sat, Feb 13, 2016 at 2:56 AM, David Lang <da...@lang.hm> wrote:
>>>
>>> On Sat, 13 Feb 2016, Andre wrote:
>>>
>>>>
>>>> The easiest way I found to do that is to have a control system and send
>>>> two
>>>> streams of data to two or more different destinations.
>>>>
>>>> In case of rsyslog processing a large message volume UDP the loss has
>>>> always been noticeable.
>>>
>>>
>>>
>>> this depends on your setup. I was able to send UDP logs at gig-E wire
>>> speed
>>> with no losses, but it required tuning the receiving sytem to not do DNS
>>> lookups, have sufficient RAM for buffering, etc
>>>
>>>
>>> I never was able to get my hands on 10G equiepment to push up from there.
>>>
>>> David Lang
>>>
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of
>>&g

[rsyslog] Identifying/quantifying log-loss or proving no-loss in a log-store

[singh.janmejay]

Inviting ideas.

Has anyone tried to quantify log-loss (#number of lines lost per day
per sender etc) for a log-store?

Let us consider the following setup:
- An environment has several application nodes. Each app node
hands-over its logs to local Rsyslog daemon(let us call it Ra,
Rsyslog-application).
- The environment has one or more Rsyslog receiver nodes (let us call
it Rr, Rsyslog-receiver).
- Rr(s) write received logs to a log-store.

The problem statement is: Quantify log-loss(defined as messages that
are successfully handed over to Ra, but can't be found in log-store)
in log-events lost per day per host.

Log-events may be lost because of any reason (in the pipe, or after
being written to log-store). It doesn't matter which of the
intermediate systems lost logs, as long as loss is bounded (by any
empirical figure, say less than 0.1%).

-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Identifying/quantifying log-loss or proving no-loss in a log-store

[singh.janmejay]

The ideal solution would be one that identifies host, log-source and
time of loss along with accurate number of messages lost.

pstats makes sense, but correlating data from stats across large
number of machines will be difficult (some machines may send stats
slightly delayed which may skew aggregation etc).

One approach I can think of: slap a stream-identifier and
sequence-number on each received message, then find gaps in sequence
number for a session-id on the other side (as a query over log-store
etc).

Large issues such as producer suddenly going silent can be detected
using macro mechanisms (like pstats).

On Sat, Feb 13, 2016 at 2:56 AM, David Lang  wrote:
> On Sat, 13 Feb 2016, Andre wrote:
>
>>
>> The easiest way I found to do that is to have a control system and send
>> two
>> streams of data to two or more different destinations.
>>
>> In case of rsyslog processing a large message volume UDP the loss has
>> always been noticeable.
>
>
> this depends on your setup. I was able to send UDP logs at gig-E wire speed
> with no losses, but it required tuning the receiving sytem to not do DNS
> lookups, have sufficient RAM for buffering, etc
>
>
> I never was able to get my hands on 10G equiepment to push up from there.
>
> David Lang
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] What would be the best place to document lookup-table and dyn-stats?

[singh.janmejay]

Done. PR: https://github.com/rsyslog/rsyslog-doc/pull/215

On Wed, Feb 10, 2016 at 2:31 PM, singh.janmejay
<singh.janme...@gmail.com> wrote:
> Makes sense.
>
> I'll link the PR here once done.
>
> --
> Regards,
> Janmejay
>
> PS: Please blame the typos in this mail on my phone's uncivilized soft
> keyboard sporting it's not-so-smart-assist technology.
>
>
> On Feb 10, 2016 10:24 AM, "David Lang" <da...@lang.hm> wrote:
>>
>> the lookup is a function, no question. It was the reload that prompted the
>> action() question. But your comments on how so many things that are standard
>> options for actions (queues, etc) make no sense for this and so a new
>> statement type is much cleaner than either disabling, or interacting with
>> all those standard things.
>>
>> we do have a handful of other non-action/input/module statements. The
>> parser and global declarations are examples. the only thing that is unique
>> in table lookups is the ability to trigger the reload based on runtime
>> logic.
>>
>> David Lang
>>
>> On Wed, 10 Feb 2016, singh.janmejay wrote:
>>
>>> Putting it next to functions makes sense. I'll start working on it today.
>>>
>>> I thought about it while implementing it. I felt action will be rather
>>> confusing for this.
>>>
>>> It is easier to think of it as a function to lookup, and supporting
>>> statements to declare,find and load-up on-disk definition, or to reload
>>> it
>>> etc.
>>>
>>> Action has controls to build a high throughput egress port with
>>> configurable formatting (like tuneable queue, parallel processing in
>>> worker
>>> pool, safety valve to discard low priority messages, template etc) which
>>> are not relevant for reload control.
>>>
>>> So I feel it's simpler to understand and use as a statement.
>>>
>>> --
>>> Regards,
>>> Janmejay
>>>
>>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>>> keyboard sporting it's not-so-smart-assist technology.
>>>
>>> On Feb 10, 2016 3:41 AM, "David Lang" <da...@lang.hm> wrote:
>>>
>>>> My thinking is that the main use is in looking things up, where it is a
>>>> function and goes everywhere that any of the other functions can go.
>>>>
>>>> The action to (re)load the table is a supporting thing, for most people
>>>> done only at startup.
>>>>
>>>> As a (probably) silly thought, should the table (re)load be an action()
>>>> clause as opposed to a different type of statement? the initial
>>>> lookup-table writeup was done a long time ago, before action() became a
>>>> significant part of the config languange.
>>>>
>>>> David Lang
>>>>
>>>> On Tue, 9 Feb 2016, singh.janmejay wrote:
>>>>
>>>> I guess everyone is OK with putting it in as a function and linking out
>>>> to
>>>>>
>>>>> details.
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Janmejay
>>>>>
>>>>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>>>>> keyboard sporting it's not-so-smart-assist technology.
>>>>>
>>>>> On Feb 8, 2016 7:50 PM, "singh.janmejay" <singh.janme...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Well, they don't really fit the definition of functions. Functions in
>>>>>>
>>>>>> our case are expressions, these are a mix of expression (usage) and
>>>>>> statement (declaration) and in case of lookup-table it involves an
>>>>>> out-of-band data-file too.
>>>>>>
>>>>>> We can document the usage part of it under functions and then add link
>>>>>> to a page that talks about the declaration (and explains how it works
>>>>>> etc).
>>>>>>
>>>>>> On Mon, Feb 8, 2016 at 5:15 PM, David Lang <da...@lang.hm> wrote:
>>>>>>
>>>>>>> On Mon, 8 Feb 2016, singh.janmejay wrote:
>>>>>>>
>>>>>>> Does it go parallel to basic_structure or action etc?
>>>>>>>>
>>>>>>>>
>>>>>>>> And where all should we link it from?
>>>>>>

Re: [rsyslog] What would be the best place to document lookup-table and dyn-stats?

[singh.janmejay]

Makes sense.

I'll link the PR here once done.

--
Regards,
Janmejay

PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.

On Feb 10, 2016 10:24 AM, "David Lang" <da...@lang.hm> wrote:

> the lookup is a function, no question. It was the reload that prompted the
> action() question. But your comments on how so many things that are
> standard options for actions (queues, etc) make no sense for this and so a
> new statement type is much cleaner than either disabling, or interacting
> with all those standard things.
>
> we do have a handful of other non-action/input/module statements. The
> parser and global declarations are examples. the only thing that is unique
> in table lookups is the ability to trigger the reload based on runtime
> logic.
>
> David Lang
>
> On Wed, 10 Feb 2016, singh.janmejay wrote:
>
> Putting it next to functions makes sense. I'll start working on it today.
>>
>> I thought about it while implementing it. I felt action will be rather
>> confusing for this.
>>
>> It is easier to think of it as a function to lookup, and supporting
>> statements to declare,find and load-up on-disk definition, or to reload it
>> etc.
>>
>> Action has controls to build a high throughput egress port with
>> configurable formatting (like tuneable queue, parallel processing in
>> worker
>> pool, safety valve to discard low priority messages, template etc) which
>> are not relevant for reload control.
>>
>> So I feel it's simpler to understand and use as a statement.
>>
>> --
>> Regards,
>> Janmejay
>>
>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>> keyboard sporting it's not-so-smart-assist technology.
>>
>> On Feb 10, 2016 3:41 AM, "David Lang" <da...@lang.hm> wrote:
>>
>> My thinking is that the main use is in looking things up, where it is a
>>> function and goes everywhere that any of the other functions can go.
>>>
>>> The action to (re)load the table is a supporting thing, for most people
>>> done only at startup.
>>>
>>> As a (probably) silly thought, should the table (re)load be an action()
>>> clause as opposed to a different type of statement? the initial
>>> lookup-table writeup was done a long time ago, before action() became a
>>> significant part of the config languange.
>>>
>>> David Lang
>>>
>>> On Tue, 9 Feb 2016, singh.janmejay wrote:
>>>
>>> I guess everyone is OK with putting it in as a function and linking out
>>> to
>>>
>>>> details.
>>>>
>>>> --
>>>> Regards,
>>>> Janmejay
>>>>
>>>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>>>> keyboard sporting it's not-so-smart-assist technology.
>>>>
>>>> On Feb 8, 2016 7:50 PM, "singh.janmejay" <singh.janme...@gmail.com>
>>>> wrote:
>>>>
>>>> Well, they don't really fit the definition of functions. Functions in
>>>>
>>>>> our case are expressions, these are a mix of expression (usage) and
>>>>> statement (declaration) and in case of lookup-table it involves an
>>>>> out-of-band data-file too.
>>>>>
>>>>> We can document the usage part of it under functions and then add link
>>>>> to a page that talks about the declaration (and explains how it works
>>>>> etc).
>>>>>
>>>>> On Mon, Feb 8, 2016 at 5:15 PM, David Lang <da...@lang.hm> wrote:
>>>>>
>>>>> On Mon, 8 Feb 2016, singh.janmejay wrote:
>>>>>>
>>>>>> Does it go parallel to basic_structure or action etc?
>>>>>>
>>>>>>>
>>>>>>> And where all should we link it from?
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> I would expect it under functions, I don't know if they need
>>>>>> additional
>>>>>> links as well.
>>>>>>
>>>>>> David Lang
>>>>>> ___
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com/professional-services/
>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>> NOTE WEL

Re: [rsyslog] What would be the best place to document lookup-table and dyn-stats?

[singh.janmejay]

Putting it next to functions makes sense. I'll start working on it today.

I thought about it while implementing it. I felt action will be rather
confusing for this.

It is easier to think of it as a function to lookup, and supporting
statements to declare,find and load-up on-disk definition, or to reload it
etc.

Action has controls to build a high throughput egress port with
configurable formatting (like tuneable queue, parallel processing in worker
pool, safety valve to discard low priority messages, template etc) which
are not relevant for reload control.

So I feel it's simpler to understand and use as a statement.

--
Regards,
Janmejay

PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.

On Feb 10, 2016 3:41 AM, "David Lang" <da...@lang.hm> wrote:

> My thinking is that the main use is in looking things up, where it is a
> function and goes everywhere that any of the other functions can go.
>
> The action to (re)load the table is a supporting thing, for most people
> done only at startup.
>
> As a (probably) silly thought, should the table (re)load be an action()
> clause as opposed to a different type of statement? the initial
> lookup-table writeup was done a long time ago, before action() became a
> significant part of the config languange.
>
> David Lang
>
> On Tue, 9 Feb 2016, singh.janmejay wrote:
>
> I guess everyone is OK with putting it in as a function and linking out to
>> details.
>>
>> --
>> Regards,
>> Janmejay
>>
>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>> keyboard sporting it's not-so-smart-assist technology.
>>
>> On Feb 8, 2016 7:50 PM, "singh.janmejay" <singh.janme...@gmail.com>
>> wrote:
>>
>> Well, they don't really fit the definition of functions. Functions in
>>> our case are expressions, these are a mix of expression (usage) and
>>> statement (declaration) and in case of lookup-table it involves an
>>> out-of-band data-file too.
>>>
>>> We can document the usage part of it under functions and then add link
>>> to a page that talks about the declaration (and explains how it works
>>> etc).
>>>
>>> On Mon, Feb 8, 2016 at 5:15 PM, David Lang <da...@lang.hm> wrote:
>>>
>>>> On Mon, 8 Feb 2016, singh.janmejay wrote:
>>>>
>>>> Does it go parallel to basic_structure or action etc?
>>>>>
>>>>> And where all should we link it from?
>>>>>
>>>>
>>>>
>>>> I would expect it under functions, I don't know if they need additional
>>>> links as well.
>>>>
>>>> David Lang
>>>> ___
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>>
>>> of
>>>
>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T
>>>> LIKE THAT.
>>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Janmejay
>>> http://codehunk.wordpress.com
>>>
>>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] What would be the best place to document lookup-table and dyn-stats?

[singh.janmejay]

Does it go parallel to basic_structure or action etc?

And where all should we link it from?

-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] What would be the best place to document lookup-table and dyn-stats?

[singh.janmejay]

Also, FYI, I am planning to copy over large portion of doc from
lookup-table proposal.

On Mon, Feb 8, 2016 at 2:59 PM, singh.janmejay <singh.janme...@gmail.com> wrote:
> Does it go parallel to basic_structure or action etc?
>
> And where all should we link it from?
>
> --
> Regards,
> Janmejay
> http://codehunk.wordpress.com



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] What would be the best place to document lookup-table and dyn-stats?

[singh.janmejay]

I guess everyone is OK with putting it in as a function and linking out to
details.

--
Regards,
Janmejay

PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.

On Feb 8, 2016 7:50 PM, "singh.janmejay" <singh.janme...@gmail.com> wrote:

> Well, they don't really fit the definition of functions. Functions in
> our case are expressions, these are a mix of expression (usage) and
> statement (declaration) and in case of lookup-table it involves an
> out-of-band data-file too.
>
> We can document the usage part of it under functions and then add link
> to a page that talks about the declaration (and explains how it works
> etc).
>
> On Mon, Feb 8, 2016 at 5:15 PM, David Lang <da...@lang.hm> wrote:
> > On Mon, 8 Feb 2016, singh.janmejay wrote:
> >
> >> Does it go parallel to basic_structure or action etc?
> >>
> >> And where all should we link it from?
> >
> >
> > I would expect it under functions, I don't know if they need additional
> > links as well.
> >
> > David Lang
> > ___
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of
> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> > LIKE THAT.
>
>
>
> --
> Regards,
> Janmejay
> http://codehunk.wordpress.com
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] What would be the best place to document lookup-table and dyn-stats?

[singh.janmejay]

Well, they don't really fit the definition of functions. Functions in
our case are expressions, these are a mix of expression (usage) and
statement (declaration) and in case of lookup-table it involves an
out-of-band data-file too.

We can document the usage part of it under functions and then add link
to a page that talks about the declaration (and explains how it works
etc).

On Mon, Feb 8, 2016 at 5:15 PM, David Lang <da...@lang.hm> wrote:
> On Mon, 8 Feb 2016, singh.janmejay wrote:
>
>> Does it go parallel to basic_structure or action etc?
>>
>> And where all should we link it from?
>
>
> I would expect it under functions, I don't know if they need additional
> links as well.
>
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog next release and lookup-tables

[singh.janmejay]

Done.

On Sun, Dec 20, 2015 at 8:52 PM, singh.janmejay
<singh.janme...@gmail.com> wrote:
> That is already the case.
>
> --
> Regards,
> Janmejay
>
> PS: Please blame the typos in this mail on my phone's uncivilized soft
> keyboard sporting it's not-so-smart-assist technology.
>
>
> On Dec 19, 2015 9:46 PM, "David Lang" <da...@lang.hm> wrote:
>>
>> remember that while the table is loading, there will be queries against it
>> (from other worker threads is nothing else), and such queries should return
>> the data from the old version of the table.
>>
>> David Lang
>>
>> On Sat, 19 Dec 2015, singh.janmejay wrote:
>>
>>> Date: Sat, 19 Dec 2015 20:43:41 +0530
>>> From: singh.janmejay <singh.janme...@gmail.com>
>>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>> Subject: Re: [rsyslog] rsyslog next release and lookup-tables
>>>
>>> Good point. I'll change it. It's not a big change, will do it whenever I
>>> can find a few hours in the coming week.
>>>
>>> For clarity, we'll return to semantics proposed in original post. I'll
>>> make
>>> it async too.
>>>
>>> --
>>> Regards,
>>> Janmejay
>>>
>>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>>> keyboard sporting it's not-so-smart-assist technology.
>>>
>>> On Dec 19, 2015 5:08 AM, "David Lang" <da...@lang.hm> wrote:
>>>
>>>> I meant to respond to this, but may have forgotten to.
>>>>
>>>> The table load can be quite lengthy[1], so we wanted it to be
>>>> asynchronous. The new two-part approach doesn't have that ability.
>>>>
>>>> I would have the load process generate a log message when it finishes
>>>> saying if it succeeds or fails and you can then detect and take action
>>>> on
>>>> that.
>>>>
>>>> David Lang
>>>>
>>>> [1] the use case I was thinking of when I designed the sparse table type
>>>> is geoip lookup, using a table such as the one provided by MaxMind (some
>>>> for free, some at some costs), this can be a multi GB table that's being
>>>> loaded, so we don't want to halt log processing while the table is
>>>> loading
>>>>
>>>>
>>>> On Fri, 18 Dec 2015, singh.janmejay wrote:
>>>>
>>>> Date: Fri, 18 Dec 2015 21:12:01 +0530
>>>>>
>>>>> From: singh.janmejay <singh.janme...@gmail.com>
>>>>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>>> Subject: Re: [rsyslog] rsyslog next release and lookup-tables
>>>>>
>>>>> This change is now in PR.
>>>>>
>>>>> There is a minor change. Actually  stub_lookup_table_value statement
>>>>> needs to know the lookup table as well, so it takes 2 arguments (not
>>>>> 1).
>>>>> So, the fragment would look like:
>>>>> if (not reload_lookup_table("foo")) then {
>>>>>  stub_lookup_table_value("foo", "reload_failed")
>>>>> }
>>>>>
>>>>> On Thu, Nov 26, 2015 at 8:16 PM, singh.janmejay
>>>>> <singh.janme...@gmail.com> wrote:
>>>>>
>>>>>> Resurrecting this thread for a discussing a deviation from old
>>>>>> lookup_table proposal.
>>>>>>
>>>>>> The PR doesn't include load_lookup_table (a rainerscript statement
>>>>>> that is supposed to reload lookup table).
>>>>>>
>>>>>> Original proposal link: http://www.rsyslog.com/doc/lookup_tables.html
>>>>>>
>>>>>> Im thinking of implementing it using a function
>>>>>> reload_lookup_table(name) and a statement
>>>>>> stub_lookup_table_value(value).
>>>>>>
>>>>>> It allows better composability and leaves room for better backward
>>>>>> compatibility.
>>>>>>
>>>>>> Its possible to get the same effect as the load_lookup_table(...) gets
>>>>>> in the proposed form using:
>>>>>>
>>>>>> if (not reload_lookup_table("foo")) then {
>>>>>>   stub_lookup_table_value("re

Re: [rsyslog] rsyslog next release and lookup-tables

[singh.janmejay]

That is already the case.

--
Regards,
Janmejay

PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.

On Dec 19, 2015 9:46 PM, "David Lang" <da...@lang.hm> wrote:

> remember that while the table is loading, there will be queries against it
> (from other worker threads is nothing else), and such queries should return
> the data from the old version of the table.
>
> David Lang
>
> On Sat, 19 Dec 2015, singh.janmejay wrote:
>
> Date: Sat, 19 Dec 2015 20:43:41 +0530
>> From: singh.janmejay <singh.janme...@gmail.com>
>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] rsyslog next release and lookup-tables
>>
>> Good point. I'll change it. It's not a big change, will do it whenever I
>> can find a few hours in the coming week.
>>
>> For clarity, we'll return to semantics proposed in original post. I'll
>> make
>> it async too.
>>
>> --
>> Regards,
>> Janmejay
>>
>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>> keyboard sporting it's not-so-smart-assist technology.
>>
>> On Dec 19, 2015 5:08 AM, "David Lang" <da...@lang.hm> wrote:
>>
>> I meant to respond to this, but may have forgotten to.
>>>
>>> The table load can be quite lengthy[1], so we wanted it to be
>>> asynchronous. The new two-part approach doesn't have that ability.
>>>
>>> I would have the load process generate a log message when it finishes
>>> saying if it succeeds or fails and you can then detect and take action on
>>> that.
>>>
>>> David Lang
>>>
>>> [1] the use case I was thinking of when I designed the sparse table type
>>> is geoip lookup, using a table such as the one provided by MaxMind (some
>>> for free, some at some costs), this can be a multi GB table that's being
>>> loaded, so we don't want to halt log processing while the table is
>>> loading
>>>
>>>
>>> On Fri, 18 Dec 2015, singh.janmejay wrote:
>>>
>>> Date: Fri, 18 Dec 2015 21:12:01 +0530
>>>
>>>> From: singh.janmejay <singh.janme...@gmail.com>
>>>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Subject: Re: [rsyslog] rsyslog next release and lookup-tables
>>>>
>>>> This change is now in PR.
>>>>
>>>> There is a minor change. Actually  stub_lookup_table_value statement
>>>> needs to know the lookup table as well, so it takes 2 arguments (not
>>>> 1).
>>>> So, the fragment would look like:
>>>> if (not reload_lookup_table("foo")) then {
>>>>  stub_lookup_table_value("foo", "reload_failed")
>>>> }
>>>>
>>>> On Thu, Nov 26, 2015 at 8:16 PM, singh.janmejay
>>>> <singh.janme...@gmail.com> wrote:
>>>>
>>>> Resurrecting this thread for a discussing a deviation from old
>>>>> lookup_table proposal.
>>>>>
>>>>> The PR doesn't include load_lookup_table (a rainerscript statement
>>>>> that is supposed to reload lookup table).
>>>>>
>>>>> Original proposal link: http://www.rsyslog.com/doc/lookup_tables.html
>>>>>
>>>>> Im thinking of implementing it using a function
>>>>> reload_lookup_table(name) and a statement
>>>>> stub_lookup_table_value(value).
>>>>>
>>>>> It allows better composability and leaves room for better backward
>>>>> compatibility.
>>>>>
>>>>> Its possible to get the same effect as the load_lookup_table(...) gets
>>>>> in the proposed form using:
>>>>>
>>>>> if (not reload_lookup_table("foo")) then {
>>>>>   stub_lookup_table_value("reload_failed")
>>>>> }
>>>>>
>>>>> However, it can be composed with other rainerscript controls. Eg. one
>>>>> can log failiures to reload table to a separate log-file, increment a
>>>>> metric which reports this failure, or send it to a remote destination.
>>>>>
>>>>> if (not reload_lookup_table("foo")) then {
>>>>>   action(type="omfile"...)
>>>>> }
>>>

Re: [rsyslog] rsyslog next release and lookup-tables

[singh.janmejay]

Good point. I'll change it. It's not a big change, will do it whenever I
can find a few hours in the coming week.

For clarity, we'll return to semantics proposed in original post. I'll make
it async too.

--
Regards,
Janmejay

PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.

On Dec 19, 2015 5:08 AM, "David Lang" <da...@lang.hm> wrote:

> I meant to respond to this, but may have forgotten to.
>
> The table load can be quite lengthy[1], so we wanted it to be
> asynchronous. The new two-part approach doesn't have that ability.
>
> I would have the load process generate a log message when it finishes
> saying if it succeeds or fails and you can then detect and take action on
> that.
>
> David Lang
>
> [1] the use case I was thinking of when I designed the sparse table type
> is geoip lookup, using a table such as the one provided by MaxMind (some
> for free, some at some costs), this can be a multi GB table that's being
> loaded, so we don't want to halt log processing while the table is loading
>
>
> On Fri, 18 Dec 2015, singh.janmejay wrote:
>
> Date: Fri, 18 Dec 2015 21:12:01 +0530
>> From: singh.janmejay <singh.janme...@gmail.com>
>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] rsyslog next release and lookup-tables
>>
>> This change is now in PR.
>>
>> There is a minor change. Actually  stub_lookup_table_value statement
>> needs to know the lookup table as well, so it takes 2 arguments (not
>> 1).
>> So, the fragment would look like:
>> if (not reload_lookup_table("foo")) then {
>>  stub_lookup_table_value("foo", "reload_failed")
>> }
>>
>> On Thu, Nov 26, 2015 at 8:16 PM, singh.janmejay
>> <singh.janme...@gmail.com> wrote:
>>
>>> Resurrecting this thread for a discussing a deviation from old
>>> lookup_table proposal.
>>>
>>> The PR doesn't include load_lookup_table (a rainerscript statement
>>> that is supposed to reload lookup table).
>>>
>>> Original proposal link: http://www.rsyslog.com/doc/lookup_tables.html
>>>
>>> Im thinking of implementing it using a function
>>> reload_lookup_table(name) and a statement
>>> stub_lookup_table_value(value).
>>>
>>> It allows better composability and leaves room for better backward
>>> compatibility.
>>>
>>> Its possible to get the same effect as the load_lookup_table(...) gets
>>> in the proposed form using:
>>>
>>> if (not reload_lookup_table("foo")) then {
>>>   stub_lookup_table_value("reload_failed")
>>> }
>>>
>>> However, it can be composed with other rainerscript controls. Eg. one
>>> can log failiures to reload table to a separate log-file, increment a
>>> metric which reports this failure, or send it to a remote destination.
>>>
>>> if (not reload_lookup_table("foo")) then {
>>>   action(type="omfile"...)
>>> }
>>>
>>> Functionally, this is all that matters.
>>>
>>> Have a look at PR(comments) for details (backward compatibility issue
>>> etc).
>>>
>>>
>>>
>>> On Tue, Nov 3, 2015 at 7:37 PM, singh.janmejay <singh.janme...@gmail.com>
>>> wrote:
>>>
>>>> Will cross-reference in the kill-feature PR.
>>>>
>>>> On Tue, Nov 3, 2015 at 7:37 PM, singh.janmejay <
>>>> singh.janme...@gmail.com> wrote:
>>>>
>>>>> Sorry for the delay. Here is the PR:
>>>>> https://github.com/rsyslog/rsyslog/pull/578
>>>>>
>>>>> On Tue, Nov 3, 2015 at 6:02 PM, Rainer Gerhards
>>>>> <rgerha...@hq.adiscon.com> wrote:
>>>>>
>>>>>> 2015-11-03 12:54 GMT+01:00  <christopher.ra...@web.de>:
>>>>>>
>>>>>>> Hello,
>>>>>>> As far as I see, today is the release data of the next rsyslog
>>>>>>> version.
>>>>>>> I did not see any changes about the lookup diffs, Janmejay promised,
>>>>>>> so I'm quite nerverous that the new release will no longer contain the
>>>>>>> lookup-tables.
>>>>>>>
>>>>>>
>>>>>> Please have a look here for status updates:
>>>>>> https://github.com/rsyslog/rsyslog/pull/544
>>>&

Re: [rsyslog] rsyslog next release and lookup-tables

[singh.janmejay]

This change is now in PR.

There is a minor change. Actually  stub_lookup_table_value statement
needs to know the lookup table as well, so it takes 2 arguments (not
1).
So, the fragment would look like:
if (not reload_lookup_table("foo")) then {
  stub_lookup_table_value("foo", "reload_failed")
}

On Thu, Nov 26, 2015 at 8:16 PM, singh.janmejay
<singh.janme...@gmail.com> wrote:
> Resurrecting this thread for a discussing a deviation from old
> lookup_table proposal.
>
> The PR doesn't include load_lookup_table (a rainerscript statement
> that is supposed to reload lookup table).
>
> Original proposal link: http://www.rsyslog.com/doc/lookup_tables.html
>
> Im thinking of implementing it using a function
> reload_lookup_table(name) and a statement
> stub_lookup_table_value(value).
>
> It allows better composability and leaves room for better backward
> compatibility.
>
> Its possible to get the same effect as the load_lookup_table(...) gets
> in the proposed form using:
>
> if (not reload_lookup_table("foo")) then {
>   stub_lookup_table_value("reload_failed")
> }
>
> However, it can be composed with other rainerscript controls. Eg. one
> can log failiures to reload table to a separate log-file, increment a
> metric which reports this failure, or send it to a remote destination.
>
> if (not reload_lookup_table("foo")) then {
>   action(type="omfile"...)
> }
>
> Functionally, this is all that matters.
>
> Have a look at PR(comments) for details (backward compatibility issue etc).
>
>
>
> On Tue, Nov 3, 2015 at 7:37 PM, singh.janmejay <singh.janme...@gmail.com> 
> wrote:
>> Will cross-reference in the kill-feature PR.
>>
>> On Tue, Nov 3, 2015 at 7:37 PM, singh.janmejay <singh.janme...@gmail.com> 
>> wrote:
>>> Sorry for the delay. Here is the PR: 
>>> https://github.com/rsyslog/rsyslog/pull/578
>>>
>>> On Tue, Nov 3, 2015 at 6:02 PM, Rainer Gerhards
>>> <rgerha...@hq.adiscon.com> wrote:
>>>> 2015-11-03 12:54 GMT+01:00  <christopher.ra...@web.de>:
>>>>> Hello,
>>>>> As far as I see, today is the release data of the next rsyslog version.
>>>>> I did not see any changes about the lookup diffs, Janmejay promised, so 
>>>>> I'm quite nerverous that the new release will no longer contain the 
>>>>> lookup-tables.
>>>>
>>>> Please have a look here for status updates:
>>>> https://github.com/rsyslog/rsyslog/pull/544
>>>>
>>>> In short: I won't remove it this release, as I have no longer been
>>>> tortured with CVEs and I think we can let it stand as is - NOT
>>>> officially existing - for a bit longer. I hope we can merge something
>>>> solid into the december or january relaese.
>>>>
>>>> Rainer
>>>>>
>>>>> Please do not remove it, as it works fine (after the last patch) and I 
>>>>> (and possibly others) use it already in production.
>>>>> If it is needed I will help to document the functionality as it exists 
>>>>> right now.
>>>>>
>>>>> Best regards,
>>>>> Christopher
>>>>>
>>>>>
>>>>> 
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Hello,
>>>>> I have never heared such a nonsense.
>>>>> Actually the number of applications that does not include features that 
>>>>> are not official documented shoult be extremly limited.
>>>>>
>>>>> The functionality is really usefull and already in big landscapes 
>>>>> productive.
>>>>> Please, please do NOT remove the lookup-table from the main branch.
>>>>> The functionaltiy works fine, I'm using this since march and I did not 
>>>>> have any issue since the latest patch of janmejay.
>>>>>
>>>>> Even the "concept" is not fully implemented (e.g. smaller things like 
>>>>> nomatch) the main part works fine.
>>>>>
>>>>>
>>>>> My suggestion would be to document everything which is currently 
>>>>> implemented and keep the "conceptual documentation" as it is.
>>>>> So the Maintainer should no longer have an issue with it.

Re: [rsyslog] rsyslog next release and lookup-tables

[singh.janmejay]

Resurrecting this thread for a discussing a deviation from old
lookup_table proposal.

The PR doesn't include load_lookup_table (a rainerscript statement
that is supposed to reload lookup table).

Original proposal link: http://www.rsyslog.com/doc/lookup_tables.html

Im thinking of implementing it using a function
reload_lookup_table(name) and a statement
stub_lookup_table_value(value).

It allows better composability and leaves room for better backward
compatibility.

Its possible to get the same effect as the load_lookup_table(...) gets
in the proposed form using:

if (not reload_lookup_table("foo")) then {
  stub_lookup_table_value("reload_failed")
}

However, it can be composed with other rainerscript controls. Eg. one
can log failiures to reload table to a separate log-file, increment a
metric which reports this failure, or send it to a remote destination.

if (not reload_lookup_table("foo")) then {
  action(type="omfile"...)
}

Functionally, this is all that matters.

Have a look at PR(comments) for details (backward compatibility issue etc).



On Tue, Nov 3, 2015 at 7:37 PM, singh.janmejay <singh.janme...@gmail.com> wrote:
> Will cross-reference in the kill-feature PR.
>
> On Tue, Nov 3, 2015 at 7:37 PM, singh.janmejay <singh.janme...@gmail.com> 
> wrote:
>> Sorry for the delay. Here is the PR: 
>> https://github.com/rsyslog/rsyslog/pull/578
>>
>> On Tue, Nov 3, 2015 at 6:02 PM, Rainer Gerhards
>> <rgerha...@hq.adiscon.com> wrote:
>>> 2015-11-03 12:54 GMT+01:00  <christopher.ra...@web.de>:
>>>> Hello,
>>>> As far as I see, today is the release data of the next rsyslog version.
>>>> I did not see any changes about the lookup diffs, Janmejay promised, so 
>>>> I'm quite nerverous that the new release will no longer contain the 
>>>> lookup-tables.
>>>
>>> Please have a look here for status updates:
>>> https://github.com/rsyslog/rsyslog/pull/544
>>>
>>> In short: I won't remove it this release, as I have no longer been
>>> tortured with CVEs and I think we can let it stand as is - NOT
>>> officially existing - for a bit longer. I hope we can merge something
>>> solid into the december or january relaese.
>>>
>>> Rainer
>>>>
>>>> Please do not remove it, as it works fine (after the last patch) and I 
>>>> (and possibly others) use it already in production.
>>>> If it is needed I will help to document the functionality as it exists 
>>>> right now.
>>>>
>>>> Best regards,
>>>> Christopher
>>>>
>>>>
>>>> 
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Hello,
>>>> I have never heared such a nonsense.
>>>> Actually the number of applications that does not include features that 
>>>> are not official documented shoult be extremly limited.
>>>>
>>>> The functionality is really usefull and already in big landscapes 
>>>> productive.
>>>> Please, please do NOT remove the lookup-table from the main branch.
>>>> The functionaltiy works fine, I'm using this since march and I did not 
>>>> have any issue since the latest patch of janmejay.
>>>>
>>>> Even the "concept" is not fully implemented (e.g. smaller things like 
>>>> nomatch) the main part works fine.
>>>>
>>>>
>>>> My suggestion would be to document everything which is currently 
>>>> implemented and keep the "conceptual documentation" as it is.
>>>> So the Maintainer should no longer have an issue with it.
>>>>
>>>>
>>>> If the main issue it the time to document the already implemented 
>>>> features, I can create a patch.
>>>>
>>>>
>>>> Chris
>>>>
>>>>
>>>>
>>>>> Gesendet: Dienstag, 06. Oktober 2015 um 07:36 Uhr
>>>>> Von: "David Lang" <da...@lang.hm>
>>>>> An: rsyslog-users <rsyslog@lists.adiscon.com>
>>>>> Betreff: Re: [rsyslog] Separation of actions based on log source - with 
>>>>> good performance
>>>>>
>>>>> a CVE for something that requires manually enabling an experimental 
>>>>> feature???
>>>>>
>>>>> it would be one thing if a defa

Re: [rsyslog] RFC: dynamic-stats support

[singh.janmejay]

I'll create 2 implementations for this. Will first build it with
hash-table (which will use exclusive lock only when adding new metric
to the table, increments will happen with shared-lock).

In the second cut, I'll change it to trie (which is what I discussed
in the proposal). After benchmarking both we'll be in a better
position to decide which one should be kept.

On Fri, Oct 9, 2015 at 3:20 AM, singh.janmejay <singh.janme...@gmail.com> wrote:
> On Thu, Oct 8, 2015 at 11:07 PM, David Lang <da...@lang.hm> wrote:
>> Atomic ops are actually rather expensive (almost as expsnsive as full
>> locks). If you want a lockless metrics capability, you should do a separate
>> set of variables per thread, gathering them a reporting time. And document
>> that there is going to be inconsistancy between the different metrics
>
> Yes, the write thing to do is to maintain thread-local counters. It'll
> require slightly wider set of changes for that, but the approach
> allows us to make those improvements later without changing the
> config-interface.
>
> Cost of atomic-ops is close to uncontended lock, but much lower than
> contended lock.
>
>>
>> unless you lock everything, you are going to have inconsistancies across the
>> different metrics.
>
> Yes, but isn't that acceptable in most cases?
>
>>
>> Also, not all platforms have the easy atomic ops you are thinking of, and
>> atomic ops can't operate on all sizes of data (for example, 32 bit systems
>> probably can't update a 64 bit value atomically)
>
> We should eventually move to thread-level accumulators for all metrics
> (static or dynamic).
>
>>
>> David Lang
>>
>>
>> On Thu, 8 Oct 2015, singh.janmejay wrote:
>>
>>> Did you mean it's not atomic across different metrics? That I think should
>>> be acceptable. A single metric however should get swapped losslessly with
>>> atomic-swap. Either an increment of m should be applied before swap
>>> (making
>>> the reading n + m, or after it leaving the accumulator at m and reading at
>>> n). But it should not be lost.
>>>
>>> Am I misunderstanding something?
>>>
>>> --
>>> Regards,
>>> Janmejay
>>>
>>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>>> keyboard sporting it's not-so-smart-assist technology.
>>>
>>> On Oct 8, 2015 12:33 PM, "Rainer Gerhards" <rgerha...@hq.adiscon.com>
>>> wrote:
>>>
>>>> 2015-10-08 8:30 GMT+02:00 singh.janmejay <singh.janme...@gmail.com>:
>>>>>>
>>>>>> Similarly, when one thread goes to output the stats, you need to lock
>>>>>
>>>>> them so that there isn't a lost increment between the time that you read
>>>>> the stat and the time you zero it.
>>>>>
>>>>> No, this involves the same shared (uncontended) lock, except
>>>>> atomic-increment is replaced by atomic-swap with 0.
>>>>>
>>>>
>>>> Just FYI: this is what the current stats system also does. It is also
>>>> where some inaccuracy stems from. Reporting stats is not atomic
>>>> without looks, so a stats counter may be read with value n, then m
>>>> atomic increments happen to it on another thread, then value n is
>>>> being reported (but we are really at n+m) and then the stats counter
>>>> is reset to 0 via an atomic swap. So m updates are lost. IMHO this is
>>>> perfectly acceptable, because otherwise we would lose almost all
>>>> concurrency.
>>>>
>>>> Rainer
>>>> ___
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T LIKE THAT.
>>>>
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>>> LIKE THAT.
>>&g

Re: [rsyslog] rsyslog next release and lookup-tables

[singh.janmejay]

ch. Then I'll remove the lookup table code, so that the code base
>>> > matches the documentation. I really don't want to create a general
>>> > principle here that we need to create CVEs (and patched) for something
>>> > that was just added as a convenience for a handful of folks who were
>>> > ready to take a risk.
>>> >
>>> > If there is sufficient interest, we can consider officially adding
>>> > this feature to the January 8.15 release iff it is ready by then.
>>> > @janmejay: please let me know if you would like to continue with your
>>> > work on lookup tables under this new situation.
>>> >
>>> > As soon as I have time, I'll check what else needs to be removed. Not
>>> > sure about the ./contributed branch, because the project cannot
>>> > guarantee at all this is bug-free. It's documented to be so, but if
>>> > that is not sufficient, it should probably live only in the
>>> > master-insecure branch.
>>> >
>>> > Rainer
>>> >
>>> > 2015-10-02 17:29 GMT+02:00 singh.janmejay <singh.janme...@gmail.com>:
>>> >> As of now it returns empty string for no-match. I guess we should go 
>>> >> ahead
>>> >> with it in its current form. We can add default value any time later
>>> >> without breaking compatibility(it'd default to "").
>>> >>
>>> >> I'll add a test for multiple tables on Monday.
>>> >>
>>> >> On Fri, Oct 2, 2015, 7:16 PM  <christopher.ra...@web.de> wrote:
>>> >>
>>> >>> Hi,
>>> >>> No, I didn't. I even didn't realize the patch.
>>> >>>
>>> >>> It seems to be exactly the related issue. So I don't expect any further
>>> >>> issues.
>>> >>> I will use the new version on 2 systems. If there is any other issue, I
>>> >>> will let you know.
>>> >>> Release data for next rsyslog version is quite far so enough time to
>>> >>> test... ;)
>>> >>>
>>> >>> The missing implementation of "nomatch" (default) entry as described at
>>> >>>  http://www.rsyslog.com/doc/lookup_tables.html
>>> >>> would from my opinion require changes:
>>> >>>
>>> >>> Arround line 132 of lookup.c file (save of value)
>>> >>> Arround line 243 of lookup.c file (search in lookuptable fails, so 
>>> >>> return
>>> >>> nomatch value.
>>> >>>
>>> >>>
>>> >>> regards
>>> >>> Chris
>>> >>>
>>> >>>
>>> >>> > Gesendet: Donnerstag, 01. Oktober 2015 um 16:57 Uhr
>>> >>> > Von: "singh.janmejay" <singh.janme...@gmail.com>
>>> >>> > An: rsyslog-users <rsyslog@lists.adiscon.com>
>>> >>> > Betreff: Re: [rsyslog] Separation of actions based on log source - 
>>> >>> > with
>>> >>> good performance
>>> >>> >
>>> >>> > Yes, if you build off master, that problem should go away (if it was 
>>> >>> > due
>>> >>> to
>>> >>> > lookup-table).
>>> >>> >
>>> >>> > On Thu, Oct 1, 2015, 7:00 PM Rainer Gerhards 
>>> >>> > <rgerha...@hq.adiscon.com>
>>> >>> > wrote:
>>> >>> >
>>> >>> > > 2015-10-01 15:14 GMT+02:00 singh.janmejay 
>>> >>> > > <singh.janme...@gmail.com>:
>>> >>> > > > If you can share output of all thread backtrace we can confirm if
>>> >>> this
>>> >>> > > > is the cause.
>>> >>> > >
>>> >>> > > let's first double-check: Christopher, did you use yesterday 
>>> >>> > > evening's
>>> >>> > > master branch? Because that contains a patch from Janmejay that I
>>> >>> > > think causes the problem for you. Or am I wrong, Janmejay?
>>> >>> > >
>>> >>> > > Rainer
>>> >>> > > >
>>> >>> > > > On Thu, Oct 1, 2015 at 2:30 PM,  <christopher.ra...@web.de> wrote:
>>> >>> > > >> Hi,
>>> >>> > > >> Up

Re: [rsyslog] rsyslog next release and lookup-tables

[singh.janmejay]

Will cross-reference in the kill-feature PR.

On Tue, Nov 3, 2015 at 7:37 PM, singh.janmejay <singh.janme...@gmail.com> wrote:
> Sorry for the delay. Here is the PR: 
> https://github.com/rsyslog/rsyslog/pull/578
>
> On Tue, Nov 3, 2015 at 6:02 PM, Rainer Gerhards
> <rgerha...@hq.adiscon.com> wrote:
>> 2015-11-03 12:54 GMT+01:00  <christopher.ra...@web.de>:
>>> Hello,
>>> As far as I see, today is the release data of the next rsyslog version.
>>> I did not see any changes about the lookup diffs, Janmejay promised, so I'm 
>>> quite nerverous that the new release will no longer contain the 
>>> lookup-tables.
>>
>> Please have a look here for status updates:
>> https://github.com/rsyslog/rsyslog/pull/544
>>
>> In short: I won't remove it this release, as I have no longer been
>> tortured with CVEs and I think we can let it stand as is - NOT
>> officially existing - for a bit longer. I hope we can merge something
>> solid into the december or january relaese.
>>
>> Rainer
>>>
>>> Please do not remove it, as it works fine (after the last patch) and I (and 
>>> possibly others) use it already in production.
>>> If it is needed I will help to document the functionality as it exists 
>>> right now.
>>>
>>> Best regards,
>>> Christopher
>>>
>>>
>>> 
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Hello,
>>> I have never heared such a nonsense.
>>> Actually the number of applications that does not include features that are 
>>> not official documented shoult be extremly limited.
>>>
>>> The functionality is really usefull and already in big landscapes 
>>> productive.
>>> Please, please do NOT remove the lookup-table from the main branch.
>>> The functionaltiy works fine, I'm using this since march and I did not have 
>>> any issue since the latest patch of janmejay.
>>>
>>> Even the "concept" is not fully implemented (e.g. smaller things like 
>>> nomatch) the main part works fine.
>>>
>>>
>>> My suggestion would be to document everything which is currently 
>>> implemented and keep the "conceptual documentation" as it is.
>>> So the Maintainer should no longer have an issue with it.
>>>
>>>
>>> If the main issue it the time to document the already implemented features, 
>>> I can create a patch.
>>>
>>>
>>> Chris
>>>
>>>
>>>
>>>> Gesendet: Dienstag, 06. Oktober 2015 um 07:36 Uhr
>>>> Von: "David Lang" <da...@lang.hm>
>>>> An: rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Betreff: Re: [rsyslog] Separation of actions based on log source - with 
>>>> good performance
>>>>
>>>> a CVE for something that requires manually enabling an experimental 
>>>> feature???
>>>>
>>>> it would be one thing if a default config had the problem, or if it was
>>>> something entirely dependent on remote data.
>>>>
>>>> I would be very tempted to respond to the CVE with "don't enable this 
>>>> incomplete
>>>> feature" as the solution. It's very common for incomplete features to be
>>>> included in released versions
>>>>
>>>> grumble, we have enough real bugs to worry about.
>>>>
>>>> David Lang
>>>>
>>>> On Tue, 6 Oct 2015, Rainer Gerhards wrote:
>>>>
>>>> > Date: Tue, 6 Oct 2015 07:15:31 +0200
>>>> > From: Rainer Gerhards <rgerha...@hq.adiscon.com>
>>>> > Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>> > To: rsyslog-users <rsyslog@lists.adiscon.com>
>>>> > Subject: Re: [rsyslog] Separation of actions based on log source - with 
>>>> > good
>>>> > performance
>>>> >
>>>> > Sorry, folks, good intent always seems to find someone who turns it
>>>> > into negative. I was yesterday contacted by a distro maintainer who
>>>> > wants to turn this bug in the officially non-existant lookup table
>>>> > feature into a CVE and insists that it is a vuln even after the
>>>> > argument that the feature never oficially existed.
>>>> >
&g

Re: [rsyslog] RFC: dynamic-stats support

[singh.janmejay]

--
Regards,
Janmejay

PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.

On Oct 7, 2015 11:25 PM, "David Lang" <da...@lang.hm> wrote:
>
> On Wed, 7 Oct 2015, singh.janmejay wrote:
>
>> --
>> Regards,
>> Janmejay
>>
>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>> keyboard sporting it's not-so-smart-assist technology.
>>
>> On Oct 7, 2015 3:26 AM, "David Lang" <da...@lang.hm> wrote:
>>>
>>>
>>> On Wed, 7 Oct 2015, singh.janmejay wrote:
>>>
>>>> --
>>>> Regards,
>>>> Janmejay
>>>>
>>>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>>>> keyboard sporting it's not-so-smart-assist technology.
>>>>
>>>> On Oct 6, 2015 10:32 PM, "David Lang" <da...@lang.hm> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Tue, 6 Oct 2015, singh.janmejay wrote:
>>>>>
>>>>>> It is possible to use global-variables (it'll require some
>>>>>> enhancements, table-support etc), but it'll be very inefficient
>>>>>> compared to this approach. For instance, choice of data-structure etc
>>>>>> allows making the solution a lot more efficient.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> As for the data structures, Rainer has been identifying inefficencies
in
>>>>
>>>>
>>>> how json-c works and working to improve them
>>>>>
>>>>>
>>>>>
>>>>
>>>> That optimizes variable system. But it still is a general propose
>>
>> variable
>>>>
>>>> system. It can't and shouldn't understand relationship between
variables.
>>>
>>>
>>>
>>> what relationships are there between the different metrics?
>>>
>>
>> The fact that they are read in one shot, reported and reset.
>
>
> I don't understand why you couldn't do $\stats!metric1, $\stats!metric2,
etc. There is no reason why this couldn't have the same locking as your new
structure.
>

If this is built over variable backend, it has to be global variables(which
is what you are suggesting). But consider the implementation of this table
($/stats). Any efficient indexing (hash or tree based data structures),
will require locks for rebalance/re-hash operations. But even counter
increment operations need to contend for the lock because of the potential
of concurrent operations that change the table. It can't have an assured
uncontended path except for initial opportunistic wide shared lock which
allows checking existence of key. But the scope of lock and contention due
to it is wider.

Also, this involves technical intricacies that all users may not be
comfortable with. It's asking them to build the stats system as opposed to
providing them a ready-made one. In this case it is without any loss in
generality or any sacrifice in capability of the feature. I like it because
it's easier to use.

>
>>>
>>>>>
>>>>>> Here its possible to locklessly increment counters in most cases, so
>>>>>> its overhead is a lot lesser than global-variables.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> how can you manage counters in multiple threads without locks?
>>
>> Especially
>>>>
>>>>
>>>> when dealing with batches.
>>>>>
>>>>>
>>>>>
>>>>
>>>> Consider a trie based implementation. With bounded fanout-factor, it's
>>
>> O(1)
>>>>
>>>> wrt metric-names cardinality. It also has very little lock contention
>>>> involved. Usually operations work with read-locks, only when new metric
>>
>> is
>>>>
>>>> initialized it requires a write lock on patent node. If recycles are
few
>>>> and far apart, lock contention would be negligible.
>>>
>>>
>>>
>>> if you have multiple threads that may need to update the same metric at
>>
>> the same time, a tree doesn't eliminate the locking.
>>>
>>>
>>
>> The only situation involving a lock that is contended for, is when a
metric
>> is to be initialized. Consider this trie:
>>
>> A -> B -> C
>>   -> D
>>
>> Now for incrementing key ABC, no contention exists, because it involves
>> read-

Re: [rsyslog] RFC: dynamic-stats support

[singh.janmejay]

Yep, makes sense. I second your opinion, absolute consistency between
metrics is not that valuable.

On Thu, Oct 8, 2015 at 8:57 PM, Rainer Gerhards
<rgerha...@hq.adiscon.com> wrote:
> 2015-10-08 17:19 GMT+02:00 singh.janmejay <singh.janme...@gmail.com>:
>> Did you mean it's not atomic across different metrics? That I think should
>> be acceptable. A single metric however should get swapped losslessly with
>> atomic-swap. Either an increment of m should be applied before swap (making
>> the reading n + m, or after it leaving the accumulator at m and reading at
>> n). But it should not be lost.
>>
>> Am I misunderstanding something?
>
> No, I haven't been precise enough. Usually, we have a set of counters
> which interdepend. And so you never get them really consistent.
>
> Rainer
>>
>> --
>> Regards,
>> Janmejay
>>
>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>> keyboard sporting it's not-so-smart-assist technology.
>>
>> On Oct 8, 2015 12:33 PM, "Rainer Gerhards" <rgerha...@hq.adiscon.com> wrote:
>>
>>> 2015-10-08 8:30 GMT+02:00 singh.janmejay <singh.janme...@gmail.com>:
>>> >> Similarly, when one thread goes to output the stats, you need to lock
>>> > them so that there isn't a lost increment between the time that you read
>>> > the stat and the time you zero it.
>>> >
>>> > No, this involves the same shared (uncontended) lock, except
>>> > atomic-increment is replaced by atomic-swap with 0.
>>> >
>>>
>>> Just FYI: this is what the current stats system also does. It is also
>>> where some inaccuracy stems from. Reporting stats is not atomic
>>> without looks, so a stats counter may be read with value n, then m
>>> atomic increments happen to it on another thread, then value n is
>>> being reported (but we are really at n+m) and then the stats counter
>>> is reset to 0 via an atomic swap. So m updates are lost. IMHO this is
>>> perfectly acceptable, because otherwise we would lose almost all
>>> concurrency.
>>>
>>> Rainer
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
>> LIKE THAT.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] RFC: dynamic-stats support

[singh.janmejay]

On Thu, Oct 8, 2015 at 11:07 PM, David Lang <da...@lang.hm> wrote:
> Atomic ops are actually rather expensive (almost as expsnsive as full
> locks). If you want a lockless metrics capability, you should do a separate
> set of variables per thread, gathering them a reporting time. And document
> that there is going to be inconsistancy between the different metrics

Yes, the write thing to do is to maintain thread-local counters. It'll
require slightly wider set of changes for that, but the approach
allows us to make those improvements later without changing the
config-interface.

Cost of atomic-ops is close to uncontended lock, but much lower than
contended lock.

>
> unless you lock everything, you are going to have inconsistancies across the
> different metrics.

Yes, but isn't that acceptable in most cases?

>
> Also, not all platforms have the easy atomic ops you are thinking of, and
> atomic ops can't operate on all sizes of data (for example, 32 bit systems
> probably can't update a 64 bit value atomically)

We should eventually move to thread-level accumulators for all metrics
(static or dynamic).

>
> David Lang
>
>
> On Thu, 8 Oct 2015, singh.janmejay wrote:
>
>> Did you mean it's not atomic across different metrics? That I think should
>> be acceptable. A single metric however should get swapped losslessly with
>> atomic-swap. Either an increment of m should be applied before swap
>> (making
>> the reading n + m, or after it leaving the accumulator at m and reading at
>> n). But it should not be lost.
>>
>> Am I misunderstanding something?
>>
>> --
>> Regards,
>> Janmejay
>>
>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>> keyboard sporting it's not-so-smart-assist technology.
>>
>> On Oct 8, 2015 12:33 PM, "Rainer Gerhards" <rgerha...@hq.adiscon.com>
>> wrote:
>>
>>> 2015-10-08 8:30 GMT+02:00 singh.janmejay <singh.janme...@gmail.com>:
>>>>>
>>>>> Similarly, when one thread goes to output the stats, you need to lock
>>>>
>>>> them so that there isn't a lost increment between the time that you read
>>>> the stat and the time you zero it.
>>>>
>>>> No, this involves the same shared (uncontended) lock, except
>>>> atomic-increment is replaced by atomic-swap with 0.
>>>>
>>>
>>> Just FYI: this is what the current stats system also does. It is also
>>> where some inaccuracy stems from. Reporting stats is not atomic
>>> without looks, so a stats counter may be read with value n, then m
>>> atomic increments happen to it on another thread, then value n is
>>> being reported (but we are really at n+m) and then the stats counter
>>> is reset to 0 via an atomic swap. So m updates are lost. IMHO this is
>>> perfectly acceptable, because otherwise we would lose almost all
>>> concurrency.
>>>
>>> Rainer
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
>>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] RFC: dynamic-stats support

[singh.janmejay]

--
Regards,
Janmejay

PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.

On Oct 7, 2015 3:26 AM, "David Lang" <da...@lang.hm> wrote:
>
> On Wed, 7 Oct 2015, singh.janmejay wrote:
>
>> --
>> Regards,
>> Janmejay
>>
>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>> keyboard sporting it's not-so-smart-assist technology.
>>
>> On Oct 6, 2015 10:32 PM, "David Lang" <da...@lang.hm> wrote:
>>>
>>>
>>> On Tue, 6 Oct 2015, singh.janmejay wrote:
>>>
>>>> It is possible to use global-variables (it'll require some
>>>> enhancements, table-support etc), but it'll be very inefficient
>>>> compared to this approach. For instance, choice of data-structure etc
>>>> allows making the solution a lot more efficient.
>>>
>>>
>>>
>>> As for the data structures, Rainer has been identifying inefficencies in
>>
>> how json-c works and working to improve them
>>>
>>>
>>
>> That optimizes variable system. But it still is a general propose
variable
>> system. It can't and shouldn't understand relationship between variables.
>
>
> what relationships are there between the different metrics?
>

The fact that they are read in one shot, reported and reset.

>
>>>
>>>> Here its possible to locklessly increment counters in most cases, so
>>>> its overhead is a lot lesser than global-variables.
>>>
>>>
>>>
>>> how can you manage counters in multiple threads without locks?
Especially
>>
>> when dealing with batches.
>>>
>>>
>>
>> Consider a trie based implementation. With bounded fanout-factor, it's
O(1)
>> wrt metric-names cardinality. It also has very little lock contention
>> involved. Usually operations work with read-locks, only when new metric
is
>> initialized it requires a write lock on patent node. If recycles are few
>> and far apart, lock contention would be negligible.
>
>
> if you have multiple threads that may need to update the same metric at
the same time, a tree doesn't eliminate the locking.
>

The only situation involving a lock that is contended for, is when a metric
is to be initialized. Consider this trie:

A -> B -> C
   -> D

Now for incrementing key ABC, no contention exists, because it involves
read-locks only. It just uses atomic-increment to bump the counter at node
C. Same for ABD.

But ABE will require a write-lock at node B, because node E doesn't exist
yet. However key with not shared parent can again be initialized
concurrently. Init operation can be amortized over a large set of increment
operations making its cost negligible(this is the knob that reset interval
exposes).

> The current json-c locking is being make intentionally over-broad right
now because it appears that some json-c code is not thread-safe and we
haven't identified it yet. Once that's tracked down and fixed (or json-c
replaced), updating one item should not require locking any more than that
item.
>
>
>>>
>>>> Recycle is precisely to allow this lockless mechanism to work. Its
>>>> basically saying, it'll track metric-names he has seen in last 1 hour.
>>>> If we kill tracking of it as soon as we don't see an increment
>>>> (between 2 reporting runs of impstats), it'll lead to unnecessary
>>>> churn when low-values are common or load is not uniform in time.
>>>
>>>
>>>
>>> that depends on the cost of initializing a metric vs the cost of
tracking
>>
>> the recycle mechanism.
>>>
>>>
>>
>> 0 value data-points can easily be filtered out. So they don't create any
>> processing overhead downstream. Cost of tracking for recycle is minimal
>> because it's a single counter bring tracked, when it reaches zero it's
>> reset to orig starting value and trie is killed after reporting
accumulated
>> stats.
>
>
> actually, filtering out 0 data-points can be a very bad thing. Far too
many monitoring tools produce stright-line graphs/estimates between
reported data-points, so it's very important to report 0 value data-points
>

I agree.

>
>>>> Implementing it on top of global-variables is not only has very high
>>>> performance-penalty(it'll be prohibitive for high-throughput
>>>> scenarios), it also exposes too much complexity to the user (where
>>>> user has to worry about reset etc).
>>>>
>>>> I don't plan to have a scheduler in this implementation.
>>&g

Re: [rsyslog] RFC: dynamic-stats support

[singh.janmejay]

It'll support dynamic-key, so any property will work.

On Wed, Oct 7, 2015 at 2:38 PM, chenlin rao <rao.chen...@gmail.com> wrote:
> I hope there is a stats about metrics based on $programname, $severity,
> $fromhost-ip etc, extends the ruleset(impstats).
>
> 2015-10-07 16:19 GMT+08:00 singh.janmejay <singh.janme...@gmail.com>:
>
>> --
>> Regards,
>> Janmejay
>>
>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>> keyboard sporting it's not-so-smart-assist technology.
>>
>> On Oct 7, 2015 3:26 AM, "David Lang" <da...@lang.hm> wrote:
>> >
>> > On Wed, 7 Oct 2015, singh.janmejay wrote:
>> >
>> >> --
>> >> Regards,
>> >> Janmejay
>> >>
>> >> PS: Please blame the typos in this mail on my phone's uncivilized soft
>> >> keyboard sporting it's not-so-smart-assist technology.
>> >>
>> >> On Oct 6, 2015 10:32 PM, "David Lang" <da...@lang.hm> wrote:
>> >>>
>> >>>
>> >>> On Tue, 6 Oct 2015, singh.janmejay wrote:
>> >>>
>> >>>> It is possible to use global-variables (it'll require some
>> >>>> enhancements, table-support etc), but it'll be very inefficient
>> >>>> compared to this approach. For instance, choice of data-structure etc
>> >>>> allows making the solution a lot more efficient.
>> >>>
>> >>>
>> >>>
>> >>> As for the data structures, Rainer has been identifying inefficencies
>> in
>> >>
>> >> how json-c works and working to improve them
>> >>>
>> >>>
>> >>
>> >> That optimizes variable system. But it still is a general propose
>> variable
>> >> system. It can't and shouldn't understand relationship between
>> variables.
>> >
>> >
>> > what relationships are there between the different metrics?
>> >
>>
>> The fact that they are read in one shot, reported and reset.
>>
>> >
>> >>>
>> >>>> Here its possible to locklessly increment counters in most cases, so
>> >>>> its overhead is a lot lesser than global-variables.
>> >>>
>> >>>
>> >>>
>> >>> how can you manage counters in multiple threads without locks?
>> Especially
>> >>
>> >> when dealing with batches.
>> >>>
>> >>>
>> >>
>> >> Consider a trie based implementation. With bounded fanout-factor, it's
>> O(1)
>> >> wrt metric-names cardinality. It also has very little lock contention
>> >> involved. Usually operations work with read-locks, only when new metric
>> is
>> >> initialized it requires a write lock on patent node. If recycles are few
>> >> and far apart, lock contention would be negligible.
>> >
>> >
>> > if you have multiple threads that may need to update the same metric at
>> the same time, a tree doesn't eliminate the locking.
>> >
>>
>> The only situation involving a lock that is contended for, is when a metric
>> is to be initialized. Consider this trie:
>>
>> A -> B -> C
>>-> D
>>
>> Now for incrementing key ABC, no contention exists, because it involves
>> read-locks only. It just uses atomic-increment to bump the counter at node
>> C. Same for ABD.
>>
>> But ABE will require a write-lock at node B, because node E doesn't exist
>> yet. However key with not shared parent can again be initialized
>> concurrently. Init operation can be amortized over a large set of increment
>> operations making its cost negligible(this is the knob that reset interval
>> exposes).
>>
>> > The current json-c locking is being make intentionally over-broad right
>> now because it appears that some json-c code is not thread-safe and we
>> haven't identified it yet. Once that's tracked down and fixed (or json-c
>> replaced), updating one item should not require locking any more than that
>> item.
>> >
>> >
>> >>>
>> >>>> Recycle is precisely to allow this lockless mechanism to work. Its
>> >>>> basically saying, it'll track metric-names he has seen in last 1 hour.
>> >>>> If we kill tracking of it as soon as we don't see an increment
>> >>>> (between 2 reporting runs of impstats), it'll lead to unnecessary
&g

Re: [rsyslog] Separation of actions based on log source - with good performance

[singh.janmejay]

Let us ship a full implementation in 8.14, let us not remove it.

--
Regards,
Janmejay

PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.

On Oct 6, 2015 1:19 PM, "singh.janmejay" <singh.janme...@gmail.com> wrote:

> How about this: I get to it next week and implement all that the proposal
> talks about and document it? It doesn't seem like a lot of work from where
> it is right now.
>
> I vote for not removing it.
>
> --
> Regards,
> Janmejay
>
> PS: Please blame the typos in this mail on my phone's uncivilized soft
> keyboard sporting it's not-so-smart-assist technology.
>
> On Oct 6, 2015 11:53 AM, <christopher.ra...@web.de> wrote:
>
>> Hello,
>> I have never heared such a nonsense.
>> Actually the number of applications that does not include features that
>> are not official documented shoult be extremly limited.
>>
>> The functionality is really usefull and already in big landscapes
>> productive.
>> Please, please do NOT remove the lookup-table from the main branch.
>> The functionaltiy works fine, I'm using this since march and I did not
>> have any issue since the latest patch of janmejay.
>>
>> Even the "concept" is not fully implemented (e.g. smaller things like
>> nomatch) the main part works fine.
>>
>>
>> My suggestion would be to document everything which is currently
>> implemented and keep the "conceptual documentation" as it is.
>> So the Maintainer should no longer have an issue with it.
>>
>>
>> If the main issue it the time to document the already implemented
>> features, I can create a patch.
>>
>>
>> Chris
>>
>>
>>
>> > Gesendet: Dienstag, 06. Oktober 2015 um 07:36 Uhr
>> > Von: "David Lang" <da...@lang.hm>
>> > An: rsyslog-users <rsyslog@lists.adiscon.com>
>> > Betreff: Re: [rsyslog] Separation of actions based on log source - with
>> good performance
>> >
>> > a CVE for something that requires manually enabling an experimental
>> feature???
>> >
>> > it would be one thing if a default config had the problem, or if it was
>> > something entirely dependent on remote data.
>> >
>> > I would be very tempted to respond to the CVE with "don't enable this
>> incomplete
>> > feature" as the solution. It's very common for incomplete features to be
>> > included in released versions
>> >
>> > grumble, we have enough real bugs to worry about.
>> >
>> > David Lang
>> >
>> > On Tue, 6 Oct 2015, Rainer Gerhards wrote:
>> >
>> > > Date: Tue, 6 Oct 2015 07:15:31 +0200
>> > > From: Rainer Gerhards <rgerha...@hq.adiscon.com>
>> > > Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>> > > To: rsyslog-users <rsyslog@lists.adiscon.com>
>> > > Subject: Re: [rsyslog] Separation of actions based on log source -
>> with good
>> > > performance
>> > >
>> > > Sorry, folks, good intent always seems to find someone who turns it
>> > > into negative. I was yesterday contacted by a distro maintainer who
>> > > wants to turn this bug in the officially non-existant lookup table
>> > > feature into a CVE and insists that it is a vuln even after the
>> > > argument that the feature never oficially existed.
>> > >
>> > > It looks like it was a bad idea to merge potentially useful yet
>> > > incomplete code into the main branch (and documenting it to be not
>> > > present). It looks like I need to re-think my stance on experimental
>> > > features.
>> > >
>> > > Anyhow, I really don't want to support the argument that something
>> > > non-existing can be a CVE. As such, I will create a new
>> > > master-insecure branch, which will be a clone of the current master
>> > > branch. Then I'll remove the lookup table code, so that the code base
>> > > matches the documentation. I really don't want to create a general
>> > > principle here that we need to create CVEs (and patched) for something
>> > > that was just added as a convenience for a handful of folks who were
>> > > ready to take a risk.
>> > >
>> > > If there is sufficient interest, we can consider officially adding
>> > > this feature to the January 8.15 release iff it is ready by then.
>> > > @janmejay: please let me know if you

Re: [rsyslog] Separation of actions based on log source - with good performance

[singh.janmejay]

How about this: I get to it next week and implement all that the proposal
talks about and document it? It doesn't seem like a lot of work from where
it is right now.

I vote for not removing it.

--
Regards,
Janmejay

PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.

On Oct 6, 2015 11:53 AM, <christopher.ra...@web.de> wrote:

> Hello,
> I have never heared such a nonsense.
> Actually the number of applications that does not include features that
> are not official documented shoult be extremly limited.
>
> The functionality is really usefull and already in big landscapes
> productive.
> Please, please do NOT remove the lookup-table from the main branch.
> The functionaltiy works fine, I'm using this since march and I did not
> have any issue since the latest patch of janmejay.
>
> Even the "concept" is not fully implemented (e.g. smaller things like
> nomatch) the main part works fine.
>
>
> My suggestion would be to document everything which is currently
> implemented and keep the "conceptual documentation" as it is.
> So the Maintainer should no longer have an issue with it.
>
>
> If the main issue it the time to document the already implemented
> features, I can create a patch.
>
>
> Chris
>
>
>
> > Gesendet: Dienstag, 06. Oktober 2015 um 07:36 Uhr
> > Von: "David Lang" <da...@lang.hm>
> > An: rsyslog-users <rsyslog@lists.adiscon.com>
> > Betreff: Re: [rsyslog] Separation of actions based on log source - with
> good performance
> >
> > a CVE for something that requires manually enabling an experimental
> feature???
> >
> > it would be one thing if a default config had the problem, or if it was
> > something entirely dependent on remote data.
> >
> > I would be very tempted to respond to the CVE with "don't enable this
> incomplete
> > feature" as the solution. It's very common for incomplete features to be
> > included in released versions
> >
> > grumble, we have enough real bugs to worry about.
> >
> > David Lang
> >
> > On Tue, 6 Oct 2015, Rainer Gerhards wrote:
> >
> > > Date: Tue, 6 Oct 2015 07:15:31 +0200
> > > From: Rainer Gerhards <rgerha...@hq.adiscon.com>
> > > Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
> > > To: rsyslog-users <rsyslog@lists.adiscon.com>
> > > Subject: Re: [rsyslog] Separation of actions based on log source -
> with good
> > > performance
> > >
> > > Sorry, folks, good intent always seems to find someone who turns it
> > > into negative. I was yesterday contacted by a distro maintainer who
> > > wants to turn this bug in the officially non-existant lookup table
> > > feature into a CVE and insists that it is a vuln even after the
> > > argument that the feature never oficially existed.
> > >
> > > It looks like it was a bad idea to merge potentially useful yet
> > > incomplete code into the main branch (and documenting it to be not
> > > present). It looks like I need to re-think my stance on experimental
> > > features.
> > >
> > > Anyhow, I really don't want to support the argument that something
> > > non-existing can be a CVE. As such, I will create a new
> > > master-insecure branch, which will be a clone of the current master
> > > branch. Then I'll remove the lookup table code, so that the code base
> > > matches the documentation. I really don't want to create a general
> > > principle here that we need to create CVEs (and patched) for something
> > > that was just added as a convenience for a handful of folks who were
> > > ready to take a risk.
> > >
> > > If there is sufficient interest, we can consider officially adding
> > > this feature to the January 8.15 release iff it is ready by then.
> > > @janmejay: please let me know if you would like to continue with your
> > > work on lookup tables under this new situation.
> > >
> > > As soon as I have time, I'll check what else needs to be removed. Not
> > > sure about the ./contributed branch, because the project cannot
> > > guarantee at all this is bug-free. It's documented to be so, but if
> > > that is not sufficient, it should probably live only in the
> > > master-insecure branch.
> > >
> > > Rainer
> > >
> > > 2015-10-02 17:29 GMT+02:00 singh.janmejay <singh.janme...@gmail.com>:
> > >> As of now it returns empty string for no-match. I