I can't fully dig into this, but I think we must *very carefully* evaluate the overall design. Some time ago we introduced arrays for the limited liblognorm use case, and it hurts us every now and then when folks want to use arrays for other use cases. It may probably make sense to re-think how the variable engine etc behaves before adding more functionality. And make sure that everything works smooth in all use cases. While anything else may take care for some use cases, I fear we may get too fragmented. At least this is what I learned in the past months discussions.
Anyone else? Rainer 2015-10-06 17:10 GMT+02:00 singh.janmejay <[email protected]>: > It is possible to use global-variables (it'll require some > enhancements, table-support etc), but it'll be very inefficient > compared to this approach. For instance, choice of data-structure etc > allows making the solution a lot more efficient. > > Here its possible to locklessly increment counters in most cases, so > its overhead is a lot lesser than global-variables. > > Recycle is precisely to allow this lockless mechanism to work. Its > basically saying, it'll track metric-names he has seen in last 1 hour. > If we kill tracking of it as soon as we don't see an increment > (between 2 reporting runs of impstats), it'll lead to unnecessary > churn when low-values are common or load is not uniform in time. > > Implementing it on top of global-variables is not only has very high > performance-penalty(it'll be prohibitive for high-throughput > scenarios), it also exposes too much complexity to the user (where > user has to worry about reset etc). > > I don't plan to have a scheduler in this implementation. > GetAllStatsLines call will purge the tree instead of reset at that > interval. Its basically a balance between freeing-up memory occupied > by stale-metric-names vs. performance (lockless handling of > increment). So it will be governed by impstat schedule. May be I > should change name to better name (equivalent of > purge_known_keys_after_they_have_been_reported_N_times). > > > On Tue, Oct 6, 2015 at 4:30 PM, David Lang <[email protected]> wrote: >> On Tue, 6 Oct 2015, singh.janmejay wrote: >> >>> Hi, >>> >>> I am working on support for stats with dynamic-name. This comes handy >>> in situations where metric-name is dependent upon value of a certain >>> attribute of the message. >>> >>> Say, for a central log-aggregation service, its valuable to know what >>> is inbound message-count distribution across application-clusters that >>> send logs to it, or for a shared-server, its valuable to know what is >>> the log-volume generation across users etc. >>> >>> Im thinking of using functions-like interface to support this. It may >>> look similar to this: >>> >>> ==================== >>> dyn_stats("user_msg_count") >>> >>> ... >>> >>> ruleset(...) { >>> ... >>> dyn_inc("user_msg_count", $.user) >>> ... >>> } >>> ==================== >>> >>> dyn_stats signature looks like: >>> dyn_stats(<name_space>, <resettable: default=true>, <max_cardinality: >>> default=10k>, <recycle_metric_names_after: default=1hr>) >>> >>> dyn_inc signature looks like: >>> dyn_inc(<name_space>, <metric_name>) >>> >>> >>> Reporting would work similar to static-metric via impstats. Mapping: >>> statsobj_s.name = name_space >>> statsobj_s.origin = "dyn" >>> ctr_s.name = "foo" (say $.user had value foo) >>> >>> >>> Thoughts / suggestions? >> >> >> how is this different/better than global variables? (although we may need to >> implement soem functions, atomic inc/dec copy+clear) If you have pstats >> output in json format, you can even piggyback on it's schedule to output the >> data. >> >> >> things like stats can very easily end up being expensive in terms of locking >> (something global variables already have figured out), and it sounds like >> you are proposing adding a scheduler of some sort to output the data. >> >> variables should not need to be 'recycled', either they contain data or they >> don't. If they contain data, you need to keep the data until you do >> something with it, if they don't, you don't have to track them. >> >> >> I am actually doing this sort of thing external to rsyslog in SEC >> >> I have a template in rsyslog that contains hostname, fromhost-ip, >> programname and I output it via improg to SEC. SEC accumulates counters and >> has scheduled outputs to files. >> >> before I started using SEC for this, I used the same template to output to a >> file and then for reports, used cut + sort + uniq -c to extract the data I >> need. When the files only contain the significant data, this is actually not >> bad to do, even at higher volumes. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. > > > > -- > Regards, > Janmejay > http://codehunk.wordpress.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
