On Tue, 6 Oct 2015, singh.janmejay wrote:
It is possible to use global-variables (it'll require some enhancements, table-support etc), but it'll be very inefficient compared to this approach. For instance, choice of data-structure etc allows making the solution a lot more efficient.
As for the data structures, Rainer has been identifying inefficencies in how json-c works and working to improve them
Here its possible to locklessly increment counters in most cases, so its overhead is a lot lesser than global-variables.
how can you manage counters in multiple threads without locks? Especially when dealing with batches.
Recycle is precisely to allow this lockless mechanism to work. Its basically saying, it'll track metric-names he has seen in last 1 hour. If we kill tracking of it as soon as we don't see an increment (between 2 reporting runs of impstats), it'll lead to unnecessary churn when low-values are common or load is not uniform in time.
that depends on the cost of initializing a metric vs the cost of tracking the recycle mechanism.
Implementing it on top of global-variables is not only has very high performance-penalty(it'll be prohibitive for high-throughput scenarios), it also exposes too much complexity to the user (where user has to worry about reset etc). I don't plan to have a scheduler in this implementation. GetAllStatsLines call will purge the tree instead of reset at that interval. Its basically a balance between freeing-up memory occupied by stale-metric-names vs. performance (lockless handling of increment). So it will be governed by impstat schedule. May be I should change name to better name (equivalent of purge_known_keys_after_they_have_been_reported_N_times).
if this is just adding additional metrics to the impstats output that eliminates the schedular/reset issue.
I think we should have a metric configuration be fairly static, allow configuring custom metrics and add to them, but don't use data from the message as part of the name of the metric, and continue reporting them forever, even if they are 0 (so no need to 'recycle' names)
David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
