-- Regards, Janmejay PS: Please blame the typos in this mail on my phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology.
On Oct 7, 2015 3:26 AM, "David Lang" <[email protected]> wrote: > > On Wed, 7 Oct 2015, singh.janmejay wrote: > >> -- >> Regards, >> Janmejay >> >> PS: Please blame the typos in this mail on my phone's uncivilized soft >> keyboard sporting it's not-so-smart-assist technology. >> >> On Oct 6, 2015 10:32 PM, "David Lang" <[email protected]> wrote: >>> >>> >>> On Tue, 6 Oct 2015, singh.janmejay wrote: >>> >>>> It is possible to use global-variables (it'll require some >>>> enhancements, table-support etc), but it'll be very inefficient >>>> compared to this approach. For instance, choice of data-structure etc >>>> allows making the solution a lot more efficient. >>> >>> >>> >>> As for the data structures, Rainer has been identifying inefficencies in >> >> how json-c works and working to improve them >>> >>> >> >> That optimizes variable system. But it still is a general propose variable >> system. It can't and shouldn't understand relationship between variables. > > > what relationships are there between the different metrics? > The fact that they are read in one shot, reported and reset. > >>> >>>> Here its possible to locklessly increment counters in most cases, so >>>> its overhead is a lot lesser than global-variables. >>> >>> >>> >>> how can you manage counters in multiple threads without locks? Especially >> >> when dealing with batches. >>> >>> >> >> Consider a trie based implementation. With bounded fanout-factor, it's O(1) >> wrt metric-names cardinality. It also has very little lock contention >> involved. Usually operations work with read-locks, only when new metric is >> initialized it requires a write lock on patent node. If recycles are few >> and far apart, lock contention would be negligible. > > > if you have multiple threads that may need to update the same metric at the same time, a tree doesn't eliminate the locking. > The only situation involving a lock that is contended for, is when a metric is to be initialized. Consider this trie: A -> B -> C -> D Now for incrementing key ABC, no contention exists, because it involves read-locks only. It just uses atomic-increment to bump the counter at node C. Same for ABD. But ABE will require a write-lock at node B, because node E doesn't exist yet. However key with not shared parent can again be initialized concurrently. Init operation can be amortized over a large set of increment operations making its cost negligible(this is the knob that reset interval exposes). > The current json-c locking is being make intentionally over-broad right now because it appears that some json-c code is not thread-safe and we haven't identified it yet. Once that's tracked down and fixed (or json-c replaced), updating one item should not require locking any more than that item. > > >>> >>>> Recycle is precisely to allow this lockless mechanism to work. Its >>>> basically saying, it'll track metric-names he has seen in last 1 hour. >>>> If we kill tracking of it as soon as we don't see an increment >>>> (between 2 reporting runs of impstats), it'll lead to unnecessary >>>> churn when low-values are common or load is not uniform in time. >>> >>> >>> >>> that depends on the cost of initializing a metric vs the cost of tracking >> >> the recycle mechanism. >>> >>> >> >> 0 value data-points can easily be filtered out. So they don't create any >> processing overhead downstream. Cost of tracking for recycle is minimal >> because it's a single counter bring tracked, when it reaches zero it's >> reset to orig starting value and trie is killed after reporting accumulated >> stats. > > > actually, filtering out 0 data-points can be a very bad thing. Far too many monitoring tools produce stright-line graphs/estimates between reported data-points, so it's very important to report 0 value data-points > I agree. > >>>> Implementing it on top of global-variables is not only has very high >>>> performance-penalty(it'll be prohibitive for high-throughput >>>> scenarios), it also exposes too much complexity to the user (where >>>> user has to worry about reset etc). >>>> >>>> I don't plan to have a scheduler in this implementation. >>>> GetAllStatsLines call will purge the tree instead of reset at that >>>> interval. Its basically a balance between freeing-up memory occupied >>>> by stale-metric-names vs. performance (lockless handling of >>>> increment). So it will be governed by impstat schedule. May be I >>>> should change name to better name (equivalent of >>>> purge_known_keys_after_they_have_been_reported_N_times). >>> >>> >>> >>> if this is just adding additional metrics to the impstats output that >> >> eliminates the schedular/reset issue. >>> >>> >>> I think we should have a metric configuration be fairly static, allow >> >> configuring custom metrics and add to them, but don't use data from the >> message as part of the name of the metric, and continue reporting them >> forever, even if they are 0 (so no need to 'recycle' names) >> >> Dynamic metrics are a real usecase for any shared system(utilisation across >> several users, several hosts, several clusters, several-subnets etc are >> easily reportable with this). The only way to report utilisation in many >> scenarios is to have dyn-metric names. The alternative is to pre-declare >> all keys, but that to me is a more indirect solution. It's not as >> flexible/adaptive. >> >> I think declarative static-key a useful feature on its own, for eg when >> classifying reportable metric into buckets known in advance, but dyn-key >> and configurable-static-key are not interchangeable. > > > dynamic systems will have pathalogical failure condiditions. Consider what happens when someone uses hostname in a dynafile template and then some system starts spewing malformed logs that put garbage data in that field. Creating hundreds or thousands of metric variables is much worse. The max-cardinality optional field in dyn-metric-namespace declaration was exactly to prevent this kind of unbounded growth. We can choose a sensible default. > > I agree that pre-declared keys are less flexible, but they are also going to be far safer and easier to deal with. > > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
