-- Regards, Janmejay PS: Please blame the typos in this mail on my phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology.
On Oct 7, 2015 11:25 PM, "David Lang" <da...@lang.hm> wrote: > > On Wed, 7 Oct 2015, singh.janmejay wrote: > >> -- >> Regards, >> Janmejay >> >> PS: Please blame the typos in this mail on my phone's uncivilized soft >> keyboard sporting it's not-so-smart-assist technology. >> >> On Oct 7, 2015 3:26 AM, "David Lang" <da...@lang.hm> wrote: >>> >>> >>> On Wed, 7 Oct 2015, singh.janmejay wrote: >>> >>>> -- >>>> Regards, >>>> Janmejay >>>> >>>> PS: Please blame the typos in this mail on my phone's uncivilized soft >>>> keyboard sporting it's not-so-smart-assist technology. >>>> >>>> On Oct 6, 2015 10:32 PM, "David Lang" <da...@lang.hm> wrote: >>>>> >>>>> >>>>> >>>>> On Tue, 6 Oct 2015, singh.janmejay wrote: >>>>> >>>>>> It is possible to use global-variables (it'll require some >>>>>> enhancements, table-support etc), but it'll be very inefficient >>>>>> compared to this approach. For instance, choice of data-structure etc >>>>>> allows making the solution a lot more efficient. >>>>> >>>>> >>>>> >>>>> >>>>> As for the data structures, Rainer has been identifying inefficencies in >>>> >>>> >>>> how json-c works and working to improve them >>>>> >>>>> >>>>> >>>> >>>> That optimizes variable system. But it still is a general propose >> >> variable >>>> >>>> system. It can't and shouldn't understand relationship between variables. >>> >>> >>> >>> what relationships are there between the different metrics? >>> >> >> The fact that they are read in one shot, reported and reset. > > > I don't understand why you couldn't do $\stats!metric1, $\stats!metric2, etc. There is no reason why this couldn't have the same locking as your new structure. > If this is built over variable backend, it has to be global variables(which is what you are suggesting). But consider the implementation of this table ($/stats). Any efficient indexing (hash or tree based data structures), will require locks for rebalance/re-hash operations. But even counter increment operations need to contend for the lock because of the potential of concurrent operations that change the table. It can't have an assured uncontended path except for initial opportunistic wide shared lock which allows checking existence of key. But the scope of lock and contention due to it is wider. Also, this involves technical intricacies that all users may not be comfortable with. It's asking them to build the stats system as opposed to providing them a ready-made one. In this case it is without any loss in generality or any sacrifice in capability of the feature. I like it because it's easier to use. > >>> >>>>> >>>>>> Here its possible to locklessly increment counters in most cases, so >>>>>> its overhead is a lot lesser than global-variables. >>>>> >>>>> >>>>> >>>>> >>>>> how can you manage counters in multiple threads without locks? >> >> Especially >>>> >>>> >>>> when dealing with batches. >>>>> >>>>> >>>>> >>>> >>>> Consider a trie based implementation. With bounded fanout-factor, it's >> >> O(1) >>>> >>>> wrt metric-names cardinality. It also has very little lock contention >>>> involved. Usually operations work with read-locks, only when new metric >> >> is >>>> >>>> initialized it requires a write lock on patent node. If recycles are few >>>> and far apart, lock contention would be negligible. >>> >>> >>> >>> if you have multiple threads that may need to update the same metric at >> >> the same time, a tree doesn't eliminate the locking. >>> >>> >> >> The only situation involving a lock that is contended for, is when a metric >> is to be initialized. Consider this trie: >> >> A -> B -> C >> -> D >> >> Now for incrementing key ABC, no contention exists, because it involves >> read-locks only. It just uses atomic-increment to bump the counter at node >> C. Same for ABD. >> >> But ABE will require a write-lock at node B, because node E doesn't exist >> yet. However key with not shared parent can again be initialized >> concurrently. Init operation can be amortized over a large set of increment >> operations making its cost negligible(this is the knob that reset interval >> exposes). > > > you are misunderstanding me > > If you have two threads that both want to change the counter at node C, you have to have locking to keep them from having problems. > Assuming the nodes A, B and C already exist, here is the seq of op: Shared-lock A (uncontended) Shared-lock B (uncontended) Shared-lock C (uncontended) Atomic-increment counter of C Shared-lock release C Shared-lock release B Shared-lock release A This sequence can be run concurrently without any additional locks. The other (much less frequent) situation is when metric doesn't exist, in which case it'll do: Shared-lock A (uncontended) Exclusive-lock B (contended) Create C Increment counter of C Exclysive-lock release B Shared-lock release A > Similarly, when one thread goes to output the stats, you need to lock them so that there isn't a lost increment between the time that you read the stat and the time you zero it. No, this involves the same shared (uncontended) lock, except atomic-increment is replaced by atomic-swap with 0. > > That's the locking that's expensive. > > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. -- Regards, Janmejay PS: Please blame the typos in this mail on my phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology. -- Regards, Janmejay PS: Please blame the typos in this mail on my phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
