Yes, it's a slide nearly 1.5 years ago. After that, we: - change to use omkafka + <https://github.com/childe/hangout> instead of omelasticsearch directly. The reason I have said in another mail days ago. - rewrite most of mmgrok into mmnormalize+rainerscript. Except PHP slowlog only. We want to translate the memory address of each line into "xxxxx", but seems can't be done in rsyslog, so a mmexternal here. - try to use streaming compress with imptcp (between shipper and rsyslog server), the bandwidth saved about 2/3. But discard msgs in peak every night. So roll back.
No experiments about ES5 now. The author of hangout above told me ES5.0.0 has some terrible problems( https://github.com/elastic/elasticsearch/issues/21612 https://github.com/elastic/elasticsearch/issues/21611), so waiting for upgrade. 2016-11-25 14:58 GMT+08:00 David Lang <[email protected]>: > reading through the slides, a couple comments. > > I've found that queue type FixedArray is slightly (but measureably) faster > than LinkedList > > I suspect that the problems you were running into with slide 52 were the > json-c threading problems that have now been solved with libjsonfast > > I'd be very interested in seeing speed comparisons between lookuptable and > your mmdblookup > > At your log volumes, I expect that creating a sting module (sm*, C version > of a template definition) would make a noticable performance difference. We > saw >10% when we changed the default templates to C definitions. > > It's a very useful slide deck. How has the 5.x version of ES changed > things there. > > David Lang > > On Fri, 25 Nov 2016, chenlin rao wrote: > > Date: Fri, 25 Nov 2016 10:46:27 +0800 >> From: chenlin rao <[email protected]> >> Reply-To: rsyslog-users <[email protected]> >> To: rsyslog-users <[email protected]> >> Subject: Re: [rsyslog] Are we building an ERK stack? >> >> >> re-upload an english version. The content was a little old though. >> >> 2016-11-23 22:39 GMT+08:00 [email protected] <[email protected]>: >> >> >>> http://www.slideshare.net/chenryn/elk-stack-at-weibocom >>> >>> I NEED the english version :P >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
