On Sun, Dec 15, 2013 at 7:42 AM, Otis Gospodnetic < [email protected]> wrote:
> Hi, > > Thanks for the info. > I was asking because having the ability to write ims and oms in different > languages would open a lot of opportunities. This is one of those > "enablement" things. I understand writing modules in other languages may > mean those using such modules may hurt performance, but some people need > certain functionality more than performance. > yeah, I agree. And if it becomes popular, that would be an indication that investing to write a native C implementation makes sense. So it would actually be good to do PoC for less-requested things. > > Take omkafka example from the other day. If there were a way to write an > om in Java it's be trivial for a lot of Java developers out there to > contribute omkafka. > > If it's trivial, is there somebody who would like to do it? That would also enable us to see limits into which the current omprog implementation runs -- and get those fixed. As I said, everything is already in place... > If omprog enables development of the ecosystem, it sounds like something to > point out clearly somewhere and nurture that a bit. I do see > http://www.rsyslog.com/doc/omprog.html because somebody shared a link, but > I don't see that on http://www.rsyslog.com/doc or on > http://www.rsyslog.com/doc/dev_oplugins.html or in the new README. > You are right. I frankly admit that it never before occured to me to think of this as an interface to other languages. I am pretty focussed on performance, and everything that's not fast enough I don't (didn't) consider as a solution. Again, there are many more talents required inside a project than software engineering, and rsyslog is missing lots of other talents. I am glad for everyone who brings in some of the missing ones! > > Coincidentally, I just came across Fluentd's instructions for writing > plugins, which could serve as guidance: > http://docs.fluentd.org/articles/plugin-development . Nice, clean, well > structured, not a lot of prose... > > Will look at it. But again, I am not the greatest writer. While I can write small code, "not a lot of prose" poses a problem to me in doc. I always think of the remote border cases that could happen... That's one of the reasons why I *really* think that the doc project is a separate entity with separate folks working on it... Thanks again for all the help, Rainer > Otis > -- > Performance Monitoring * Log Analytics * Search Analytics > Solr & Elasticsearch Support * http://sematext.com/ > > > On Sat, Dec 14, 2013 at 1:39 PM, David Lang <[email protected]> wrote: > > > On Sat, 14 Dec 2013, RB wrote: > > > > On Sat, Dec 14, 2013 at 5:24 AM, Rainer Gerhards > >> <[email protected]> wrote: > >> > >>> well, technically it's for sure possible, it's just another of these > 24h > >>> things. Technically, it's a question of interface, and insofar of which > >>> types of modules. Obviously, these will be slower, and how slow is > >>> another > >>> interface/effort question. > >>> > >>> Thinking about this, one could probably also claim the answer is "yes, > >>> you > >>> can write OUTPUT modules in any language", it's just a doc issue. In > >>> fact, > >>> omprog can be used as an interface here. It's actually not even a bad > >>> interface... > >>> > >>> Again, something learned ;) > >>> > >> > >> Probably the cheapest (implementation) "binding" for rsyslog would be > >> a system() like call. Execute the subprogram with /bin/sh -c and > >> communicate with structured messages on STDIO. > >> > > > > a real module binding would be far more complex. It would allow the > module > > (in whatever language) access to the rsyslog queues and other data > > structures. This is possible, but not easy by any means. > > > > One big problem is that currently rsyslog does all this work in a > threaded > > environment. It may make sense in v9 or v10 to shift from a default > > shared-everything threading model to a explicit shared memory > multiprocess > > model. At that point having one of the processes use a different language > > would not be that hard. > > > > But in the current threaded model, having one thread run a different > > language would be very, very hard. > > > > The other issue here is performance. Rsyslog goes to a LOT of effort to > be > > fast. Some of the things that have made very noticable diffences in > > performance in rsyslog are things that seem like they should be very > minor. > > Think about these things and then think about what would be involved to > > define interfaces in a multi-language safe way. > > > > things that have resulted in noticable speedups have been: > > > > removing gettimeofday() calls. > > > > it used to be that rsyslog recorded when a message arrived, when it was > > put on the main queue, when it was moved to an action queue, when it was > > pulled from the action queue, and when it was delivered > > > > now, high performance users configure rsyslog so that it only does one > > gettimeofday() call per hundred (ot thousand) messages that arrive and > use > > that one time for every message > > > > string modules > > > > it used to be that the default template (<%pri%>%timestamp% %hostname% > > %syslogtag%%msg%) was interpreted by the rsyslog engine for every message > > that was output > > > > now string modules written in C create these strings rather than > > interpreting the template. This resulted in a double-digit % performance > > improvement > > > > With optimizations like these in use, changing things to allow for a > > module written in a different language to have access to the rsyslog > > internals as would be needed for a high-performance interface seems like > it > > will probably end up hurting the rsyslog performance overall. > > > > > > That being said, I am very much in favor of multi-process with explicit > > sharing rather than multi-threaded with implicit sharing, but getting all > > the interfaces correct and fast would be a VERY hard task. > > > > David Lang > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
