2016-11-23 12:52 GMT+01:00 Bob Gregory <bob.greg...@made.com>: > There've been a few discussions over the last few days that are all > pointing in the same direction: > > * Is it better to use Rsyslog's omelasticsearch rather than pushing to > logstash? > * Should we have a minimal log shipper component as distinct from rsyslog's > processing capabilities? > * Ought we to have an imhiredis module? > > Really what we're talking about is replacing Logstash (and the various > beats) with rsyslog. I'm perfectly happy with that, Logstash is a > resource-expensive and fickle beast that spoils my otherwise pristine log > pipeline, but I do think the community ought to think about whether this is > the direction they want to take. > > For my part, I'm quite happy to help build an imhiredis (and imkafka?) > module but only if I can actually dogfood it, which means replacing > Logstash in our own environment. > > For that, I'd like to see better support for GeoIP tagging, a Riemann > output plugin, some better guidance on "failed message queues", etc. etc. > etc. > > Are we jointly interested in building the REK stack and, if so, can we > start to work out the feature set we're missing, and the documentation we'd > need for this to work? I'm a little concerned that if we tackle the usecase > piece-meal, we'll end up with lots of disjointed parts that don't really > solve the problem: logstash is not an adequate logstash.
I am really extremely interested in this proposal and would appreciate if we could go forward with it. Just let me explain my situation a bit,which hopefully helps to understand how I act and what are my limits. I don't like disappointed people, and so I think talking about limits is essential to get to an agreement. Sorry that the posting is a bit length! I am with Adiscon, and Adiscon still sponsors most of the development for rsyslog. Adiscon is a very small shop (less than 10 folks) and we do have a big budget. That's fine with all of us, as we do not aim at getting rich but aim at having a satisfactory and happy life, which is unequal to being rich in our PoV ;) We still need to pay bills, and so we a) sell closed-source Windows products and b) sell consulting and support contracts. Rsyslog revenue is small, it typically (barely) funds me and half a support engineer. I put in quite a bit of my free time as I am personally interested in this project. Besides rsyslog, I also have some other appointments, for example I am currently working towards two academic research projects, where one is targeted towards logging. Development-wise, this boils down to me being the development ressource, and often not at 100%. If we receive sponsored or custom work, I can add development ressources inside Adiscon, so this actually increases development capability. More important is that Adiscon does not monetize rsyslog in any other way: we do not sell appliances, we do not offer logging as a service and we do not run a large network that we monitor with rsyslog. We really do one thing (development and support for rsyslog) and we do that thing well. Among others, this means we do not have need for Kibana, redis, kafka, ... So we also do not use it. So we do not know it. And learning *everything* just to develop rsyslog is out of reach giving the ressources we have. So far the reality check. The good news is the rsyslog community. It may not be the fastest growing open source community on earth, but it is very healthy and very knowledgable. And we have seen good, quality growth especially in the past two years. We have a lot of different talents, and we have folks that actually use all these subsystems that Adiscon doesn't even know before someone asked a question. As a community, I think we can make the ERK stack a reality. I am very open to changing things, and rsyslog has been refactored more than once since it's inception. Another round is not a problem. If the community helps to shape what actually *needs* to be done (leaving out the "nice to have" to go to a doable workload), and if some folks inside the community help to implement it, I think we can come very far, and can even do so quickly. What is now hopefully obvious from my initial remarks is that I *alone* cannot do all of the big hauling. But again, we had great contributions and we have great contributors! So, yes we can ;-) For example and to be honest, I frankly admit that I didn't know about Riemann until 10 minutes ago. So developing any integration into it will take a lot of time first learning and understanding how it works. This usually is prohibitive expensive for me to do. If, however, we have someone who already knows the ins and outs, we can either work together on getting something done (with me doing the rsyslog bits), or I can educate that person to know the bare minimum required to integrate into rsyslog. Rsyslog integration is not very hard if you do not insist on knowing every detail. And I can fine-tune it afterwards. But it must be a team effort, for any one person, learning the "other part" is probably too time consuming. This is why I mean we need to act as a community. If we can form such (virtual) teams, I would be extremely interested in participating and moving rsyslog forward towards new goals. I think I may even get Adiscon to put in some extra effort for a while. And I personally would find such a community effort uber-cool ;-) What do you think? Sorry again for the long posting, Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
