Logstash-forward/Filebeat just cut logs in preparation for processing elsewhere. It doesn't process logs just forward it to another processor ( Logstash / Heka / Fluentd ). It do not have any processing filter like Logstash. At least, we need some thing tool like grok, syslog intput etc.
what we need is: * listen on syslog like socket to collect logs * processing plugin, like logstash grok does. I do not think fielbeat meet this requirement. So finally, we need <service> -> filebeat ( maybe, log forward ) -> Logstash/heka/Fluentd ( log processing ) -> ES ( log storage ) -> grafana ( log ui ) On Mon, Nov 28, 2016 at 4:45 AM, Steven Dake (stdake) <[email protected]> wrote: > Jeffrey, > > Logstash-forwarder is deprecated upstream, so we can’t rely on that. > Elastic's replacement is filebeat. > > I’m not sure which one meets the requirements – filebeat or fluentd. In > kolla-kubernetes fluentd is being used, and is well maintained. Both > implementations are pretty green IMO. Not sure if fluentd also does log > processing. I think its crucial to pick a component that just does log > forwarding since that is the part that was deprecated. > > Our system has no log stash at all in it, and I’d like to keep it that > way. Logstash is unnecessary for our use case. What we want is > forwarder->es->cabana. Whatever forwarder is chosen, recommend picking the > best of the two choices. I’d start with defining best as “does it solve > the same problem as Heka does in our current implementation” then sprinkle > throughput and minimal cpu and network utilization on top. If we can’t > make a decision from there, not sure I have any further suggestions as I am > not writing the code. > > Regards > -steve > > > From: Jeffrey Zhang <[email protected]> > Reply-To: "OpenStack Development Mailing List (not for usage questions)" < > [email protected]> > Date: Sunday, November 27, 2016 at 9:40 AM > To: "OpenStack Development Mailing List (not for usage questions)" < > [email protected]> > Subject: Re: [openstack-dev] [kolla] the alternative of log processing > tool > > So filebeat is working with Logstash right? We need split the logs into > pieces by using logstash. IMU, Filebeat do not a variety of processing > plugins, like Logstash[0]. > > [0] https://www.elastic.co/guide/en/logstash/current/filter-plugins.html > > On Sun, Nov 27, 2016 at 11:30 PM, Ian Cordasco <[email protected]> > wrote: > >> File beat is maintained be elastic and a part of their product line just >> like ELK. It's a fantastic tool and quite flexible given its age and size >> of codebase >> >> On Nov 26, 2016 11:59 PM, "Jeffrey Zhang" <[email protected]> >> wrote: >> >>> Heka is marked deprecated in Kolla during Newton cycle[0]. And Now we >>> have a >>> blueprint for this[1]. Two alternatives, fluentd[3] and Filebeat. >>> >>> For Filebeat, it is just a replacement of logstash-forward[2]. It is not >>> intent >>> to replace the Logstash at all. >>> >>> > Filebeat is based on the Logstash Forwarder source code and replaces >>> Logstash >>> > Forwarder as the method to use for tailing log files and forwarding >>> them to >>> > Logstash. >>> >>> Fillebeat is a log transport tool rather than log processing too. I do >>> not >>> treat it as an alternative at all. >>> >>> To be honest, I'd like back to Logstash, and Logstash 5.x is released >>> with high >>> performance improvement[4]. >>> >>> > In our performance testing, we've seen consistent throughput increases >>> > across multiple configurations. In some cases, we observed up to 75% >>> > increase in events processed through Logstash. >>> >>> another benefit to using Logstash is the whole ELK stack is maintained >>> by one >>> community/company. It is well tested and easy to upgrade the whole stack >>> at the >>> same time. Using other tools may force us on certain elasticsearch >>> release. >>> >>> So, I think we have to alternative tools. >>> >>> * Fluentd >>> * Logstash >>> >>> IMO, we need to make the decision and at least prepare the migration >>> solution now. >>> >>> [1] https://blueprints.launchpad.net/kolla/+spec/heka-deprecation >>> [2] https://www.elastic.co/guide/en/beats/filebeat/current/migra >>> ting-from-logstash-forwarder.html >>> [3] http://www.fluentd.org/ >>> [4] https://www.elastic.co/blog/logstash-5-0-0-released >>> >>> -- >>> Regards, >>> Jeffrey Zhang >>> Blog: http://xcodest.me >>> >>> ____________________________________________________________ >>> ______________ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: [email protected] >>> enstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >> ____________________________________________________________ >> ______________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: [email protected]?subject:unsubscrib >> e >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > > -- > Regards, > Jeffrey Zhang > Blog: http://xcodest.me > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- Regards, Jeffrey Zhang Blog: http://xcodest.me
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
