Hi Suresh, If logstash see back-pressure then it stops processing the messages so its always good to have a persistant storage as the input side of the logstash, so logstash can crash anytime and kafka offset works very nicely with the cases where logstash crashes and you just have to restart the logstash and missed logs will be loaded to ES cluster.
There's nothing like sending the logs *to* logstash because logstash is simply a data pipeline and not a data store. The other reason is that if we push to kafka we can just handle moving the data to ES with minimum number of logstash (theoretically one logstash) where we have the full configuration with what topics to listen and where to send those logs. Regards Lahiru On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru <[email protected]> wrote: > Hi Lahiru, > > Can you please elaborate on the role of the messaging system (Kafka in > your example) vs directly using log stash plugins [1] as an example? > > Suresh > [1] - https://www.elastic.co/guide/en/logstash/current/ > plugins-inputs-log4j.html > > On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige <[email protected]> > wrote: > > Hi Devs, > > I've been working on a feature in Airavata to send its logs to a Kafka > topic and finally move them to elastic search using logstash. Please find > all the work[1] and the code[2] and I've added a documentation in to wiki > [3](since we do not have a development location for wiki). > > After finishing the issue, Me and Ajinkya did a setup with Airavata and > elastic search and Kafka cluster and we were able to visualize the logs in > kibana. > > Please provide your valuable feedback on this feature. > > [1]https://issues.apache.org/jira/browse/AIRAVATA-2065 > [2]https://github.com/apache/airavata/commit/ > 8eea17fb9c8c073784795dfc56618ec556a12ad1 > [3]https://cwiki.apache.org/confluence/display/AIRAVATA/ > Tutorial+09+-+How+to+setup+centralized+logging+for+Airavata > > > Regards > Lahiru > > >
