Hi Lahiru, I didn't see any architectural mail regarding this hence asking, Can you explain your design and deployment strategy here? few follow-up questions, how many kafka producers, partitions and consumers in the system?, why you choose kafka over other messaging systems?
Regards, Shameera. On Tue, Oct 4, 2016 at 1:58 PM Lahiru Ginnaliya Gamathige <[email protected]> wrote: > Hi Suresh, > > If logstash see back-pressure then it stops processing the messages so its > always good to have a persistant storage as the input side of the logstash, > so logstash can crash anytime and kafka offset works very nicely with the > cases where logstash crashes and you just have to restart the logstash and > missed logs will be loaded to ES cluster. > > There's nothing like sending the logs *to* logstash because logstash is > simply a data pipeline and not a data store. The other reason is that if we > push to kafka we can just handle moving the data to ES with minimum number > of logstash (theoretically one logstash) where we have the full > configuration with what topics to listen and where to send those logs. > > Regards > Lahiru > > > > On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru <[email protected]> wrote: > > Hi Lahiru, > > Can you please elaborate on the role of the messaging system (Kafka in > your example) vs directly using log stash plugins [1] as an example? > > Suresh > [1] - > https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html > > On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige <[email protected]> > wrote: > > Hi Devs, > > I've been working on a feature in Airavata to send its logs to a Kafka > topic and finally move them to elastic search using logstash. Please find > all the work[1] and the code[2] and I've added a documentation in to wiki > [3](since we do not have a development location for wiki). > > After finishing the issue, Me and Ajinkya did a setup with Airavata and > elastic search and Kafka cluster and we were able to visualize the logs in > kibana. > > Please provide your valuable feedback on this feature. > > [1]https://issues.apache.org/jira/browse/AIRAVATA-2065 > [2] > https://github.com/apache/airavata/commit/8eea17fb9c8c073784795dfc56618ec556a12ad1 > [3] > https://cwiki.apache.org/confluence/display/AIRAVATA/Tutorial+09+-+How+to+setup+centralized+logging+for+Airavata > > > Regards > Lahiru > > > > -- Shameera Rathnayaka
