On Fri, Oct 7, 2016 at 10:15 AM Lahiru Ginnaliya Gamathige < [email protected]> wrote:
> Created a Jira for RabbitMQ Appender[1], please implement it if you like. > It is always good to plug differnt implementations, If I am going to implement a new logging appender(RabbitMQAppender or any other). How I can plug it to Airavata? Can we easily plug differnt implementations? It seem you have code agains the kafka logging implementation instead of an Interface, correct me if I am wrong. I think if you can provide better solution to plug different appenders this task would be more easier. > > > [1]https://issues.apache.org/jira/browse/AIRAVATA-2145 > > On Fri, Oct 7, 2016 at 7:05 AM, Lahiru Ginnaliya Gamathige < > [email protected]> wrote: > > > > On Thu, Oct 6, 2016 at 6:33 AM, Shameera Rathnayaka < > [email protected]> wrote: > > > > On Tue, Oct 4, 2016 at 2:28 PM Lahiru Ginnaliya Gamathige < > [email protected]> wrote: > > On Tue, Oct 4, 2016 at 11:07 AM, Shameera Rathnayaka < > [email protected]> wrote: > > Hi Lahiru, > > I didn't see any architectural mail regarding this hence asking, Can you > explain your design and deployment strategy here? few follow-up questions, > how many kafka producers, partitions and consumers in the system?, why you > choose kafka over other messaging systems? > > One producer in-process with Airavata JVM(if we deploy gfac, orchestrator, > airavata-api separate then one produce per each component), consumer is in > logstash which reads the logs from preconfigured topics. Once we push to > Kafka its totally up the the developers who deploy airavata to use it > however they want. Using logstash and ES is just one approach which me and > Ajinkya tried and a very common solution people use to visualize their logs > coming from multiple components. > > > How many instances need to deploy your logging solution in productions? > How Simple/Complex compare to Airavata deployment? > > These are completely deployment level details which someone needs to > experiment with based on the load of Airavata deployment, for the setup I > tested I used 3 node kafka cluster with 25GB in each and I think its more > than enough but completely depends on the deployment. > > > > > Regarding why Kafka, I've used similar setup and it never gave me a > problem and worked with thousands of logs without giving performance > problems or issue with the scale of the logs. I did a demo/presentation > about this to the IU folks few weeks back and attached the presentation to > Jira[1]. > > > Just because Kafka works doesn't mean kafka is the best solution for > Airavata. > > Yes obvious statement. > > If you explain little bit the setup you have worked on, that would help us > to get some ideas. What is the other advantage make Kafka is the best one > here instead of RabbitMQ (Which we already use in Airavata for messaging) ? > > Yes, having RabbitMQ appender is also good, so if some user wants to use > RabbitMQ they can use if someone wants Kafka they can use Kafka. I think we > should support Kafka for internal communication too. Having KafkaAppender > will not stop you from implementing more Appenders, there's more > improvements can be done in this area to provide better infrastructure with > Airavata. > > Regards, > Shameera. > > > > [1]https://issues.apache.org/jira/browse/AIRAVATA-2065 > Regards > Lahiru > > > > > Regards, > Shameera. > > On Tue, Oct 4, 2016 at 1:58 PM Lahiru Ginnaliya Gamathige < > [email protected]> wrote: > > Hi Suresh, > > If logstash see back-pressure then it stops processing the messages so its > always good to have a persistant storage as the input side of the logstash, > so logstash can crash anytime and kafka offset works very nicely with the > cases where logstash crashes and you just have to restart the logstash and > missed logs will be loaded to ES cluster. > > There's nothing like sending the logs *to* logstash because logstash is > simply a data pipeline and not a data store. The other reason is that if we > push to kafka we can just handle moving the data to ES with minimum number > of logstash (theoretically one logstash) where we have the full > configuration with what topics to listen and where to send those logs. > > Regards > Lahiru > > > > On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru <[email protected]> wrote: > > Hi Lahiru, > > Can you please elaborate on the role of the messaging system (Kafka in > your example) vs directly using log stash plugins [1] as an example? > > Suresh > [1] - > https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html > > On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige <[email protected]> > wrote: > > Hi Devs, > > I've been working on a feature in Airavata to send its logs to a Kafka > topic and finally move them to elastic search using logstash. Please find > all the work[1] and the code[2] and I've added a documentation in to wiki > [3](since we do not have a development location for wiki). > > After finishing the issue, Me and Ajinkya did a setup with Airavata and > elastic search and Kafka cluster and we were able to visualize the logs in > kibana. > > Please provide your valuable feedback on this feature. > > [1]https://issues.apache.org/jira/browse/AIRAVATA-2065 > [2] > https://github.com/apache/airavata/commit/8eea17fb9c8c073784795dfc56618ec556a12ad1 > [3] > https://cwiki.apache.org/confluence/display/AIRAVATA/Tutorial+09+-+How+to+setup+centralized+logging+for+Airavata > > > Regards > Lahiru > > > > -- > Shameera Rathnayaka > > -- > Shameera Rathnayaka > > > > -- Shameera Rathnayaka
