On Thu, Oct 6, 2016 at 6:33 AM, Shameera Rathnayaka <[email protected]> wrote:
> > > On Tue, Oct 4, 2016 at 2:28 PM Lahiru Ginnaliya Gamathige < > [email protected]> wrote: > >> On Tue, Oct 4, 2016 at 11:07 AM, Shameera Rathnayaka < >> [email protected]> wrote: >> >> Hi Lahiru, >> >> I didn't see any architectural mail regarding this hence asking, Can you >> explain your design and deployment strategy here? few follow-up questions, >> how many kafka producers, partitions and consumers in the system?, why you >> choose kafka over other messaging systems? >> >> One producer in-process with Airavata JVM(if we deploy gfac, >> orchestrator, airavata-api separate then one produce per each component), >> consumer is in logstash which reads the logs from preconfigured topics. >> Once we push to Kafka its totally up the the developers who deploy airavata >> to use it however they want. Using logstash and ES is just one approach >> which me and Ajinkya tried and a very common solution people use to >> visualize their logs coming from multiple components. >> > > How many instances need to deploy your logging solution in productions? > How Simple/Complex compare to Airavata deployment? > These are completely deployment level details which someone needs to experiment with based on the load of Airavata deployment, for the setup I tested I used 3 node kafka cluster with 25GB in each and I think its more than enough but completely depends on the deployment. > > >> Regarding why Kafka, I've used similar setup and it never gave me a >> problem and worked with thousands of logs without giving performance >> problems or issue with the scale of the logs. I did a demo/presentation >> about this to the IU folks few weeks back and attached the presentation to >> Jira[1]. >> > > Just because Kafka works doesn't mean kafka is the best solution for > Airavata. > Yes obvious statement. > If you explain little bit the setup you have worked on, that would help us > to get some ideas. What is the other advantage make Kafka is the best one > here instead of RabbitMQ (Which we already use in Airavata for messaging) ? > > Yes, having RabbitMQ appender is also good, so if some user wants to use RabbitMQ they can use if someone wants Kafka they can use Kafka. I think we should support Kafka for internal communication too. Having KafkaAppender will not stop you from implementing more Appenders, there's more improvements can be done in this area to provide better infrastructure with Airavata. > Regards, > Shameera. > > >> >> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065 >> Regards >> Lahiru >> >> >> >> >> Regards, >> Shameera. >> >> On Tue, Oct 4, 2016 at 1:58 PM Lahiru Ginnaliya Gamathige < >> [email protected]> wrote: >> >> Hi Suresh, >> >> If logstash see back-pressure then it stops processing the messages so >> its always good to have a persistant storage as the input side of the >> logstash, so logstash can crash anytime and kafka offset works very nicely >> with the cases where logstash crashes and you just have to restart the >> logstash and missed logs will be loaded to ES cluster. >> >> There's nothing like sending the logs *to* logstash because logstash is >> simply a data pipeline and not a data store. The other reason is that if we >> push to kafka we can just handle moving the data to ES with minimum number >> of logstash (theoretically one logstash) where we have the full >> configuration with what topics to listen and where to send those logs. >> >> Regards >> Lahiru >> >> >> >> On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru <[email protected]> wrote: >> >> Hi Lahiru, >> >> Can you please elaborate on the role of the messaging system (Kafka in >> your example) vs directly using log stash plugins [1] as an example? >> >> Suresh >> [1] - https://www.elastic.co/guide/en/logstash/current/ >> plugins-inputs-log4j.html >> >> On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige < >> [email protected]> wrote: >> >> Hi Devs, >> >> I've been working on a feature in Airavata to send its logs to a Kafka >> topic and finally move them to elastic search using logstash. Please find >> all the work[1] and the code[2] and I've added a documentation in to wiki >> [3](since we do not have a development location for wiki). >> >> After finishing the issue, Me and Ajinkya did a setup with Airavata and >> elastic search and Kafka cluster and we were able to visualize the logs in >> kibana. >> >> Please provide your valuable feedback on this feature. >> >> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065 >> [2]https://github.com/apache/airavata/commit/ >> 8eea17fb9c8c073784795dfc56618ec556a12ad1 >> [3]https://cwiki.apache.org/confluence/display/AIRAVATA/ >> Tutorial+09+-+How+to+setup+centralized+logging+for+Airavata >> >> >> Regards >> Lahiru >> >> >> >> -- >> Shameera Rathnayaka >> >> -- > Shameera Rathnayaka >
