Created a Jira for RabbitMQ Appender[1], please implement it if you like.
[1]https://issues.apache.org/jira/browse/AIRAVATA-2145 On Fri, Oct 7, 2016 at 7:05 AM, Lahiru Ginnaliya Gamathige < [email protected]> wrote: > > > On Thu, Oct 6, 2016 at 6:33 AM, Shameera Rathnayaka < > [email protected]> wrote: > >> >> >> On Tue, Oct 4, 2016 at 2:28 PM Lahiru Ginnaliya Gamathige < >> [email protected]> wrote: >> >>> On Tue, Oct 4, 2016 at 11:07 AM, Shameera Rathnayaka < >>> [email protected]> wrote: >>> >>> Hi Lahiru, >>> >>> I didn't see any architectural mail regarding this hence asking, Can you >>> explain your design and deployment strategy here? few follow-up questions, >>> how many kafka producers, partitions and consumers in the system?, why you >>> choose kafka over other messaging systems? >>> >>> One producer in-process with Airavata JVM(if we deploy gfac, >>> orchestrator, airavata-api separate then one produce per each component), >>> consumer is in logstash which reads the logs from preconfigured topics. >>> Once we push to Kafka its totally up the the developers who deploy airavata >>> to use it however they want. Using logstash and ES is just one approach >>> which me and Ajinkya tried and a very common solution people use to >>> visualize their logs coming from multiple components. >>> >> >> How many instances need to deploy your logging solution in productions? >> How Simple/Complex compare to Airavata deployment? >> > These are completely deployment level details which someone needs to > experiment with based on the load of Airavata deployment, for the setup I > tested I used 3 node kafka cluster with 25GB in each and I think its more > than enough but completely depends on the deployment. > >> >> > >>> Regarding why Kafka, I've used similar setup and it never gave me a >>> problem and worked with thousands of logs without giving performance >>> problems or issue with the scale of the logs. I did a demo/presentation >>> about this to the IU folks few weeks back and attached the presentation to >>> Jira[1]. >>> >> >> Just because Kafka works doesn't mean kafka is the best solution for >> Airavata. >> > Yes obvious statement. > >> If you explain little bit the setup you have worked on, that would help >> us to get some ideas. What is the other advantage make Kafka is the best >> one here instead of RabbitMQ (Which we already use in Airavata for >> messaging) ? >> >> Yes, having RabbitMQ appender is also good, so if some user wants to use > RabbitMQ they can use if someone wants Kafka they can use Kafka. I think we > should support Kafka for internal communication too. Having KafkaAppender > will not stop you from implementing more Appenders, there's more > improvements can be done in this area to provide better infrastructure with > Airavata. > >> Regards, >> Shameera. >> >> >>> >>> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065 >>> Regards >>> Lahiru >>> >>> >>> >>> >>> Regards, >>> Shameera. >>> >>> On Tue, Oct 4, 2016 at 1:58 PM Lahiru Ginnaliya Gamathige < >>> [email protected]> wrote: >>> >>> Hi Suresh, >>> >>> If logstash see back-pressure then it stops processing the messages so >>> its always good to have a persistant storage as the input side of the >>> logstash, so logstash can crash anytime and kafka offset works very nicely >>> with the cases where logstash crashes and you just have to restart the >>> logstash and missed logs will be loaded to ES cluster. >>> >>> There's nothing like sending the logs *to* logstash because logstash is >>> simply a data pipeline and not a data store. The other reason is that if we >>> push to kafka we can just handle moving the data to ES with minimum number >>> of logstash (theoretically one logstash) where we have the full >>> configuration with what topics to listen and where to send those logs. >>> >>> Regards >>> Lahiru >>> >>> >>> >>> On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru <[email protected]> wrote: >>> >>> Hi Lahiru, >>> >>> Can you please elaborate on the role of the messaging system (Kafka in >>> your example) vs directly using log stash plugins [1] as an example? >>> >>> Suresh >>> [1] - https://www.elastic.co/guide/en/logstash/current/plugins- >>> inputs-log4j.html >>> >>> On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige < >>> [email protected]> wrote: >>> >>> Hi Devs, >>> >>> I've been working on a feature in Airavata to send its logs to a Kafka >>> topic and finally move them to elastic search using logstash. Please find >>> all the work[1] and the code[2] and I've added a documentation in to wiki >>> [3](since we do not have a development location for wiki). >>> >>> After finishing the issue, Me and Ajinkya did a setup with Airavata and >>> elastic search and Kafka cluster and we were able to visualize the logs in >>> kibana. >>> >>> Please provide your valuable feedback on this feature. >>> >>> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065 >>> [2]https://github.com/apache/airavata/commit/8eea17fb9c8c073 >>> 784795dfc56618ec556a12ad1 >>> [3]https://cwiki.apache.org/confluence/display/AIRAVATA/Tuto >>> rial+09+-+How+to+setup+centralized+logging+for+Airavata >>> >>> >>> Regards >>> Lahiru >>> >>> >>> >>> -- >>> Shameera Rathnayaka >>> >>> -- >> Shameera Rathnayaka >> > >
