Hi Srinath, We need Elasticsearch server to store the metrics data. Kibana is just a client side application to visualize data.
Kibana & Grafana, both have feature to select data and visualize as we want. We can have our own visualizations and it support many panels [1]. I'm not sure whether we can extend it with additional panel types. When running Elasticsearch in a standalone application, with another application sending continuous metrics data, I observe 0-20% CPU usage and ~120MB Heap Usage. I also configured to use file system for data. This is why I meant this as a temporary approach. I will discuss with Chanaka and show this prototype. Thanks! Best Regards, [1] http://www.elasticsearch.org/guide/en/kibana/current/panels.html On Tue, Jan 13, 2015 at 6:00 AM, Srinath Perera <[email protected]> wrote: > Hi Isuru, > > Why do we need elastic server? > What is the footprint of it? In Memory and CPU wise. (check that running > it standalone)? > Does Kibana let us choose a metric and visualize? can we extend it with > our own visualisations? > > Could you show and chat with Chanaka as well. > > --Srinath > > On Mon, Jan 12, 2015 at 8:27 PM, Cyril Rognon <[email protected]> > wrote: > >> Hi Isuru, >> >> using ES+Kibana is a very good idea. We also are on the way to provide >> some kibana dashboard to the devops of one of our client. >> Why would you need some embedded ElasticSearch ? is it a security issue ? >> >> For the architecture list, the elastic suite is very nice and could >> provide a nice solution to provide rich dashboards. It would be very simple >> to target ES as a stream store to benefit from kibana time-series >> dashboard. We are thinking of using the complete ELK stack (adding logtash >> as well) to provide easy admin stats dashboard to multiple wso2 product, >> gaining the fact is that it can also aggregate data from other sources than >> wos2 stat (using logtash to gather other logs). >> >> The apache licence of the whole products is something that makes it >> possible. >> >> Cheers, >> Cyril >> >> >> >>> ---------- Forwarded message ---------- >>> From: Isuru Perera <[email protected]> >>> Date: Mon, Jan 12, 2015 at 2:18 PM >>> Subject: [Architecture] Metrics Visualization: Using Kibana & >>> Elasticsearch >>> To: Srinath Perera <[email protected]>, WSO2 Architecture < >>> [email protected]> >>> >>> >>> Hi all, >>> >>> As part of Metrics [1] integration with WSO2 Platform [2], we are >>> planning to have a separate UI for Metrics visualization. >>> >>> One of the main requirements is that the UI application should be >>> standalone.There are Metrics reporters [3] for Ganglia & Graphite, which >>> require to setup separate servers. >>> >>> Since we need a UI application, I looked at current work on such >>> visualization applications. >>> >>> I mainly looked at Kibana [4] & Grafana [5]. Both applications are based >>> on HTML, JavaScript, etc. I could run both in WSO2 Application Server >>> without any issue. >>> >>> I chose Kibana, as it supports Elasticsearch and we can run >>> Elasticsearch as an embedded server [6]. Elasticsearch also provides a >>> reporter for Metrics [7]. >>> >>> Grafana also looks good and as I found out, we need to write a custom >>> data source to provide the Metrics data. >>> >>> Using mentioned software, I was able to run Kibana in a standalone >>> application and visualize some sample Metrics. >>> >>> This was just a prototype and I will create a proper web application. >>> With Kibana, we can easily create dashboards depending on our requirements. >>> I'm going down this path, as we can quickly create dashboards and identify >>> exact visualizations we need. Since we are in the initial stages of Metrics >>> integration, we have to see how we can use the data provided by Metrics. >>> >>> With Metrics Reporters, each metric will be reported with the configured >>> time period and there will be general statistical information, such as min, >>> max, mean, percentiles etc for Timers. (See JSON format for metrics in >>> README of [7]). We also need to figure out how we can visualize these data. >>> >>> I think we can use Kibana initially for our visualization purposes. All >>> of the mentioned software and dependencies are based on Apache License. >>> >>> Only concern here would be the use of an embedded Elasticsearch server. >>> IMO, Elasticsearch provides good features such as indexing data and smooth >>> integration with Kibana. >>> >>> As I mentioned earlier, I would like to think of this as a temporary >>> approach and we can create a separate dashboard once we have solid >>> requirements. There is another work going on to create custom dashboards >>> [8] and we can also make use of that later. >>> >>> We have not started the Metrics integration with products yet and we >>> might have more concerns when we start to integrate. >>> >>> Please share your suggestions. >>> >>> Thanks! >>> >>> Best Regards, >>> >>> [1] https://dropwizard.github.io/metrics/ >>> [2] http://markmail.org/message/akqyreopmxhcd6hx >>> [3] >>> https://dropwizard.github.io/metrics/3.1.0/manual/core/#other-reporters >>> [4] http://www.elasticsearch.org/overview/kibana/ >>> [5] http://grafana.org/ >>> [6] >>> http://www.elasticsearch.org/guide/en/elasticsearch/client/java-api/current/client.html >>> [7] https://github.com/elasticsearch/elasticsearch-metrics-reporter-java >>> [8] http://markmail.org/message/74ogpnup5uqfqbps >>> >>> >>> -- >>> Isuru Perera >>> Senior Software Engineer | WSO2, Inc. | http://wso2.com/ >>> Lean . Enterprise . Middleware >>> >>> about.me/chrishantha >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >>> >> > > > -- > ============================ > Blog: http://srinathsview.blogspot.com twitter:@srinath_perera > Site: http://people.apache.org/~hemapani/ > Photos: http://www.flickr.com/photos/hemapani/ > Phone: 0772360902 > -- Isuru Perera Senior Software Engineer | WSO2, Inc. | http://wso2.com/ Lean . Enterprise . Middleware about.me/chrishantha
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
