I would suggest that its useful to think of metrics as separate from logs/events. Similarly, I would treat exception handling (eg. Sentry) to be a separate leg of service monitoring, but that's beside the point.
Metrics have a rather more fixed cardinality and by design sacrifice detail, whereas logs tend to retain their richness (and often get further enriched). Metrics are great for figuring out there are problems, but are quite limited in delving into why (eg. because your site is being crawled by a new type of spider), but you end up with a smaller mount of data. Logs are great for figuring out why there are problems, and modern tools such as Elasticsearch make it possible to create performant dashboards even when you have mountains of data. You also tend not to worry about cardinality as much as you would with Prometheus You can of course expose metrics within your log_processing, but from my experience, if you're using something like ElasticSearch, the aggregations and dashboards are quite performant, although there can be some overlap. I've been using Elasticsearch for a few years now (Prometheus only for a few months long), and my most comprehensive dashboards are to do with web-sites. Before using Elasticsearch I was (and still am) very skilled at pulling apart logs using grep/sed/awk etc. but ELK (Elasticsearch, Logstash and Kibana) give me pretty much all the insight I've needed to be able to quickly drill down into a problems and help make data-informed improvements. I should also say, however, that this insight tends to come from a very verbose logging format I've specified in httpd (also nginx, which emits cleaner JSON) with a generous amount of enrichment from Logstash. Indeed, you could even dispense with Elasticsearch and Kibana and just use Logstash for enriching logs and synthesizing metrics to be sent to... (well, typically statsd) Coupled with this, I have used elastalert (albeit, not much) to look for things to alert about. According to the following, it can also integrate with alertmanager, although I haven't testing this myself yet. https://github.com/Yelp/elastalert/pull/2228 Note also that Grafana can use Elasticsearch as a datasource too. Hope that helps, Cameron On Thursday, 5 March 2020 04:35:26 UTC+13, Nabil L. wrote: > > Hi Folks, > > > I am currently looking for an exporter which will allow me to monitors > somes logs (from WebLogic, Apaches or other application). > The aim is to find some specific pattern in the logs an raise an alert > trough the Alertmanager. > > Anyone can advise me which exporter is the best to perform this task > (ideally an exporter that your are using)? > > I've a look on the grok_exporter, but it seem a little bit hard to > configure/handle... > > > Thanks a lot > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c4bf46c5-de4a-499f-b822-2fba9c5cb3b2%40googlegroups.com.
