nicusX commented on code in PR #766: URL: https://github.com/apache/flink-web/pull/766#discussion_r1858936119
########## docs/content/posts/2024-11-26-introducing-new-prometheus-connector.md: ########## @@ -0,0 +1,201 @@ +--- +title: "Introducing the new Prometheus connector" +date: "2024-11-26T00:00:00.000Z" +authors: +- nicusX: + name: "Lorenzo Nicora" +--- + + +We are excited to announce a new sink connector that enables writing data to Prometheus ([FLIP-312](https://cwiki.apache.org/confluence/display/FLINK/FLIP-312:+Prometheus+Sink+Connector)). This articles introduces the main features of the connector, the reasoning behind design decisions. + +This connector allows writing data to Prometheus using the [Remote-Write](https://prometheus.io/docs/specs/remote_write_spec/) push interface, which lets you write time-series data to Prometheus at scale. + +## Motivations for a Prometheus connector + +Prometheus is an efficient time-series database optimized for building real-time dashboards and alerts, typically in combination with Grafana or other visualization tools. + +Prometheus is commonly used to monitor compute resources, IT infrastructure, Kubernetes clusters, applications, and cloud resources. It can also be used to observe your Flink cluster and Flink jobs. Flink already has [Metric Reporters](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/metric_reporters/) to export metrics to Prometheus. + +So, why do we need a connector? + +Prometheus can serve as a general-purpose observability time-series database, beyond traditional infrastructure monitoring. For example, it can be used to monitor IoT devices, sensors, connected cars, media streaming devices, and any resource that streams events or measurements continuously. + +Observability data from these use cases differs from metrics generated by compute resources. Events are pushed by devices instead of being scraped, resulting in irregular frequency. Devices may be connected via mobile networks or even Bluetooth, causing events from each device to follow different paths and arrive at different times. The frequency and cardinality of events emitted by these devices can be very high, making it challenging to derive insights directly. Finally, events often lack contextual information and require enrichment to add additional dimensions before being sent to a time-series database. Review Comment: This is actually mixing a challenges and the solutions. I will rephrase into bulletpoints, listing the challenges, and what you need to do to overcome them -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
