Re: How to visualize the results of Flink processing or aggregation?

[Xiong Qiang]

Thank you, @David Anderson and @Fuyao Li. This answered my question and
cleared my confusions.

On Fri, Mar 26, 2021 at 11:08 AM David Anderson 
wrote:

> Prometheus is a metrics system; you can use Flink's Prometheus metrics
> reporter to send metrics to Prometheus.
>
> Grafana can also be connected to influxdb, and to databases like mysql and
> postgresql, for which sinks are available.
>
> And the Elasticsearch sink can be used to create visualizations with
> Kibana.
>
> I'm sure there are other solutions as well, but these are some of the
> popular ones.
>
> Regards,
> David
>
> On Fri, Mar 26, 2021 at 5:15 PM Xiong Qiang 
> wrote:
>
>> Hi All,
>>
>> I am new to Flink, so forgive me if it is a naive question.
>>
>> The context is:
>> We have a data streaming coming in, and we will use Flink applications to
>> do the processing or aggregations. After the processing or aggregation, we
>> need some approaches to visualize the results, to either build a dashboard
>> or setup alerts, for example, using Prometheus and Grafana.
>> However, after reading the documents (
>> https://flink.apache.org/flink-architecture.html and more links) and
>> examples (
>> https://ci.apache.org/projects/flink/flink-docs-release-1.12/try-flink/datastream_api.html)
>> (
>> https://github.com/ververica/flink-training/blob/master/long-ride-alerts/src/solution/java/org/apache/flink/training/solutions/longrides/LongRidesSolution.java)
>> , *I am still not able to close the gap between Flink and a
>> monitoring/dashboard tool, e.g. Prometheus/Grafana. *
>>
>> *The question is:*
>> *How are processing results connected/sinked from Flink to
>> Prometheus/Grafana? *for example, in the fraud detection example, how is
>> the account id = 3, send to Prometheus and Grafana, so that I have a
>> dashboard showing there is one suspected account? In the taxi long rides
>> example, how do I send the count of long rides from Flink to
>> Prometheus/Grafana?
>>
>> I understand there are sinks (
>> https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/connectors/).
>> However, I didn't see sinks for Prometheus.
>>
>> Hope I made my question clear.
>>
>> Thanks
>>
>

Re: How to visualize the results of Flink processing or aggregation?

[David Anderson]

Prometheus is a metrics system; you can use Flink's Prometheus metrics
reporter to send metrics to Prometheus.

Grafana can also be connected to influxdb, and to databases like mysql and
postgresql, for which sinks are available.

And the Elasticsearch sink can be used to create visualizations with Kibana.

I'm sure there are other solutions as well, but these are some of the
popular ones.

Regards,
David

On Fri, Mar 26, 2021 at 5:15 PM Xiong Qiang 
wrote:

> Hi All,
>
> I am new to Flink, so forgive me if it is a naive question.
>
> The context is:
> We have a data streaming coming in, and we will use Flink applications to
> do the processing or aggregations. After the processing or aggregation, we
> need some approaches to visualize the results, to either build a dashboard
> or setup alerts, for example, using Prometheus and Grafana.
> However, after reading the documents (
> https://flink.apache.org/flink-architecture.html and more links) and
> examples (
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/try-flink/datastream_api.html)
> (
> https://github.com/ververica/flink-training/blob/master/long-ride-alerts/src/solution/java/org/apache/flink/training/solutions/longrides/LongRidesSolution.java)
> , *I am still not able to close the gap between Flink and a
> monitoring/dashboard tool, e.g. Prometheus/Grafana. *
>
> *The question is:*
> *How are processing results connected/sinked from Flink to
> Prometheus/Grafana? *for example, in the fraud detection example, how is
> the account id = 3, send to Prometheus and Grafana, so that I have a
> dashboard showing there is one suspected account? In the taxi long rides
> example, how do I send the count of long rides from Flink to
> Prometheus/Grafana?
>
> I understand there are sinks (
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/connectors/).
> However, I didn't see sinks for Prometheus.
>
> Hope I made my question clear.
>
> Thanks
>

Re: [External] : How to visualize the results of Flink processing or aggregation?

[Fuyao Li]

Hello Xiong,

You can expose monitors through Metric system of Flink.
https://ci.apache.org/projects/flink/flink-docs-stable/ops/metrics.html
Metrics can be exposed by metric reporter:
https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/metric_reporters.html
That includes Prometheus.

For the DataStream API, you can build customized metrics.
For the Flink SQL/Table API, you can only use the listed predefined metrics in 
Flink. There could be work arounds ways, but no direct way to supply Flink SQL 
customized metrics.

Best regards,
Fuyao

From: Xiong Qiang 
Date: Friday, March 26, 2021 at 09:15
To: user@flink.apache.org 
Subject: [External] : How to visualize the results of Flink processing or 
aggregation?
Hi All,

I am new to Flink, so forgive me if it is a naive question.

The context is:
We have a data streaming coming in, and we will use Flink applications to do 
the processing or aggregations. After the processing or aggregation, we need 
some approaches to visualize the results, to either build a dashboard or setup 
alerts, for example, using Prometheus and Grafana.
However, after reading the documents 
(https://flink.apache.org/flink-architecture.html
 and more links) and examples 
(https://ci.apache.org/projects/flink/flink-docs-release-1.12/try-flink/datastream_api.html)
 
(https://github.com/ververica/flink-training/blob/master/long-ride-alerts/src/solution/java/org/apache/flink/training/solutions/longrides/LongRidesSolution.java)
 , I am still not able to close the gap between Flink and a 
monitoring/dashboard tool, e.g. Prometheus/Grafana.

The question is:
How are processing results connected/sinked from Flink to Prometheus/Grafana? 
for example, in the fraud detection example, how is the account id = 3, send to 
Prometheus and Grafana, so that I have a dashboard showing there is one 
suspected account? In the taxi long rides example, how do I send the count of 
long rides from Flink to Prometheus/Grafana?

I understand there are sinks 
(https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/connectors/).
 However, I didn't see sinks for Prometheus.

Hope I made my question clear.

Thanks

How to visualize the results of Flink processing or aggregation?

[Xiong Qiang]

Hi All,

I am new to Flink, so forgive me if it is a naive question.

The context is:
We have a data streaming coming in, and we will use Flink applications to
do the processing or aggregations. After the processing or aggregation, we
need some approaches to visualize the results, to either build a dashboard
or setup alerts, for example, using Prometheus and Grafana.
However, after reading the documents (
https://flink.apache.org/flink-architecture.html and more links) and
examples (
https://ci.apache.org/projects/flink/flink-docs-release-1.12/try-flink/datastream_api.html)
(
https://github.com/ververica/flink-training/blob/master/long-ride-alerts/src/solution/java/org/apache/flink/training/solutions/longrides/LongRidesSolution.java)
, *I am still not able to close the gap between Flink and a
monitoring/dashboard tool, e.g. Prometheus/Grafana. *

*The question is:*
*How are processing results connected/sinked from Flink to
Prometheus/Grafana? *for example, in the fraud detection example, how is
the account id = 3, send to Prometheus and Grafana, so that I have a
dashboard showing there is one suspected account? In the taxi long rides
example, how do I send the count of long rides from Flink to
Prometheus/Grafana?

I understand there are sinks (
https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/connectors/).
However, I didn't see sinks for Prometheus.

Hope I made my question clear.

Thanks