Re: java.lang.IllegalStateException: The RPC connection is already closed

ent-tabpanel#comment-16946580 > > On Mon, May 4, 2020 at 5:52 AM Manish G > wrote: > >> Hi, >> >> I have set up flink and kafka locally. When I start my flink >> program(configured ot read messages from kafka topic), I get error as: >> >&

Re: java.lang.IllegalStateException: The RPC connection is already closed

https://issues.apache.org/jira/browse/FLINK-16373 On Mon, May 4, 2020 at 9:37 PM Manish G wrote: > I found another similar issue: > > > On Mon, May 4, 2020 at 9:28 PM Steven Wu wrote: > >> Manish, might be related to this bug, which is fixed in 1.10.1. >> >>

InvalidTypesException: Input mismatch while consuming Kafka messages

I have following code: // Properties properties = new Properties(); properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, MyCustomClassDeserializer.class.getName()); FlinkKafkaConsumer kafkaConsumer = new FlinkKafkaConsumer(

Autoscaling flink application

Hi, I understand task parallelism in flink, but is it possible to configure dynamic horizontal scaling also. Manish

Re: InvalidTypesException: Input mismatch while consuming Kafka messages

ies.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, > MyCustomClassDeserializer.class.getName()); > > You need to use Flink's DeserializationSchema. > > On Mon, May 4, 2020 at 10:26 AM Manish G > wrote: > >> I have following code: >> >> // >> Propert

java.lang.IllegalStateException: The RPC connection is already closed

Hi, I have set up flink and kafka locally. When I start my flink program(configured ot read messages from kafka topic), I get error as: 2020-05-04 18:17:58.035 INFO 23516 --- [lt-dispatcher-2] o.a.f.r.taskexecutor.JobLeaderService: Successful registration at job manager

Re: Autoscaling flink application

k at Netflix [1] by Timothy Farkas. > > [1] https://www.youtube.com/watch?v=NV0jvA5ZDNc > > Regards, > David > > > > On Mon, May 4, 2020 at 1:14 PM Manish G > wrote: > >> Hi, >> >> I understand task parallelism in flink, but is it possible to configure >> dynamic horizontal scaling also. >> >> Manish >> >

Autoscaling vs backpressure

Hi, As flink doesn't provide out-of-box support for autoscaling, can backpressure be considered as an alternative to it? Autoscaling allows us to add/remove nodes as load goes up/down. With backpressure, if load goes up system would signal upstream to release data slowly. So we don't need to add

Metrics for the overall pipeline

Adding metrics to individual RichMapFunction implementation classes would give metrics information about that particular class. As a pipeline consists of multiple such classes, how can we have metrics for the overall data pipeline?Are there any best practices for it? With regards

Metrics for number of events in a timeframe

Hi, Meter gives throughput while counter gives number of events since system started. What if I have to find number of events processed in say, last 5 minutes. Is there an inbuilt metrics for it OR do I need to do meter * timeframe? With regards

Re: Metrics for number of events in a timeframe

t; > On 04/08/2020 15:04, Manish G wrote: > > Hi, > > > > Meter gives throughput while counter gives number of events since > > system started. What if I have to find number of events processed in > > say, last 5 minutes. > > > > Is there an inbuilt metrics for it OR do I need to do meter * timeframe? > > > > With regards > > > > > >

A query on Flink metrics in kubernetes

Hi, I have a query regarding prometheus scraping Flink metrics data with application running in kubernetes cluster. If taskmanager is running on multiple nodes, and prometheus requests for the metrics data, then is that request directed to one of the nodes(based on some strategy, like

Limiting metrics logs to custom metric

Hi, I have added a Meter in my code and pushing it to app logs using slf4j reporter. I observe that apart from my custometrics, lots of other metrics like gauge, histogram etc is also published. It makes it difficult to filter out data for generating splunk graphs. Is there a way to limit

Re: Limiting metrics logs to custom metric

Ok.Any resource on same? On Wed, Jul 8, 2020, 9:38 PM Chesnay Schepler wrote: > There's no built-in functionality for this. You could customize the > reporter though. > > On 08/07/2020 17:19, Manish G wrote: > > Hi, > > > > I have added a Meter in my code and push

Re: Integrating prometheus

ons. > You need to go with the documented approach. > But of course, you can try to build your own metrics abstraction based on > Flink's metric system. > > On Fri, Jul 3, 2020 at 9:35 AM Manish G > wrote: > >> Hi, >> >> I am following this >> <https://c

Re: Integrating prometheus

Also, it seems custom metrics can only be recorded if we extend RichFunction, as it allows us to override open wherein we can get hold of context and metrics constructs. Please let me know if there are other ways too. On Fri, Jul 3, 2020 at 10:05 PM Manish G wrote: > Hi, > > I am

Re: Logging Flink metrics

lf4jreporter > > On 06/07/2020 13:49, Manish G wrote: > > Hi, > > > > Is it possible to log Flink metrics in application logs apart from > > publishing it to Prometheus? > > > > With regards > > >

Logging Flink metrics

Hi, Is it possible to log Flink metrics in application logs apart from publishing it to Prometheus? With regards

Re: Logging Flink metrics

run for, and what is the configured interval? > > > On 06/07/2020 15:51, Manish G wrote: > > Hi, > > Thanks for this. > > I did the configuration as mentioned at the link(changes in > flink-conf.yml, copying the jar in lib directory), and registered the Meter > with metrics

Re: Logging Flink metrics

Yes. On Mon, Jul 6, 2020 at 10:43 PM Chesnay Schepler wrote: > Are you running Flink is WSL by chance? > > On 06/07/2020 19:06, Manish G wrote: > > In flink-conf.yaml: > *metrics.reporter.prom.port: 9250-9260* > > This is based on information provided here > <

Re: Logging Flink metrics

> > ./bin/jobmanager.sh start > > ./bin/taskmanager.sh start > > On 06/07/2020 19:16, Manish G wrote: > > Yes. > > On Mon, Jul 6, 2020 at 10:43 PM Chesnay Schepler > wrote: > >> Are you running Flink is WSL by chance? >> >> On 06/07/2020 19:06,

Re: Logging Flink metrics

e done based on several tutorials and blogs available online. On Mon, Jul 6, 2020 at 10:20 PM Chesnay Schepler wrote: > These are all JobManager metrics; have you configured prometheus to also > scrape the task manager processes? > > On 06/07/2020 18:35, Manish G wrote: > > T

Re: Logging Flink metrics

h for warnings from the metric > groups/registry/reporter. > > If you cannot find anything suspicious, you can also send the foll log to > me directly. > > On 06/07/2020 16:29, Manish G wrote: > > Job is an infinite streaming one, so it keeps going. Flink configuration > i

Re: Logging Flink metrics

hepler wrote: > You've said elsewhere that you do see some metrics in prometheus, which > are those? > > Why are you configuring the host for the prometheus reporter? This option > is only for the PrometheusPushGatewayReporter. > > On 06/07/2020 18:01, Man

Integrating prometheus

Hi, I am following this link on how to integrate prometheus with flink. Going by the code sample, I would need to insert related metrics code in the main logic. Is it avoidable, like by using some annotations on

Prometheus not showing custom metrics for flink

Hi, I have integrated prometheus and flink for monitoring custom flink metrics. But I am not able to get those metrics displayed on prometheus dashboard or on Grafana dashboard. I can see default flink metrics, but not custom ones. I have put details here SO question

org.aspectj.lang.annotation annotation not working with flink

I have created a custom annotation to log time consumed by a method using aspectj library. I tested it in a spring boot application for one of the rest endpoint and it works fine. But when I annotate map method in my flink job class, it doesn't work. Anyone having any inputs on it? With regards

Re: Making sense of Meter metrics graph on Grafana

gt; second over the last minute (basically, rate(T) = (count(T) - > count(T-60)) / 60; so short spikes tend to be flattened quite a bit. > > On 29/07/2020 15:13, Manish G wrote: > > I have added Meter metrics to my flink job code,and in grafana I can > > see the graph

Simple MDC logs don't show up

Hi All, I have some very simple MDC logs in my flink job: MDC.put("methodName", new Object() {}.getClass().getEnclosingMethod().getName()); MDC.put("className", this.getClass().getSimpleName()); When I run flink job locally, I can see them in the application logs. But when I run the same job

Metrics for average time taken by flatMap function

Hi All, I have implemented a flatmap function and I want to collect metrics for average time for this function which I plan to monitor via prometheus. What would be good approach for it? I have added a gauge to the method(extending Gauge interface from flink API). Would it work for my needs?

Re: Metrics for average time taken by flatMap function

: > A gauge just returns a value, and Flink exposes it as is. As such you need > to calculate the average over time yourself, taking 2 time measurements > (before and after the processing of each). > > On 1/12/2021 11:31 AM, Manish G wrote: > > startTime is set at start of functio

Re: Metrics for average time taken by flatMap function

mpared to the logic within a function, usually rather expensive and > may impact performance. > > On 1/12/2021 10:57 AM, Manish G wrote: > > Hi All, > > > > I have implemented a flatmap function and I want to collect metrics > > for average time for this function w

Re: Metrics for average time taken by flatMap function

startTime is set at start of function: long startTime = System.currentTimeMillis(); On Tue, Jan 12, 2021 at 3:59 PM Manish G wrote: > My code is: > > public class SimpleGauge implements Gauge { > > private T mValue; > > @Override > public T getValue() {

Re: Metrics for average time taken by flatMap function

t to look into exponential moving averages. > Alternatively, just expose the diff as an absolute value and calculate the > average in prometheus. > > On 1/12/2021 11:50 AM, Manish G wrote: > > OK, got it. > So I would need to accumulate the time value over the calls as well as

Re: Metrics for average time taken by flatMap function

021 12:30 PM, Chesnay Schepler wrote: > > I mean the difference itself, not cumulative. > > On 1/12/2021 12:08 PM, Manish G wrote: > > Can you elaborate the second approach more? > Currently I am exposing the difference itself. OR do you mean the > cumulative difference?ie I maint

Counter metrics for prometheus having unexepcted gaps in grafana

Hi All, I am facing an issue with counter metrics I have added to a flatmap function. My application is deployed in kubernetes, and hence the prometheus metrics generated has pod id as one of its label. Now if pod dies and a new pod comes up, we have a brand new metrics starting from 0. As a

Gauges generating same graph

Hi All, I have few RichFlatMapFunction classes, and I have gauge added to each one of them. For a particular usecase I am updating these gauges incrementally. I have a class member variable in each of these classes which keeps increasing as flapMap function in these classes is called, and then I

Re: Metrics for average time taken by flatMap function

This approach has an issue. Even for those periods when there is no activity, still the latest gauge value is used for calculations and this generates graphs which are not correct representation of the situation. On Tue, Jan 12, 2021 at 7:01 PM Manish G wrote: > Prometheus provi

Re: Gauges generating same graph

task manager then have the same value? > > Why do you need the variable? What's not working when updating the gauge > directly? > > On Fri, Jan 15, 2021 at 4:46 AM Manish G > wrote: > >> Hi All, >> >> I have few RichFlatMapFunction classes, and I have gauge added to

Histogram has count data but not sum

Hi All, I have added histogram to code following here . But I observe that on prometheus board I get only count metrics, not sum. Metrics itself is missing. I have used classes: com.codahale.metrics.Histogram

Re: Histogram has count data but not sum

gt; What do you see in Prometheus, and what do you consider to be missing? > > On 1/13/2021 4:10 PM, Manish G wrote: > > Hi All, > > I have added histogram to code following here > <https://ci.apache.org/projects/flink/flink-docs-stable/ops/metrics.html#histogram> > . >