Re: ConnectedStream keyby issues

Hello I am little confused on when the state will be gc. For example, Example 1: Class abc extends RichProcessFunction,Tuple<>> { public void processElement(..) { if(timer never set) { ctx.timerService().registerEventTimeTimer(...);

Are configs stored as part of savepoints

Hi,  I was wondering if configs applied while creating a flink application are also stored as part of savepoint? If yes, an app is restored from a savepoint, does it start with the same configs? Thanks 

CRD compatible with native and standalone mode

Hi,  I plan to create a flink K8s operator which supports standalone mode, and and switch to native mode sometime later. I was wondering what are some of the approaches to ensure that CRD is compatible with both native and standalone mode?  Thanks 

Re: CRD compatible with native and standalone mode

. And you could easily parse them in your K8s operator.":  Do you mean having fields that are needed for both native and standalone mode in the CRD (probably making them optional in the CRD) and each operator type (standalone/native) using fields that are relevant for it?  Thanks,Gaurav

Official flink java client

Hi,  Is there any official flink client in java that's available? I came across RestClusterClient, but I am not sure if its official. I can create my own client, but just wanted to check if there is anything official available already that I can leverage.  Thanks,G

Re: Official flink java client

be great if there is an official client available or if the client can be generated automatically.  Thanks,Gaurav On Friday, April 23, 2021, 06:18:35 AM PDT, Flavio Pompermaier wrote: Yes, that's a known risk. Indeed it would be awesome if the REST API would be published also

Influxdb reporter not honouring the metrics scope

ith a default configuration of " .taskmanager" We tried following configuration and none of them worked ".taskmanager..." ".taskmanager...constant_value." None of them worked and task_name continues to be part of the tags of the measurement sent by influxdb reporter. Thanks, Gaurav Singhania

Re: Influxdb reporter not honouring the metrics scope

Thanks for the response and the fix. On Wed, 22 Jan 2020 at 01:43, Chesnay Schepler wrote: > The solution for 1.9 and below is to create a customized version of the > influx db reporter which excludes certain tags. > > On 21/01/2020 19:27, Yun Tang wrote: > > Hi, Gaurav &g

Batch Processing

Hello, I am looking for batch processing framework which will read data in batches from MongoDb and enrich it using another data source and then upload them in ElasticSearch, is Flink a good framework for such a use case. Regards, Gaurav

Regarding implementation of aggregate function using a ProcessFunction

to implement it. So, kindly share your feedback on this. As I need to implement this. Thanks & Regards Gaurav Luthra -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Regarding implementation of aggregate function using a ProcessFunction

ReturnType( function, input.getType(), null, false); return aggregate(function, accumulatorType, resultType); } Kindly, check above snapshot of flink;s aggregate() method, that got applied on windowed stream. Thanks & Regards Gaurav Luthra Mob:- +91-9901945206 On Fri, Sep 28, 2018 at 1:40 PM vino y

Re: Regarding implementation of aggregate function using a ProcessFunction

ot my point. Thanks & Regards Gaurav Luthra On Fri, Sep 28, 2018 at 4:22 PM Chesnay Schepler wrote: > Please see: https://issues.apache.org/jira/browse/FLINK-10250 > > On 28.09.2018 11:27, vino yang wrote: > > Hi Gaurav, > > Yes, you are right. It is really not allow

Re: Regarding implementation of aggregate function using a ProcessFunction

() method. So, I am looking for inputs if anyone has tried implementing aggregation using ProcessFunction and process() function. As it very much needed thing with flink. Thanks and Regards, Gaurav Luthra Mob:- +91-9901945206 On Sun, Sep 30, 2018 at 5:12 AM Ken Krugler wrote: > Hi Gau

Re: Regarding implementation of aggregate function using a ProcessFunction

Hi Fabian, Thanks for explaining in detail. But we know and you also mentioned the issues in 1) and 2). So, I am continuing with point 3). Thanks & Regards Gaurav Luthra Mob:- +91-9901945206 On Mon, Oct 1, 2018 at 3:11 PM Fabian Hueske wrote: > Hi, > > There are basically three

Partitioning by composite key, But type and number of keys are dynamic

There is a data stream of some records, Lets call them "input records". Now, I want to partition this data stream by using keyBy(). I want partitioning based on one or more fields of "input record", But the number and type of fields are not fixed. So, Kindly tell me how should I achieve this partit

Re: Partitioning by composite key, But type and number of keys are dynamic

Hi Chesnay, My End user will be aware about the fields of "input records" (GenericRecord). In configuration my end user only will tell me the name and number of the fields, based on these fields of GenericRecord I will have to partition the DataStream and make Keyed Stream. Currently, I have impl

Identify metrics belonging to the "same" task manager in kubernetes

suffix/prefix to the names).  Appreciate your help.  Thanks,Gaurav

Re: Identify metrics belonging to the "same" task manager in kubernetes

cords processed, processing rate, overall latency etc)? Thanks,GauravOn Thursday, August 26, 2021, 04:02:10 AM PDT, Chesnay Schepler wrote: 1) Can you clarify what you mean with "same"? Are you referring to something like an index, i.e., "This is TaskManager #2"

Re: Identify metrics belonging to the "same" task manager in kubernetes

Thanks for the response! For #2, custom labels should work too for our case.  Thanks,Gaurav On Thursday, August 26, 2021, 08:28:27 AM PDT, Chesnay Schepler wrote: 1) As is there is no way to accomplish this. 2) Yes (datadog, graphite) but if you are happy with Prometheus I

Re: Identify metrics belonging to the "same" task manager in kubernetes

Hi,  I have another question: What mechanisms are usually used to correlate prometheus flink metrics for kubernetes?  Thanks,Gaurav  On Thursday, August 26, 2021, 10:06:30 AM PDT, gaurav kulkarni wrote: Thanks for the response! For #2, custom labels should work too for our case

Need the way to create custom metrics

AggregateFunction in aggregate() method. So, Kindly guide me, how can I create custom metrics in my code? Note: As we know we can not user RichAggregateFunction with aggregate() method Thanks & Regards Gaurav Luthra Mob:- +91-9901945206

Getting "ProducerFenced" exception while using flink kafka producer

erstanding the reason of this exception. And kindly tell how can we get different transactional ID for two jobs. Thanks & Regards Gaurav Luthra Mob:- +91-9901945206

Counter Metrics not getting removed from Flink GUI after close()

ated using myMetricGroup shall be removed from Flink GUI. Thanks & Regards Gaurav Luthra Mob:- +91-9901945206

Re: Counter Metrics not getting removed from Flink GUI after close()

Hi Chesnay, If removing the metrics is not possible from Flink GUI, while the job is running. Then kindly tell me how to update a metric counter. Explaination: Suppose I created a metric Counter with key "chesnay" and incremented the counter to 20, by code mentioned below. getRuntimeContext().get

Re: Counter Metrics not getting removed from Flink GUI after close()

ric by storing a reference to it in your function. > Why do you want to avoid the member variable? > > On 08.01.2019 17:24, Gaurav Luthra wrote: > > Hi Chesnay, > > If removing the metrics is not possible from Flink GUI, while the job is > running. > Then kindly tell me

Issue with counter metrics for large number of keys

values of myKey. Is there any alternate solution? I am looking for a solution where I achieve above functionality to maintain approx. 100 thousands counter metrics without keeping their reference in map (or any other data structure). Thanks & Regards Gaurav Luthra Mob:- +91-9901945206

Re: Issue with counter metrics for large number of keys

. And will restrict the maximum value to 1000 so that no mishap happens about memory. and will tune this max value with memories of JobManager and my application. And try to explore other solutions in flink. Thanks & Regards Gaurav Luthra Mob:- +91-9901945206 On Thu, Jan 17, 2019 at 9:40 AM

Registration order with Kryo

anged? If so, what would be the general recommendation if we need to remove or add classes registered with Kryo? Thanks Gaurav

Collector.collect

Hello I am working on RichProcessFunction and I want to emit multiple records at a time. To achieve this, I am currently doing : while(condition) { Collector.collect(new Tuple<>...); } I was wondering, is this the correct way or there is any other alternative.