Order of Execution in KeyedBroadcastProcessFunction

in the *processElement* function? Can someone explain or point me to some docs on how the synchronisation of Broadcasted and non Broadcasted messages works in KeyedBroadcastProcessFunction? Kind regards Anil

Re: Question About Histograms

Hello Prasanna, Thanks for your response, Could you elaborate on what you meant by "overriding the Prometheus Histogram class provided "? if possible with any samples? Regards, Anil On Tue, Apr 5, 2022 at 1:11 AM Prasanna kumar wrote: > Anil, > > Flink Histograms are actual

Question About Histograms

. If Histograms are not checkpointed and are reset, are there any other possible ways to calculate Histograms/P95 on a large window like 30days? Thanks & Regards Anil

Tracking ID in log4j MDC

am seeing tracking id is not getting propagated to the next function. I am using flink 1.9 running in k8. Thanks, Anil

Re: FlinkKafakaProducer with Confluent SchemaRegistry and KafkaSerializationSchema

); return new ProducerRecord<>(topic, value); } } Then used a new object of GenericSerializer in the FlinkKafkaProducer FlinkKafkaProducer producer = new FlinkKafkaProducer<>(topic, new GenericSerializer(topic, schema, schemaRegistryUrl), kafkaConfig, Semantic.AT_LEAST_ONCE); T

FlinkKafakaProducer with Confluent SchemaRegistry and KafkaSerializationSchema

erence/documentation i could use? Thanks , Anil.

Custom Exception Handling

. Regards Anil

Re: Approach to Auto Scaling Flink Job

is what if the Yarn Master node goes down. Is the Yarn cluster running with Multi-master or in case of failure do you migrate your job do a different cluster. If so is this failover to a different cluster built into Athenax. Regards, Anil. -- Sent from: http://apache-flink-user-mailing-list-archive

Re: Approach to Auto Scaling Flink Job

Manager which monitors the job. What additional functionality does Docker provide here. Also when the jobs are deployed on Yarn, the Master Node becomes a Single point of failure. Are you using a Multi-Master setup or have taken another approach to handle failover. Regards, Anil. -- Sent from

Re: Approach to Auto Scaling Flink Job

Thanks for the reply Rong. Can you please let me know the design for the auto-scaling part, if possible. Or guide me in the direction so that I could create this feature myself. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Approach to Auto Scaling Flink Job

I'm using Uber Open Source project Athenax. As mentioned in it's docs[1] it supports `Auto scaling for AthenaX jobs`. I went through the source code on Github but didn't find the auto scaling part. Can someone aware of this project please point me in the right direction here. I'm using Flink's

Re: Is the eval method invoked in a thread safe manner in Fink UDF

Thanks Rong!. The UDF makes a http call to another service synchronously. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Is the eval method invoked in a thread safe manner in Fink UDF

Thanks Hequn!. Is it also thread safe when the same UDF is called multiple times in the same record. Is the UDF called sequentially for each fields a single record, I have a query like - select GetName(data.id, 'city'), GetName(data.id, 'zone') from .. -- Sent from:

Is the eval method invoked in a thread safe manner in Fink UDF

Is the eval method invoked in a thread safe manner in Fink UDF. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Running Flink on Yarn

Hi Andrey. Thanks for the reply. Apologies about the late follow up, I was out of office. Suppose I have 3 TM and each has 3 task slot and each kafka stream has 9 partitions each. Each thread will consumer from stream 1 (a1) and stream 2 (a2). Considering the query, data will need to be buffered

Re: Running Flink on Yarn

I am using time-windowed join only. Here's a sample query - SELECT a1.order_id, a2.order.restaurant_id FROM awz_s3_stream1 a1 INNER JOIN awz_s3_stream2 a2 ON CAST(a1.order_id AS VARCHAR) = a2.order_id AND a1.to_state = 'PLACED' AND a1.proctime BETWEEN a2.proctime - INTERVAL '2' HOUR AND

Re: Running Flink on Yarn

Thanks for the quick response Andrey. I'm doing a SQL time-windowed join on non-keyed stream. So all the thread in various task slot in the same TM will share this state. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Running Flink on Yarn

I have a setup for Flink(1.4.2) with YARN. I'm using Flink Yarn Client for deploying my jobs to Yarn Cluster. In the current setup parallelism was directly mapped to the number of cores, with each parallel instance of the job running in one container. So for a parallelism of 9, there are 10

Re: Kafka offset auto-commit stops after timeout

I had the same issue and enabling checkpoint seems to solve the problem. Can you please explain how does enabling checkpoint fixes the issue. Thanks! -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Error deploying task manager after failure in Yarn

I'm using Flink 1.4.2 and running Flink on Yarn. Job runs with a parallelism of 2. Each task manager is allocated 1 core. When the container memory exceeds the allocated memory yarn kills the container as expected. {"debug_level":"INFO","debug_timestamp":"2018-12-04

Counter Implementation in Flink

I'm using Flink 1.4.2 and deploying job on Yarn Cluster. I have a streaming job, which flattens the data and outputs it. It basically takes a input record and produces n output record. I'm using Table Function for this. The logic to flatten the data is implemented in a UDF. The UDF has a counter

Re: akka timeout exception

Thanks for the reply Dawid. The Flink jobs are deployed in Yarn cluster. I am seeing the error in Job Manager log for some jobs too frequently. I'm using Flink 1.4.2. I'm running only Streaming Jobs. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: InterruptedException when async function is cancelled

Hi Till, Thanks for the reply. Is there any particular patch I can use as upgrading to Flink 1.6 is not an option for me at the moment. Regards, Anil. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Kafka offset behaviour when restarting job from savepoint

Hey Paul. Can you please point me to the code in Flink. Thanks! -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

akka timeout exception

getting this error in my job manager too frequently. any help. Thanks! java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/jobmanager#1927353472]] after [1 ms]. Sender[null] sent message of type

Re: InterruptedException when async function is cancelled

I do see the same error but in case different situation. I'm not cancelling the job. Below is my error stack trace. SwiglobeZoneFromLatLong is my UDF name. Is this error something ignorable. I'm using flink 1.4.2. Thanks in advance. ``` {"debug_level":"ERROR","debug_timestamp":"2018-10-28

Flink Error - Remote system has been silent for too long

The Flink jobs are deployed in Yarn cluster. I am seeing the following log for some of my jobs in Job Manager. I'm using Flink 1.4. The job has, taskmanager.exit-on-fatal-akka-error=true. But I don't see the task manager being restarted. I made the following observations - 1. One job does a

Re: Manual savepoint trigger

I think I didn't make myself clear. Sorry. What I want to know is, when we trigger the savepoint, which checkpoint barrier will it consider to trigger the savepoint. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Manual savepoint trigger

A checkpoint is completed when the nth the checkpoint barrier crosses the last operator. When we trigger a savepoint manually, which checkpoint barrier is considered. Is it the next checkpoint barrier(next relative to the last barrier that was considered to create the last checkpoint or

Re: JDBCAppendTableSink Metric

Thanks Hequn. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

JDBCAppendTableSink Metric

I need to monitor the latency and other similar metric for the Mysql Sink that Flink writes to. I'm using JDBCAppendTableSink. Any way how I can check these. Thanks! -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Flink Checkpoint Barrier in case of Join

I'm trying to understand when will Flink's Stream Barrier (for checkpoint) be emitted by the join operator. Consider a query like - select * from stream_1 a1 INNER JOIN stream_2 a2 on a2.orderId = a1.orderId group by HOP(a1.proctime, INTERVAL '1' HOUR, INTERVAL '1' DAY), a1.restaurantId

Re: Flink dynamic scaling 1.5

Thanks for the reply Till. Resubmitting the job is an option. I was wondering if there's any way that Flink could be configured to detect issues like a memory issue and rescale without me submitting the job again. -- Sent from:

Flink dynamic scaling 1.5

This is regarding dynamic rescaling in Flink 1.5 I am using Yarn for running Flink jobs. I start these jobs with a static resource. Is there any option to scale out these job by itself in specific conditions like if there's a memory issues. In Flink 1.5 release notes - Applications can be

Flink dynamic scaling 1.5

This is regarding dynamic rescaling in Flink 1.5 I am using Yarn for running Flink jobs. I start these jobs with a static resource. Is there any option to scale out these job by itself in specific conditions like if there's a memory issues. In Flink 1.5 release notes - Applications can be

Re: Trigerring Savepoint for the Flink Job

Sorry about the late reply. This reply is more specific to the Uber's AthenaX project . To trigger the savepoint we need to simply create an instance of YarnClusterClient. This class has implementation to trigger savepoint. To trigger the savepoint for any job

Savepoint with S3

I'm using RocksDB and S3 to for Savepoint. Flink version is 1.4. Currently I'm creating a savepoint for the jobs every 10 mins. When the job starts the data is saved as expected but after a while I see these message in my log and the savepoint is not saved anymore. 2018-06-19 12:23:18,992

Re: Trigerring Savepoint for the Flink Job

Just out of curiosity how do you save your checkpoint. Currently I'm using filesystem but I'm migrating it to RocksDB which allows for async operations to avoid latency at higher scale as we grow. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Trigerring Savepoint for the Flink Job

Hi Rongs. Thanks for your help. I was about to look into the CLI API but then I figured out how to trigger savepoint and restore a job's savepoint with the job running in detached mode. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Submit Flink Job on Yarn cluster to restores its state from it's last savepoint

I am running these Flink job in a Yarn cluster. When the flink job fails or I restart it I want the Flink job to use the savepoint that was created before it restarted. How do can restart the Flink job with this savepoint. I am using Flink 1.4.2. -- Sent from:

Re: Trigerring Savepoint for the Flink Job

Thanks for the reply Rong. We had updated Athenax to version 1.4. I had checked Flink 1.4, it's rest endpoint dose not support only creating Savepoint. It has cancel With Savepoint. I think creating Savepoint is supported in 1.5. Since we can't upgrade to 1.5 at the moment it would like to find

Trigerring Savepoint for the Flink Job

I am using Flink 1.4.2. I have forker Uber's AthenaX project . The Flink jobs are deployed in Yarn cluster. I needed to save the Savepoint for all the jobs everyday. ClusterClient