sanity check in production

Hello everyone, Our team ran into an issue that testing new deployment of flink job is difficult as explained below Goal: When we are deploying new version of a flink job in production. we want to be able to have the job process some test messages and verify the output to make sure that the

Ever increasing key space

Hi every one, We are building a flink job that keys on a dynamic value. Only a few events share the same key and events with new keys are consumed constantly. For each key, there are some keyedState created the first time it is seen. And we clean up the keyedState if the key has not been seen

Increased Size of Incremental Checkpoint

Hi guys, I enabled incremental flink checkpoint for my flink job. I had the job read messages at a stable rate. For each message, the flink job store something in the keyed state. My question is: For every minute, the increased state size is the same, shouldn't the incremental checkpoint size

Multiple operators to the same sink

Hi Guys! I'm designing a topology where multiple operators should forward the messages to the same sink. For example I have Operator A,B,C,D,E. I want A,B,C to forward to Sink1 and D, E to forward to Sink2. My options are 1. Union A, B and C. then add Sink1 to them. Similarly for D and E.

Singleton in a taskmanager

Hi Guys, I am running into a problem. I have 2 jobs running on the same taskmanager. Each Job creates a singleton of the same class, say MySingleton class. Are they actually sharing the same singleton? Hope my question is clear. Best, Burgess Chen -- Sent from:

Expire records in FLINK dynamic tables?

Hi everyone, I'm trying out the SQL API in flink 1.6 I first convert a data stream to a table (let's call it tableA) using tableEnv.registerDataStream, then perform queries on tableA and finally convert the result to a retract stream. My question is: Is there a way to clean up/ remove

Re: Queryable State race condition or serialization errors?

Hi Gary. Thanks for the reply. I am using RocksDBStateBackend though. Best, Chen -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Queryable State race condition or serialization errors?

Hi Guys, I observed some strange behaviors while using Queryable state with Flink 1.6.2. Here is the story: My state is of type MapState[String, Map[String, String]]. the inner map is frequently updated. Upon querying, sometimes the returned inner map can miss some fields. What's more, sometimes

host name in REST API

Hi everyone, I am having a problem and hope some one can provide insights here. I am trying to make use of the queryable state feature. due to https://issues.apache.org/jira/browse/FLINK-10225, I have to know on which task managers the job is deployed. My idea is to use REST API

Slots Leak Observed when

Hi guys, Out team is observing a stability issue on our Standalone Flink clusters. Background: The kafka cluster our flink jobs read from/ write to have some issues and every 10 to15 mins one of the partition leaders switch. This causes jobs that write to/ read from that topic fail and restart.