Making external calls from a FlinkKafkaPartitioner

We have a system where the Kafka partition a message should go into is a function of a value in the message. Often, it’s value % # partitions, but for some values it’s not - it’s a specified list of partitions that changes over time. Our “simple Java library” that produces messages for this

Incremental checkpointing documentation

There doesn't appear to be much in the way of documentation for incremental checkpointing other than how to turn it on. That leaves a lot of questions unanswered. What is the interaction of incremental checkpointing and external checkpoints? Any interaction with the

Initialise side input state

Hi Flink users, I'm struggling with some basic concept and would appreciate some help. I have 2 Input streams, one is fast event stream and one is slow changing dimension. They have the same key and I use CoProcessFunction to store slow data in state and enrich fast data from this state.

Re: Docker-Flink Project: TaskManagers can't talk to JobManager if they are on different nodes

I think I may be getting somewhere with this. I have opened the blob.server.port and the query.server.port on the TaskManager service, and I can now connect to JobManager from nodes in the same subnet. However, nodes that are located in different clouds don't seem to be able to resolve the

Re: Docker-Flink Project: TaskManagers can't talk to JobManager if they are on different nodes

Hi Piotr, Thank you very much for your reply. Yes, I have tried to open these ports when I create the services. If I create them with: docker service create --name jobmanager --env JOB_MANAGER_RPC_ADDRESS=jobmanager -p 8081:8081 -p 6123:6123 -p 48081:48081 --network overlay

Re: Flink send checkpointing message in IT

Chesnay, thanks for your reply, it was very helpful, but I took logic from this test template and tried to reuse it in my IT case, but found one more issue. I’ve registered an accumulator in my source function, and for it’s value, as specified in the specified example. When accumulator has an

Negative values using latency marker

Hi group, Can someone maybe elaborate how can latency gauge shown by latency marker be negative? 2017-11-02 18:54:56,842 INFO com.citi.artemis.flink.reporters.ArtemisReporter - [Flink-MetricRegistry-1] 127.0.0.1.taskmanager.f40ca57104c8642218e5b8979a380ee4.Flink Streaming Job.Sink:

Re: Batch job per stream message?

Hi Tomas, I'm not familiar with the details of the AsyncFunction, but I'd interpret this as follows: - you can make one async call in the asyncInvoke method. - this call will result in a callback and from that one callback you can emit a single result by calling AsyncCollector.collect() The

Re: Docker-Flink Project: TaskManagers can't talk to JobManager if they are on different nodes

Did you try to expose required ports that are listed in the README when starting the containers? https://github.com/apache/flink/tree/master/flink-contrib/docker-flink Ports: • The Web Client is on port 48081

Docker-Flink Project: TaskManagers can't talk to JobManager if they are on different nodes

I am using the Docker-Flink project in: https://github.com/apache/flink/tree/master/flink-contrib/docker-flink I am creating the services with the following commands: docker network create -d overlay overlay docker service create --name jobmanager --env JOB_MANAGER_RPC_ADDRESS=jobmanager -p

Re: Use keyBy to deterministically hash each record to a processor/task/slot

Hello Dongwon, Thanks a lot for your excellent reply! Seems we have the same problem. Still your solution is less hard coded than mine. Thanks a lot! I am also looking forward to see a capability of creating a custom partitioner for keyBy() in Flink. Best, Max -- Sent from:

Re: Use keyBy to deterministically hash each record to a processor/task/slot

Hello, As I need to generate the same number of keys as that of partitions, I also suffer from this problem [1]: My current solution is to generate enough keys until I have at least one key per partition, which looks very stupid to me (I copy and paste my code below). If Flink changes its way to

Re: Use keyBy to deterministically hash each record to a processor/task/slot

Hello Tony, Thanks a lot for your answer. Now I know exactly what happens with keyBy function, yet still I haven't figured out a proper (non hard coded way) to deterministically send a tuple to each key. If somenone from the Flink team could help it would be great! Max -- Sent from: