Re: Correlation between data streams/operators and threads

Bump. On Wed, Nov 15, 2017 at 12:34 AM, Shailesh Jain wrote: > 1. Single data source because I have one kafka topic where all events get > published. But I am creating multiple data streams by applying a series of > filter operations on the single input stream, to

Task manager suddenly lost connection to JM

Hi team, I see an wired issue that one of my TM suddenly lost connection to JM. Once the job running on the TM relocated to a new TM, it can reconnect to JM again. And after a while, the new TM running the same job will repeat the same process. It is not guaranteed the troubled TMs can reconnect

Re: Job Manager Configuration

I have an IO-dominated batch job with 471 distinct tasks (3786 tasks with parallelism) running on 8 nodes with 12 GiB of memory and 4 CPUs each. I haven’t had any problems adding additional tasks except for 1) tasks timing out the first time the cluster is started (I suppose the JVM needs to

Building flink and libraries from source

Hello, A quick question. I am building flink from source after applying some patch. There are some changes in CEP library also. So I want to use that source generated lib instead of release maven repo from central maven. Is anybody has direct script for publishing in local? is generated jars in

Re: org.apache.flink.runtime.io.network.NetworkEnvironment causing memory leak?

Sorry, the "killed" I mean here is JM lost the TM. The TM instance is still running inside kubernetes, but it is not responding to any requests, probably due to high load. And from JM side, JM lost heartbeat tracking of the TM, so it marked the TM as died. The „volume“ of Kafka topics, I mean,

Re: org.apache.flink.runtime.io.network.NetworkEnvironment causing memory leak?

Hi, > In addition to your comments, what are the items retained by > NetworkEnvironment? They grew seems like indefinitely, do they ever reduce? > Mostly the network buffers, which should be ok. They are always recycled and should not be released until the network environment is GCed. > I

Re: org.apache.flink.runtime.io.network.NetworkEnvironment causing memory leak?

Thanks a lot! This is very helpful. In addition to your comments, what are the items retained by NetworkEnvironment? They grew seems like indefinitely, do they ever reduce? I think there is a GC issue because my task manager is killed somehow after a job run. The duration correlates to the volume

Re: R/W traffic estimation between Flink and Zookeeper

Great, thanks for the info, Stefan. On Thu, Nov 16, 2017, 01:59 Stefan Richter wrote: > Hi, > > I think Zookeeper is only used as a meta data store in HA mode. > Interactions with ZK are not part of the per-record stream processing code > paths of Flink. Things that

Re: [Flink] merge-sort for a DataStream

Hi Jiewen, Since a DataStream can have infinite number of elements, you can't globally sorted all the elements. If the number of element is finite, you can use the DataSet API, which will look smth like this DataSet a; DataSet aFlatten = a.flatMap(..); DataSet aSorted =

Re: [Flink] How to Converting DataStream to Dataset or Table?

Hi Richard, in general it is difficult to sort a DataStream that is potentially neverending. However, if you use Flink's event-time semantics with watermarks that indicate that you stream is complete until a certain point you can sort it. The Table API will offer a a sort option in 1.4

Re: [Flink] How to Converting DataStream to Dataset or Table?

Hi, does this answer your question: https://flink.apache.org/news/2017/03/29/table-sql-api-update.html ? Best, Stefan > Am 15.11.2017 um 20:33 schrieb Richard Xin : > > I have DataStream, is there a

Re: Apache Flink - Question about TriggerResult.FIRE

Yes, all of this is correct. Sliding windows in fact look like completely separate windows to the windowing system. Best, Aljoscha > On 16. Nov 2017, at 10:15, Stefan Richter wrote: > > Hi, > > I think the effect is pretty straight forward, the elements in a

Akka configuration setting missing if RemoteEnvironment job is started from CLI

Good morning, I have some problems using a Flink RemoteEnvironment in my Java application. If I run the following code [1] directly from the IDE it runs as expected. However, if I package a jar and run this from a CLI via `java -cp ...` I get the following error [2]. I want to use the

Re: R/W traffic estimation between Flink and Zookeeper

Hi, I think Zookeeper is only used as a meta data store in HA mode. Interactions with ZK are not part of the per-record stream processing code paths of Flink. Things that are written to ZK can (also depending on your job) include e.g. the job graph, Kafka offsets, or the meta data about

Re: org.apache.flink.runtime.io.network.NetworkEnvironment causing memory leak?

Hi, I cannot spot anything that indicates a leak from your screenshots. Maybe you misinterpret the numbers? In your heap dump, there is only a single instance of org.apache.flink.runtime.io.network.NetworkEnvironment and it retains about 400,000,000 bytes from being GCed because it holds

Re: RestartStrategies & checkpoints

Hi Stefan, thanks for your answer. I should have probably understand this behavior from https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/restart_strategies.html#overview "If checkpointing is not enabled, the “no restart” strategy is used. If checkpointing is activated and the

Re: Apache Flink - Question about TriggerResult.FIRE

Hi, I think the effect is pretty straight forward, the elements in a window are not purged if the trigger is only FIRE and not FIRE_AND_PURGE. Unfortunately, your question is a bit unclear about what exactly you mean by „new window“: a truly „new“ window or another triggering of the previous

Re: RestartStrategies & checkpoints

Hi, if the restart strategy that you set is configured to do restarts and you have checkpoints enabled, it will try to recover from the latest checkpoint. Is there any confusing point in the documentation that made this unclear for you, which we could improve? Best, Stefan > Am 15.11.2017

Re: external checkpoints

Hi, thanks for the answer. I can use the first option (REST API). for some reason it is undocumented in flink documentation (https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html) regarding the second option, configure each job with an externalized checkpoint