Re: Flink 1.1.3 OOME Permgen

Hi, could you somehow provide us a heap dump from a TM that run for a while (ideally, shortly before an OOME)? This would greatly help us to figure out if there is a classloader leak that causes the problem. Best, Stefan > Am 29.11.2016 um 18:39 schrieb Konstantin Knauf > : > > Hi everyone,

Query regarding tumbling event time windows with ingestion time

Hi, Suppose we have a stream like this. someStream.timeWindow(Time.minutes(15)).apply { operation A }.keyby("").window(TumblingEventTimeWindows.ofseconds(5)).apply { operation B }.keyby("").window(TumblingEventTimeWindows.ofseconds(5)).apply { operation C } Say operation A emits som

Re: Problem - Negative currentWatermark if the watermark assignment is made before connecting the streams

Hi Vinay , I'm simply using Netbeans Debugger. Regards, Pedro - Best Regards, Pedro Chaves -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Problem-Negative-currentWatermark-if-the-watermark-assignment-is-made-before-connecting-the-strea

Re: Collect() freeze on yarn cluster on strange recover/deserialization error

Hey Arnaud, could this be a left over job that is recovered from ZooKeeper? Recovery only happens if the configured ZK root contains data. A job is removed from ZooKeeper only if it terminates (e.g. finishes, fails terminally w/o restarting, cancelled). If you just shut down the cluster this i

Flink 1.1.3 OOME Permgen

Hi everyone, since upgrading to Flink 1.1.3 we observe frequent OOME Permgen Taskmanager Failures. Monitoring the permgen size on one of the Taskamanagers you can see that each Job (New Job and Restarts) adds a few MB, which can not be collected. Eventually, the OOME happens. This happens with

Collect() freeze on yarn cluster on strange recover/deserialization error

Hello, I have a Flink 1.1.3 batch application that makes a simple aggregation but freezes when collect() is called when the app is deployed on a ha-enabled yarn cluster (it works on a local cluster). Just before it hangs, I have the following deserialization error in the logs : (...) 2016-11-29

Re: JVM Non Heap Memory

Hey Daniel! Thanks for reporting this. Unbounded growth of non-heap memory is not expected.  What kind of Threads are you seeing being spawned/lingering around? As a first step, could you try to disable checkpointing and see how it behaves afterwards? – Ufuk On 29 November 2016 at 17:32:32, D

Re: JVM Non Heap Memory

Hello, Nope I am using Hadoop HDFS, as state backend, Kafka, as source, and a HttpClient as a Sink, also Kafka as Sink. So it's possible that the state backend is the culprit? Curious thing is even when no jobs are running streaming or otherwise, the JVM Non-HEAP stays the same. Which I find

Re: JVM Non Heap Memory

Are you using the RocksDB backend in native mode? If so then the off-heap memory may be there. On Tue, Nov 29, 2016 at 9:54 AM, wrote: > i have the same problem,but i put the flink job into yarn. > but i put the job into yarn on the computer 22,and the job can success > run,and the jobmanager is

RE: Problems with RollingSink

Hi Fabian and Kostas: Thanks! Both approaches look good, although I prefer using union, since then I can apply next steps to a single stream. Here is my code now: //For each data source val normalizedStreams = dataSources.map(source => { val sourceName = source._1 val dataSource = source.

回复：JVM Non Heap Memory

i have the same problem,but i put the flink job into yarn. but i put the job into yarn on the computer 22,and the job can success run,and the jobmanager is 79 and taskmanager is 69,they three different compu345ter, however,on computer 22,the pid=3463,which is the job that put into yarn,is have 2.

Re: Problems with RollingSink

Hi Diego, If you want the data of all streams to be written to the same files, you can also union the streams before sending them to the sink. Best, Fabian 2016-11-29 15:50 GMT+01:00 Kostas Kloudas : > Hi Diego, > > You cannot prefix each stream with a different > string so that the paths do no

Re: Problems with RollingSink

Hi Diego, You cannot prefix each stream with a different string so that the paths do not collide? If I understand your use-case correctly, this might work. Cheers, Kostas > On Nov 29, 2016, at 10:04 AM, Diego Fustes Villadóniga > wrote: > > Hi Kostas, > > Thanks for your reply. > > The p

About delta awareness caches

Hi all, Recently I tried to transfer some old applications from Storm to Flink. In Storm, the window implementation (TupleWindow) gets two methods named getNew() and getExpired() which supply the delta information of a window and therefore we wrote some stateful caches that are aware of them. Howe

JVM Non Heap Memory

Hello, Is it common to have high usage of Non-Heap in JVM ? I am running flink in stand-alone cluster and in docker, with each docker bieng capped at 6G of memory. I have been struggling to keep memory usage in check. The non-heap increases to no end. It start with just 100MB of usage and a

CEP issue

Hello, I am currently building a multi-tenant monitoring application and exploring the effectiveness of different Complex Event Processors (CEP) and whether or not this would be a potential solution for what I want to achieve. I have created a small test application which utilises Flink and its

Re: How to let 1.1.3 not drop late events as verion 1.0.3 does

Hi Aljoscha, Yes that it true, but user who are new to flink does not know how to handle the case where they cannot discard late data, so they end up writing custom trigger which was not the case in 1.0.3 Regards, Vinay Patil On Tue, Nov 29, 2016 at 4:29 PM, Aljoscha Krettek [via Apache Flink U

flink-job-in-yarn,has max memory

Hi, i have a flink job,and abt assembly to get a jar file,so i put it to yarn and run it,use the follow commend:/home/www/flink-1.1.1/bin/flink run \-m yarn-cluster \-yn 1 \-ys 2 \-yjm 4096 \-ytm 4096 \--class skRecomm.

Re: How to let 1.1.3 not drop late events as verion 1.0.3 does

Yes, but the allowedLateness() setting is that flag, no? On Tue, 29 Nov 2016 at 11:04 vinay patil wrote: > Yes, That's what I have done. > > IMO flink should allow user to decide whether to discard late data or not > by having some flag, because for some cases users cannot afford to loose > any

Re: How to let 1.1.3 not drop late events as verion 1.0.3 does

Yes, That's what I have done. IMO flink should allow user to decide whether to discard late data or not by having some flag, because for some cases users cannot afford to loose any data. Regards, Vinay Patil On Tue, Nov 29, 2016 at 3:16 PM, Maximilian Michels [via Apache Flink User Mailing List

Re: How to let 1.1.3 not drop late events as verion 1.0.3 does

Setting allowedLateness to Long.MAX_VALUE and returning TriggerResult.FIRE_AND_PURGE in the custom trigger should do the trick. -Max On Mon, Nov 28, 2016 at 2:57 PM, vinay patil wrote: > Hi Sendoh, > > I have used the Custom Trigger which is same as 1.0.3 EventTimeTrigger, and > kept the allowe

RE: Problems with RollingSink

Hi Kostas, Thanks for your reply. The problem is at the initialization of the job. The reason was that I was using the same HDFS path as sink for 3 different streams, which is something that I would like. I can fix it by using different paths for each stream. Maybe there is a way to achieve t