Re: Kafka Monitoring

Hi Daniel, Yes Now I am able to see it , this was just a dummy code I was running on local VM. However on cluster , I had enabled checkpointing, still I was not able to see the consumers, I guess I have to put the brokerPath as you have provided in the properties. >From where did you get the

An idea for a parallel AllWindowedStream

Hi, As a self training exercise I've defined a class extending WindowedStream for implementing a proof of concept for a parallel version of AllWindowStream /** * Tries to create a parallel version of a AllWindowStream for a DataStream * by creating a KeyedStream by using as key the hash of the

Re: Why did the Flink Cluster JM crash?

Ok. There is an OOM exception...but this used to work fine with the same configurations.There are four nodes: beam1 through 4.The Kafka partitions are 4096 > 3584 deg of parallelism. jobmanager.rpc.address: beam1jobmanager.rpc.port: 6123jobmanager.heap.mb: 1024taskmanager.heap.mb:

Re: Why did the Flink Cluster JM crash?

OOps! sorry Till.I replicated it and I see exceptions in JM logs.How can I send the logs to you? or what "interesting" part of it do you need so I can copy/paste it here...Thanks From: Till Rohrmann To: user@flink.apache.org; amir bahmanyari

Re: Kinesis Connector Dependency Problems

Thanks for checking Steffen and Craig! If the master does not build with 3.0.3, we should updated the docs. 2016-11-08 23:38 GMT+01:00 Foster, Craig : > Yes, with Maven 3.0.5-based jar I’m seeing the same error. I cannot seem > to get the runtime to build with 3.0.3. > > On

Re: Why did the Flink Cluster JM crash?

Clean .No errors...no exceptions :-(Thanks Till. From: Till Rohrmann To: user@flink.apache.org; amir bahmanyari Sent: Tuesday, November 8, 2016 2:11 PM Subject: Re: Why did the Flink Cluster JM crash? Hi Amir, what does the JM logs

Re: Why did the Flink Cluster JM crash?

Hi Amir, what does the JM logs say? Cheers, Till On Tue, Nov 8, 2016 at 9:33 PM, amir bahmanyari wrote: > Hi colleagues, > I started the cluster all fine. Started the Beam app running in the Flink > Cluster fine. > Dashboard showed all tasks being consumed and open for

Too few memory segments provided. Hash Table needs at least 33 memory segments.

Dear community, I have a problem which I hope you'll be able to help with. I apologize in advance for the verbosity of the post. I am running the Flink standalone cluster (not even storing to the filesystem) with 2 Docker containers. I set the image of the Dockerfile for Flink 1.1.2, which was

Why did the Flink Cluster JM crash?

Hi colleagues,I started the cluster all fine. Started the Beam app running in the Flink Cluster fine.Dashboard showed all tasks being consumed and open for business.I started sending data to the Beam app, and all of the sudden the Flink JM crashed.Exceptions below.Thanks+regardsAmir

Re: Kafka Monitoring

Hello, On flink do you have the checkpoint enabled ? env.enableCheckpointing(interval = CHKPOINT_INTERVAL) Regards, Daniel Santos On 11/08/2016 12:30 PM, vinay patil wrote: Yes Kafka and Flink connect to that zookeeper only. Not sure why it is not listing the consumer Regards, Vinay

Re: Last event of each window belongs to the next window - Wrong

Hi Aljoscha, Thanks for the question. I key by source ID, because I want to isolate users per source. If I would key by User ID, I would need to have a logic to create sessions based on time. But I would like to create my sessions based on user ID changes in the events stream for each source.

Re: Last event of each window belongs to the next window - Wrong

Hi Till, Thanks for your answer and the hint. However, the trigger must be based on user ID changes and not time. I tried this approach too, but I end-up having some events with the same userID that belong to the next window. I finally solved the problem by implementing a custom WindowFunction

Re: Cannot see all events in window apply() for big input

Flink does not support per key watermarks or type sensitive watermarks. The underlying assumption is that you have a global watermark which defines the progress wrt to event time in your topology. The easiest way would be to have an input which has a monotonically increasing timestamp.

Re: Window PURGE Behaviour 1.0.2 vs 1.1.3

Hi Aljoscha, interesting, this explains it. Well, in our case the PURGE in the onProcessingTimeTimer is only used to clear KeyValueStates*, and at this point there are usually no records in the window state. Any Ideas? I do have a workaround with an evictor, but it seemed to be unnecessarily

Window PURGE Behaviour 1.0.2 vs 1.1.3

Hi everyone, I just migrated a streaming Job from 1.0.2 to 1.1.3 and stumbled across a problem concerning one of our custom triggers. The trigger basically FIRE_AND_PURGEs multiple times in onElement() and the window is PURGEd onProcessingTimeTimer(), but it seems that the all registered

Custom Window Assigner With Lateness

Is it possible in a custom window assigner to determine if an object has appeared after the watermark has passed? I want to have a standard event time tumbling window but custom logic for late data. From what I can tell there is no way from within the WindowAssigner interface to determine if an

Re: Cannot see all events in window apply() for big input

Thank you for confirming. What would you think an efficient way not having global watermark? The following logic fails to build Watermark per KeyStream: jsonStreams.keyBy(new JsonKeySelector()).assignTimestampsAndWatermarks(new JsonWatermark()).keyBy(JsonKeySelector()).window( So, using

Re: Kinesis Connector Dependency Problems

Hi Fabian, I can confirm that the behaviour is reproducible with both, Maven 3.3.9 and Maven 3.0.5. Cheers, Steffen Am 8. November 2016 11:11:19 MEZ, schrieb Fabian Hueske : >Hi, > >I encountered this issue before as well. > >Which Maven version are you using? >Maven 3.3.x

Re: Cannot see all events in window apply() for big input

Hi Sendoh, Flink should actually never lose data unless it is so late that it arrives after the allowed lateness. This should be independent of the total data size. The watermarks are indeed global and not bound to a specific input element or a group. So for example if you create the watermarks

Re: Cannot see all events in window apply() for big input

Hi, Would the issue be events are too out of ordered and the watermark is global? We want to count event per event type per day, and the data looks like: eventA, 10-29-XX eventB,, 11-02-XX eventB,, 11-02-XX eventB,, 11-03-XX eventB,, 11-04-XX eventA, 10-29-XX eventA, 10-30-XX eventA,

Re: Listening to timed-out patterns in Flink CEP

Hi David, sorry for my late reply. I just found time to look into the problem. You were right with your observation that the CEP operator did not behave as I've described it. The problem was that the time of the underlying NFA was not advanced if there were no events buffered in the CEP operator

Re: Kafka Monitoring

Yes Kafka and Flink connect to that zookeeper only. Not sure why it is not listing the consumer Regards, Vinay Patil On Tue, Nov 8, 2016 at 5:36 PM, Daniel Santos [via Apache Flink User Mailing List archive.] wrote: > Hi, > > brokerPath is just

Re: Kafka Monitoring

Hi, brokerPath is just optional. Used if you want to have multile kafka clusters. Each kafka cluster would connect to the same brokerPath. Since I have multiple clusters I use the brokerPath. From the looks of it you dont. So never mind it doesn't matter. You only have one zookeeper correct

Re: Last-Event-Only Timer (Custom Trigger)

Hi Till, thank you for your reply. This is exactly what I was looking for! Flink continues to surprise me with its versatility. :) Best Regards, Julian 2016-11-07 16:47 GMT+01:00 Till Rohrmann : > Hi Julian, > > you can use the TriggerContext to register and unregister

Re: Kafka Monitoring

Hi Daniel, Yes I have specified the zookeeper host in server.properties file , so the broker is connected to zookeeper. https://kafka.apache.org/documentation#brokerconfigs -> according to this link, I guess all these configs are done in server.prop , so from where did you get kafka09 as

Re: Kafka Monitoring

Hi, Your kafka broker is connected to zookeeper I believe. I am using kafka 0.9.0.1 my self too. On kafka broker 0.9.0.1 I have configured the zookeeper connect to a path, for instances : zk1:2181,zk2:2181,zk3:2181/kafka09 https://kafka.apache.org/documentation#brokerconfigs Now on the

Re: Kafka Monitoring

Hi Daniel, I have the same properties set for the consumer and the same code *brokerspath only needed if you have set it on kafka config* -> I did not get this, do you mean to check the brokerspath in conf/server.properties file ? I have even tried by setting offset.storage property to

Re: Cannot see all events in window apply() for big input

Yes. the other job performs event time window and we tried 1.2-SNAPSHOT and 1.1.3. The old version 1.0.3 we lost much much less data. We tried both windowAll() and keyBy() window() already, and tried very tiny lag and window(1 millisecond). My doubt comes from smaller input works while bigger

Re: Kafka Monitoring

Hello, This is my config. On kafka props : val props = new Properties() props.setProperty("zookeeper.connect", zkHosts) props.setProperty("bootstrap.servers", kafHosts) props.setProperty("group.id", "prod") props.setProperty("auto.offset.reset", "earliest") Now for

Re: Flink on Yarn delegation token renewal

Thank you for the prompt reply Stefan! Cheers, Theo On 2016-11-08 11:29, Stefan Richter wrote: Hi, I think this problem tracked in this issue: https://issues.apache.org/jira/browse/FLINK-3670 . This means that the current master and upcoming release 1.2 should work correctly. Best, Stefan

Re: Flink on Yarn delegation token renewal

Hi, I think this problem tracked in this issue: https://issues.apache.org/jira/browse/FLINK-3670 . This means that the current master and upcoming release 1.2 should work correctly. Best, Stefan > Am 08.11.2016 um 10:25 schrieb Theofilos

Re: Kinesis Connector Dependency Problems

Yes this definitely looks like a similar issue. Once we shade the aws dependencies in the Kinesis connector, the problem should be (hopefully) resolved. I've added your problem description to the JIRA. Thanks for reporting it. Cheers, Till On Mon, Nov 7, 2016 at 8:01 PM, Foster, Craig

Flink on Yarn delegation token renewal

Hi everyone, I'm using Flink 1.1.3 with Hadoop 2.7.3 and was wondering about delegation token renewal when running Flink on Yarn. Yarn demands services to renew delegation tokens on their own and if not, Yarn jobs will fail after one week.

Re: Csv to windows?

Hi Yassine, thanks that explains it :) Best regards, Felix On Nov 7, 2016 21:28, "Yassine MARZOUGUI" wrote: > Hi Flelix, > > As I see in kddcup.newtestdata_small_unlabeled_index >

Re: Kafka Monitoring

Hi Limbo, I am using 0.9, I am not able to see updated results even after refreshing. There is some property that we have to set in order to make this work Regards, Vinay Patil On Tue, Nov 8, 2016 at 12:32 PM, limbo [via Apache Flink User Mailing List archive.]

Re: Memory on Aggr

thanks ¡¡ Now its clear for me 2016-11-08 9:23 GMT+01:00 Fabian Hueske : > Given the semantics described in the document the query can be computed in > principle. > However, if the query is not bounded by time, the required state might > grow very large if the number of

Re: Memory on Aggr

Given the semantics described in the document the query can be computed in principle. However, if the query is not bounded by time, the required state might grow very large if the number of distinct xx values grows over time. That's why we will probably enforce a time predicate or meta data that