re: About Sliding window

Hi Kostas： Thanks for your answer. So in your previous figure (yesterday) when e3 arrives, also e2 should be included in the result, right? --zhangrucong: In Oct 11 email, e2 is coming at 9:02, e3 is coming at 9:07, and the aging time is 5 mins. So When e3 coming, e2 is aged. E2 is not in

Re: java.lang.IllegalArgumentException: JDBC-Class not found. - org.postgresql.jdbc.Driver

Hi Sunny, As stated by Fabian try to see whether including the postgres classes in the shaded jar solves the problem. If it doesn't, you're probably hitting the same problem i had with an older version of Flink ( https://issues.apache.org/jira/plugins/servlet/mobile#issue/FLINK-4061) and this you

question about making a temporal Graph with Gelly

​​Hi there, I'm currently working on making a Temporal Graph with Gelly, a Temporal graph is a graph where edges have 2 extra values namely a beginning and ending time. I started with this project a couple of weeks ago, since i don't have much experience with Gelly or Flink i wanted to ask you

Re: Error with table sql query - No match found for function signature TUMBLE(, )

Hi Pedro, support for window aggregations in SQL and Table API is currently work in progress. We have a pull request for the Table API and will add this feature for the next release. For SQL we depend on Apache Calcite to include the TUMBLE keyword in its parser and optimizer. At the moment the

Re: Distributing Tasks over Task manager

Hi Robert, Thanks for your suggestions. We are using the DataStream API and I tried it with disabling it completely, but that didn't help. I attached the plan and to add some context, it starts with a Kafka source followed by a map operation ( parallelism 4). The next map is the expensive

Re: Flink Kafka Consumer Behaviour

Hi Janardhan/Stephan, I just figured out what the issue is (Talking about Flink KafkaConnector08, don't know about Flink KafkaConnector09) The reason why- bin/kafka-consumer-groups.sh --zookeeper --describe --group is not showing any result is because of the absence of the

Re: Flink Kafka connector08 not updating the offsets with the zookeeper

Hi Robert, Thanks for your response. I just figured out what the issue is. The reason why- bin/kafka-consumer-groups.sh --zookeeper --describe --group is not showing any result is because of the absence of the /consumers//owners/ in the zookeeper. The flink application is creating and

Error with table sql query - No match found for function signature TUMBLE(, )

Hello, I am trying to build an query using the StreamTableEnvironment API. I Am trying to build this queries with tableEnvironment.sql("QUERY") so that I can in the future load those queries from a file. Code source: Table accesses = tableEnvironment.sql

Re: bucketing in RollingSink

Hi Robert, Thanks! I’ll likely pursue option #2 and see if I can copy over the code from org.apache.flink….fs.bucketing. Do you know a general timeline for when 1.2 will be released or perhaps a location where I could follow its progress? Thanks again! From: Robert Metzger

Re: bucketing in RollingSink

Hi Robert, I see two possible workarounds: 1) You use the unreleased Flink 1.2-SNAPSHOT version. From time to time, there are some unstable commits in that version, but most of the time, its quite stable. We provide nightly binaries and maven artifacts for snapshot versions here:

Re: Distributing Tasks over Task manager

Hi Jürgen, Are you using the DataStream or the DataSet API? Maybe the operator chaining is causing too many operations to be "packed" into one task. Check out this documentation page: https://ci.apache.org/projects/flink/flink-docs-master/dev/datastream_api.html#task-chaining-and-resource-groups

Re: About Sliding window

Hello, So in your previous figure (yesterday) when e3 arrives, also e2 should be included in the result, right? In this case, I think that what you need is a Session window with gap equal to your event aging duration and an evictor that evicts the elements that lag behind more than the gap

Keyed join Flink Streaming

Hi, I have 2 streams which are partitioned based on key field. I want to join those streams based on key fields on windows. This is an example I saw in the flink website: val firstInput: DataStream[MyType] = ... val secondInput: DataStream[AnotherType] = ... val firstKeyed =

Re: Tumbling window rich functionality

Hi, apply() will be called for each key. On Wed, Oct 12, 2016 at 2:25 PM, Swapnil Chougule wrote: > Thanks Aljoscha. > > Whenever I am using WindowFunction.apply() on keyed stream, apply() will > be called once or multiple times (equal to number of keys in that windowed

re: About Sliding window

Hi Kostas: It doesn’t matter. Can you see the picture? My user case is: 1、The events are coming according to the following order [cid:image004.png@01D224D0.A15CB290] At 9:01 e1 is coming At 9:02 e2 is coming At 9:06 e3 is coming At 9:08 e4 is coming The time is system time. 2、And

Re: About Sliding window

Hello again, Sorry for the delay but I cannot really understand your use case. Could you explain a bit more what do you mean by “out-of-date” event and “aging” an event? Also your windows are of a certain duration or global? Thanks, Kostas > On Oct 11, 2016, at 3:04 PM, Zhangrucong

Re: Exception while running Flink jobs (1.0.0)

Ok, thanks for the update Ufuk! Let me know if you need test or anything! Best, Flavio On Wed, Oct 12, 2016 at 11:26 AM, Ufuk Celebi wrote: > No, sorry. I was waiting for Tarandeep's feedback before looking into > it further. I will do it over the next days in any case. > > On

Re: Tumbling window rich functionality

Thanks Aljoscha. Whenever I am using WindowFunction.apply() on keyed stream, apply() will be called once or multiple times (equal to number of keys in that windowed stream)? Ex: DataStream dataStream = env .socketTextStream("localhost", ) .flatMap(new

bucketing in RollingSink

Hi Flinksters, At one stage in my data stream, I want to save the stream to a set of rolling files where the file name used (i.e. the bucket) is chosen based on an attribute of each data record. Specifically, I’m using a windowing function to create aggregates of certain metrics and I want to

AW: What is the best way to load/add patterns dynamically (at runtime) with Flink?

Hey, I face the same problem and decided to go with your third solution. I use Groovy as the scripting language, which has access to Java classes and therefore also to Flink constructs like Time.seconds(10). See below for an example of a pattern definition with Groovy: private static Binding

Distributing Tasks over Task manager

Hi, we currently have an issue with Flink, as it allocates many tasks to the same task manager and as a result it overloads it. I reduced the amount of task slots per task manager (keeping the CPU count) and added some more servers but that did not help to distribute the load. Is there some

Re: jdbc.JDBCInputFormat

Hi guys, I am facing following error message in flink scala JDBC wordcount. could you please advise me on this? *Information:12/10/2016, 10:43 - Compilation completed with 2 errors and 0 warnings in 1s 903ms* */Users/janaidu/faid/src/main/scala/fgid/JDBC.scala* *Error:(17, 67) can't expand

Re: Flink Kafka connector08 not updating the offsets with the zookeeper

Hi Anchit, Can you re-run your job with the debug level for Flink set to DEBUG? Then, you should see the following log message every time the offset is committed of Zookeeper: "Committing offsets to Kafka/ZooKeeper for checkpoint" Alternatively, can you check whether the offsets are available

Re: "Slow ReadProcessor" warnings when using BucketSink

Hi, I haven't seen this error before. Also, I didn't find anything helpful searching for the error on Google. Did you check the GC times also for Flink? Is your Flink job doing any heavy tasks (like maintaining large windows, or other operations involving a lot of heap space?) Regards, Robert

Re: Exception while running Flink jobs (1.0.0)

No, sorry. I was waiting for Tarandeep's feedback before looking into it further. I will do it over the next days in any case. On Wed, Oct 12, 2016 at 10:49 AM, Flavio Pompermaier wrote: > Hi Ufuk, > any news on this? > > On Thu, Oct 6, 2016 at 1:30 PM, Ufuk Celebi

Re: Data Transfer between TM should be encrypted

Hi, I think that pull request will be merged for 1.2. On Fri, Oct 7, 2016 at 6:26 PM, vinay patil wrote: > Hi Stephan, > > https://github.com/apache/flink/pull/2518 > Is this pull request going to be part of 1.2 release ? Just wanted to get > an idea on timelines so

Re: Exception while running Flink jobs (1.0.0)

Hi Ufuk, any news on this? On Thu, Oct 6, 2016 at 1:30 PM, Ufuk Celebi wrote: > I guess that this is caused by a bug in the checksum calculation. Let > me check that. > > On Thu, Oct 6, 2016 at 1:24 PM, Flavio Pompermaier > wrote: > > I've ran the job

Re: mapreduce.HadoopOutputFormat config value issue

Hi Shannon, I tried to reproduce the problem in a unit test without success. My test configures a HadoopOutputFormat object, serializes and deserializes it, cally open, and verifies that a configured String property is present in the getRecordWriter() method. Next I would try to reproduce the

Re: What is the best way to load/add patterns dynamically (at runtime) with Flink?

I've been thinking in several options to solve this problem: 1. I can use Flink savepoints in order to save the application state , change the jar file and submit a new job (as the new jar file with the patterns added/changed). The problem in this case is to be able to correctly handle the