Flink Streaming Core 0.10 in maven repos

Hello, Small question: I can't find the Streaming Core component in 0.10 version in the maven repo : http://mvnrepository.com/artifact/org.apache.flink/flink-streaming-core Thus in my pom file this artifact is the only part of my Flink's dependencies to stay in 0.10-SNAPSHOT version. Is there

Re: Flink Streaming Core 0.10 in maven repos

Hi Arnaud! In 0.10 , we renamed the dependency to "flink-streaming-java" (and flink-streaming-scala"), to be more in line with the structure of the dependencies on the batch side. Just replace "flink-streaming-core" with "flink-streaming-java"... Greetings, Stephan On Mon, Nov 23, 2015 at

Re: ReduceByKeyAndWindow in Flink

Thanks! @Fabian: Yepp, but this still results in multiple outputs per window, because the maximum is emitted for every key. @Gyula: Yepp, that's the second bullet point from my question ;) The way I implemented it, it basically doubles the latency, because the timeWindowAll has to wait for the

Re: ReduceByKeyAndWindow in Flink

Yes, you are right I think we should have some nice abstractions for doing this. Before the rewrite of the windowing runtime to support out-of-order events, we had abstractions for supporting this but that code was not feasible from performance perspective. (The result of a keyed window reduce

Re: YARN High Availability

Hey Gwenhaël, the restarting jobs are most likely old job submissions. They are not cleaned up when you shut down the cluster, but only when they finish (either regular finish or after cancelling). The workaround is to use the command line frontend: bin/flink cancel JOBID for each RESTARTING

Re: Flink Streaming Core 0.10 in maven repos

There is also this guide: https://cwiki.apache.org/confluence/display/FLINK/Migration+Guide%3A+0.9.x+to+0.10.x On Monday, 23 November 2015, Stephan Ewen wrote: > Hi Arnaud! > > In 0.10 , we renamed the dependency to "flink-streaming-java" (and > flink-streaming-scala"), to be

Re: ReduceByKeyAndWindow in Flink

Hi, Can't you use a second keyed window (with the same size) and apply .max(...)? -Matthias On 11/23/2015 11:00 AM, Konstantin Knauf wrote: > Hi Fabian, > > thanks for your answer. Yes, that's what I want. > > The solution you suggest is what I am doing right now (see last of the > bullet

Re: ReduceByKeyAndWindow in Flink

One addition: You can set the system to use "ingestion time", which gives you event time with auto-generated timestamps and watermarks, based on the time that the events are seen in the sources. That way you have the same simplicity as processing time, and you get the window alignment that

Re: ReduceByKeyAndWindow in Flink

Hi, @Konstantin: are you using event-time or processing-time windows. If you are using processing time, then you can only do it the way Fabian suggested. The problem here is, however, that the .keyBy().reduce() combination would emit a new maximum for every element that arrives there and you

[VOTE] Release Apache Flink 0.10.1 (release-0.10.0-rc1)

Hi All, this is the first bugfix release for the 0.10 series of Flink. I've CC'ed the user@ list if users are interested in helping to verify the release. It contains fixes for critical issues, in particular: - FLINK-3021 Fix class loading issue for streaming sources - FLINK-2974 Add periodic

RE: YARN High Availability

OK, I understand. Maybe we are not really using flink as you intended. The way we are using it, one cluster equals one job. That way we are sure to isolate the different jobs as much as possible and in case of crashes / bugs / (etc) can completely kill one cluster without interfering with the

Re: ReduceByKeyAndWindow in Flink

If you set the key to the time attribute, the "old" key is no longer valid. The streams are organized by time and only one aggregate for each window-time should be computed. This should do what you are looking for: DataStream .keyBy(_._1) // key by orginal key .timeWindow(..) .apply(...)

Re: How to pass hdp.version to flink on yarn

Hi, In Flink the configuration parameter for passing custom JVM options is "env.java.opts". I would recommend to put it into the conf/flink-config.yaml like this: env.java.opts: "-Dhdp.version=2.3.0.0-2557 -Dhdp.version=2.3.0.0-2557" Please let me know if this works. Maybe you are the first

Re: YARN High Availability

The problem is the execution graph handle which is stored in ZooKeeper. You can manually remove it via the ZooKeeper shell by simply deleting everything below your `recovery.zookeeper.path.root` ZNode. But you should be sure that the cluster has been stopped before. Do you start the different

Cancel Streaming Job

Hi All, Is there any way to stop/cancel the job that's restarting ? I have already stop the cluster and start it again but seems it's still restarting in dashboard. I also try to cancel the job via CLI by running bin/flink cancel but it's not working. Cheers -- Welly Tambunan Triplelands

Re: Cancel Streaming Job

Hi All, Finally i've found the solution for killing the job manager. https://flink.apache.org/faq.html#i-cant-stop-flink-with-the-provided-stop-scripts-what-can-i-do But i do really hope that we have that cancel button for restarting job. Cheers On Tue, Nov 24, 2015 at 8:30 AM, Welly

Running Flink in Cloudfoundry Environment

Hi Is it possible to run Flink in Cloudfoundry Environment? if yes, How can we achive this? Any help is appreciated. Thanks in Advance. Thanks, Madhu

Re: Cancel Streaming Job

Hi! This issue has been fixed very recently and the fix will go into the upcoming bugfix release. (0.10.1) Should be out in the next few days :) Cheers Gyula On Tue, Nov 24, 2015 at 4:49 AM Welly Tambunan wrote: > Hi All, > > Finally i've found the solution for killing the

RE: YARN High Availability

We are not yet using HA in our cluster instances. But yes, we will have to change the zookeeper.path.root ☺ We package our jobs with their own config folder (we don’t rely on flink’s config folder); we can put the maven project name into this property then they will have different values ☺