Re: Adding a new operator

2015-04-26 Thread Markus Holzemer
Hey Andrea, perhaps you are looking at the wrong ReduceDriver? As you can see in the DriverStrategy enum there is several different ReduceDrivers depending on the strategy the optimizer chooses. best, Markus 2015-04-26 12:26 GMT+02:00 Andra Lungu lungu.an...@gmail.com: Hey guys, I am trying

Re: Fault Tolerance for Flink Iterations

2015-04-22 Thread Markus Holzemer
) and then start the program again with the source that refers to those files. That way, you go through the proper stack (optimizer and jobgraph generator) that inserts all the necessary partition and sort operations. Greetings, Stephan On Tue, Apr 21, 2015 at 8:58 AM, Markus Holzemer

Fault Tolerance for Flink Iterations

2015-04-21 Thread Markus Holzemer
Hi everybody, I am writing my master thesis about making flink iterations / iterative flink algorithms fault tolerant. The first approach I implemented is a basic checkpointing, where every N iterations the current state is saved into HDFS. To do this I enabled data sinks inside of iterations,

[jira] [Created] (FLINK-1846) Sinks inside of iterations

2015-04-08 Thread Markus Holzemer (JIRA)
Markus Holzemer created FLINK-1846: -- Summary: Sinks inside of iterations Key: FLINK-1846 URL: https://issues.apache.org/jira/browse/FLINK-1846 Project: Flink Issue Type: Improvement