Note #2 on the list

2015-06-19 Thread Ted Dunning
http://www.datanami.com/2015/06/12/8-new-big-data-projects-to-watch/

Re: Flink Runtime Exception

2015-06-19 Thread Till Rohrmann
I think Andra wrote that there is *no deadlock*. On Fri, Jun 19, 2015 at 3:18 PM Fabian Hueske fhue...@gmail.com http://mailto:fhue...@gmail.com wrote: Hi Andra, The system should never deadlock. There is a bug somewhere if that happens. Can you check if the program is really stuck?

Re: Flink Runtime Exception

2015-06-19 Thread Fabian Hueske
woops, sorry! Whenever I read the word deadlock I getting a bit nervous and distracted ;-) 2015-06-19 15:21 GMT+02:00 Till Rohrmann trohrm...@apache.org: I think Andra wrote that there is *no deadlock*. On Fri, Jun 19, 2015 at 3:18 PM Fabian Hueske fhue...@gmail.com

Re: Flink Runtime Exception

2015-06-19 Thread Ufuk Celebi
On 19 Jun 2015, at 14:53, Andra Lungu lungu.an...@gmail.com wrote: Another problem that I encountered during the same set of experiments(sorry if I am asking too many questions, I am eager to get things fixed): - for the same configuration, a piece of code runs perfectly on 10GB of input,

Re: Flink Runtime Exception

2015-06-19 Thread Till Rohrmann
Hi Andra, the problem seems to be that the deployment of some tasks takes longer than 100s. From the stack trace it looks as if you're not using the latest master. We had problems with previous version where the deployment call waited for the TM to completely download the user code jars. For

[jira] [Created] (FLINK-2245) Programs that contain collect() reported as multiple jobs in the Web frontend

2015-06-19 Thread Theodore Vasiloudis (JIRA)
Theodore Vasiloudis created FLINK-2245: -- Summary: Programs that contain collect() reported as multiple jobs in the Web frontend Key: FLINK-2245 URL: https://issues.apache.org/jira/browse/FLINK-2245

[jira] [Created] (FLINK-2248) Allow disabling of sdtout logging output

2015-06-19 Thread Theodore Vasiloudis (JIRA)
Theodore Vasiloudis created FLINK-2248: -- Summary: Allow disabling of sdtout logging output Key: FLINK-2248 URL: https://issues.apache.org/jira/browse/FLINK-2248 Project: Flink Issue

[jira] [Created] (FLINK-2249) ExecutionEnvironment: Ignore calls to execute() if no data sinks defined

2015-06-19 Thread Maximilian Michels (JIRA)
Maximilian Michels created FLINK-2249: - Summary: ExecutionEnvironment: Ignore calls to execute() if no data sinks defined Key: FLINK-2249 URL: https://issues.apache.org/jira/browse/FLINK-2249

[jira] [Created] (FLINK-2250) Backtracking of intermediate results

2015-06-19 Thread Maximilian Michels (JIRA)
Maximilian Michels created FLINK-2250: - Summary: Backtracking of intermediate results Key: FLINK-2250 URL: https://issues.apache.org/jira/browse/FLINK-2250 Project: Flink Issue Type: New

[jira] [Created] (FLINK-2244) Add ability to start and stop persistent IaaS cluster

2015-06-19 Thread Theodore Vasiloudis (JIRA)
Theodore Vasiloudis created FLINK-2244: -- Summary: Add ability to start and stop persistent IaaS cluster Key: FLINK-2244 URL: https://issues.apache.org/jira/browse/FLINK-2244 Project: Flink

Reduce combiner not chained

2015-06-19 Thread Ufuk Celebi
Hey all, on the current master running the WordCount example with a text file input/output results and a manual reduce function (instead of the sum(1)) results in a combiner, which is not chained. The corresponding issue is here: https://issues.apache.org/jira/browse/FLINK-2246 Can someone

Re: Flink Runtime Exception

2015-06-19 Thread Andra Lungu
Yes, so I am using flink-0.9.0-milestone-1. Was it a problem for this version? I'll just fetch the latest master if this is the case. On Fri, Jun 19, 2015 at 2:12 PM, Till Rohrmann trohrm...@apache.org wrote: Hi Andra, the problem seems to be that the deployment of some tasks takes longer

Re: Failing Builds on Travis

2015-06-19 Thread Matthias J. Sax
+1 On 06/19/2015 10:35 AM, Ufuk Celebi wrote: On 19 Jun 2015, at 10:29, Maximilian Michels m...@apache.org wrote: Bringing this up again because of a recently discovered issue: https://issues.apache.org/jira/browse/FLINK-2235 Flink 0.9.0 will still support Java 6. Just wanted to clarify

Re: Reduce combiner not chained

2015-06-19 Thread Fabian Hueske
This is not a bug. Chained combiners are not supported for ReduceFunctions yet. :-( I updated the JIRA accordingly. 2015-06-19 13:04 GMT+02:00 Ufuk Celebi u...@apache.org: Hey all, on the current master running the WordCount example with a text file input/output results and a manual reduce

execute() and collect()/print()/count()

2015-06-19 Thread Maximilian Michels
Dear Flink community, I have stopped to count how many people on the user list and during Flink trainings have asked why their Flink program throws an Exception when they just one to print a DataSet. The reason for this is that print() now executes eagerly, thus, executes the Flink program.

Re: Flink Runtime Exception

2015-06-19 Thread Till Rohrmann
Yes, it was an issue for the milestone release. On Fri, Jun 19, 2015 at 2:18 PM Andra Lungu lungu.an...@gmail.com wrote: Yes, so I am using flink-0.9.0-milestone-1. Was it a problem for this version? I'll just fetch the latest master if this is the case. On Fri, Jun 19, 2015 at 2:12 PM,

[jira] [Created] (FLINK-2251) Implement job session management

2015-06-19 Thread Maximilian Michels (JIRA)
Maximilian Michels created FLINK-2251: - Summary: Implement job session management Key: FLINK-2251 URL: https://issues.apache.org/jira/browse/FLINK-2251 Project: Flink Issue Type:

Re: execute() and collect()/print()/count()

2015-06-19 Thread Chiwan Park
+1 for ignoring execute() call with warning. But I'm concerned for how the user catches the error in program without any data sinks. By the way, eager execution is not well documented in data sinks section but is in program skeleton section. [1] This makes the user’s confusion. We should clean

[jira] [Created] (FLINK-2247) Improve the way memory is reported in the web frontend

2015-06-19 Thread Theodore Vasiloudis (JIRA)
Theodore Vasiloudis created FLINK-2247: -- Summary: Improve the way memory is reported in the web frontend Key: FLINK-2247 URL: https://issues.apache.org/jira/browse/FLINK-2247 Project: Flink

Re: Failing Builds on Travis

2015-06-19 Thread Ufuk Celebi
On 19 Jun 2015, at 10:29, Maximilian Michels m...@apache.org wrote: Bringing this up again because of a recently discovered issue: https://issues.apache.org/jira/browse/FLINK-2235 Flink 0.9.0 will still support Java 6. Just wanted to clarify again that we drop support for Java 6 for any

Re: Quickstart POMs

2015-06-19 Thread Ufuk Celebi
On 19 Jun 2015, at 10:19, Maximilian Michels m...@apache.org wrote: @Robert @Stephan Thanks for clarifying! Of course it would be better to have a concise quickstart pom.xml but the necessary workarounds justify the current state. +1

Re: Quickstart POMs

2015-06-19 Thread Maximilian Michels
@Robert @Stephan Thanks for clarifying! Of course it would be better to have a concise quickstart pom.xml but the necessary workarounds justify the current state. On Thu, Jun 18, 2015 at 11:48 PM, Stephan Ewen se...@apache.org wrote: I remember we had a similar discussion a while back. The

Re: Flink Runtime Exception

2015-06-19 Thread Asterios Katsifodimos
Hi Andra, I would try increasing the memory per task manager, i.e. on a machine with 8 CPUs and 16GBs of memory, instead of spawning 8 TMs with 2GB each, I would try to spawn 2 TMs of 8GBs each. This might help with the spilling problem (in case that the CPU is not your bottleneck, this might

[jira] [Created] (FLINK-2253) ALS fails if the data flow is split up and the input is non-deterministic

2015-06-19 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-2253: Summary: ALS fails if the data flow is split up and the input is non-deterministic Key: FLINK-2253 URL: https://issues.apache.org/jira/browse/FLINK-2253 Project:

Re: [VOTE] Release Apache Flink 0.9.0 (release-0.9.0-rc4)

2015-06-19 Thread Henry Saputra
Signature file look good Checksum files look good LICENSE file look good NOTICE file look good Code compile and pass tests No 3rd party exes in the source artifacts Run local mode for examples Start standalone mode Deploy in YARN for session and non-session +1 (binding) - Henry On Thu, Jun 18,