Re: The household of the Kafka connector

I would like to consolidate those as well. Biggest blocker is, however, that the PersistentKafkaSource never commits to zookeeper when checkpointing is not enabled. It should at least group commit periodically in those cases. Concerning (4), I though the high-level consumer (that we build the

Re: Question: SourceFunction

Hi, There is one more tricky issue here if the variable is not volatile, which can cause a problem on any architecture: If the compiler determines that the code inside the loop will never modify isRunning, then it might optimize the exit condition into just while(true). And this can actually

Re: execute() and collect()/print()/count()

We have two situations to trade off here, and fixing one will make the other worse: 1) env.execute() after collect() - see Max's mail 2) env.execute() on empty sinks program. Not throwing an exception makes people wonder why nothing happens (if they write the program to just test whether it runs

Re: Removing reduce/aggregations from non-grouped data streams

+1 totally agreed On Mon, Jun 22, 2015 at 5:32 PM, Gyula Fóra gyf...@apache.org wrote: Hey all, Currently we have reduce and aggregation methods for non-grouped DataStreams as well, which will produce local aggregates depending on the parallelism of the operator. This behaviour is neither

Removing reduce/aggregations from non-grouped data streams

Hey all, Currently we have reduce and aggregation methods for non-grouped DataStreams as well, which will produce local aggregates depending on the parallelism of the operator. This behaviour is neither intuitive nor useful as it only produces sensible results if the user specifically sets the

Re: Failing Builds on Travis

+1 On Fri, Jun 19, 2015 at 10:48 AM, Matthias J. Sax mj...@informatik.hu-berlin.de wrote: +1 On 06/19/2015 10:35 AM, Ufuk Celebi wrote: On 19 Jun 2015, at 10:29, Maximilian Michels m...@apache.org wrote: Bringing this up again because of a recently discovered issue:

[jira] [Created] (FLINK-2262) ParameterTool API misnamed function

Greg Hogan created FLINK-2262: - Summary: ParameterTool API misnamed function Key: FLINK-2262 URL: https://issues.apache.org/jira/browse/FLINK-2262 Project: Flink Issue Type: Bug

The household of the Kafka connector

Hey, Due to the effort invested to the Kafka connector mainly by Robert and Gabor Hermann we are going to ship a fairly nice solution for reading from and writing to Kafka with 0.9.0. This is the most prominent streaming connector currently, and rightfully so as pipeline level end-to-end exactly

Known minor streaming issue in 0.9.0

Hey, I have found that open and close methods of streaming RichWindowFunctions are not called. I have the fix [1] as I did implement a fix for a similar issue some time ago, [2] sorry for not realizing it back then. [1] https://github.com/apache/flink/pull/855 [2]

[jira] [Created] (FLINK-2256) Handling Broker failures on the KafkaSink side

Márton Balassi created FLINK-2256: - Summary: Handling Broker failures on the KafkaSink side Key: FLINK-2256 URL: https://issues.apache.org/jira/browse/FLINK-2256 Project: Flink Issue Type:

Re: Known minor streaming issue in 0.9.0

Hi Marton, Thanks for spotting this issue. It is a bug we should list under known issues for the upcoming release. I agree with you that we can live with it until the next bug fix release. Cheers, Max On Mon, Jun 22, 2015 at 1:04 PM, Márton Balassi balassi.mar...@gmail.com wrote: Hey, I

Re: execute() and collect()/print()/count()

+1 for cleaning up the documentation +1 for adding a link to the documentation (should be a permalink) +1 for printing a warning instead of an exception On Sun, Jun 21, 2015 at 12:25 AM, Robert Metzger rmetz...@apache.org wrote: We could also add a link to the documentation into the exception

Re: Removing reduce/aggregations from non-grouped data streams

I opened a PR https://github.com/apache/flink/pull/860 for this. Stephan Ewen se...@apache.org ezt írta (időpont: 2015. jún. 22., H, 19:25): +1 totally agreed On Mon, Jun 22, 2015 at 5:32 PM, Gyula Fóra gyf...@apache.org wrote: Hey all, Currently we have reduce and aggregation methods

Re: Thoughts About Streaming

Hi Aljoscha, Thanks for the nice summary, this is a very good initiative. I added some comments to the respective sections (where I didnt fully agree :).). At some point I think it would be good to have a public hangout session on this, which could make a more dynamic discussion. Cheers, Gyula

Thoughts About Streaming

Hi, with people proposing changes to the streaming part I also wanted to throw my hat into the ring. :D During the last few months, while I was getting acquainted with the streaming system, I wrote down some thoughts I had about how things could be improved. Hopefully, they are in somewhat

[jira] [Created] (FLINK-2257) Open and close of RichWindowFunctions is not called

Márton Balassi created FLINK-2257: - Summary: Open and close of RichWindowFunctions is not called Key: FLINK-2257 URL: https://issues.apache.org/jira/browse/FLINK-2257 Project: Flink Issue

Re: Known minor streaming issue in 0.9.0

Added a ticket, so we can refer to it. https://issues.apache.org/jira/browse/FLINK-2257 On Mon, Jun 22, 2015 at 2:14 PM, Ufuk Celebi u...@apache.org wrote: On 22 Jun 2015, at 14:00, Maximilian Michels m...@apache.org wrote: Hi Marton, Thanks for spotting this issue. It is a bug we

[jira] [Created] (FLINK-2258) Add hyperparameter optimization to FlinkML

Theodore Vasiloudis created FLINK-2258: -- Summary: Add hyperparameter optimization to FlinkML Key: FLINK-2258 URL: https://issues.apache.org/jira/browse/FLINK-2258 Project: Flink Issue

[jira] [Created] (FLINK-2260) Have a complete model evaluation and selection framework for FlinkML

Theodore Vasiloudis created FLINK-2260: -- Summary: Have a complete model evaluation and selection framework for FlinkML Key: FLINK-2260 URL: https://issues.apache.org/jira/browse/FLINK-2260