DataStream API - What does "Beta" mean?

2015-08-19 Thread ankurcha
Hi, This is a more of a semantics / scope question. What does it mean by the "beta" status? Main questions are around: * API stability - How stable is the API? * Regression / testing coverage? * Feature completeness? * Production readiness? Also, a pointer to the relevant roadmaps and/or plan

Re: add some new api to the scheduler in the job manager

2015-08-19 Thread ankurcha
Hi Robert, I agree with everything that you and Stephan are saying. I haven't looked into the flink codebase and the papers/design docs to comment at a finer level so maybe that's the first piece of homework that I need to do ( need pointers for that). And, yes I would definitely be interested i

Re: add some new api to the scheduler in the job manager

2015-08-19 Thread Aljoscha Krettek
Hi, one thing to also keep in mind here is that TaskManagers might have to keep the intermediate data even after a job is finished. For example, if the user has an interactive scala-shell session where they are exploring same data and transforming it in several steps as they go. Cheers, Aljoscha

[jira] [Created] (FLINK-2549) Add topK operator for DataSet

2015-08-19 Thread Chengxiang Li (JIRA)
Chengxiang Li created FLINK-2549: Summary: Add topK operator for DataSet Key: FLINK-2549 URL: https://issues.apache.org/jira/browse/FLINK-2549 Project: Flink Issue Type: New Feature

Re: add some new api to the scheduler in the job manager

2015-08-19 Thread Robert Metzger
Hi, I'm sorry for the late reply, I'm still working through a long email backlog from a one week vacation ;) Thank you for the long reply in this thread. Your observations are very good. I think we should indeed work towards a more fine grained "intra-job" elasticity. Let me comment on some of you

Re: [jira](FLINK-2493) Simplify names of example program JARs

2015-08-19 Thread Liang Chen
Sorry,made you confusion. I meant, may only use scala to write machine learning examples in future. (java doesn't include these examples). --- Can you explain what you mean by "BTW, my personal opinion: current

[jira] [Created] (FLINK-2548) VertexCentricIteration should avoid doing a coGroup with the edges and the solution set

2015-08-19 Thread Gabor Gevay (JIRA)
Gabor Gevay created FLINK-2548: -- Summary: VertexCentricIteration should avoid doing a coGroup with the edges and the solution set Key: FLINK-2548 URL: https://issues.apache.org/jira/browse/FLINK-2548 Pro

[jira] [Created] (FLINK-2547) Provide Cluster Status Info

2015-08-19 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-2547: --- Summary: Provide Cluster Status Info Key: FLINK-2547 URL: https://issues.apache.org/jira/browse/FLINK-2547 Project: Flink Issue Type: Sub-task Compon

[jira] [Created] (FLINK-2546) The webclient does not allow users to save the execution plan

2015-08-19 Thread Andra Lungu (JIRA)
Andra Lungu created FLINK-2546: -- Summary: The webclient does not allow users to save the execution plan Key: FLINK-2546 URL: https://issues.apache.org/jira/browse/FLINK-2546 Project: Flink Issu

Re: [jira](FLINK-2493) Simplify names of example program JARs

2015-08-19 Thread Stephan Ewen
Hi! Can you explain what you mean by "BTW, my personal opinion: currently scala be acknowledged by more and more developers, in future there may have some different examples for scala." ? It would be fine to have different Scala and Java examples, there is no need to have them in sync (although t

Re: [jira](FLINK-2493) Simplify names of example program JARs

2015-08-19 Thread Liang Chen
Ya,If takes proposal 2, the the name overlap is one issue from java and scala examples. Your suggestion sounds good,gives different package for java and scala examples. but drops the subpackages, i have some different points: there are many examples use the similar directory structure such as:stor

RE: A Question About Execution Schedule

2015-08-19 Thread MaGuoWei
Thanks Till! > Date: Wed, 19 Aug 2015 09:31:49 +0200 > Subject: Re: A Question About Execution Schedule > From: trohrm...@apache.org > To: dev@flink.apache.org > > Hi MaGuoWei, > > this is not a problem. If you look at the implementation of > SlotAllocationFuture.setFutureAction, you’ll see that

Re: A Question About Execution Schedule

2015-08-19 Thread Till Rohrmann
Hi MaGuoWei, this is not a problem. If you look at the implementation of SlotAllocationFuture.setFutureAction, you’ll see that the method is synchronized on a lock which is also used to complete the future. Furthermore, you’ll see that the slot variable is checked upon setting an action and if it’