Re: Questions re: ExecutionGraph & ResultPartitions for interactive use a la Spark

2016-01-15 Thread Ufuk Celebi
> On 14 Jan 2016, at 22:00, kovas boguta wrote: > > On Thu, Jan 14, 2016 at 5:52 AM, Ufuk Celebi wrote: > Hey Kovas > > sorry for the long delay. > > It was worth the wait! Thanks for the detailed response. > > > Ideally, I could force certain

Re: Questions re: ExecutionGraph & ResultPartitions for interactive use a la Spark

2016-01-14 Thread Ufuk Celebi
Hey Kovas sorry for the long delay. > On 10 Jan 2016, at 06:20, kovas boguta wrote: > 1) How can I prevent ResultPartitions from being released? > > In interactive use, RPs should not necessarily be released when there are no > pending tasks to consume them. Max

Re: Questions re: ExecutionGraph & ResultPartitions for interactive use a la Spark

2016-01-14 Thread kovas boguta
On Thu, Jan 14, 2016 at 4:00 PM, kovas boguta wrote: > > For a "real" solution, the REPL needs seem related to the WebUI, which I > haven't studied yet. One would want a fairly detailed view into the running > execution graph, possibly but not necessarily as an HTTP api. >

Re: Questions re: ExecutionGraph & ResultPartitions for interactive use a la Spark

2016-01-14 Thread kovas boguta
On Thu, Jan 14, 2016 at 5:52 AM, Ufuk Celebi wrote: > Hey Kovas > > sorry for the long delay. > It was worth the wait! Thanks for the detailed response. > Ideally, I could force certain ResultPartitions to only be manually > releasable, so I can consume them over and over. > >

Re: Questions re: ExecutionGraph & ResultPartitions for interactive use a la Spark

2016-01-05 Thread Aljoscha Krettek
Hi, these are certainly valid use cases. As far is I know, the people who know most in this area are on vacation right now. They should be back in a week, I think. They should be able to give you a proper description of the current situation and some pointers. Cheers, Aljoscha > On 04 Jan

Questions re: ExecutionGraph & ResultPartitions for interactive use a la Spark

2016-01-04 Thread kovas boguta
I'm impressed with the Flink API, it seems simpler and more composable than what I've seen elsewhere. I'm trying to see how to achieve a more interactive, REPL-driven experience, similar to Spark. I'm consuming Flink from Clojure. For now I'm only interested in smaller clusters & interactive