Let's start by identifying some scenarios that the client is used: - Users trying to build an application which needs to interactively integrate with Spark, where the client is used to submit a piece of code and expects an indication that the full results are back to continue doing what it needs to do. In this scenario, the client is outside the Jupyter realm and should be shielded from any protocol details.
Regarding onStream, is that used on the scope of Spark Streaming, or just results (big or small) being sent via output stream? And how do we know when the full results have been received by the client side? On Thu, Feb 2, 2017 at 7:35 AM, Corey Stubbs <[email protected]> wrote: > So the client programming model is meant to reflect the Jupyter Protocol, > which has taken into account issues like streaming. However, the client > could still be cleaned up. Maybe something along the lines of: > > val exRes: DeferredExecution = client.execute(code) > .onError(executeReplyError =>{ > ... > }).onSuccess(Option(executeResult) => { > ... > }).onStream(streamResult => { > ... > }) > > Key thing being the optional execute result as a parameter to onSuccess. > This would be a better interface to handle when there are/are not results > from an execution. > > Kind Regards, > Corey Stubbs > > On Tue, Jan 31, 2017 at 10:29 AM Luciano Resende <[email protected]> > wrote: > > The Toree client seems to be a client utility to enable other applications > to interact with Toree/Spark. These applications are not based or aware of > the ZeroMQ protocol, but nevertheless, we are imposing them on a very > similar programming model while evaluating code using the client. > > See an example below, where the application programmer has to handle > different options, where more than one option can be called per eval > result, and the order cannot be known prior to the execution time : > > val exRes: DeferredExecution = client.execute(code) > .onResult(executeResult => { > ... > }).onError(executeReplyError =>{ > ... > }).onSuccess(executeReplyOk => { > ... > }).onStream(streamResult => { > ... > }) > > > I was wondering what are the community thoughts on enhancing the client to > abstract its programming model, and having all this abstracted and handled > on the client implementation, and the eval would return one promise (e.g. > onResult which one option would be status success or error) or at most two > promises (e.g. onSuccess and onError). > > The only issue I haven't fully understood yet is how streaming is handled > and what are the implication of these changes on this scenario. > > Thoughts? > > -- > Luciano Resende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
