That scenario is handled in my above sample API, maybe with a better interface of a Future/Promise result instead of the option. The client library would handle waiting for execution to be completed as well as waiting for the results to be returned. I feel like I am not understanding your point, I am just getting over a cold so maybe I need to let it sink in for a bit before it hits me.
The streaming results have nothing to do with Spark Streaming. They are just messages sent via the output stream, and there can be 0 or more of these per execution request. The stream results can also continue after execution has completed (in the event the prints happen in a new thread). Just to provide examples of what clients currently exist and to see how they differ from the Toree Client: Python Client: https://jupyter-client.readthedocs.io/en/latest/messaging.html#python-api JavaScript Client: http://jupyterlab.github.io/services/interfaces/kernel.ikernel.html On Thu, Feb 2, 2017 at 9:45 AM Luciano Resende <[email protected]> wrote: > Let's start by identifying some scenarios that the client is used: > > - Users trying to build an application which needs to interactively > integrate with Spark, where the client is used to submit a piece of code > and expects an indication that the full results are back to continue doing > what it needs to do. In this scenario, the client is outside the Jupyter > realm and should be shielded from any protocol details. > > Regarding onStream, is that used on the scope of Spark Streaming, or just > results (big or small) being sent via output stream? And how do we know > when the full results have been received by the client side? > > > On Thu, Feb 2, 2017 at 7:35 AM, Corey Stubbs <[email protected]> wrote: > > > So the client programming model is meant to reflect the Jupyter Protocol, > > which has taken into account issues like streaming. However, the client > > could still be cleaned up. Maybe something along the lines of: > > > > val exRes: DeferredExecution = client.execute(code) > > .onError(executeReplyError =>{ > > ... > > }).onSuccess(Option(executeResult) => { > > ... > > }).onStream(streamResult => { > > ... > > }) > > > > Key thing being the optional execute result as a parameter to onSuccess. > > This would be a better interface to handle when there are/are not results > > from an execution. > > > > Kind Regards, > > Corey Stubbs > > > > On Tue, Jan 31, 2017 at 10:29 AM Luciano Resende <[email protected]> > > wrote: > > > > The Toree client seems to be a client utility to enable other > applications > > to interact with Toree/Spark. These applications are not based or aware > of > > the ZeroMQ protocol, but nevertheless, we are imposing them on a very > > similar programming model while evaluating code using the client. > > > > See an example below, where the application programmer has to handle > > different options, where more than one option can be called per eval > > result, and the order cannot be known prior to the execution time : > > > > val exRes: DeferredExecution = client.execute(code) > > .onResult(executeResult => { > > ... > > }).onError(executeReplyError =>{ > > ... > > }).onSuccess(executeReplyOk => { > > ... > > }).onStream(streamResult => { > > ... > > }) > > > > > > I was wondering what are the community thoughts on enhancing the client > to > > abstract its programming model, and having all this abstracted and > handled > > on the client implementation, and the eval would return one promise (e.g. > > onResult which one option would be status success or error) or at most > two > > promises (e.g. onSuccess and onError). > > > > The only issue I haven't fully understood yet is how streaming is handled > > and what are the implication of these changes on this scenario. > > > > Thoughts? > > > > -- > > Luciano Resende > > http://twitter.com/lresende1975 > > http://lresende.blogspot.com/ > > > > > > -- > Luciano Resende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ >
