Jeff you are correct. .NET for Spark behaves similarly to how pyspark and sparkR interacts with spark. Much like its counterparts, .NET for Spark acts as a wrapper to the underlying spark apis. Besides adding a SparkDotnetInterpreter to the livy codebase, a user will need to install a REPL on their system and set either an ENV variable or a spark conf to point to the binary (similar to python and R). In our case, we are focusing on supporting the dotnet-try <https://github.com/dotnet/try> REPL.
Regards, -Steve On Mon, Jan 13, 2020 at 8:09 PM Jeff Zhang <zjf...@gmail.com> wrote: > IIUC, donet spark is just a wrapper of spark. Besides adding > DotNetSparkInterpreter, what kind of other things user need to do ? For > now, user also need to specify SPARK_HOME at least, but to support donet, > what other things user need to do. > > > Saisai Shao <sai.sai.s...@gmail.com> 于2020年1月14日周二 上午11:01写道： > > > I see your point. Personally I don't have a strong opinion on this, I'm > not > > sure what others think about it. Why don't you start a design doc and > call > > for a vote about this feature. > > > > Thanks > > Saisai > > > > Steve Suh <suhst...@gmail.com> 于2020年1月14日周二 上午10:12写道： > > > > > @Saisai > > > > > > This will not be a wrapper around the REST API. The plan is to > support a > > > Livy POST /sessions request where kind == "sparkdotnet". Livy will > > > eventually push this request down to the ReplDriver, and in turn the > > > ReplDriver will create a new SparkDotnet Interpreter class. This will > be > > > similar to how the PythonInterpreter, SparkRInterpreter, and > > > SQLInterpreter classes get instantiated and used. > > > > > > > > > Regards, > > > -Steve > > > > > > On Sun, Jan 12, 2020 at 5:45 PM Saisai Shao <sai.sai.s...@gmail.com> > > > wrote: > > > > > > > Is this just a wrapper of Livy REST API, or it could support Livy Job > > > API? > > > > > > > > From my opinion, if it is just a wrapper of REST API, then it would > be > > > > better to maintain out of Livy, since REST API is language > independent, > > > if > > > > we're going to have all the languages support in Livy, then it is > hard > > to > > > > maintain. > > > > > > > > Just my two cents. > > > > > > > > Thanks > > > > Saisai > > > > > > > > Steve Suh <suhst...@gmail.com> 于2020年1月12日周日 下午3:49写道： > > > > > > > > > Hi, > > > > > > > > > > I contribute to the .NET for Apache Spark < > > > > https://github.com/dotnet/spark > > > > > > > > > > > project and we have seen *a lot* of *user interest* in providing > > first > > > > > class notebook support for *.NET for Apache Spark*. > > > > > > > > > > Livy currently supports *Scala*, *Python *and *R **interactive > > > sessions*. > > > > > We have a working prototype > > > > > < > > > > > > > > > > > > > > > https://github.com/dotnet/spark/tree/master/deployment/HDI-Spark/Notebooks > > > > > > > > > > > available > > > > > that adds support for a Spark *Dotnet **interactive session* and I > > > would > > > > > like to know if the Livy community would be interested in adding > this > > > > > feature to the main code base. If so, please let me know and I can > > > work > > > > on > > > > > creating this PR. > > > > > > > > > > For now, I've created a Jira item > > > > > <https://issues.apache.org/jira/browse/LIVY-742> to track this. > > > > > > > > > > > > > > > Regards, > > > > > -Steve > > > > > > > > > > > > > > > > > -- > Best Regards > > Jeff Zhang >
