I'm afraid there's no template for this. I think you could use Spark's SPIP template.
BTW, I'm still not sure how you achieved dotnet support for Spark. We have python, R support because Spark supports them natively, not just an interpreter binding, but also some other pyspark, R works. But obviously Apache Spark doesn't have a dotnet support. So is only an interpreter binding enough? Do we need to use some other versions of Spark for dotnet support? Thanks Saisai Steve Suh <suhst...@gmail.com> 于2020年1月18日周六 上午5:29写道： > @Saisai , Is there a design doc for the python/r/sql interpreter that I can > use as a template? It will be very similar (most similar to R interpreter > in fact) in design. > > > Regards, > -Steve > > On Tue, Jan 14, 2020 at 6:13 PM Steve Suh <suhst...@gmail.com> wrote: > > > Jeff you are correct. .NET for Spark behaves similarly to how pyspark > and > > sparkR interacts with spark. Much like its counterparts, .NET for Spark > > acts as a wrapper to the underlying spark apis. Besides adding a > > SparkDotnetInterpreter to the livy codebase, a user will need to install > a > > REPL on their system and set either an ENV variable or a spark conf to > > point to the binary (similar to python and R). In our case, we are > > focusing on supporting the dotnet-try <https://github.com/dotnet/try> > > REPL. > > > > > > Regards, > > -Steve > > > > On Mon, Jan 13, 2020 at 8:09 PM Jeff Zhang <zjf...@gmail.com> wrote: > > > >> IIUC, donet spark is just a wrapper of spark. Besides adding > >> DotNetSparkInterpreter, what kind of other things user need to do ? For > >> now, user also need to specify SPARK_HOME at least, but to support > donet, > >> what other things user need to do. > >> > >> > >> Saisai Shao <sai.sai.s...@gmail.com> 于2020年1月14日周二 上午11:01写道： > >> > >> > I see your point. Personally I don't have a strong opinion on this, > I'm > >> not > >> > sure what others think about it. Why don't you start a design doc and > >> call > >> > for a vote about this feature. > >> > > >> > Thanks > >> > Saisai > >> > > >> > Steve Suh <suhst...@gmail.com> 于2020年1月14日周二 上午10:12写道： > >> > > >> > > @Saisai > >> > > > >> > > This will not be a wrapper around the REST API. The plan is to > >> support a > >> > > Livy POST /sessions request where kind == "sparkdotnet". Livy will > >> > > eventually push this request down to the ReplDriver, and in turn the > >> > > ReplDriver will create a new SparkDotnet Interpreter class. This > >> will be > >> > > similar to how the PythonInterpreter, SparkRInterpreter, and > >> > > SQLInterpreter classes get instantiated and used. > >> > > > >> > > > >> > > Regards, > >> > > -Steve > >> > > > >> > > On Sun, Jan 12, 2020 at 5:45 PM Saisai Shao <sai.sai.s...@gmail.com > > > >> > > wrote: > >> > > > >> > > > Is this just a wrapper of Livy REST API, or it could support Livy > >> Job > >> > > API? > >> > > > > >> > > > From my opinion, if it is just a wrapper of REST API, then it > would > >> be > >> > > > better to maintain out of Livy, since REST API is language > >> independent, > >> > > if > >> > > > we're going to have all the languages support in Livy, then it is > >> hard > >> > to > >> > > > maintain. > >> > > > > >> > > > Just my two cents. > >> > > > > >> > > > Thanks > >> > > > Saisai > >> > > > > >> > > > Steve Suh <suhst...@gmail.com> 于2020年1月12日周日 下午3:49写道： > >> > > > > >> > > > > Hi, > >> > > > > > >> > > > > I contribute to the .NET for Apache Spark < > >> > > > https://github.com/dotnet/spark > >> > > > > > > >> > > > > project and we have seen *a lot* of *user interest* in providing > >> > first > >> > > > > class notebook support for *.NET for Apache Spark*. > >> > > > > > >> > > > > Livy currently supports *Scala*, *Python *and *R **interactive > >> > > sessions*. > >> > > > > We have a working prototype > >> > > > > < > >> > > > > > >> > > > > >> > > > >> > > >> > https://github.com/dotnet/spark/tree/master/deployment/HDI-Spark/Notebooks > >> > > > > > > >> > > > > available > >> > > > > that adds support for a Spark *Dotnet **interactive session* > and I > >> > > would > >> > > > > like to know if the Livy community would be interested in adding > >> this > >> > > > > feature to the main code base. If so, please let me know and I > >> can > >> > > work > >> > > > on > >> > > > > creating this PR. > >> > > > > > >> > > > > For now, I've created a Jira item > >> > > > > <https://issues.apache.org/jira/browse/LIVY-742> to track this. > >> > > > > > >> > > > > > >> > > > > Regards, > >> > > > > -Steve > >> > > > > > >> > > > > >> > > > >> > > >> > >> > >> -- > >> Best Regards > >> > >> Jeff Zhang > >> > > >
