Re: .NET for Apache Spark interactive interpreter support

[Saisai Shao]

I'm afraid there's no template for this. I think you could use Spark's SPIP
template.

BTW, I'm still not sure how you achieved dotnet support for Spark. We have
python, R support because Spark supports them natively, not just an
interpreter binding, but also some other pyspark, R works. But obviously
Apache Spark doesn't have a dotnet support. So is only an interpreter
binding enough? Do we need to use some other versions of Spark for dotnet
support?

Thanks
Saisai


Steve Suh  于2020年1月18日周六 上午5:29写道：

> @Saisai , Is there a design doc for the python/r/sql interpreter that I can
> use as a template? It will be very similar (most similar to R interpreter
> in fact) in design.
>
>
> Regards,
> -Steve
>
> On Tue, Jan 14, 2020 at 6:13 PM Steve Suh  wrote:
>
> > Jeff you are correct.  .NET for Spark behaves similarly to how pyspark
> and
> > sparkR interacts with spark.  Much like its counterparts, .NET for Spark
> > acts as a wrapper to the underlying spark apis.  Besides adding a
> > SparkDotnetInterpreter to the livy codebase, a user will need to install
> a
> > REPL on their system and set either an ENV variable or a spark conf to
> > point to the binary (similar to python and R).  In our case, we are
> > focusing on supporting the dotnet-try 
> > REPL.
> >
> >
> > Regards,
> > -Steve
> >
> > On Mon, Jan 13, 2020 at 8:09 PM Jeff Zhang  wrote:
> >
> >> IIUC, donet spark is just a wrapper of spark. Besides adding
> >> DotNetSparkInterpreter, what kind of other things user need to do ? For
> >> now, user also need to specify SPARK_HOME at least, but to support
> donet,
> >> what other things user need to do.
> >>
> >>
> >> Saisai Shao  于2020年1月14日周二 上午11:01写道：
> >>
> >> > I see your point. Personally I don't have a strong opinion on this,
> I'm
> >> not
> >> > sure what others think about it. Why don't you start a design doc and
> >> call
> >> > for a vote about this feature.
> >> >
> >> > Thanks
> >> > Saisai
> >> >
> >> > Steve Suh  于2020年1月14日周二 上午10:12写道：
> >> >
> >> > > @Saisai
> >> > >
> >> > > This will not be a wrapper around the REST API.  The plan is to
> >> support a
> >> > > Livy POST /sessions request where kind == "sparkdotnet".  Livy will
> >> > > eventually push this request down to the ReplDriver, and in turn the
> >> > > ReplDriver will create a new SparkDotnet Interpreter class.  This
> >> will be
> >> > > similar to how the  PythonInterpreter, SparkRInterpreter, and
> >> > > SQLInterpreter classes get instantiated and used.
> >> > >
> >> > >
> >> > > Regards,
> >> > > -Steve
> >> > >
> >> > > On Sun, Jan 12, 2020 at 5:45 PM Saisai Shao  >
> >> > > wrote:
> >> > >
> >> > > > Is this just a wrapper of Livy REST API, or it could support Livy
> >> Job
> >> > > API?
> >> > > >
> >> > > > From my opinion, if it is just a wrapper of REST API, then it
> would
> >> be
> >> > > > better to maintain out of Livy, since REST API is language
> >> independent,
> >> > > if
> >> > > > we're going to have all the languages support in Livy, then it is
> >> hard
> >> > to
> >> > > > maintain.
> >> > > >
> >> > > > Just my two cents.
> >> > > >
> >> > > > Thanks
> >> > > > Saisai
> >> > > >
> >> > > > Steve Suh  于2020年1月12日周日 下午3:49写道：
> >> > > >
> >> > > > > Hi,
> >> > > > >
> >> > > > > I contribute to the .NET for Apache Spark <
> >> > > > https://github.com/dotnet/spark
> >> > > > > >
> >> > > > > project and we have seen *a lot* of *user interest* in providing
> >> > first
> >> > > > > class notebook support for *.NET for Apache Spark*.
> >> > > > >
> >> > > > > Livy currently supports *Scala*, *Python *and *R **interactive
> >> > > sessions*.
> >> > > > > We have a working prototype
> >> > > > > <
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/dotnet/spark/tree/master/deployment/HDI-Spark/Notebooks
> >> > > > > >
> >> > > > > available
> >> > > > > that adds support for a Spark *Dotnet **interactive session*
> and I
> >> > > would
> >> > > > > like to know if the Livy community would be interested in adding
> >> this
> >> > > > > feature to the main code base.  If so, please let me know and I
> >> can
> >> > > work
> >> > > > on
> >> > > > > creating this PR.
> >> > > > >
> >> > > > > For now, I've created a Jira item
> >> > > > >  to track this.
> >> > > > >
> >> > > > >
> >> > > > > Regards,
> >> > > > > -Steve
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >>
> >> --
> >> Best Regards
> >>
> >> Jeff Zhang
> >>
> >
>

Re: .NET for Apache Spark interactive interpreter support

[Steve Suh]

@Saisai , Is there a design doc for the python/r/sql interpreter that I can
use as a template? It will be very similar (most similar to R interpreter
in fact) in design.


Regards,
-Steve

On Tue, Jan 14, 2020 at 6:13 PM Steve Suh  wrote:

> Jeff you are correct.  .NET for Spark behaves similarly to how pyspark and
> sparkR interacts with spark.  Much like its counterparts, .NET for Spark
> acts as a wrapper to the underlying spark apis.  Besides adding a
> SparkDotnetInterpreter to the livy codebase, a user will need to install a
> REPL on their system and set either an ENV variable or a spark conf to
> point to the binary (similar to python and R).  In our case, we are
> focusing on supporting the dotnet-try 
> REPL.
>
>
> Regards,
> -Steve
>
> On Mon, Jan 13, 2020 at 8:09 PM Jeff Zhang  wrote:
>
>> IIUC, donet spark is just a wrapper of spark. Besides adding
>> DotNetSparkInterpreter, what kind of other things user need to do ? For
>> now, user also need to specify SPARK_HOME at least, but to support donet,
>> what other things user need to do.
>>
>>
>> Saisai Shao  于2020年1月14日周二 上午11:01写道：
>>
>> > I see your point. Personally I don't have a strong opinion on this, I'm
>> not
>> > sure what others think about it. Why don't you start a design doc and
>> call
>> > for a vote about this feature.
>> >
>> > Thanks
>> > Saisai
>> >
>> > Steve Suh  于2020年1月14日周二 上午10:12写道：
>> >
>> > > @Saisai
>> > >
>> > > This will not be a wrapper around the REST API.  The plan is to
>> support a
>> > > Livy POST /sessions request where kind == "sparkdotnet".  Livy will
>> > > eventually push this request down to the ReplDriver, and in turn the
>> > > ReplDriver will create a new SparkDotnet Interpreter class.  This
>> will be
>> > > similar to how the  PythonInterpreter, SparkRInterpreter, and
>> > > SQLInterpreter classes get instantiated and used.
>> > >
>> > >
>> > > Regards,
>> > > -Steve
>> > >
>> > > On Sun, Jan 12, 2020 at 5:45 PM Saisai Shao 
>> > > wrote:
>> > >
>> > > > Is this just a wrapper of Livy REST API, or it could support Livy
>> Job
>> > > API?
>> > > >
>> > > > From my opinion, if it is just a wrapper of REST API, then it would
>> be
>> > > > better to maintain out of Livy, since REST API is language
>> independent,
>> > > if
>> > > > we're going to have all the languages support in Livy, then it is
>> hard
>> > to
>> > > > maintain.
>> > > >
>> > > > Just my two cents.
>> > > >
>> > > > Thanks
>> > > > Saisai
>> > > >
>> > > > Steve Suh  于2020年1月12日周日 下午3:49写道：
>> > > >
>> > > > > Hi,
>> > > > >
>> > > > > I contribute to the .NET for Apache Spark <
>> > > > https://github.com/dotnet/spark
>> > > > > >
>> > > > > project and we have seen *a lot* of *user interest* in providing
>> > first
>> > > > > class notebook support for *.NET for Apache Spark*.
>> > > > >
>> > > > > Livy currently supports *Scala*, *Python *and *R **interactive
>> > > sessions*.
>> > > > > We have a working prototype
>> > > > > <
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/dotnet/spark/tree/master/deployment/HDI-Spark/Notebooks
>> > > > > >
>> > > > > available
>> > > > > that adds support for a Spark *Dotnet **interactive session* and I
>> > > would
>> > > > > like to know if the Livy community would be interested in adding
>> this
>> > > > > feature to the main code base.  If so, please let me know and I
>> can
>> > > work
>> > > > on
>> > > > > creating this PR.
>> > > > >
>> > > > > For now, I've created a Jira item
>> > > > >  to track this.
>> > > > >
>> > > > >
>> > > > > Regards,
>> > > > > -Steve
>> > > > >
>> > > >
>> > >
>> >
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>

Re: .NET for Apache Spark interactive interpreter support

[Steve Suh]

Jeff you are correct.  .NET for Spark behaves similarly to how pyspark and
sparkR interacts with spark.  Much like its counterparts, .NET for Spark
acts as a wrapper to the underlying spark apis.  Besides adding a
SparkDotnetInterpreter to the livy codebase, a user will need to install a
REPL on their system and set either an ENV variable or a spark conf to
point to the binary (similar to python and R).  In our case, we are
focusing on supporting the dotnet-try  REPL.


Regards,
-Steve

On Mon, Jan 13, 2020 at 8:09 PM Jeff Zhang  wrote:

> IIUC, donet spark is just a wrapper of spark. Besides adding
> DotNetSparkInterpreter, what kind of other things user need to do ? For
> now, user also need to specify SPARK_HOME at least, but to support donet,
> what other things user need to do.
>
>
> Saisai Shao  于2020年1月14日周二 上午11:01写道：
>
> > I see your point. Personally I don't have a strong opinion on this, I'm
> not
> > sure what others think about it. Why don't you start a design doc and
> call
> > for a vote about this feature.
> >
> > Thanks
> > Saisai
> >
> > Steve Suh  于2020年1月14日周二 上午10:12写道：
> >
> > > @Saisai
> > >
> > > This will not be a wrapper around the REST API.  The plan is to
> support a
> > > Livy POST /sessions request where kind == "sparkdotnet".  Livy will
> > > eventually push this request down to the ReplDriver, and in turn the
> > > ReplDriver will create a new SparkDotnet Interpreter class.  This will
> be
> > > similar to how the  PythonInterpreter, SparkRInterpreter, and
> > > SQLInterpreter classes get instantiated and used.
> > >
> > >
> > > Regards,
> > > -Steve
> > >
> > > On Sun, Jan 12, 2020 at 5:45 PM Saisai Shao 
> > > wrote:
> > >
> > > > Is this just a wrapper of Livy REST API, or it could support Livy Job
> > > API?
> > > >
> > > > From my opinion, if it is just a wrapper of REST API, then it would
> be
> > > > better to maintain out of Livy, since REST API is language
> independent,
> > > if
> > > > we're going to have all the languages support in Livy, then it is
> hard
> > to
> > > > maintain.
> > > >
> > > > Just my two cents.
> > > >
> > > > Thanks
> > > > Saisai
> > > >
> > > > Steve Suh  于2020年1月12日周日 下午3:49写道：
> > > >
> > > > > Hi,
> > > > >
> > > > > I contribute to the .NET for Apache Spark <
> > > > https://github.com/dotnet/spark
> > > > > >
> > > > > project and we have seen *a lot* of *user interest* in providing
> > first
> > > > > class notebook support for *.NET for Apache Spark*.
> > > > >
> > > > > Livy currently supports *Scala*, *Python *and *R **interactive
> > > sessions*.
> > > > > We have a working prototype
> > > > > <
> > > > >
> > > >
> > >
> >
> https://github.com/dotnet/spark/tree/master/deployment/HDI-Spark/Notebooks
> > > > > >
> > > > > available
> > > > > that adds support for a Spark *Dotnet **interactive session* and I
> > > would
> > > > > like to know if the Livy community would be interested in adding
> this
> > > > > feature to the main code base.  If so, please let me know and I can
> > > work
> > > > on
> > > > > creating this PR.
> > > > >
> > > > > For now, I've created a Jira item
> > > > >  to track this.
> > > > >
> > > > >
> > > > > Regards,
> > > > > -Steve
> > > > >
> > > >
> > >
> >
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: .NET for Apache Spark interactive interpreter support

[Jeff Zhang]

IIUC, donet spark is just a wrapper of spark. Besides adding
DotNetSparkInterpreter, what kind of other things user need to do ? For
now, user also need to specify SPARK_HOME at least, but to support donet,
what other things user need to do.


Saisai Shao  于2020年1月14日周二 上午11:01写道：

> I see your point. Personally I don't have a strong opinion on this, I'm not
> sure what others think about it. Why don't you start a design doc and call
> for a vote about this feature.
>
> Thanks
> Saisai
>
> Steve Suh  于2020年1月14日周二 上午10:12写道：
>
> > @Saisai
> >
> > This will not be a wrapper around the REST API.  The plan is to support a
> > Livy POST /sessions request where kind == "sparkdotnet".  Livy will
> > eventually push this request down to the ReplDriver, and in turn the
> > ReplDriver will create a new SparkDotnet Interpreter class.  This will be
> > similar to how the  PythonInterpreter, SparkRInterpreter, and
> > SQLInterpreter classes get instantiated and used.
> >
> >
> > Regards,
> > -Steve
> >
> > On Sun, Jan 12, 2020 at 5:45 PM Saisai Shao 
> > wrote:
> >
> > > Is this just a wrapper of Livy REST API, or it could support Livy Job
> > API?
> > >
> > > From my opinion, if it is just a wrapper of REST API, then it would be
> > > better to maintain out of Livy, since REST API is language independent,
> > if
> > > we're going to have all the languages support in Livy, then it is hard
> to
> > > maintain.
> > >
> > > Just my two cents.
> > >
> > > Thanks
> > > Saisai
> > >
> > > Steve Suh  于2020年1月12日周日 下午3:49写道：
> > >
> > > > Hi,
> > > >
> > > > I contribute to the .NET for Apache Spark <
> > > https://github.com/dotnet/spark
> > > > >
> > > > project and we have seen *a lot* of *user interest* in providing
> first
> > > > class notebook support for *.NET for Apache Spark*.
> > > >
> > > > Livy currently supports *Scala*, *Python *and *R **interactive
> > sessions*.
> > > > We have a working prototype
> > > > <
> > > >
> > >
> >
> https://github.com/dotnet/spark/tree/master/deployment/HDI-Spark/Notebooks
> > > > >
> > > > available
> > > > that adds support for a Spark *Dotnet **interactive session* and I
> > would
> > > > like to know if the Livy community would be interested in adding this
> > > > feature to the main code base.  If so, please let me know and I can
> > work
> > > on
> > > > creating this PR.
> > > >
> > > > For now, I've created a Jira item
> > > >  to track this.
> > > >
> > > >
> > > > Regards,
> > > > -Steve
> > > >
> > >
> >
>


-- 
Best Regards

Jeff Zhang

Re: .NET for Apache Spark interactive interpreter support

[Saisai Shao]

I see your point. Personally I don't have a strong opinion on this, I'm not
sure what others think about it. Why don't you start a design doc and call
for a vote about this feature.

Thanks
Saisai

Steve Suh  于2020年1月14日周二 上午10:12写道：

> @Saisai
>
> This will not be a wrapper around the REST API.  The plan is to support a
> Livy POST /sessions request where kind == "sparkdotnet".  Livy will
> eventually push this request down to the ReplDriver, and in turn the
> ReplDriver will create a new SparkDotnet Interpreter class.  This will be
> similar to how the  PythonInterpreter, SparkRInterpreter, and
> SQLInterpreter classes get instantiated and used.
>
>
> Regards,
> -Steve
>
> On Sun, Jan 12, 2020 at 5:45 PM Saisai Shao 
> wrote:
>
> > Is this just a wrapper of Livy REST API, or it could support Livy Job
> API?
> >
> > From my opinion, if it is just a wrapper of REST API, then it would be
> > better to maintain out of Livy, since REST API is language independent,
> if
> > we're going to have all the languages support in Livy, then it is hard to
> > maintain.
> >
> > Just my two cents.
> >
> > Thanks
> > Saisai
> >
> > Steve Suh  于2020年1月12日周日 下午3:49写道：
> >
> > > Hi,
> > >
> > > I contribute to the .NET for Apache Spark <
> > https://github.com/dotnet/spark
> > > >
> > > project and we have seen *a lot* of *user interest* in providing first
> > > class notebook support for *.NET for Apache Spark*.
> > >
> > > Livy currently supports *Scala*, *Python *and *R **interactive
> sessions*.
> > > We have a working prototype
> > > <
> > >
> >
> https://github.com/dotnet/spark/tree/master/deployment/HDI-Spark/Notebooks
> > > >
> > > available
> > > that adds support for a Spark *Dotnet **interactive session* and I
> would
> > > like to know if the Livy community would be interested in adding this
> > > feature to the main code base.  If so, please let me know and I can
> work
> > on
> > > creating this PR.
> > >
> > > For now, I've created a Jira item
> > >  to track this.
> > >
> > >
> > > Regards,
> > > -Steve
> > >
> >
>

Re: .NET for Apache Spark interactive interpreter support

[Steve Suh]

@Saisai

This will not be a wrapper around the REST API.  The plan is to support a
Livy POST /sessions request where kind == "sparkdotnet".  Livy will
eventually push this request down to the ReplDriver, and in turn the
ReplDriver will create a new SparkDotnet Interpreter class.  This will be
similar to how the  PythonInterpreter, SparkRInterpreter, and
SQLInterpreter classes get instantiated and used.


Regards,
-Steve

On Sun, Jan 12, 2020 at 5:45 PM Saisai Shao  wrote:

> Is this just a wrapper of Livy REST API, or it could support Livy Job API?
>
> From my opinion, if it is just a wrapper of REST API, then it would be
> better to maintain out of Livy, since REST API is language independent, if
> we're going to have all the languages support in Livy, then it is hard to
> maintain.
>
> Just my two cents.
>
> Thanks
> Saisai
>
> Steve Suh  于2020年1月12日周日 下午3:49写道：
>
> > Hi,
> >
> > I contribute to the .NET for Apache Spark <
> https://github.com/dotnet/spark
> > >
> > project and we have seen *a lot* of *user interest* in providing first
> > class notebook support for *.NET for Apache Spark*.
> >
> > Livy currently supports *Scala*, *Python *and *R **interactive sessions*.
> > We have a working prototype
> > <
> >
> https://github.com/dotnet/spark/tree/master/deployment/HDI-Spark/Notebooks
> > >
> > available
> > that adds support for a Spark *Dotnet **interactive session* and I would
> > like to know if the Livy community would be interested in adding this
> > feature to the main code base.  If so, please let me know and I can work
> on
> > creating this PR.
> >
> > For now, I've created a Jira item
> >  to track this.
> >
> >
> > Regards,
> > -Steve
> >
>

Re: .NET for Apache Spark interactive interpreter support

[Saisai Shao]

Is this just a wrapper of Livy REST API, or it could support Livy Job API?

>From my opinion, if it is just a wrapper of REST API, then it would be
better to maintain out of Livy, since REST API is language independent, if
we're going to have all the languages support in Livy, then it is hard to
maintain.

Just my two cents.

Thanks
Saisai

Steve Suh  于2020年1月12日周日 下午3:49写道：

> Hi,
>
> I contribute to the .NET for Apache Spark  >
> project and we have seen *a lot* of *user interest* in providing first
> class notebook support for *.NET for Apache Spark*.
>
> Livy currently supports *Scala*, *Python *and *R **interactive sessions*.
> We have a working prototype
> <
> https://github.com/dotnet/spark/tree/master/deployment/HDI-Spark/Notebooks
> >
> available
> that adds support for a Spark *Dotnet **interactive session* and I would
> like to know if the Livy community would be interested in adding this
> feature to the main code base.  If so, please let me know and I can work on
> creating this PR.
>
> For now, I've created a Jira item
>  to track this.
>
>
> Regards,
> -Steve
>

.NET for Apache Spark interactive interpreter support

[Steve Suh]

Hi,

I contribute to the .NET for Apache Spark 
project and we have seen *a lot* of *user interest* in providing first
class notebook support for *.NET for Apache Spark*.

Livy currently supports *Scala*, *Python *and *R **interactive sessions*.
We have a working prototype

available
that adds support for a Spark *Dotnet **interactive session* and I would
like to know if the Livy community would be interested in adding this
feature to the main code base.  If so, please let me know and I can work on
creating this PR.

For now, I've created a Jira item
 to track this.


Regards,
-Steve