Re: Ammonite as REPL for Spark Connect

[John Zhuge]

+1 on better notebook and other REPL experience

On Thu, Mar 23, 2023 at 9:17 AM Dongjoon Hyun 
wrote:

> I also support Herman's `SPARK-42884 Add Ammonite REPL integration` PR.
>
> Thanks,
> Dongjoon.
>
>
> On Thu, Mar 23, 2023 at 7:51 AM Mridul Muralidharan 
> wrote:
>
>>
>> Sounds good, thanks for clarifying !
>>
>> Regards,
>> Mridul
>>
>> On Thu, Mar 23, 2023 at 9:09 AM Herman van Hovell 
>> wrote:
>>
>>> The goal of adding this, is to make it easy for a user to connect a
>>> scala REPL to a Spark Connect server. Just like Spark shell makes it easy
>>> to work with a regular Spark environment.
>>>
>>> It is not meant as a Spark shell replacement. They represent two
>>> different modes of working with Spark, and they have very different API
>>> surfaces (Connect being a subset of what regular Spark has to offer). I do
>>> think we should consider using ammonite for Spark shell at some point,
>>> since this has better UX and does not require us to fork a REPL. That
>>> discussion is for another day though.
>>>
>>> I guess you can use it as an example of building an integration. In
>>> itself I wouldn't call it that because I think this a key part of getting
>>> started with connect, and/or doing debugging.
>>>
>>> On Thu, Mar 23, 2023 at 4:00 AM Mridul Muralidharan 
>>> wrote:
>>>

 What is unclear to me is why we are introducing this integration, how
 users will leverage it.

 * Are we replacing spark-shell with it ?
 Given the existing gaps, this is not the case.

 * Is it an example to showcase how to build an integration ?
 That could be interesting, and we can add it to external/

 Anything else I am missing ?

 Regards,
 Mridul



 On Wed, Mar 22, 2023 at 6:58 PM Herman van Hovell <
 her...@databricks.com> wrote:

> Ammonite is maintained externally by Li Haoyi et al. We are including
> it as a 'provided' dependency. The integration bits and pieces (1 file) 
> are
> included in Apache Spark.
>
> On Wed, Mar 22, 2023 at 7:53 PM Mridul Muralidharan 
> wrote:
>
>>
>> Will this be maintained externally or included into Apache Spark ?
>>
>> Regards ,
>> Mridul
>>
>>
>>
>> On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell
>>  wrote:
>>
>>> Hi All,
>>>
>>> For Spark Connect Scala Client we are working on making the REPL
>>> experience a bit nicer .
>>> In a nutshell we want to give users a turn key scala REPL, that works 
>>> even
>>> if you don't have a Spark distribution on your machine (through
>>> coursier ). We are using Ammonite
>>>  instead of the standard scala REPL for this,
>>> the main reason for going with Ammonite is that it is easier to 
>>> customize,
>>> and IMO has a superior user experience.
>>>
>>> Does anyone object to doing this?
>>>
>>> Kind regards,
>>> Herman
>>>
>>>

-- 
John Zhuge

Re: Ammonite as REPL for Spark Connect

[Dongjoon Hyun]

I also support Herman's `SPARK-42884 Add Ammonite REPL integration` PR.

Thanks,
Dongjoon.


On Thu, Mar 23, 2023 at 7:51 AM Mridul Muralidharan 
wrote:

>
> Sounds good, thanks for clarifying !
>
> Regards,
> Mridul
>
> On Thu, Mar 23, 2023 at 9:09 AM Herman van Hovell 
> wrote:
>
>> The goal of adding this, is to make it easy for a user to connect a scala
>> REPL to a Spark Connect server. Just like Spark shell makes it easy to work
>> with a regular Spark environment.
>>
>> It is not meant as a Spark shell replacement. They represent two
>> different modes of working with Spark, and they have very different API
>> surfaces (Connect being a subset of what regular Spark has to offer). I do
>> think we should consider using ammonite for Spark shell at some point,
>> since this has better UX and does not require us to fork a REPL. That
>> discussion is for another day though.
>>
>> I guess you can use it as an example of building an integration. In
>> itself I wouldn't call it that because I think this a key part of getting
>> started with connect, and/or doing debugging.
>>
>> On Thu, Mar 23, 2023 at 4:00 AM Mridul Muralidharan 
>> wrote:
>>
>>>
>>> What is unclear to me is why we are introducing this integration, how
>>> users will leverage it.
>>>
>>> * Are we replacing spark-shell with it ?
>>> Given the existing gaps, this is not the case.
>>>
>>> * Is it an example to showcase how to build an integration ?
>>> That could be interesting, and we can add it to external/
>>>
>>> Anything else I am missing ?
>>>
>>> Regards,
>>> Mridul
>>>
>>>
>>>
>>> On Wed, Mar 22, 2023 at 6:58 PM Herman van Hovell 
>>> wrote:
>>>
 Ammonite is maintained externally by Li Haoyi et al. We are including
 it as a 'provided' dependency. The integration bits and pieces (1 file) are
 included in Apache Spark.

 On Wed, Mar 22, 2023 at 7:53 PM Mridul Muralidharan 
 wrote:

>
> Will this be maintained externally or included into Apache Spark ?
>
> Regards ,
> Mridul
>
>
>
> On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell
>  wrote:
>
>> Hi All,
>>
>> For Spark Connect Scala Client we are working on making the REPL
>> experience a bit nicer .
>> In a nutshell we want to give users a turn key scala REPL, that works 
>> even
>> if you don't have a Spark distribution on your machine (through
>> coursier ). We are using Ammonite
>>  instead of the standard scala REPL for this,
>> the main reason for going with Ammonite is that it is easier to 
>> customize,
>> and IMO has a superior user experience.
>>
>> Does anyone object to doing this?
>>
>> Kind regards,
>> Herman
>>
>>

Re: Ammonite as REPL for Spark Connect

[Mridul Muralidharan]

Sounds good, thanks for clarifying !

Regards,
Mridul

On Thu, Mar 23, 2023 at 9:09 AM Herman van Hovell 
wrote:

> The goal of adding this, is to make it easy for a user to connect a scala
> REPL to a Spark Connect server. Just like Spark shell makes it easy to work
> with a regular Spark environment.
>
> It is not meant as a Spark shell replacement. They represent two different
> modes of working with Spark, and they have very different API surfaces
> (Connect being a subset of what regular Spark has to offer). I do think we
> should consider using ammonite for Spark shell at some point, since this
> has better UX and does not require us to fork a REPL. That discussion is
> for another day though.
>
> I guess you can use it as an example of building an integration. In itself
> I wouldn't call it that because I think this a key part of getting started
> with connect, and/or doing debugging.
>
> On Thu, Mar 23, 2023 at 4:00 AM Mridul Muralidharan 
> wrote:
>
>>
>> What is unclear to me is why we are introducing this integration, how
>> users will leverage it.
>>
>> * Are we replacing spark-shell with it ?
>> Given the existing gaps, this is not the case.
>>
>> * Is it an example to showcase how to build an integration ?
>> That could be interesting, and we can add it to external/
>>
>> Anything else I am missing ?
>>
>> Regards,
>> Mridul
>>
>>
>>
>> On Wed, Mar 22, 2023 at 6:58 PM Herman van Hovell 
>> wrote:
>>
>>> Ammonite is maintained externally by Li Haoyi et al. We are including it
>>> as a 'provided' dependency. The integration bits and pieces (1 file) are
>>> included in Apache Spark.
>>>
>>> On Wed, Mar 22, 2023 at 7:53 PM Mridul Muralidharan 
>>> wrote:
>>>

 Will this be maintained externally or included into Apache Spark ?

 Regards ,
 Mridul



 On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell
  wrote:

> Hi All,
>
> For Spark Connect Scala Client we are working on making the REPL
> experience a bit nicer .
> In a nutshell we want to give users a turn key scala REPL, that works even
> if you don't have a Spark distribution on your machine (through
> coursier ). We are using Ammonite
>  instead of the standard scala REPL for this,
> the main reason for going with Ammonite is that it is easier to customize,
> and IMO has a superior user experience.
>
> Does anyone object to doing this?
>
> Kind regards,
> Herman
>
>

Re: Ammonite as REPL for Spark Connect

[Herman van Hovell]

The goal of adding this, is to make it easy for a user to connect a scala
REPL to a Spark Connect server. Just like Spark shell makes it easy to work
with a regular Spark environment.

It is not meant as a Spark shell replacement. They represent two different
modes of working with Spark, and they have very different API surfaces
(Connect being a subset of what regular Spark has to offer). I do think we
should consider using ammonite for Spark shell at some point, since this
has better UX and does not require us to fork a REPL. That discussion is
for another day though.

I guess you can use it as an example of building an integration. In itself
I wouldn't call it that because I think this a key part of getting started
with connect, and/or doing debugging.

On Thu, Mar 23, 2023 at 4:00 AM Mridul Muralidharan 
wrote:

>
> What is unclear to me is why we are introducing this integration, how
> users will leverage it.
>
> * Are we replacing spark-shell with it ?
> Given the existing gaps, this is not the case.
>
> * Is it an example to showcase how to build an integration ?
> That could be interesting, and we can add it to external/
>
> Anything else I am missing ?
>
> Regards,
> Mridul
>
>
>
> On Wed, Mar 22, 2023 at 6:58 PM Herman van Hovell 
> wrote:
>
>> Ammonite is maintained externally by Li Haoyi et al. We are including it
>> as a 'provided' dependency. The integration bits and pieces (1 file) are
>> included in Apache Spark.
>>
>> On Wed, Mar 22, 2023 at 7:53 PM Mridul Muralidharan 
>> wrote:
>>
>>>
>>> Will this be maintained externally or included into Apache Spark ?
>>>
>>> Regards ,
>>> Mridul
>>>
>>>
>>>
>>> On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell
>>>  wrote:
>>>
 Hi All,

 For Spark Connect Scala Client we are working on making the REPL
 experience a bit nicer .
 In a nutshell we want to give users a turn key scala REPL, that works even
 if you don't have a Spark distribution on your machine (through
 coursier ). We are using Ammonite
  instead of the standard scala REPL for this,
 the main reason for going with Ammonite is that it is easier to customize,
 and IMO has a superior user experience.

 Does anyone object to doing this?

 Kind regards,
 Herman

Re: Ammonite as REPL for Spark Connect

[Mridul Muralidharan]

What is unclear to me is why we are introducing this integration, how users
will leverage it.

* Are we replacing spark-shell with it ?
Given the existing gaps, this is not the case.

* Is it an example to showcase how to build an integration ?
That could be interesting, and we can add it to external/

Anything else I am missing ?

Regards,
Mridul



On Wed, Mar 22, 2023 at 6:58 PM Herman van Hovell 
wrote:

> Ammonite is maintained externally by Li Haoyi et al. We are including it
> as a 'provided' dependency. The integration bits and pieces (1 file) are
> included in Apache Spark.
>
> On Wed, Mar 22, 2023 at 7:53 PM Mridul Muralidharan 
> wrote:
>
>>
>> Will this be maintained externally or included into Apache Spark ?
>>
>> Regards ,
>> Mridul
>>
>>
>>
>> On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell
>>  wrote:
>>
>>> Hi All,
>>>
>>> For Spark Connect Scala Client we are working on making the REPL
>>> experience a bit nicer . In
>>> a nutshell we want to give users a turn key scala REPL, that works even if
>>> you don't have a Spark distribution on your machine (through coursier
>>> ). We are using Ammonite
>>>  instead of the standard scala REPL for this, the
>>> main reason for going with Ammonite is that it is easier to customize, and
>>> IMO has a superior user experience.
>>>
>>> Does anyone object to doing this?
>>>
>>> Kind regards,
>>> Herman
>>>
>>>

Re: Ammonite as REPL for Spark Connect

[Holden Karau]

I am +1 to the general concept of including Ammonite magic 🪄.

On Wed, Mar 22, 2023 at 4:58 PM Herman van Hovell
 wrote:

> Ammonite is maintained externally by Li Haoyi et al. We are including it
> as a 'provided' dependency. The integration bits and pieces (1 file) are
> included in Apache Spark.
>
> On Wed, Mar 22, 2023 at 7:53 PM Mridul Muralidharan 
> wrote:
>
>>
>> Will this be maintained externally or included into Apache Spark ?
>>
>> Regards ,
>> Mridul
>>
>>
>>
>> On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell
>>  wrote:
>>
>>> Hi All,
>>>
>>> For Spark Connect Scala Client we are working on making the REPL
>>> experience a bit nicer . In
>>> a nutshell we want to give users a turn key scala REPL, that works even if
>>> you don't have a Spark distribution on your machine (through coursier
>>> ). We are using Ammonite
>>>  instead of the standard scala REPL for this, the
>>> main reason for going with Ammonite is that it is easier to customize, and
>>> IMO has a superior user experience.
>>>
>>> Does anyone object to doing this?
>>>
>>> Kind regards,
>>> Herman
>>>
>>> --
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: Ammonite as REPL for Spark Connect

[Herman van Hovell]

Ammonite is maintained externally by Li Haoyi et al. We are including it as
a 'provided' dependency. The integration bits and pieces (1 file) are
included in Apache Spark.

On Wed, Mar 22, 2023 at 7:53 PM Mridul Muralidharan 
wrote:

>
> Will this be maintained externally or included into Apache Spark ?
>
> Regards ,
> Mridul
>
>
>
> On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell
>  wrote:
>
>> Hi All,
>>
>> For Spark Connect Scala Client we are working on making the REPL
>> experience a bit nicer . In
>> a nutshell we want to give users a turn key scala REPL, that works even if
>> you don't have a Spark distribution on your machine (through coursier
>> ). We are using Ammonite 
>> instead of the standard scala REPL for this, the main reason for going with
>> Ammonite is that it is easier to customize, and IMO has a superior user
>> experience.
>>
>> Does anyone object to doing this?
>>
>> Kind regards,
>> Herman
>>
>>

Re: Ammonite as REPL for Spark Connect

[Mridul Muralidharan]

Will this be maintained externally or included into Apache Spark ?

Regards ,
Mridul



On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell
 wrote:

> Hi All,
>
> For Spark Connect Scala Client we are working on making the REPL
> experience a bit nicer . In a
> nutshell we want to give users a turn key scala REPL, that works even if
> you don't have a Spark distribution on your machine (through coursier
> ). We are using Ammonite 
> instead of the standard scala REPL for this, the main reason for going with
> Ammonite is that it is easier to customize, and IMO has a superior user
> experience.
>
> Does anyone object to doing this?
>
> Kind regards,
> Herman
>
>

Ammonite as REPL for Spark Connect

[Herman van Hovell]

Hi All,

For Spark Connect Scala Client we are working on making the REPL experience
a bit nicer . In a nutshell we
want to give users a turn key scala REPL, that works even if you don't have
a Spark distribution on your machine (through coursier
). We are using Ammonite 
instead of the standard scala REPL for this, the main reason for going with
Ammonite is that it is easier to customize, and IMO has a superior user
experience.

Does anyone object to doing this?

Kind regards,
Herman