Re: IDE suitable for Spark : Monitoring & Debugging Spark Jobs
The definitive guide
Chapter 18:
Monitoring and Debugging
"This chapter covers the key details you need to monitor and debug your
Spark Applications. To do this , we will walk through the spark UI with an
example query designed to help you understand how to trace your own jobs
through the executions life cycle. The example we'll look at will also help
you to understand how to debug your jobs and where errors are likely to
occur."
On Tue, 7 Apr 2020, 18:28 Pat Ferrel, wrote:
> IntelliJ Scala works well when debugging master=local. Has anyone used it
> for remote/cluster debugging? I’ve heard it is possible...
>
>
> From: Luiz Camargo
> Reply: Luiz Camargo
> Date: April 7, 2020 at 10:26:35 AM
> To: Dennis Suhari
>
> Cc: yeikel valdes ,
> zahidr1...@gmail.com ,
> user@spark.apache.org
> Subject: Re: IDE suitable for Spark
>
> I have used IntelliJ Spark/Scala with the sbt tool
>
> On Tue, Apr 7, 2020 at 1:18 PM Dennis Suhari
> wrote:
>
>> We are using Pycharm resp. R Studio with Spark libraries to submit Spark
>> Jobs.
>>
>> Von meinem iPhone gesendet
>>
>> Am 07.04.2020 um 18:10 schrieb yeikel valdes :
>>
>>
>>
>> Zeppelin is not an IDE but a notebook. It is helpful to experiment but
>> it is missing a lot of the features that we expect from an IDE.
>>
>> Thanks for sharing though.
>>
>> On Tue, 07 Apr 2020 04:45:33 -0400 * zahidr1...@gmail.com
>> * wrote
>>
>> When I first logged on I asked if there was a suitable IDE for Spark.
>> I did get a couple of responses.
>> *Thanks.*
>>
>> I did actually find one which is suitable IDE for spark.
>> That is *Apache Zeppelin.*
>>
>> One of many reasons it is suitable for Apache Spark is.
>> The *up and running Stage* which involves typing *bin/zeppelin-daemon.sh
>> start*
>> Go to browser and type *http://localhost:8080 <http://localhost:8080>*
>> That's it!
>>
>> Then to
>> * Hit the ground running*
>> There are also ready to go Apache Spark examples
>> showing off the type of functionality one will be using in real life
>> production.
>>
>> Zeppelin comes with embedded Apache Spark and scala as default
>> interpreter with 20 + interpreters.
>> I have gone on to discover there are a number of other advantages for
>> real time production
>> environment with Zeppelin offered up by other Apache Products.
>>
>> Backbutton.co.uk
>> ¯\_(ツ)_/¯
>> ♡۶Java♡۶RMI ♡۶
>> Make Use Method {MUM}
>> makeuse.org
>> <http://www.backbutton.co.uk>
>>
>>
>>
>
> --
>
>
> Prof. Luiz Camargo
> Educador - Computação
>
>
>
Re: IDE suitable for Spark
I think as long as you set the master in the code as the correct cluster URL,
everything should work as expected. So you should be able to place breakpoints
in IntelliJ as if running in local mode.
Get Outlook for iOS<https://aka.ms/o0ukef>
From: Pat Ferrel
Sent: Tuesday, April 7, 2020 10:58:39 PM
To: Luiz Camargo ; Dennis Suhari
Cc: zahidr1...@gmail.com ; user@spark.apache.org
; yeikel valdes
Subject: Re: IDE suitable for Spark
IntelliJ Scala works well when debugging master=local. Has anyone used it for
remote/cluster debugging? I’ve heard it is possible...
From: Luiz Camargo <mailto:camar...@gmail.com>
Reply: Luiz Camargo <mailto:camar...@gmail.com>
Date: April 7, 2020 at 10:26:35 AM
To: Dennis Suhari
<mailto:d.suh...@icloud.com.invalid>
Cc: yeikel valdes <mailto:em...@yeikel.com>,
zahidr1...@gmail.com <mailto:zahidr1...@gmail.com>,
user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: IDE suitable for Spark
I have used IntelliJ Spark/Scala with the sbt tool
On Tue, Apr 7, 2020 at 1:18 PM Dennis Suhari
wrote:
We are using Pycharm resp. R Studio with Spark libraries to submit Spark Jobs.
Von meinem iPhone gesendet
Am 07.04.2020 um 18:10 schrieb yeikel valdes
mailto:em...@yeikel.com>>:
Zeppelin is not an IDE but a notebook. It is helpful to experiment but it is
missing a lot of the features that we expect from an IDE.
Thanks for sharing though.
On Tue, 07 Apr 2020 04:45:33 -0400
zahidr1...@gmail.com<mailto:zahidr1...@gmail.com>wrote
When I first logged on I asked if there was a suitable IDE for Spark.
I did get a couple of responses.
Thanks.
I did actually find one which is suitable IDE for spark.
That is Apache Zeppelin.
One of many reasons it is suitable for Apache Spark is.
The up and running Stage which involves typing bin/zeppelin-daemon.sh start
Go to browser and type http://localhost:8080
That's it!
Then to
Hit the ground running
There are also ready to go Apache Spark examples
showing off the type of functionality one will be using in real life production.
Zeppelin comes with embedded Apache Spark and scala as default interpreter
with 20 + interpreters.
I have gone on to discover there are a number of other advantages for real time
production
environment with Zeppelin offered up by other Apache Products.
Backbutton.co.uk<http://Backbutton.co.uk>
¯_(ツ)_/¯
♡۶Java♡۶RMI ♡۶
Make Use Method {MUM}
makeuse.org<http://makeuse.org>
<http://www.backbutton.co.uk>
--
Prof. Luiz Camargo
Educador - Computação
Re: IDE suitable for Spark
IntelliJ Scala works well when debugging master=local. Has anyone used it for
remote/cluster debugging? I’ve heard it is possible...
From: Luiz Camargo
Reply: Luiz Camargo
Date: April 7, 2020 at 10:26:35 AM
To: Dennis Suhari
Cc: yeikel valdes , zahidr1...@gmail.com
, user@spark.apache.org
Subject: Re: IDE suitable for Spark
I have used IntelliJ Spark/Scala with the sbt tool
On Tue, Apr 7, 2020 at 1:18 PM Dennis Suhari
wrote:
We are using Pycharm resp. R Studio with Spark libraries to submit Spark Jobs.
Von meinem iPhone gesendet
Am 07.04.2020 um 18:10 schrieb yeikel valdes :
Zeppelin is not an IDE but a notebook. It is helpful to experiment but it is
missing a lot of the features that we expect from an IDE.
Thanks for sharing though.
On Tue, 07 Apr 2020 04:45:33 -0400 zahidr1...@gmail.com wrote
When I first logged on I asked if there was a suitable IDE for Spark.
I did get a couple of responses.
Thanks.
I did actually find one which is suitable IDE for spark.
That is Apache Zeppelin.
One of many reasons it is suitable for Apache Spark is.
The up and running Stage which involves typing bin/zeppelin-daemon.sh start
Go to browser and type http://localhost:8080
That's it!
Then to
Hit the ground running
There are also ready to go Apache Spark examples
showing off the type of functionality one will be using in real life production.
Zeppelin comes with embedded Apache Spark and scala as default interpreter
with 20 + interpreters.
I have gone on to discover there are a number of other advantages for real time
production
environment with Zeppelin offered up by other Apache Products.
Backbutton.co.uk
¯\_(ツ)_/¯
♡۶Java♡۶RMI ♡۶
Make Use Method {MUM}
makeuse.org
--
Prof. Luiz Camargo
Educador - Computação
Re: IDE suitable for Spark
I have used IntelliJ Spark/Scala with the sbt tool
On Tue, Apr 7, 2020 at 1:18 PM Dennis Suhari
wrote:
> We are using Pycharm resp. R Studio with Spark libraries to submit Spark
> Jobs.
>
> Von meinem iPhone gesendet
>
> Am 07.04.2020 um 18:10 schrieb yeikel valdes :
>
>
>
> Zeppelin is not an IDE but a notebook. It is helpful to experiment but it
> is missing a lot of the features that we expect from an IDE.
>
> Thanks for sharing though.
>
> On Tue, 07 Apr 2020 04:45:33 -0400 * zahidr1...@gmail.com
> * wrote
>
> When I first logged on I asked if there was a suitable IDE for Spark.
> I did get a couple of responses.
> *Thanks.*
>
> I did actually find one which is suitable IDE for spark.
> That is *Apache Zeppelin.*
>
> One of many reasons it is suitable for Apache Spark is.
> The *up and running Stage* which involves typing *bin/zeppelin-daemon.sh
> start*
> Go to browser and type *http://localhost:8080 <http://localhost:8080>*
> That's it!
>
> Then to
> * Hit the ground running*
> There are also ready to go Apache Spark examples
> showing off the type of functionality one will be using in real life
> production.
>
> Zeppelin comes with embedded Apache Spark and scala as default
> interpreter with 20 + interpreters.
> I have gone on to discover there are a number of other advantages for real
> time production
> environment with Zeppelin offered up by other Apache Products.
>
> Backbutton.co.uk
> ¯\_(ツ)_/¯
> ♡۶Java♡۶RMI ♡۶
> Make Use Method {MUM}
> makeuse.org
> <http://www.backbutton.co.uk>
>
>
>
--
Prof. Luiz Camargo
Educador - Computação
Re: IDE suitable for Spark
We are using Pycharm resp. R Studio with Spark libraries to submit Spark Jobs.
Von meinem iPhone gesendet
> Am 07.04.2020 um 18:10 schrieb yeikel valdes :
>
>
>
> Zeppelin is not an IDE but a notebook. It is helpful to experiment but it is
> missing a lot of the features that we expect from an IDE.
>
> Thanks for sharing though.
>
> On Tue, 07 Apr 2020 04:45:33 -0400 zahidr1...@gmail.com wrote
>
> When I first logged on I asked if there was a suitable IDE for Spark.
> I did get a couple of responses.
> Thanks.
>
> I did actually find one which is suitable IDE for spark.
> That is Apache Zeppelin.
>
> One of many reasons it is suitable for Apache Spark is.
> The up and running Stage which involves typing bin/zeppelin-daemon.sh start
> Go to browser and type http://localhost:8080
> That's it!
>
> Then to
> Hit the ground running
> There are also ready to go Apache Spark examples
> showing off the type of functionality one will be using in real life
> production.
>
> Zeppelin comes with embedded Apache Spark and scala as default interpreter
> with 20 + interpreters.
> I have gone on to discover there are a number of other advantages for real
> time production
> environment with Zeppelin offered up by other Apache Products.
>
> Backbutton.co.uk
> ¯\_(ツ)_/¯
> ♡۶Java♡۶RMI ♡۶
> Make Use Method {MUM}
> makeuse.org
>
Re: IDE suitable for Spark
I have been using Idea for both scala/spark and pyspark projects since
2013. It required fair amount of fiddling that first year but has been
stable since early 2015. For pyspark projects only Pycharm naturally also
works v well.
Am Di., 7. Apr. 2020 um 09:10 Uhr schrieb yeikel valdes :
>
> Zeppelin is not an IDE but a notebook. It is helpful to experiment but it
> is missing a lot of the features that we expect from an IDE.
>
> Thanks for sharing though.
>
> On Tue, 07 Apr 2020 04:45:33 -0400 * zahidr1...@gmail.com
> * wrote
>
> When I first logged on I asked if there was a suitable IDE for Spark.
> I did get a couple of responses.
> *Thanks.*
>
> I did actually find one which is suitable IDE for spark.
> That is *Apache Zeppelin.*
>
> One of many reasons it is suitable for Apache Spark is.
> The *up and running Stage* which involves typing *bin/zeppelin-daemon.sh
> start*
> Go to browser and type *http://localhost:8080 <http://localhost:8080>*
> That's it!
>
> Then to
> * Hit the ground running*
> There are also ready to go Apache Spark examples
> showing off the type of functionality one will be using in real life
> production.
>
> Zeppelin comes with embedded Apache Spark and scala as default
> interpreter with 20 + interpreters.
> I have gone on to discover there are a number of other advantages for real
> time production
> environment with Zeppelin offered up by other Apache Products.
>
> Backbutton.co.uk
> ¯\_(ツ)_/¯
> ♡۶Java♡۶RMI ♡۶
> Make Use Method {MUM}
> makeuse.org
> <http://www.backbutton.co.uk>
>
>
>
Re: IDE suitable for Spark
Zeppelin is not an IDE but a notebook. It is helpful to experiment but it is
missing a lot of the features that we expect from an IDE.
Thanks for sharing though.
On Tue, 07 Apr 2020 04:45:33 -0400 zahidr1...@gmail.com wrote
When I first logged on I asked if there was a suitable IDE for Spark.
I did get a couple of responses.
Thanks.
I did actually find one which is suitable IDE for spark.
That is Apache Zeppelin.
One of many reasons it is suitable for Apache Spark is.
The up and running Stage which involves typing bin/zeppelin-daemon.sh start
Go to browser and type http://localhost:8080
That's it!
Then to
Hit the ground running
There are also ready to go Apache Spark examples
showing off the type of functionality one will be using in real life production.
Zeppelin comes with embedded Apache Spark and scala as default interpreter
with 20 + interpreters.
I have gone on to discover there are a number of other advantages for real time
production
environment with Zeppelin offered up by other Apache Products.
Backbutton.co.uk
¯\_(ツ)_/¯
♡۶Java♡۶RMI ♡۶
Make Use Method {MUM}
makeuse.org
IDE suitable for Spark
When I first logged on I asked if there was a suitable IDE for Spark.
I did get a couple of responses.
*Thanks.*
I did actually find one which is suitable IDE for spark.
That is *Apache Zeppelin.*
One of many reasons it is suitable for Apache Spark is.
The *up and running Stage* which involves typing *bin/zeppelin-daemon.sh
start*
Go to browser and type *http://localhost:8080 <http://localhost:8080>*
That's it!
Then to
* Hit the ground running*
There are also ready to go Apache Spark examples
showing off the type of functionality one will be using in real life
production.
Zeppelin comes with embedded Apache Spark and scala as default
interpreter with 20 + interpreters.
I have gone on to discover there are a number of other advantages for real
time production
environment with Zeppelin offered up by other Apache Products.
Backbutton.co.uk
¯\_(ツ)_/¯
♡۶Java♡۶RMI ♡۶
Make Use Method {MUM}
makeuse.org
<http://www.backbutton.co.uk>
