Re: Spark Web UI port 4040 not working

[Marius Soutier]

That's to be expected - the application UI is not started by the master, but by 
the driver. So the UI will run on the machine that submits the job.


> On 26.07.2016, at 15:49, Jestin Ma  wrote:
> 
> I did netstat -apn | grep 4040 on machine 6, and I see
> 
> tcp0  0 :::4040 :::*
> LISTEN  30597/java
> 
> What does this mean?
> 
> On Tue, Jul 26, 2016 at 6:47 AM, Jestin Ma  > wrote:
> I do not deploy using cluster mode and I don't use EC2.
> 
> I just read that launching as client mode: "the driver is launched directly 
> within the spark-submit process which acts as a client to the cluster."
> 
> My current setup is that I have cluster machines 1, 2, 3, 4, 5, with 1 being 
> the master. 
> I submit from another cluster machine 6 in client mode. So I'm taking that 
> the driver is launched in my machine 6.
> 
> On Tue, Jul 26, 2016 at 6:38 AM, Jacek Laskowski  > wrote:
> Hi,
> 
> Do you perhaps deploy using cluster mode? Is this EC2? You'd need to
> figure out where the driver runs and use the machine's IP.
> 
> Pozdrawiam,
> Jacek Laskowski
> 
> https://medium.com/@jaceklaskowski/ 
> Mastering Apache Spark http://bit.ly/mastering-apache-spark 
> 
> Follow me at https://twitter.com/jaceklaskowski 
> 
> 
> 
> On Tue, Jul 26, 2016 at 3:36 PM, Jestin Ma  > wrote:
> > I tried doing that on my master node.
> > I got nothing.
> > However, I grep'd port 8080 and I got the standalone UI.
> >
> > On Tue, Jul 26, 2016 at 12:39 AM, Chanh Le  > > wrote:
> >>
> >> You’re running in StandAlone Mode?
> >> Usually inside active task it will show the address of current job.
> >> or you can check in master node by using netstat -apn | grep 4040
> >>
> >>
> >>
> >> > On Jul 26, 2016, at 8:21 AM, Jestin Ma  >> > >
> >> > wrote:
> >> >
> >> > Hello, when running spark jobs, I can access the master UI (port 8080
> >> > one) no problem. However, I'm confused as to how to access the web UI to 
> >> > see
> >> > jobs/tasks/stages/etc.
> >> >
> >> > I can access the master UI at http://:8080. But port 4040
> >> > gives me a -connection cannot be reached-.
> >> >
> >> > Is the web UI http:// with a port of 4040?
> >> >
> >> > I'm running my Spark job on a cluster machine and submitting it to a
> >> > master node part of the cluster. I heard of ssh tunneling; is that 
> >> > relevant
> >> > here?
> >> >
> >> > Thank you!
> >>
> >
> 
> 

Re: Spark Web UI port 4040 not working

[Andy Davidson]

Yup in cluster mode you need to figure out what machine the driver is
running on. That is the machine the UI will run on

https://issues.apache.org/jira/browse/SPARK-15829

You may also have fire wall issues

Here are some notes I made about how to figure out what machine the driver
is running on when using cluster mode

Application UI Bug work around
If our application is running in cluster mode the Master Console UI will
incorrectly assume the Application UI¹s are running on the master machine.
The UI is actually running on the same machine as the driver
To find the URL to the apps running in cluster mode
1. find the worker id for the app you are interested in.
* on the master console UI go to the Running App¹s section
* you should see a column ŒWorker¹
* a worker is something like worker-20160322041632-172.31.23.201-34909
2. got to the AWS ec2 console
3. from the private ip find the public DNS name
* the private ip in our example is 172.31.23.201

From:  Jacek Laskowski <ja...@japila.pl>
Date:  Tuesday, July 26, 2016 at 6:38 AM
To:  Jestin Ma <jestinwith.a...@gmail.com>
Cc:  Chanh Le <giaosu...@gmail.com>, "user @spark" <user@spark.apache.org>
Subject:  Re: Spark Web UI port 4040 not working

> Hi,
> 
> Do you perhaps deploy using cluster mode? Is this EC2? You'd need to
> figure out where the driver runs and use the machine's IP.
> 
> Pozdrawiam,
> Jacek Laskowski
> 
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
> 
> 
> On Tue, Jul 26, 2016 at 3:36 PM, Jestin Ma <jestinwith.a...@gmail.com> wrote:
>>  I tried doing that on my master node.
>>  I got nothing.
>>  However, I grep'd port 8080 and I got the standalone UI.
>> 
>>  On Tue, Jul 26, 2016 at 12:39 AM, Chanh Le <giaosu...@gmail.com> wrote:
>>> 
>>>  You¹re running in StandAlone Mode?
>>>  Usually inside active task it will show the address of current job.
>>>  or you can check in master node by using netstat -apn | grep 4040
>>> 
>>> 
>>> 
>>>>  > On Jul 26, 2016, at 8:21 AM, Jestin Ma <jestinwith.a...@gmail.com>
>>>>  > wrote:
>>>>  >
>>>>  > Hello, when running spark jobs, I can access the master UI (port 8080
>>>>  > one) no problem. However, I'm confused as to how to access the web UI to
>>>> see
>>>>  > jobs/tasks/stages/etc.
>>>>  >
>>>>  > I can access the master UI at http://:8080. But port 4040
>>>>  > gives me a -connection cannot be reached-.
>>>>  >
>>>>  > Is the web UI http:// with a port of 4040?
>>>>  >
>>>>  > I'm running my Spark job on a cluster machine and submitting it to a
>>>>  > master node part of the cluster. I heard of ssh tunneling; is that
>>>> relevant
>>>>  > here?
>>>>  >
>>>>  > Thank you!
>>> 
>> 
> 
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
> 
> 

Re: Spark Web UI port 4040 not working

[Jestin Ma]

I did netstat -apn | grep 4040 on machine 6, and I see

tcp0  0 :::4040 :::*
 LISTEN  30597/java

What does this mean?

On Tue, Jul 26, 2016 at 6:47 AM, Jestin Ma 
wrote:

> I do not deploy using cluster mode and I don't use EC2.
>
> I just read that launching as client mode: "the driver is launched
> directly within the spark-submit process which acts as a *client* to the
> cluster."
>
> My current setup is that I have cluster machines 1, 2, 3, 4, 5, with 1
> being the master.
> I submit from another cluster machine 6 in client mode. So I'm taking that
> the driver is launched in my machine 6.
>
> On Tue, Jul 26, 2016 at 6:38 AM, Jacek Laskowski  wrote:
>
>> Hi,
>>
>> Do you perhaps deploy using cluster mode? Is this EC2? You'd need to
>> figure out where the driver runs and use the machine's IP.
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> 
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>>
>> On Tue, Jul 26, 2016 at 3:36 PM, Jestin Ma 
>> wrote:
>> > I tried doing that on my master node.
>> > I got nothing.
>> > However, I grep'd port 8080 and I got the standalone UI.
>> >
>> > On Tue, Jul 26, 2016 at 12:39 AM, Chanh Le  wrote:
>> >>
>> >> You’re running in StandAlone Mode?
>> >> Usually inside active task it will show the address of current job.
>> >> or you can check in master node by using netstat -apn | grep 4040
>> >>
>> >>
>> >>
>> >> > On Jul 26, 2016, at 8:21 AM, Jestin Ma 
>> >> > wrote:
>> >> >
>> >> > Hello, when running spark jobs, I can access the master UI (port 8080
>> >> > one) no problem. However, I'm confused as to how to access the web
>> UI to see
>> >> > jobs/tasks/stages/etc.
>> >> >
>> >> > I can access the master UI at http://:8080. But port
>> 4040
>> >> > gives me a -connection cannot be reached-.
>> >> >
>> >> > Is the web UI http:// with a port of 4040?
>> >> >
>> >> > I'm running my Spark job on a cluster machine and submitting it to a
>> >> > master node part of the cluster. I heard of ssh tunneling; is that
>> relevant
>> >> > here?
>> >> >
>> >> > Thank you!
>> >>
>> >
>>
>
>

Re: Spark Web UI port 4040 not working

[Jestin Ma]

I do not deploy using cluster mode and I don't use EC2.

I just read that launching as client mode: "the driver is launched directly
within the spark-submit process which acts as a *client* to the cluster."

My current setup is that I have cluster machines 1, 2, 3, 4, 5, with 1
being the master.
I submit from another cluster machine 6 in client mode. So I'm taking that
the driver is launched in my machine 6.

On Tue, Jul 26, 2016 at 6:38 AM, Jacek Laskowski  wrote:

> Hi,
>
> Do you perhaps deploy using cluster mode? Is this EC2? You'd need to
> figure out where the driver runs and use the machine's IP.
>
> Pozdrawiam,
> Jacek Laskowski
> 
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
>
> On Tue, Jul 26, 2016 at 3:36 PM, Jestin Ma 
> wrote:
> > I tried doing that on my master node.
> > I got nothing.
> > However, I grep'd port 8080 and I got the standalone UI.
> >
> > On Tue, Jul 26, 2016 at 12:39 AM, Chanh Le  wrote:
> >>
> >> You’re running in StandAlone Mode?
> >> Usually inside active task it will show the address of current job.
> >> or you can check in master node by using netstat -apn | grep 4040
> >>
> >>
> >>
> >> > On Jul 26, 2016, at 8:21 AM, Jestin Ma 
> >> > wrote:
> >> >
> >> > Hello, when running spark jobs, I can access the master UI (port 8080
> >> > one) no problem. However, I'm confused as to how to access the web UI
> to see
> >> > jobs/tasks/stages/etc.
> >> >
> >> > I can access the master UI at http://:8080. But port
> 4040
> >> > gives me a -connection cannot be reached-.
> >> >
> >> > Is the web UI http:// with a port of 4040?
> >> >
> >> > I'm running my Spark job on a cluster machine and submitting it to a
> >> > master node part of the cluster. I heard of ssh tunneling; is that
> relevant
> >> > here?
> >> >
> >> > Thank you!
> >>
> >
>

Re: Spark Web UI port 4040 not working

[Jestin Ma]

I tried doing that on my master node.
I got nothing.
However, I grep'd port 8080 and I got the standalone UI.

On Tue, Jul 26, 2016 at 12:39 AM, Chanh Le  wrote:

> You’re running in StandAlone Mode?
> Usually inside active task it will show the address of current job.
> or you can check in master node by using netstat -apn | grep 4040
>
>
>
> > On Jul 26, 2016, at 8:21 AM, Jestin Ma 
> wrote:
> >
> > Hello, when running spark jobs, I can access the master UI (port 8080
> one) no problem. However, I'm confused as to how to access the web UI to
> see jobs/tasks/stages/etc.
> >
> > I can access the master UI at http://:8080. But port 4040
> gives me a -connection cannot be reached-.
> >
> > Is the web UI http:// with a port of 4040?
> >
> > I'm running my Spark job on a cluster machine and submitting it to a
> master node part of the cluster. I heard of ssh tunneling; is that relevant
> here?
> >
> > Thank you!
>
>

Re: Spark Web UI port 4040 not working

[Jacek Laskowski]

Hi,

Do you perhaps deploy using cluster mode? Is this EC2? You'd need to
figure out where the driver runs and use the machine's IP.

Pozdrawiam,
Jacek Laskowski

https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Tue, Jul 26, 2016 at 3:36 PM, Jestin Ma  wrote:
> I tried doing that on my master node.
> I got nothing.
> However, I grep'd port 8080 and I got the standalone UI.
>
> On Tue, Jul 26, 2016 at 12:39 AM, Chanh Le  wrote:
>>
>> You’re running in StandAlone Mode?
>> Usually inside active task it will show the address of current job.
>> or you can check in master node by using netstat -apn | grep 4040
>>
>>
>>
>> > On Jul 26, 2016, at 8:21 AM, Jestin Ma 
>> > wrote:
>> >
>> > Hello, when running spark jobs, I can access the master UI (port 8080
>> > one) no problem. However, I'm confused as to how to access the web UI to 
>> > see
>> > jobs/tasks/stages/etc.
>> >
>> > I can access the master UI at http://:8080. But port 4040
>> > gives me a -connection cannot be reached-.
>> >
>> > Is the web UI http:// with a port of 4040?
>> >
>> > I'm running my Spark job on a cluster machine and submitting it to a
>> > master node part of the cluster. I heard of ssh tunneling; is that relevant
>> > here?
>> >
>> > Thank you!
>>
>

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Spark Web UI port 4040 not working

[Jacek Laskowski]

Hi,

Go to 8080 and under Running Applications click the Application ID.
You're on the page with Application Detail UI just before Executor
Summary table. Use it to access the web UI.

Pozdrawiam,
Jacek Laskowski

https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Tue, Jul 26, 2016 at 3:21 AM, Jestin Ma  wrote:
> Hello, when running spark jobs, I can access the master UI (port 8080 one)
> no problem. However, I'm confused as to how to access the web UI to see
> jobs/tasks/stages/etc.
>
> I can access the master UI at http://:8080. But port 4040 gives
> me a -connection cannot be reached-.
>
> Is the web UI http:// with a port of 4040?
>
> I'm running my Spark job on a cluster machine and submitting it to a master
> node part of the cluster. I heard of ssh tunneling; is that relevant here?
>
> Thank you!

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Spark Web UI port 4040 not working

[Chanh Le]

You’re running in StandAlone Mode?
Usually inside active task it will show the address of current job.
or you can check in master node by using netstat -apn | grep 4040



> On Jul 26, 2016, at 8:21 AM, Jestin Ma  wrote:
> 
> Hello, when running spark jobs, I can access the master UI (port 8080 one) no 
> problem. However, I'm confused as to how to access the web UI to see 
> jobs/tasks/stages/etc.
> 
> I can access the master UI at http://:8080. But port 4040 gives 
> me a -connection cannot be reached-.
> 
> Is the web UI http:// with a port of 4040?
> 
> I'm running my Spark job on a cluster machine and submitting it to a master 
> node part of the cluster. I heard of ssh tunneling; is that relevant here?
> 
> Thank you!


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org