RE: a question about web ui log
If the application history is turned on, it should work, even through ssh tunnel. Can you elaborate on what you mean by “it does not work?” Also, are you able to see the application web UI while an application is executing a job? Mohammed Author: Big Data Analytics with Spark<http://www.amazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/> From: Philip Lee [mailto:philjj...@gmail.com] Sent: Tuesday, January 26, 2016 5:12 AM To: Mohammed Guller Cc: user@spark.apache.org Subject: Re: a question about web ui log Yes, I tried it, but it simply does not work. so, my concern is to use "ssh tunnel" to forward a port of cluster to localhost port. But in Spark UI, there are two ports which I should forward using "ssh tunnel". Considering a default port, 8080 is web-ui port to come into web-ui, and 4040 is web-monitoring port to see time execution like DAG in application details UI, as you probably know. But after finishing a job, I can see the list of a job on the web-ui on 8080, but when I click "application details UI" on port 4040 to see time excution, it does not work. Any suggestion? I really need to see execution of DAG. Best, Phil On Tue, Jan 26, 2016 at 12:04 AM, Mohammed Guller mailto:moham...@glassbeam.com>> wrote: I am not sure whether you can copy the log files from Spark workers to your local machine and view it from the Web UI. In fact, if you are able to copy the log files locally, you can just view them directly in any text editor. I suspect what you really want to see is the application history. Here is the relevant information from Spark’s monitoring page (http://spark.apache.org/docs/latest/monitoring.html) To view the web UI after the fact, set spark.eventLog.enabled to true before starting the application. This configures Spark to log Spark events that encode the information displayed in the UI to persisted storage. Mohammed Author: Big Data Analytics with Spark<http://www.amazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/> From: Philip Lee [mailto:philjj...@gmail.com<mailto:philjj...@gmail.com>] Sent: Monday, January 25, 2016 9:51 AM To: user@spark.apache.org<mailto:user@spark.apache.org> Subject: Re: a question about web ui log As I mentioned before, I am tryint to see the spark log on a cluster via ssh-tunnel 1) The error on application details UI is probably from monitoring porting 4044. Web UI port is 8088, right? so how could I see job web ui view and application details UI view in the web ui on my local machine? 2) still wondering how to see the log after copyting log file to my local. The error was metioned in previous mail. Thanks, Phil On Mon, Jan 25, 2016 at 5:36 PM, Philip Lee mailto:philjj...@gmail.com>> wrote: Hello, a questino about web UI log. I could see web interface log after forwarding the port on my cluster to my local and click completed application, but when I clicked "application detail UI" [Inline image 1] It happened to me. I do not know why. I also checked the specific log folder. It has a log file in it. Actually, that's why I could click the completed application link, right? So is it okay for me to copy the log file in my cluster to my local machine. And after turning on spark Job Manger on my local by myself, I could see application deatils UI in my local machine? Best, Phil
Re: a question about web ui log
Yes, I tried it, but it simply does not work. so, my concern is *to use "ssh tunnel" to forward a port of cluster to localhost port. * But in Spark UI, there are two ports which I should forward using "*ssh tunnel*". Considering a default port, 8080 is web-ui port to come into web-ui, and 4040 is web-monitoring port to see time execution like DAG in application details UI, as you probably know. But after finishing a job, I can see the list of a job on the web-ui on 8080, but when I click "application details UI" on port 4040 to see time excution, it does not work. Any suggestion? I really need to see execution of DAG. Best, Phil On Tue, Jan 26, 2016 at 12:04 AM, Mohammed Guller wrote: > I am not sure whether you can copy the log files from Spark workers to > your local machine and view it from the Web UI. In fact, if you are able to > copy the log files locally, you can just view them directly in any text > editor. > > > > I suspect what you really want to see is the application history. Here is > the relevant information from Spark’s monitoring page ( > http://spark.apache.org/docs/latest/monitoring.html) > > > > To view the web UI after the fact, set spark.eventLog.enabled to true > before starting the application. This configures Spark to log Spark events > that encode the information displayed in the UI to persisted storage. > > > > Mohammed > > Author: Big Data Analytics with Spark > <http://www.amazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/> > > > > *From:* Philip Lee [mailto:philjj...@gmail.com] > *Sent:* Monday, January 25, 2016 9:51 AM > *To:* user@spark.apache.org > *Subject:* Re: a question about web ui log > > > > As I mentioned before, I am tryint to see the spark log on a cluster via > ssh-tunnel > > > > 1) The error on application details UI is probably from monitoring porting > 4044. Web UI port is 8088, right? so how could I see job web ui view and > application details UI view in the web ui on my local machine? > > > > 2) still wondering how to see the log after copyting log file to my local. > > > > The error was metioned in previous mail. > > > > Thanks, > > Phil > > > > > > > > On Mon, Jan 25, 2016 at 5:36 PM, Philip Lee wrote: > > Hello, a questino about web UI log. > > > > I could see web interface log after forwarding the port on my cluster to > my local and click completed application, but when I clicked "application > detail UI" > > > > [image: Inline image 1] > > > > It happened to me. I do not know why. I also checked the specific log > folder. It has a log file in it. Actually, that's why I could click the > completed application link, right? > > > > So is it okay for me to copy the log file in my cluster to my local > machine. > > And after turning on spark Job Manger on my local by myself, I could see > application deatils UI in my local machine? > > > > Best, > > Phil > > >
RE: a question about web ui log
I am not sure whether you can copy the log files from Spark workers to your local machine and view it from the Web UI. In fact, if you are able to copy the log files locally, you can just view them directly in any text editor. I suspect what you really want to see is the application history. Here is the relevant information from Spark’s monitoring page (http://spark.apache.org/docs/latest/monitoring.html) To view the web UI after the fact, set spark.eventLog.enabled to true before starting the application. This configures Spark to log Spark events that encode the information displayed in the UI to persisted storage. Mohammed Author: Big Data Analytics with Spark<http://www.amazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/> From: Philip Lee [mailto:philjj...@gmail.com] Sent: Monday, January 25, 2016 9:51 AM To: user@spark.apache.org Subject: Re: a question about web ui log As I mentioned before, I am tryint to see the spark log on a cluster via ssh-tunnel 1) The error on application details UI is probably from monitoring porting 4044. Web UI port is 8088, right? so how could I see job web ui view and application details UI view in the web ui on my local machine? 2) still wondering how to see the log after copyting log file to my local. The error was metioned in previous mail. Thanks, Phil On Mon, Jan 25, 2016 at 5:36 PM, Philip Lee mailto:philjj...@gmail.com>> wrote: Hello, a questino about web UI log. I could see web interface log after forwarding the port on my cluster to my local and click completed application, but when I clicked "application detail UI" [Inline image 1] It happened to me. I do not know why. I also checked the specific log folder. It has a log file in it. Actually, that's why I could click the completed application link, right? So is it okay for me to copy the log file in my cluster to my local machine. And after turning on spark Job Manger on my local by myself, I could see application deatils UI in my local machine? Best, Phil
Re: a question about web ui log
As I mentioned before, I am tryint to see the spark log on a cluster via ssh-tunnel 1) The error on application details UI is probably from monitoring porting 4044. Web UI port is 8088, right? so how could I see job web ui view and application details UI view in the web ui on my local machine? 2) still wondering how to see the log after copyting log file to my local. The error was metioned in previous mail. Thanks, Phil On Mon, Jan 25, 2016 at 5:36 PM, Philip Lee wrote: > Hello, a questino about web UI log. > > I could see web interface log after forwarding the port on my cluster to > my local and click completed application, but when I clicked "application > detail UI" > > [image: Inline image 1] > > It happened to me. I do not know why. I also checked the specific log > folder. It has a log file in it. Actually, that's why I could click the > completed application link, right? > > So is it okay for me to copy the log file in my cluster to my local > machine. > And after turning on spark Job Manger on my local by myself, I could see > application deatils UI in my local machine? > > Best, > Phil >
a question about web ui log
Hello, a questino about web UI log. I could see web interface log after forwarding the port on my cluster to my local and click completed application, but when I clicked "application detail UI" [image: Inline image 1] It happened to me. I do not know why. I also checked the specific log folder. It has a log file in it. Actually, that's why I could click the completed application link, right? So is it okay for me to copy the log file in my cluster to my local machine. And after turning on spark Job Manger on my local by myself, I could see application deatils UI in my local machine? Best, Phil
