[jira] [Commented] (SPARK-5925) YARN - Spark progress bar stucks at 10% but after finishing shows 100%
[ https://issues.apache.org/jira/browse/SPARK-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15584969#comment-15584969 ] Steve Loughran commented on SPARK-5925: --- looking at this, I'm confused about what I'd written myself. I concur with Marco, with one little point: if this is the YARN UI, then somewhere under YARN-896 there was discussion on moving the view on from "percentage done" —valid for simple batch runs— to, for long lived services, "percentage in use". That is, if the Spark AM knows its desired cluster size, it can tell the YARN RM that it's at 100% capacity; if below that it could set a different value. That is, the field will move from "percentage done" to "percentage that cluster size matches desired". There's no change in the YARN API needed here; the main changes are (a) YARN RM to display allocation value of long-lived services different from short-lived executions (different text, colour), and for the Spark AM to update the RM with the current percentage value, changing it as containers come and go. > YARN - Spark progress bar stucks at 10% but after finishing shows 100% > -- > > Key: SPARK-5925 > URL: https://issues.apache.org/jira/browse/SPARK-5925 > Project: Spark > Issue Type: Improvement > Components: YARN >Affects Versions: 1.2.1 >Reporter: Laszlo Fesus >Priority: Minor > > I did set up a yarn cluster (CDH5) and spark (1.2.1), and also started Spark > History Server. Now I am able to click on more details on yarn's web > interface and get redirected to the appropriate spark logs during both job > execution and also after the job has finished. > My only concern is that while a spark job is being executed (either > yarn-client or yarn-cluster), the progress bar stucks at 10% and doesn't > increase as for MapReduce jobs. After finishing, it shows 100% properly, but > we are loosing the real-time tracking capability of the status bar. > Also tested yarn restful web interface, and it retrieves again 10% during > (yarn) spark job execution, and works well again after finishing. (I suppose > for the while being I should have a look on Spark Job Server and see if it's > possible to track the job via its restful web interface.) > Did anyone else experience this behaviour? Thanks in advance. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5925) YARN - Spark progress bar stucks at 10% but after finishing shows 100%
[ https://issues.apache.org/jira/browse/SPARK-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144460#comment-15144460 ] Steve Loughran commented on SPARK-5925: --- I think this should be fixed by SPARK-7889; linking to it. > YARN - Spark progress bar stucks at 10% but after finishing shows 100% > -- > > Key: SPARK-5925 > URL: https://issues.apache.org/jira/browse/SPARK-5925 > Project: Spark > Issue Type: Improvement > Components: YARN >Affects Versions: 1.2.1 >Reporter: Laszlo Fesus >Priority: Minor > > I did set up a yarn cluster (CDH5) and spark (1.2.1), and also started Spark > History Server. Now I am able to click on more details on yarn's web > interface and get redirected to the appropriate spark logs during both job > execution and also after the job has finished. > My only concern is that while a spark job is being executed (either > yarn-client or yarn-cluster), the progress bar stucks at 10% and doesn't > increase as for MapReduce jobs. After finishing, it shows 100% properly, but > we are loosing the real-time tracking capability of the status bar. > Also tested yarn restful web interface, and it retrieves again 10% during > (yarn) spark job execution, and works well again after finishing. (I suppose > for the while being I should have a look on Spark Job Server and see if it's > possible to track the job via its restful web interface.) > Did anyone else experience this behaviour? Thanks in advance. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5925) YARN - Spark progress bar stucks at 10% but after finishing shows 100%
[ https://issues.apache.org/jira/browse/SPARK-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14941112#comment-14941112 ] Steve Loughran commented on SPARK-5925: --- I've actually thought about how to do this as part of SPARK-1537, but was waiting to get the core patch done first. What needs to be done is *relatively* straightforward: for every incomplete app loaded into the history server, something needs to GETs updated events from the history server and forwards them to the app UI. That's the core concept; implementation details need to consider * completion of app * removal of app from cache & need to ensure no leaks by retaining app ui links elsewhere * the fact that ATS doesn't do blocking reads, so polling on a schedule is required * the need to keep load on the history server down so as to avoid overloading it on a large cluster. > YARN - Spark progress bar stucks at 10% but after finishing shows 100% > -- > > Key: SPARK-5925 > URL: https://issues.apache.org/jira/browse/SPARK-5925 > Project: Spark > Issue Type: Improvement > Components: YARN >Affects Versions: 1.2.1 >Reporter: Laszlo Fesus >Priority: Minor > > I did set up a yarn cluster (CDH5) and spark (1.2.1), and also started Spark > History Server. Now I am able to click on more details on yarn's web > interface and get redirected to the appropriate spark logs during both job > execution and also after the job has finished. > My only concern is that while a spark job is being executed (either > yarn-client or yarn-cluster), the progress bar stucks at 10% and doesn't > increase as for MapReduce jobs. After finishing, it shows 100% properly, but > we are loosing the real-time tracking capability of the status bar. > Also tested yarn restful web interface, and it retrieves again 10% during > (yarn) spark job execution, and works well again after finishing. (I suppose > for the while being I should have a look on Spark Job Server and see if it's > possible to track the job via its restful web interface.) > Did anyone else experience this behaviour? Thanks in advance. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5925) YARN - Spark progress bar stucks at 10% but after finishing shows 100%
[ https://issues.apache.org/jira/browse/SPARK-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649688#comment-14649688 ] Kevin Cox commented on SPARK-5925: -- Could a simple `stage / number of stages` be a good enough estimate? It is nice to just get a rough idea. YARN - Spark progress bar stucks at 10% but after finishing shows 100% -- Key: SPARK-5925 URL: https://issues.apache.org/jira/browse/SPARK-5925 Project: Spark Issue Type: Improvement Components: YARN Affects Versions: 1.2.1 Reporter: Laszlo Fesus Priority: Minor I did set up a yarn cluster (CDH5) and spark (1.2.1), and also started Spark History Server. Now I am able to click on more details on yarn's web interface and get redirected to the appropriate spark logs during both job execution and also after the job has finished. My only concern is that while a spark job is being executed (either yarn-client or yarn-cluster), the progress bar stucks at 10% and doesn't increase as for MapReduce jobs. After finishing, it shows 100% properly, but we are loosing the real-time tracking capability of the status bar. Also tested yarn restful web interface, and it retrieves again 10% during (yarn) spark job execution, and works well again after finishing. (I suppose for the while being I should have a look on Spark Job Server and see if it's possible to track the job via its restful web interface.) Did anyone else experience this behaviour? Thanks in advance. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5925) YARN - Spark progress bar stucks at 10% but after finishing shows 100%
[ https://issues.apache.org/jira/browse/SPARK-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329303#comment-14329303 ] Laszlo Fesus commented on SPARK-5925: - Yes, but I thought it would be quite useful if the progress bar could be forwarded from the more detailed view (spark web interface) to the master yarn interface. That could fix the problem. Maybe this feature could be implemented also for the _spark.yarn.historyServer.address_ functionality, which actually does redirect us to the proper job details on the spark web interface. (And would be even better, if we could retrieve this _updated_ progress bar also for the yarn restful interface) YARN - Spark progress bar stucks at 10% but after finishing shows 100% -- Key: SPARK-5925 URL: https://issues.apache.org/jira/browse/SPARK-5925 Project: Spark Issue Type: Improvement Components: YARN Affects Versions: 1.2.1 Reporter: Laszlo Fesus Priority: Minor I did set up a yarn cluster (CDH5) and spark (1.2.1), and also started Spark History Server. Now I am able to click on more details on yarn's web interface and get redirected to the appropriate spark logs during both job execution and also after the job has finished. My only concern is that while a spark job is being executed (either yarn-client or yarn-cluster), the progress bar stucks at 10% and doesn't increase as for MapReduce jobs. After finishing, it shows 100% properly, but we are loosing the real-time tracking capability of the status bar. Also tested yarn restful web interface, and it retrieves again 10% during (yarn) spark job execution, and works well again after finishing. (I suppose for the while being I should have a look on Spark Job Server and see if it's possible to track the job via its restful web interface.) Did anyone else experience this behaviour? Thanks in advance. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
