[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-19 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-70458587
  
  [Test build #25748 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25748/consoleFull)
 for   PR 4043 at commit 
[`3b11803`](https://github.com/apache/spark/commit/3b11803ae9b64acba2d64ad02d1e31d756783eaf).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-70458595
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25748/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-19 Thread asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/4043


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-18 Thread ScrapCodes
Github user ScrapCodes commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-70453703
  
@pwendell - patch updated to latest master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-18 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-70453629
  
  [Test build #25748 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25748/consoleFull)
 for   PR 4043 at commit 
[`3b11803`](https://github.com/apache/spark/commit/3b11803ae9b64acba2d64ad02d1e31d756783eaf).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-17 Thread pwendell
Github user pwendell commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-70395195
  
@ScrapCodes mind bringing up to date? The current form LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-16 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-70220093
  
  [Test build #25644 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25644/consoleFull)
 for   PR 4043 at commit 
[`c7b3332`](https://github.com/apache/spark/commit/c7b33329eda2093adc415dacc7941f9ca6d4fbe8).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-16 Thread pwendell
Github user pwendell commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-70220172
  
@ScrapCodes Rather than change this to +LinkedHashMap+ can you just check 
if it contains it before removing it? It might not be obvious to developers 
that +remove+ has that specific behavior. I think it's better to just be 
explicit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-16 Thread ScrapCodes
Github user ScrapCodes commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-70220772
  
LinkedHashMap was introduced to maintain the insertion order of 
stagesIds(Hasmap will lead to arbitrary order), all map(s) in scala has return 
an `Option` on `remove` behavior. Even in java calling remove on inexistent key 
returns null. Do you still want me to change it? Also may be changing the name 
pendingStages to pendingStagesMap will avoid that confusion ? and should the 
same apply to - already existing field activeStages ? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-16 Thread pwendell
Github user pwendell commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-70220952
  
ah I see - if the existing remove call is safe, then I think it's fine.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-16 Thread ScrapCodes
Github user ScrapCodes commented on a diff in the pull request:

https://github.com/apache/spark/pull/4043#discussion_r23068344
  
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllStagesPage.scala 
---
@@ -37,12 +37,18 @@ private[ui] class AllStagesPage(parent: StagesTab) 
extends WebUIPage() {
   val numCompletedStages = listener.numCompletedStages
   val failedStages = listener.failedStages.reverse.toSeq
   val numFailedStages = listener.numFailedStages
+  val pendingStages = listener.pendingStages.values.toSeq
+  val numWaitingStages = pendingStages.size
   val now = System.currentTimeMillis
 
   val activeStagesTable =
 new StageTableBase(activeStages.sortBy(_.submissionTime).reverse,
   parent.basePath, parent.listener, isFairScheduler = 
parent.isFairScheduler,
   killEnabled = parent.killEnabled)
+  val pendingStagesTable =
+new StageTableBase(pendingStages.sortBy(_.submissionTime),
--- End diff --

@pwendell Can I use Sorting.stableSort here ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-16 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-70227033
  
  [Test build #25644 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25644/consoleFull)
 for   PR 4043 at commit 
[`c7b3332`](https://github.com/apache/spark/commit/c7b33329eda2093adc415dacc7941f9ca6d4fbe8).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-70227040
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25644/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-16 Thread pwendell
Github user pwendell commented on a diff in the pull request:

https://github.com/apache/spark/pull/4043#discussion_r23070055
  
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllStagesPage.scala 
---
@@ -37,12 +37,18 @@ private[ui] class AllStagesPage(parent: StagesTab) 
extends WebUIPage() {
   val numCompletedStages = listener.numCompletedStages
   val failedStages = listener.failedStages.reverse.toSeq
   val numFailedStages = listener.numFailedStages
+  val pendingStages = listener.pendingStages.values.toSeq
+  val numWaitingStages = pendingStages.size
   val now = System.currentTimeMillis
 
   val activeStagesTable =
 new StageTableBase(activeStages.sortBy(_.submissionTime).reverse,
   parent.basePath, parent.listener, isFairScheduler = 
parent.isFairScheduler,
   killEnabled = parent.killEnabled)
+  val pendingStagesTable =
+new StageTableBase(pendingStages.sortBy(_.submissionTime),
--- End diff --

Why not just keep the sorting the same as it was? I think the submission 
time is unlikely to be tied in most cases. It would be good to just make it 
consistent with the existing ones.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-16 Thread pwendell
Github user pwendell commented on a diff in the pull request:

https://github.com/apache/spark/pull/4043#discussion_r23070066
  
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllStagesPage.scala 
---
@@ -37,12 +37,18 @@ private[ui] class AllStagesPage(parent: StagesTab) 
extends WebUIPage() {
   val numCompletedStages = listener.numCompletedStages
   val failedStages = listener.failedStages.reverse.toSeq
   val numFailedStages = listener.numFailedStages
+  val pendingStages = listener.pendingStages.values.toSeq
--- End diff --

can you move this up with `activeStages` to make the declarations grouped 
properly?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-16 Thread pwendell
Github user pwendell commented on a diff in the pull request:

https://github.com/apache/spark/pull/4043#discussion_r23070082
  
--- Diff: 
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -17,7 +17,7 @@
 
 package org.apache.spark.ui.jobs
 
-import scala.collection.mutable.{HashMap, HashSet, ListBuffer}
+import scala.collection.mutable.{HashMap, HashSet, ListBuffer, 
LinkedHashMap}
--- End diff --

if it's safe to do `remove` on a HashMap, I think it's fine to revert this 
back to using a HashMap. It's more consistent in that case.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-16 Thread ScrapCodes
Github user ScrapCodes commented on a diff in the pull request:

https://github.com/apache/spark/pull/4043#discussion_r23074849
  
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllStagesPage.scala 
---
@@ -37,12 +37,18 @@ private[ui] class AllStagesPage(parent: StagesTab) 
extends WebUIPage() {
   val numCompletedStages = listener.numCompletedStages
   val failedStages = listener.failedStages.reverse.toSeq
   val numFailedStages = listener.numFailedStages
+  val pendingStages = listener.pendingStages.values.toSeq
+  val numWaitingStages = pendingStages.size
   val now = System.currentTimeMillis
 
   val activeStagesTable =
 new StageTableBase(activeStages.sortBy(_.submissionTime).reverse,
   parent.basePath, parent.listener, isFairScheduler = 
parent.isFairScheduler,
   killEnabled = parent.killEnabled)
+  val pendingStagesTable =
+new StageTableBase(pendingStages.sortBy(_.submissionTime),
--- End diff --

I was thinking about usability, like the most interesting stage which is 
next to be executed will appear at last. Anyway will change it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-16 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-70240729
  
  [Test build #25656 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25656/consoleFull)
 for   PR 4043 at commit 
[`15cdda4`](https://github.com/apache/spark/commit/15cdda44960f94e8218e6d922eb8c61fdfbb).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-16 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-70247730
  
  [Test build #25656 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25656/consoleFull)
 for   PR 4043 at commit 
[`15cdda4`](https://github.com/apache/spark/commit/15cdda44960f94e8218e6d922eb8c61fdfbb).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-70247735
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25656/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-15 Thread ScrapCodes
Github user ScrapCodes commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-70212456
  
Hi Imran, Thanks for taking a look. @pwendell please take a look ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-15 Thread pwendell
Github user pwendell commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-70214406
  
I'm not sure this can be merged as-is. The state clean-up here is based on 
the assumption that every stage that is pending will at some later time be 
submitted. Is that definitely true? What happens if a stage is aborted or 
failed, won't its dependent stages remain pending indefinitely for the entire 
history of the application? I think at a minimum you need to make sure you 
remove all associated stages with a given job if the job ends. There may also 
be other corner cases I'm not thinking of.

A second assumption this makes is that the job start event will always 
occur before the stage is submitted. Is that definitely true? It would be good 
to dig through the reporting API and make sure that is a safe assumption. I 
think the guarantees of the listener around event ordering are pretty minimal.

Finally, what about putting pending stages after active stages in the 
display page? My concern is you may have dozens or more pending stages in 
production jobs, I think people will be frustrated if they open the UI and they 
have to scroll way down every time they need to e.g. refresh the page. It's a 
big odd to go (active - pending - completed), but I think better for 
usability.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-15 Thread ScrapCodes
Github user ScrapCodes commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-70216469
  
Thanks Patrick !
Have two questions inline.

 I'm not sure this can be merged as-is. The state clean-up here is based 
on the assumption that every stage that is pending will at some later time be 
submitted. Is that definitely true? What happens if a stage is aborted or 
failed, won't its dependent stages remain pending indefinitely for the entire 
history of the application? I think at a minimum you need to make sure you 
remove all associated stages with a given job if the job ends. There may also 
be other corner cases I'm not thinking of.

If I am understanding this correctly, All stages for a particular job Id 
on jobEnd event - should be cleaned up. I am certainly missing something, is 
this not already achieved by 
[this](https://github.com/apache/spark/pull/4043/files#diff-1f32bcb61f51133bd0959a4177a066a5R191)
 ?

A second assumption this makes is that the job start event will always 
occur before the stage is submitted. Is that definitely true? It would be good 
to dig through the reporting API and make sure that is a safe assumption. I 
think the guarantees of the listener around event ordering are pretty minimal.

This can lead to an anomaly that a stage will appear in progress in both 
active stages section and pending section. But it will still be obvious that - 
that stage is in progress. Is this safe to ignore, or is it a  concern that 
should be addressed at the cost of minute complexity ?

Finally, what about putting pending stages after active stages in the 
display page? My concern is you may have dozens or more pending stages in 
production jobs, I think people will be frustrated if they open the UI and they 
have to scroll way down every time they need to e.g. refresh the page. It's a 
big odd to go (active - pending - completed), but I think better for 
usability.

This is thoughtful, I am going to incorporate this asap. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-15 Thread pwendell
Github user pwendell commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-70219039
  
Ah I see - so on the first point, the issue may be covered by the code you 
referenced. For some reason the diff originally rendered in a way where I 
didn't notice that.

For the second issue. My concern is that this line could throw an exception:

https://github.com/apache/spark/pull/4043/files#diff-1f32bcb61f51133bd0959a4177a066a5R263

I.e. if you had a stage submitted, but you never received the job started 
event.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-15 Thread ScrapCodes
Github user ScrapCodes commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-70219564
  
Good point, in map in scala remove returns an Option and does not throw 
exception. However in lists what you said holds. See 
https://github.com/scala/scala/blob/2.11.x/src/library/scala/collection/mutable/LinkedHashMap.scala#L76



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-14 Thread squito
Github user squito commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-70015231
  
lgtm.

I was going to suggest that pending stages should be sorted with oldest 
submission time first, not reversed ... but I guess we want the completed 
stages sorted with oldest last, and probably makes sense to keep those tables 
consistent with each other.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-14 Thread ScrapCodes
GitHub user ScrapCodes opened a pull request:

https://github.com/apache/spark/pull/4043

SPARK-5217 Spark UI should report pending stages during job execution on 
AllStagesPage.


![pending_stages](https://cloud.githubusercontent.com/assets/992952/5738019/70ee913e-9c0d-11e4-9970-bfd89cae7a65.png)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ScrapCodes/spark-1 
SPARK-5216/5217-show-waiting-stages

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/4043.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4043


commit d335f8fa8fe9740281f6ce05af292ca108485bfb
Author: Prashant Sharma prashan...@imaginea.com
Date:   2015-01-14T11:18:31Z

SPARK-5217 Spark UI should report pending stages during job execution on 
AllStagesPage.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-14 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-69902838
  
  [Test build #25544 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25544/consoleFull)
 for   PR 4043 at commit 
[`d335f8f`](https://github.com/apache/spark/commit/d335f8fa8fe9740281f6ce05af292ca108485bfb).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-14 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-69904323
  
  [Test build #25545 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25545/consoleFull)
 for   PR 4043 at commit 
[`f6dc584`](https://github.com/apache/spark/commit/f6dc584e65a36c281677cf39a3e7bb369cc9f80b).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-69909651
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25544/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-14 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-69909642
  
  [Test build #25544 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25544/consoleFull)
 for   PR 4043 at commit 
[`d335f8f`](https://github.com/apache/spark/commit/d335f8fa8fe9740281f6ce05af292ca108485bfb).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-14 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-69911073
  
  [Test build #25545 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25545/consoleFull)
 for   PR 4043 at commit 
[`f6dc584`](https://github.com/apache/spark/commit/f6dc584e65a36c281677cf39a3e7bb369cc9f80b).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: SPARK-5217 Spark UI should report pending stag...

2015-01-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4043#issuecomment-69911085
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25545/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org