Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1940#issuecomment-55364614
@andrewor14 I think you're right that there's a deeper problem here. I
haven't tested this but here's what I think is going on:
(1
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2342#issuecomment-3741
@sarutak yes it represents task ID. The pie chart sounds cool!
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1940#issuecomment-55696539
This seems like it could be ok -- my only concern is about the semantics of
when we tell the user we've killed their job. Currently I think we invoke
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2035#discussion_r17710076
--- Diff: core/pom.xml ---
@@ -42,6 +42,10 @@
groupIdjavax.servlet/groupId
artifactIdservlet-api/artifactId
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2440#issuecomment-56116856
Removing this sounds good to me!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2087#issuecomment-56307027
@aarondav @sryza Did you consider using reader.getPos() to get the correct
metrics for older versions of Hadoop (as in here:
https://github.com/kayousterhout/spark
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2226#issuecomment-57243214
I've merged this into master. Sorry for the delay -- unicode characters in
the commit author names were causing our merge script to crash!
---
If your project
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2599#issuecomment-57423834
As discussed offline, this looks good if you add a comment about why
NonFatal.
---
If your project is set up for it, you can reply to this email and have your
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57746732
Comment out the print statement in merge_pr that causes the failure.
On Thu, Oct 2, 2014 at 7:58 PM, Michael Armbrust notificati...@github.com
wrote
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1462#issuecomment-49661602
@tgravescs I actually mentioned this race condition in the previous PR:
https://github.com/apache/spark/pull/900#diff-for-comment-14205738 . In the
future we
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1462#issuecomment-49669448
No worries I should have just made a top-level comment -- the code-level
comments are easy to miss once they get compressed because the code is out
of date
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1507#discussion_r15238649
--- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala
---
@@ -90,19 +94,18 @@ class TaskMetrics extends Serializable
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1507#discussion_r15238692
--- Diff:
core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleReader.scala ---
@@ -20,6 +20,8 @@ package org.apache.spark.shuffle.hash
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1507#discussion_r15238657
--- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala
---
@@ -19,6 +19,7 @@ package org.apache.spark.executor
import
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1507#discussion_r15238878
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala ---
@@ -31,6 +31,7 @@ import
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1507#discussion_r15238860
--- Diff:
core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleReader.scala ---
@@ -35,8 +37,15 @@ class HashShuffleReader[K, C
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1507#discussion_r15238942
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala ---
@@ -131,7 +122,9 @@ object BlockFetcherIterator
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1507#discussion_r15242942
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala ---
@@ -191,7 +183,7 @@ object BlockFetcherIterator
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1507#discussion_r15244576
--- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala
---
@@ -75,9 +76,12 @@ class TaskMetrics extends Serializable
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1507#issuecomment-49776893
At a high level, this depends on one of your other patches (#1056?) to
incrementally send updates right? Is the idea that mergeShuffleReadMetrics
will get called
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1507#discussion_r15247072
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala ---
@@ -131,7 +122,9 @@ object BlockFetcherIterator
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1507#discussion_r15250406
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala ---
@@ -154,14 +147,15 @@ object BlockFetcherIterator
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1507#discussion_r15250426
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -29,7 +29,7 @@ import akka.actor.{ActorSystem, Cancellable, Props
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1507#discussion_r15250466
--- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala
---
@@ -73,11 +75,16 @@ class TaskMetrics extends Serializable {
var
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1507#issuecomment-49789216
Thanks Sandy!! Just a few more small things.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1525#discussion_r15308341
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
---
@@ -36,6 +36,7 @@ private[spark] class
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1525#issuecomment-49921298
I find it a bit confusing that totalRegisteredResources can refer to
cores (in standalone mode) or executors (in Yarn mode). Can we just use
different variables
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1545#discussion_r15317700
--- Diff:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -43,13 +43,16 @@ class JobProgressListener(conf: SparkConf
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1545#discussion_r15317731
--- Diff:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -43,13 +43,16 @@ class JobProgressListener(conf: SparkConf
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1545#discussion_r15318181
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -35,7 +35,8 @@ private[ui] class StagePage(parent: JobProgressTab
GitHub user kayousterhout opened a pull request:
https://github.com/apache/spark/pull/1566
[SPARK-1726] [SPARK-2567] Eliminate zombie stages in UI.
Due to problems with when we update runningStages (in DAGScheduler.scala)
and how we decide to send a SparkListenerStageCompleted
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1566#discussion_r15330087
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -710,7 +710,6 @@ class DAGScheduler(
if (missing == Nil
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1566#discussion_r15330105
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -753,11 +752,14 @@ class DAGScheduler(
null
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1516#issuecomment-49968180
I looked into this a bit more and this change doesn't quite fix the problem
in the right way, partially because of what @markhamstra pointed out (that we
shouldn't
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1566#issuecomment-49968323
Thanks for the quick review @markhamstra !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1561#issuecomment-49970120
Total nit but would you mind adding to the description (so it ends up in
the commit) the point of this change (which, I think, is as part of a scheduler
cleanup
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1561#discussion_r15330894
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -223,7 +217,7 @@ class DAGScheduler(
new Stage(id, rdd
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1561#discussion_r15331043
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -355,14 +351,13 @@ class DAGScheduler
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1561#discussion_r15331172
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -395,16 +389,9 @@ class DAGScheduler(
activeJobs -= job
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1561#discussion_r15331280
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -395,16 +389,9 @@ class DAGScheduler(
activeJobs -= job
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1561#discussion_r15331310
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -355,14 +351,13 @@ class DAGScheduler
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1561#discussion_r15331465
--- Diff: core/src/main/scala/org/apache/spark/scheduler/Stage.scala ---
@@ -22,6 +22,8 @@ import org.apache.spark.rdd.RDD
import
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1561#discussion_r15331488
--- Diff: core/src/main/scala/org/apache/spark/scheduler/Stage.scala ---
@@ -56,6 +58,16 @@ private[spark] class Stage(
val numPartitions
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1561#discussion_r15331519
--- Diff: core/src/main/scala/org/apache/spark/scheduler/Stage.scala ---
@@ -56,6 +58,16 @@ private[spark] class Stage(
val numPartitions
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1561#issuecomment-49971683
Love the cleanup here!!!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1541#issuecomment-50073250
When is this accessed concurrently? I looked quickly and can only find
updates from the (single-threaded) DAGScheduler event loop. Is the issue that
it can
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1572#issuecomment-50074102
My understanding is that it can help to let the remaining tasks run --
because they may hit Fetch failures from different map outputs than the
original fetch
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1566#issuecomment-50246187
Yeah that seems fine to me -- thanks Matei!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1498#discussion_r15497921
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -17,7 +17,7 @@
package org.apache.spark.scheduler
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1498#discussion_r15498341
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -691,25 +689,41 @@ class DAGScheduler
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1498#discussion_r15498524
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -691,25 +689,41 @@ class DAGScheduler
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1498#discussion_r15498569
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -691,25 +689,41 @@ class DAGScheduler
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1498#discussion_r15498909
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -17,7 +17,7 @@
package org.apache.spark.scheduler
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1498#discussion_r15499281
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -691,25 +689,41 @@ class DAGScheduler
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1525#issuecomment-50855070
I will take a look at this tomorrow.
On Thu, Jul 31, 2014 at 10:37 PM, Zhihui Li notificati...@github.com
wrote:
@tgravescs https
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1525#discussion_r15725825
--- Diff:
yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterSchedulerBackend.scala
---
@@ -40,6 +41,10 @@ private[spark] class
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1525#discussion_r15725840
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -47,19 +47,19 @@ class
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1525#issuecomment-50954397
My preference is also to remove this for standalone mode (as mentioned in
the original PR, #900) -- but adding @tgravescs who looked quite a bit at the
original PR
GitHub user kayousterhout opened a pull request:
https://github.com/apache/spark/pull/1762
Remove support for waiting for executors in standalone mode.
Current code waits until some minimum fraction of expected executors
have registered before beginning scheduling. The current
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1462#issuecomment-51024346
@pwendell I created https://github.com/apache/spark/pull/1762 for your
judgment of what the right thing to do here is!
---
If your project is set up for it, you
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1762#discussion_r15742607
--- Diff:
yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
---
@@ -0,0 +1,63 @@
+/*
+ * Licensed
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1481#discussion_r15742765
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -29,7 +29,7 @@ import akka.actor.{ActorSystem, Cancellable, Props
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1309#discussion_r15769145
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala ---
@@ -42,6 +44,13 @@ class TaskInfo(
var gettingResultTime: Long = 0
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1481#issuecomment-51389141
wooh!
On Wed, Aug 6, 2014 at 1:04 PM, Apache Spark QA notificati...@github.com
wrote:
QA results for PR 1481:
- This patch
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1507#discussion_r15905777
--- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala
---
@@ -73,11 +75,16 @@ class TaskMetrics extends Serializable {
var
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1525#issuecomment-51531440
I think the verdict here is to leave this feature in, and this patch looks
good to me if it looks good to you, Tom.
On Thu, Aug 7, 2014 at 1:54 PM
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1869#issuecomment-51710728
Looks great!! +1 on this being useful.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18920574
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18920718
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18920780
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18920907
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18920962
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18922212
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -101,7 +101,7 @@ private[spark] class TaskSetManager
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18922282
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -110,19 +110,26 @@ private[spark] class TaskSetManager
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18922413
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18922597
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -110,19 +110,26 @@ private[spark] class TaskSetManager
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18923728
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18924277
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -110,19 +110,26 @@ private[spark] class TaskSetManager
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59280725
@andrewor14 this looks cool! Two high level comments:
(1) Right now, there are a bunch of different kinds of timers that can be
set and then triggered
GitHub user kayousterhout opened a pull request:
https://github.com/apache/spark/pull/2832
[SPARK-3984] [SPARK-3983] Improve UI task metrics.
This commit fixes the scheduler delay in the UI (which previously
included things that are not scheduler delay, like time
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2832#issuecomment-59851411
I'm holding off on this until I finish
https://issues.apache.org/jira/browse/SPARK-4016 due to the concern that
otherwise these new metrics will add confusion
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59854068
@sryza What's the policy used by MR/Tez? When talking to Patrick/Andrew
offline, I'd argued for allocating executors such that the total executors
given
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59857927
I see that's a good point @andrewor14
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
GitHub user kayousterhout opened a pull request:
https://github.com/apache/spark/pull/2867
[SPARK-4016] Allow user to show/hide UI metrics.
This commit adds a set of checkboxes to the stage detail
page that the user can use to show additional task metrics,
including the GC
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2852#discussion_r19165361
--- Diff: core/src/main/scala/org/apache/spark/ui/UITables.scala ---
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2852#discussion_r19165764
--- Diff: core/src/main/scala/org/apache/spark/ui/UITables.scala ---
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2852#discussion_r19165811
--- Diff: core/src/main/scala/org/apache/spark/ui/UITables.scala ---
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2852#issuecomment-59969443
This looks awesome! Will it be hard to add classes to rows / columns (as is
needed in #2867, for example) with this? That's one thing that was pretty
cumbersome
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2865#issuecomment-59973580
@andrewor14 this looks good, and definitely seems to represent the
expected use of activeExecutorIds.
---
If your project is set up for it, you can reply
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2867#issuecomment-60011996
Haha @andrewor14 that's exactly how I had it originally but @pwendell
wanted it to look like this. I see arguments for both versions so happy with
whatever
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2867#issuecomment-60041059
@ash211 that's a great idea, I'll add that and incorporate @andrewor14's
suggestion about a drop-down menu. Thanks all!
---
If your project is set up for it, you
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2867#issuecomment-60155916
Ok I updated this, as well as the photos in the description at the top. As
per @andrewor14's suggestion, there's now an expandable menu of options. Based
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2867#issuecomment-60296814
@andrewor14 I can add tooltips to each metric name, or just list it in line
(e.g., Scheduler Delay: A metric that explains ). Which do you think is
better
Github user kayousterhout closed the pull request at:
https://github.com/apache/spark/pull/2867
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2867#issuecomment-60301451
@aarondav I wonder if I should change scheduler delay to task launch
overhead or similar. It's (time when schedule receives task completion - time
from when
GitHub user kayousterhout reopened a pull request:
https://github.com/apache/spark/pull/2867
[SPARK-4016] Allow user to show/hide UI metrics.
This commit adds a set of checkboxes to the stage detail
page that the user can use to show additional task metrics,
including the GC
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2867#issuecomment-60443288
I can add the tooltips in an hour or so -- was waiting to modify this to
hear if there were more review comments, but it sounds like there are not!
Will post
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2867#issuecomment-60653883
@JoshRosen @andrewor14 does one of you have time to look at this today? I
added the tool tips to each additional metric. Would be great to get this in
so that I
GitHub user kayousterhout opened a pull request:
https://github.com/apache/spark/pull/2966
[SPARK-4102] Remove unused ShuffleReader.stop() method.
This method is not implemented by the only subclass
(HashShuffleReader), nor is it ever called. While the
use of Scala's fancy
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r19440194
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -329,6 +329,15 @@ class SparkContext(config: SparkConf) extends
SparkStatusAPI
1 - 100 of 1796 matches
Mail list logo
