This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new ce3a157 [SPARK-26939][CORE][DOC] Fix some outdated comments about
task schedulers
ce3a157 is described below
commit ce3a157f00e052ca4dc0c6d4ef282cb21588837e
Author: seancxmao <seancx...@gmail.com>
AuthorDate: Sat Feb 23 10:52:53 2019 -0600
[SPARK-26939][CORE][DOC] Fix some outdated comments about task schedulers
## What changes were proposed in this pull request?
This PR aims to fix some outdated comments about task schedulers.
1. Change "ClusterScheduler" to "YarnScheduler" in comments of
`YarnClusterScheduler`
According to [SPARK-1140 Remove references to
ClusterScheduler](https://issues.apache.org/jira/browse/SPARK-1140),
ClusterScheduler is not used anymore.
I also searched "ClusterScheduler" within the whole project, no other
occurrences are found in comments or test cases. Note classes like
`YarnClusterSchedulerBackend` or `MesosClusterScheduler` are not relevant.
2. Update comments about `statusUpdate` from `TaskSetManager`
`statusUpdate` has been moved to `TaskSchedulerImpl`. StatusUpdate event
handling is delegated to `handleSuccessfulTask`/`handleFailedTask`.
## How was this patch tested?
N/A. Fix comments only.
Closes #23844 from seancxmao/taskscheduler-comments.
Authored-by: seancxmao <seancx...@gmail.com>
Signed-off-by: Sean Owen <sean.o...@databricks.com>
---
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala | 3 ++-
.../org/apache/spark/scheduler/cluster/YarnClusterScheduler.scala | 2 +-
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git
a/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala
b/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala
index b7bf069..453939a 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala
@@ -38,7 +38,8 @@ import org.apache.spark.util.collection.MedianHeap
* each task, retries tasks if they fail (up to a limited number of times), and
* handles locality-aware scheduling for this TaskSet via delay scheduling.
The main interfaces
* to it are resourceOffer, which asks the TaskSet whether it wants to run a
task on one node,
- * and statusUpdate, which tells it that one of its tasks changed state (e.g.
finished).
+ * and handleSuccessfulTask/handleFailedTask, which tells it that one of its
tasks changed state
+ * (e.g. finished/failed).
*
* THREADING: This class is designed to only be called from code with a lock
on the
* TaskScheduler (e.g. its event handlers). It should not be called from other
threads.
diff --git
a/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterScheduler.scala
b/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterScheduler.scala
index 96c9151..1f622a0 100644
---
a/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterScheduler.scala
+++
b/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterScheduler.scala
@@ -21,7 +21,7 @@ import org.apache.spark._
import org.apache.spark.deploy.yarn.ApplicationMaster
/**
- * This is a simple extension to ClusterScheduler - to ensure that appropriate
initialization of
+ * This is a simple extension to YarnScheduler - to ensure that appropriate
initialization of
* ApplicationMaster, etc is done
*/
private[spark] class YarnClusterScheduler(sc: SparkContext) extends
YarnScheduler(sc) {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
