[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/1487#discussion_r20179173 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala --- @@ -170,6 +170,20 @@ private[spark] class MesosSchedulerBackend( } } + def toWorkerOffer(offer: Offer) = new WorkerOffer( --- End diff -- Actually I just removed it altogether :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user tnachen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1487#discussion_r20179375
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala
---
@@ -170,6 +170,20 @@ private[spark] class MesosSchedulerBackend(
}
}
+ def toWorkerOffer(offer: Offer) = new WorkerOffer(
+offer.getSlaveId.getValue,
+offer.getHostname,
+getResource(offer.getResourcesList, cpus).toInt)
+
+ private def inClassLoader()(fun: = Unit) = {
--- End diff --
This is actually just a refactor of what's already in the backend, I don't
actualy know why we need to set and restore class loaders
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-62619270 @andrewor14 thanks for the review, I just rebased and should address all your comments too --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-62619717 [Test build #23222 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23222/consoleFull) for PR 1487 at commit [`4ea5dec`](https://github.com/apache/spark/commit/4ea5dec9a95c846e5eb3938210e4acbf389db4bf). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-62633467 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23222/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-62633457 [Test build #23222 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23222/consoleFull) for PR 1487 at commit [`4ea5dec`](https://github.com/apache/spark/commit/4ea5dec9a95c846e5eb3938210e4acbf389db4bf). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-62633746 Ok LGTM I'm merging this into master and 1.2 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1487 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-62458376 @andrewor14 can someone help merge this? It's been here for 2 months --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1487#discussion_r20116914
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala
---
@@ -170,6 +170,20 @@ private[spark] class MesosSchedulerBackend(
}
}
+ def toWorkerOffer(offer: Offer) = new WorkerOffer(
+offer.getSlaveId.getValue,
+offer.getHostname,
+getResource(offer.getResourcesList, cpus).toInt)
+
+ private def inClassLoader()(fun: = Unit) = {
--- End diff --
Can you add a comment on why this is needed?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1487#discussion_r20116941
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala
---
@@ -180,53 +194,40 @@ private[spark] class MesosSchedulerBackend(
* tasks are balanced across the cluster.
*/
override def resourceOffers(d: SchedulerDriver, offers: JList[Offer]) {
-val oldClassLoader = setClassLoader()
-try {
- synchronized {
-// Build a big list of the offerable workers, and remember their
indices so that we can
-// figure out which Offer to reply to for each worker
-val offerableWorkers = new ArrayBuffer[WorkerOffer]
-val offerableIndices = new HashMap[String, Int]
-
-def enoughMemory(o: Offer) = {
- val mem = getResource(o.getResourcesList, mem)
- val slaveId = o.getSlaveId.getValue
- mem = sc.executorMemory ||
slaveIdsWithExecutors.contains(slaveId)
-}
-
-for ((offer, index) - offers.zipWithIndex if enoughMemory(offer))
{
- offerableIndices.put(offer.getSlaveId.getValue, index)
- offerableWorkers += new WorkerOffer(
-offer.getSlaveId.getValue,
-offer.getHostname,
-getResource(offer.getResourcesList, cpus).toInt)
-}
-
-// Call into the TaskSchedulerImpl
-val taskLists = scheduler.resourceOffers(offerableWorkers)
-
-// Build a list of Mesos tasks for each slave
-val mesosTasks = offers.map(o = new JArrayList[MesosTaskInfo]())
-for ((taskList, index) - taskLists.zipWithIndex) {
- if (!taskList.isEmpty) {
-for (taskDesc - taskList) {
- val slaveId = taskDesc.executorId
- val offerNum = offerableIndices(slaveId)
- slaveIdsWithExecutors += slaveId
- taskIdToSlaveId(taskDesc.taskId) = slaveId
- mesosTasks(offerNum).add(createMesosTask(taskDesc, slaveId))
-}
- }
-}
-
-// Reply to the offers
-val filters = Filters.newBuilder().setRefuseSeconds(1).build() //
TODO: lower timeout?
-for (i - 0 until offers.size) {
- d.launchTasks(Collections.singleton(offers(i).getId),
mesosTasks(i), filters)
+inClassLoader() {
+ val (acceptedOffers, declinedOffers) = offers.partition(o = {
+val mem = getResource(o.getResourcesList, mem)
+val slaveId = o.getSlaveId.getValue
+mem = sc.executorMemory || slaveIdsWithExecutors.contains(slaveId)
+ })
--- End diff --
Style should be
```
... = offers.partition { o =
...
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1487#discussion_r20117032
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala
---
@@ -180,53 +194,40 @@ private[spark] class MesosSchedulerBackend(
* tasks are balanced across the cluster.
*/
override def resourceOffers(d: SchedulerDriver, offers: JList[Offer]) {
-val oldClassLoader = setClassLoader()
-try {
- synchronized {
-// Build a big list of the offerable workers, and remember their
indices so that we can
-// figure out which Offer to reply to for each worker
-val offerableWorkers = new ArrayBuffer[WorkerOffer]
-val offerableIndices = new HashMap[String, Int]
-
-def enoughMemory(o: Offer) = {
- val mem = getResource(o.getResourcesList, mem)
- val slaveId = o.getSlaveId.getValue
- mem = sc.executorMemory ||
slaveIdsWithExecutors.contains(slaveId)
-}
-
-for ((offer, index) - offers.zipWithIndex if enoughMemory(offer))
{
- offerableIndices.put(offer.getSlaveId.getValue, index)
- offerableWorkers += new WorkerOffer(
-offer.getSlaveId.getValue,
-offer.getHostname,
-getResource(offer.getResourcesList, cpus).toInt)
-}
-
-// Call into the TaskSchedulerImpl
-val taskLists = scheduler.resourceOffers(offerableWorkers)
-
-// Build a list of Mesos tasks for each slave
-val mesosTasks = offers.map(o = new JArrayList[MesosTaskInfo]())
-for ((taskList, index) - taskLists.zipWithIndex) {
- if (!taskList.isEmpty) {
-for (taskDesc - taskList) {
- val slaveId = taskDesc.executorId
- val offerNum = offerableIndices(slaveId)
- slaveIdsWithExecutors += slaveId
- taskIdToSlaveId(taskDesc.taskId) = slaveId
- mesosTasks(offerNum).add(createMesosTask(taskDesc, slaveId))
-}
- }
-}
-
-// Reply to the offers
-val filters = Filters.newBuilder().setRefuseSeconds(1).build() //
TODO: lower timeout?
-for (i - 0 until offers.size) {
- d.launchTasks(Collections.singleton(offers(i).getId),
mesosTasks(i), filters)
+inClassLoader() {
+ val (acceptedOffers, declinedOffers) = offers.partition(o = {
+val mem = getResource(o.getResourcesList, mem)
+val slaveId = o.getSlaveId.getValue
+mem = sc.executorMemory || slaveIdsWithExecutors.contains(slaveId)
+ })
+
+ val offerableWorkers = acceptedOffers.map(toWorkerOffer)
+
+ val slaveIdToOffer = acceptedOffers.map(o = o.getSlaveId.getValue
- o).toMap
+
+ val mesosTasks = new HashMap[String, JArrayList[MesosTaskInfo]]
+
+ // Call into the TaskSchedulerImpl
+ scheduler.resourceOffers(offerableWorkers)
+.filter(!_.isEmpty)
+.foreach(_.foreach(taskDesc = {
+val slaveId = taskDesc.executorId
+slaveIdsWithExecutors += slaveId
+taskIdToSlaveId(taskDesc.taskId) = slaveId
+mesosTasks.getOrElseUpdate(slaveId, new JArrayList[MesosTaskInfo])
+ .add(createMesosTask(taskDesc, slaveId))
+ }))
+
+ // Reply to the offers
+ val filters = Filters.newBuilder().setRefuseSeconds(1).build() //
TODO: lower timeout?
+
+ mesosTasks.foreach {
+case (slaveId, tasks) = {
+
d.launchTasks(Collections.singleton(slaveIdToOffer(slaveId).getId), tasks,
filters)
}
--- End diff --
Style should be
```
mesosTasks.foreach { case (slaveId, tasks) =
d.launchTasks(...)
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1487#discussion_r20117053 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala --- @@ -170,6 +170,20 @@ private[spark] class MesosSchedulerBackend( } } + def toWorkerOffer(offer: Offer) = new WorkerOffer( --- End diff -- should this be private def? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1487#discussion_r20117137 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala --- @@ -170,6 +170,20 @@ private[spark] class MesosSchedulerBackend( } } + def toWorkerOffer(offer: Offer) = new WorkerOffer( --- End diff -- I see this is used for testing. We should probably add a comment `// Exposed for testing` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1487#discussion_r20117206
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala
---
@@ -180,53 +194,40 @@ private[spark] class MesosSchedulerBackend(
* tasks are balanced across the cluster.
*/
override def resourceOffers(d: SchedulerDriver, offers: JList[Offer]) {
-val oldClassLoader = setClassLoader()
-try {
- synchronized {
-// Build a big list of the offerable workers, and remember their
indices so that we can
-// figure out which Offer to reply to for each worker
-val offerableWorkers = new ArrayBuffer[WorkerOffer]
-val offerableIndices = new HashMap[String, Int]
-
-def enoughMemory(o: Offer) = {
- val mem = getResource(o.getResourcesList, mem)
- val slaveId = o.getSlaveId.getValue
- mem = sc.executorMemory ||
slaveIdsWithExecutors.contains(slaveId)
-}
-
-for ((offer, index) - offers.zipWithIndex if enoughMemory(offer))
{
- offerableIndices.put(offer.getSlaveId.getValue, index)
- offerableWorkers += new WorkerOffer(
-offer.getSlaveId.getValue,
-offer.getHostname,
-getResource(offer.getResourcesList, cpus).toInt)
-}
-
-// Call into the TaskSchedulerImpl
-val taskLists = scheduler.resourceOffers(offerableWorkers)
-
-// Build a list of Mesos tasks for each slave
-val mesosTasks = offers.map(o = new JArrayList[MesosTaskInfo]())
-for ((taskList, index) - taskLists.zipWithIndex) {
- if (!taskList.isEmpty) {
-for (taskDesc - taskList) {
- val slaveId = taskDesc.executorId
- val offerNum = offerableIndices(slaveId)
- slaveIdsWithExecutors += slaveId
- taskIdToSlaveId(taskDesc.taskId) = slaveId
- mesosTasks(offerNum).add(createMesosTask(taskDesc, slaveId))
-}
- }
-}
-
-// Reply to the offers
-val filters = Filters.newBuilder().setRefuseSeconds(1).build() //
TODO: lower timeout?
-for (i - 0 until offers.size) {
- d.launchTasks(Collections.singleton(offers(i).getId),
mesosTasks(i), filters)
+inClassLoader() {
+ val (acceptedOffers, declinedOffers) = offers.partition(o = {
+val mem = getResource(o.getResourcesList, mem)
+val slaveId = o.getSlaveId.getValue
+mem = sc.executorMemory || slaveIdsWithExecutors.contains(slaveId)
+ })
+
+ val offerableWorkers = acceptedOffers.map(toWorkerOffer)
+
+ val slaveIdToOffer = acceptedOffers.map(o = o.getSlaveId.getValue
- o).toMap
+
+ val mesosTasks = new HashMap[String, JArrayList[MesosTaskInfo]]
+
+ // Call into the TaskSchedulerImpl
+ scheduler.resourceOffers(offerableWorkers)
+.filter(!_.isEmpty)
+.foreach(_.foreach(taskDesc = {
+val slaveId = taskDesc.executorId
+slaveIdsWithExecutors += slaveId
+taskIdToSlaveId(taskDesc.taskId) = slaveId
+mesosTasks.getOrElseUpdate(slaveId, new JArrayList[MesosTaskInfo])
+ .add(createMesosTask(taskDesc, slaveId))
+ }))
--- End diff --
This is difficult to read. Style should be:
```
scheduler.resourceOffers(...)
.filter(...)
.foreach { offer =
offer.foreach { taskDesk =
val slaveId = taskDesc.executorId
...
}
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-62464948 Hey @tnachen sorry for leaving this sitting. This seems like a small enough refactor get into 1.2 at this point. Once you rebase to master and address the minor comments I will merge this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-55344675 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-55344724 @tnachen What is the state of this patch? Is the existing logic relevant after the recent changes? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-55344901 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20180/consoleFull) for PR 1487 at commit [`f86768b`](https://github.com/apache/spark/commit/f86768b2404b674b457d1900ac1f466bdf566145). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-55349080 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20180/consoleFull) for PR 1487 at commit [`f86768b`](https://github.com/apache/spark/commit/f86768b2404b674b457d1900ac1f466bdf566145). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class CreateTableAsSelect(` * `case class CreateTableAsSelect(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-55354150 hi @andrewor14 yes the logic is still the same, it is good to merge as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-50115251 @pwendell this seems to pass tests now, mind to take a look? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-49704715 Jenkins, retest this please. Jenkins, add to whitelist. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-49705027 QA tests have started for PR 1487. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16946/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-49713371 QA results for PR 1487:br- This patch PASSES unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16946/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-49651731 @pwendell The console said the test failed but in a very unrelated way, is CI failing in general? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-49575030 Jenkins, test this please. Naw this is a message from jenkins. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1487#discussion_r15156158
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/mesos/MesosSchedulerBackendSuite.scala
---
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler.mesos
+
+import org.scalatest.FunSuite
+import org.apache.spark.{SparkConf, SparkContext, LocalSparkContext}
+import org.apache.spark.scheduler.{TaskDescription, WorkerOffer,
TaskSchedulerImpl}
+import org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend
+import org.apache.mesos.SchedulerDriver
+import org.apache.mesos.Protos._
+import org.scalatest.mock.EasyMockSugar
+import org.apache.mesos.Protos.Value.Scalar
+import org.easymock.{Capture, EasyMock}
+import java.nio.ByteBuffer
+import java.util.Collections
+import java.util
+
+class MesosSchedulerBackendSuite extends FunSuite with LocalSparkContext
with EasyMockSugar {
+ test(mesos resource offer is launching tasks) {
+def createOffer(id: Int, mem: Int, cpu: Int) = {
+ val builder = Offer.newBuilder()
+ builder.addResourcesBuilder()
+.setName(mem)
+.setType(Value.Type.SCALAR)
+.setScalar(Scalar.newBuilder().setValue(mem))
+ builder.addResourcesBuilder()
+.setName(cpus)
+.setType(Value.Type.SCALAR)
+.setScalar(Scalar.newBuilder().setValue(cpu))
+
builder.setId(OfferID.newBuilder().setValue(id.toString).build()).setFrameworkId(FrameworkID.newBuilder().setValue(f1))
+
.setSlaveId(SlaveID.newBuilder().setValue(s1)).setHostname(localhost).build()
+}
+
+val driver = EasyMock.createMock(classOf[SchedulerDriver])
+val taskScheduler = EasyMock.createMock(classOf[TaskSchedulerImpl])
+val offers = new java.util.ArrayList[Offer]
+offers.add(createOffer(1, 101, 1))
+offers.add(createOffer(1, 99, 1))
+
+val conf = new SparkConf
+conf.set(spark.executor.memory, 100m)
+conf.set(spark.home, /path)
+val sc = new SparkContext(local-cluster[2 , 1 , 512], test, conf)
--- End diff --
it would be better not to create a whole SparkContext here for the tests
(creating a local context like this is really expensive). Instead, could this
test just create a MesosSchedulerBackend directly? I think you can even just
pass it `null` for the SparkContext (or you could mock one, depending on how it
is used).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-49575213 QA tests have started for PR 1487. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16899/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-49575273 Don't have a ton of time to look ATM but I kicked off the tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-49578260 @pwendell sounds good, I updated the PR addressing your comments --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-49580185 QA results for PR 1487:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16899/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2269 Refactor mesos scheduler resourceOf...
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/1487#issuecomment-49553051 Do I need to specify someone to verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
