[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93882452 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-17 Thread jongyoul
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93898567 @andrewor14 I've fixed what you issue. Please review and merge this. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/5063 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-17 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-94112146 LGTM I'm merging this finally thanks everyone. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-16 Thread jongyoul
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93731580 I've rebased it from current master at first. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-16 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93732843 [Test build #30419 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30419/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-16 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93748187 [Test build #30419 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30419/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93748213 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-16 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93730968 [Test build #30418 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30418/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-16 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93751671 [Test build #30418 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30418/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93751686 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-16 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93882446 [Test build #30448 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30448/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-16 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93872612 [Test build #30448 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30448/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-16 Thread jongyoul
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93872085 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-14 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93011001 Should we reuse the existing `spark.executor.cores` property? It would be good to make all the cluster managers use a consistent configuration than to introduce a

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-14 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/5063#discussion_r28385856 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala --- @@ -67,6 +67,8 @@ private[spark] class

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-14 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/5063#discussion_r28386441 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala --- @@ -233,9 +234,7 @@ private[spark] class

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-14 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93132743 I talked to @sryza and @tnachen offline about the potential sources of confusion here. It seems that this code used to mistakenly use `spark.task.cpus` as the number

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-14 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/5063#discussion_r28386662 --- Diff: docs/running-on-mesos.md --- @@ -211,6 +211,16 @@ See the [configuration page](configuration.html) for information on Spark config /td

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-14 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/5063#discussion_r28386348 --- Diff: docs/running-on-mesos.md --- @@ -211,6 +211,16 @@ See the [configuration page](configuration.html) for information on Spark config /td

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-14 Thread jongyoul
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93206397 @andrewor14 Thanks for overall reviewing. I'll handle what you issue. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-14 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93040635 @andrewor14 this was a concern I had as well, and there's a long discussion above. The reason for not using `spark.executor.cores` is that the property introduced here

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-14 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/5063#discussion_r28365258 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala --- @@ -220,10 +222,9 @@ private[spark] class

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-14 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93056104 @sryza @tnachen just so I understand, doesn't `spark.mesos.executor.cores` refer to the number of cores used by the Spark executor launched to run a given task? Why

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-14 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93058391 My understanding based on the discussion here is that `spark.mesos.executor.cores` is the number of cores reserved by an executor *not* for use in running tasks. So if

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-14 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93105439 What @sryza is correct. spark.mesos.executor.cores is different than spark.executor.cores, as it's only the cpu cores used for the mesos executor, which is used to

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-14 Thread jongyoul
Github user jongyoul commented on a diff in the pull request: https://github.com/apache/spark/pull/5063#discussion_r28381413 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala --- @@ -220,10 +222,9 @@ private[spark] class

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-04-06 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-90266435 @andrewor14 do you think you can merge this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-31 Thread jongyoul
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-88293005 @sryza Could you please merge this PR for spark 1.3.1? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-30 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-87791403 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-87513975 [Test build #29377 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29377/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-87532197 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-87532194 [Test build #29377 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29377/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-29 Thread jongyoul
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-87544727 @sryza I hope this will be a final review :-) I'm sorry for making a typo again and again. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-27 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86841449 [Test build #29285 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29285/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86841450 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-27 Thread jongyoul
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86841550 @sryza Review it again, please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-27 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86973622 One minor nit and otherwise LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-27 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/5063#discussion_r27302728 --- Diff: docs/running-on-mesos.md --- @@ -211,6 +211,16 @@ See the [configuration page](configuration.html) for information on Spark config /td

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-26 Thread jongyoul
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86795209 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86812076 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86796016 [Test build #29272 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29272/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86812055 [Test build #29272 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29272/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-26 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/5063#discussion_r27274938 --- Diff: docs/running-on-mesos.md --- @@ -211,6 +211,14 @@ See the [configuration page](configuration.html) for information on Spark config /td

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-26 Thread jongyoul
Github user jongyoul commented on a diff in the pull request: https://github.com/apache/spark/pull/5063#discussion_r27275980 --- Diff: docs/running-on-mesos.md --- @@ -211,6 +211,14 @@ See the [configuration page](configuration.html) for information on Spark config /td

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86824172 [Test build #29285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29285/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86356704 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86356693 [Test build #29214 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29214/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-26 Thread jongyoul
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86357482 @sryza Who Do you know help this build issue? I just edited docs only. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-26 Thread jongyoul
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86357586 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86357920 [Test build #29215 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29215/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86365655 [Test build #29215 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29215/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86365666 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-25 Thread jongyoul
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85920174 @tnachen @elyast I've updated it to support fractional number of cores for executors. Please review this PR. I've tested my cluster. My setting is ```

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-25 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85921782 [Test build #29159 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29159/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-25 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85965524 [Test build #29159 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29159/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85965536 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-25 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/5063#discussion_r27149062 --- Diff: docs/running-on-mesos.md --- @@ -211,6 +211,14 @@ See the [configuration page](configuration.html) for information on Spark config /td

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-25 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/5063#discussion_r27149333 --- Diff: docs/running-on-mesos.md --- @@ -211,6 +211,14 @@ See the [configuration page](configuration.html) for information on Spark config /td

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-25 Thread elyast
Github user elyast commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86113945 @jongyoul looks good to me thanks :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-25 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86155273 Had some doc comments, but after they're fixed this LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-25 Thread jongyoul
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86345441 I've rebased from master. Don't mind that I have a mistake to write a commit log which I've worked another issue. --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-25 Thread jongyoul
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86344972 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-25 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86345756 [Test build #29214 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29214/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86354455 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-25 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86354447 [Test build #29212 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29212/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-25 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86327593 [Test build #29204 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29204/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-25 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86341311 [Test build #29204 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29204/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86341349 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-25 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86343836 [Test build #29212 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29212/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-25 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86309543 [Test build #29202 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29202/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-25 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86326197 [Test build #29202 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29202/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86326206 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-25 Thread jongyoul
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86327232 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-24 Thread jongyoul
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85500449 @elyast @tnachen Do you think `CPUS_PER_TASK` also support fractional value? If it's not, I may be support executorCores as fractional value without huge changes. ---

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-24 Thread elyast
Github user elyast commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85594000 @jongyoul @tnachen imho its fine if CPU_PER_TASK will remain Integers, however if your job is less CPU intensive, that might be beneficial to optimize it, although I

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-24 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85596738 Yes let's keep CPUS_PER_TASK as is, since it's shared among different things. Executor cores I think can still be a double that is only used in

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-23 Thread elyast
Github user elyast commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85002672 @sryza you can request from Mesos fraction of CPU, however I haven't realized that we have wrong type in this patch, we should change it to Double instead of Int. ---

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-23 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85096699 @sryza When creating a Mesos Task, one usually define the resources required for the execution of the task and the resources required to run the Mesos executor. Again

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-23 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85096806 Fractional is definitely supported, since it's just cpu shares in the end. We should make it a double --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-23 Thread jongyoul
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85317877 @tnachen I see. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-23 Thread elyast
Github user elyast commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85336463 @jongyoul I can try to test it out with allocating CPU to 0, but cannot promise it will work, otherwise it's just doesn't make much sense imho, I'd discuss with project

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84587905 **[Test build #28959 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28959/consoleFull)** for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84587909 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84564916 [Test build #28959 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28959/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-22 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84712739 I see. So frameworkCores isn't ideal because framework already has a meaning in the Mesos world. Also, executor already has some inherent ambiguity because Mesos uses it

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-22 Thread elyast
Github user elyast commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84733162 @sryza I don't think it actually need more than a single core, the issue is you cannot give less than 1 CPU. --- If your project is set up for it, you can reply to this

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-22 Thread jongyoul
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84761674 Jekins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84762169 [Test build #28971 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28971/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-22 Thread jongyoul
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84769168 I rebased this from master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-22 Thread jongyoul
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84769305 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84770282 [Test build #28974 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28974/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84801569 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84801541 [Test build #28971 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28971/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84801446 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84801414 [Test build #28974 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28974/consoleFull) for PR 5063 at commit

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-22 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84814176 @elyast can you explain what you mean in a little more detail? Are you saying that the patch somehow enables users to request fractional cores? --- If your project is

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-21 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84272183 @sryza I believe this setting is a bit unique for Mesos. I think it's better to clarify the terminology here, of what does framework and executor means for Mesos.

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-20 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84055432 I'd like to better understand the reason this is needed in Mesos fine-grained mode. We've talked about eventually adding a fine-grained mode for YARN as well. So if we

[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...

2015-03-20 Thread jongyoul
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84258060 @elyast It's enough to explain why we can set this property. @sryza This feature exists only in Mesos mode now. How about setting this property specified in Mesos now,

  1   2   >