[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2018-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19194 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2018-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19194 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93367/ Test PASSed. ---

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2018-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19194 **[Test build #93367 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93367/testReport)** for PR 19194 at commit

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2018-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19194 **[Test build #93367 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93367/testReport)** for PR 19194 at commit

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2018-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19194 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2018-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19194 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2018-07-20 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/19194 test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2018-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19194 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2018-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19194 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93350/ Test FAILed. ---

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2018-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19194 **[Test build #93350 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93350/testReport)** for PR 19194 at commit

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2018-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19194 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2018-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19194 **[Test build #93350 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93350/testReport)** for PR 19194 at commit

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2018-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19194 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2018-07-20 Thread dhruve
Github user dhruve commented on the issue: https://github.com/apache/spark/pull/19194 @squito I have made the changes as requested. Can you have a look at this again. Thanks. --- - To unsubscribe, e-mail:

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2018-07-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19194 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93187/ Test PASSed. ---

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2018-07-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19194 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2018-07-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19194 **[Test build #93187 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93187/testReport)** for PR 19194 at commit

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2018-07-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19194 **[Test build #93187 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93187/testReport)** for PR 19194 at commit

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2018-07-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19194 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2018-07-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19194 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-10-26 Thread dhruve
Github user dhruve commented on the issue: https://github.com/apache/spark/pull/19194 Hi @squito. Sorry for the late response. I want to get back on this. As soon as I get a chance I will work on it and update the PR. ---

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-10-23 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19194 @dhruve is this still active? Sorry I was out a while, catching up on everything. --- - To unsubscribe, e-mail:

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-10-05 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19194 btw I'm going to have really intermittent access for a few weeks, so you don't need to wait for me to proceed with this @tgravescs ---

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-29 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/19194 I agree with that its going to be a toss-up from the users perspective as to which setting will be applied. But if the TaskSetManager is using the active job properties and that is what its

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-28 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19194 Sorry if I was unclear earlier on the issue w/ the active Job ID. So, I agree, that if a user actually gets into this situation, where they've got two different jobs for the same stage, with

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-27 Thread dhruve
Github user dhruve commented on the issue: https://github.com/apache/spark/pull/19194 Configuring at the stage level seems to be the appropriate and more deterministic choice. If we agree on changing the API, we can start another effort looking in that direction. Till then we can

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-27 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19194 btw I'm not opposed to this feature, just want to make sure that given (a) complexity and (b) the fact that it doesn't quite meet the original ask anyway (configuring at the stage & no need to break

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-27 Thread dhruve
Github user dhruve commented on the issue: https://github.com/apache/spark/pull/19194 @squito Thanks for pointing that out. What you mentioned makes sense and I did dig some more on the `DAGScheduler` and `activeJobForStage` to gather more context. We could take into account the

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-26 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19194 one more thought -- have you considered adding stage properties? I know that is more involved since its an API change, but I dunno how many times I've wanted something like that, eg. for labelling

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19194 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82092/ Test PASSed. ---

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19194 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19194 **[Test build #82092 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82092/testReport)** for PR 19194 at commit

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19194 **[Test build #82092 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82092/testReport)** for PR 19194 at commit

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-22 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/19194 Jenkins, test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19194 **[Test build #82090 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82090/testReport)** for PR 19194 at commit

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19194 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19194 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82052/ Test PASSed. ---

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19194 **[Test build #82052 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82052/testReport)** for PR 19194 at commit

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19194 **[Test build #82052 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82052/testReport)** for PR 19194 at commit

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19194 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81894/ Test PASSed. ---

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19194 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19194 **[Test build #81894 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81894/testReport)** for PR 19194 at commit

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-18 Thread dhruve
Github user dhruve commented on the issue: https://github.com/apache/spark/pull/19194 @tgraves I have addressed the comments and tried to cover the possible cases in the existing test for job groups and speculation. Kindly let me know if we need to add or address more use cases.

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19194 **[Test build #81894 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81894/testReport)** for PR 19194 at commit

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19194 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81645/ Test PASSed. ---

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19194 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19194 **[Test build #81645 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81645/testReport)** for PR 19194 at commit

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19194 **[Test build #81645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81645/testReport)** for PR 19194 at commit

[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage

2017-09-11 Thread dhruve
Github user dhruve commented on the issue: https://github.com/apache/spark/pull/19194 Rebased this PR with current master and have squashed the earlier commits. --- - To unsubscribe, e-mail: