Github user zhzhan commented on the issue:
https://github.com/apache/spark/pull/15218
@wangmiao1981 Thanks for reviewing this. I will open another PR solving
these comments soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/15218
The test case design is pretty good. It covers all the scenarios.
- Could you add a check for the negative case? That means, when users do
not provide the right TaskAssigner name, we
Github user zhzhan commented on the issue:
https://github.com/apache/spark/pull/15218
@rxin Thanks a lot for the detail review. I will update the patch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/15218
@zhzhan in general it'd be great to have proper documentation on the
classes. For example, it is important to document the behavior of the various
assigners, and even more importantly, document the
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/15218
@mridulm @zhzhan I liked the idea here, but unfortunately I think it's
merged prematurely. There are insufficient documentation and basic styles that
don't align with rest of Spark. I'm going to
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/15218
@zhzhan and @mridulm all the classes need to be private[scheduler]
shouldn't they?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user zhzhan commented on the issue:
https://github.com/apache/spark/pull/15218
@mridulm Thanks for reviewing this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15218
Merged to master, thanks @zhzhan !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15218
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67021/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15218
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15218
**[Test build #67021 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67021/consoleFull)**
for PR 15218 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15218
**[Test build #67021 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67021/consoleFull)**
for PR 15218 at commit
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15218
I am assuming @kayousterhout does not have comments on this.
Can you please fix the conflict @zhzhan ? I will merge it in after that to
master.
---
If your project is set up for it, you can
Github user zhzhan commented on the issue:
https://github.com/apache/spark/pull/15218
@mridulm You are right. This patch is mainly for the job that has multiple
stages, which is very common in production pipeline. As you mentioned, if there
is shuffle involved,
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15218
@zhzhan I am curious why this is the case for the jobs being mentioned.
This pr should have an impact if the locality preference of the taskset
being run is fairly suboptimal to begin with, no ?
Github user zhzhan commented on the issue:
https://github.com/apache/spark/pull/15218
@mridulm Thanks for the comments. Your concern regarding the locality is
right. The patch does not change this behavior, which takes priority of
locality preference. But if multiple executors
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15218
Btw, taking a step back, I am not sure this will work as you expect it to.
Other than a few taskset's - those without locality information - the
schedule is going to be highly biased
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15218
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66465/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15218
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15218
**[Test build #66465 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66465/consoleFull)**
for PR 15218 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15218
**[Test build #66465 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66465/consoleFull)**
for PR 15218 at commit
Github user zhzhan commented on the issue:
https://github.com/apache/spark/pull/15218
@mridulm Thanks for review this. Will wait for a while in case there are
more comments before solving it.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15218
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66328/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15218
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15218
**[Test build #66328 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66328/consoleFull)**
for PR 15218 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15218
**[Test build #66328 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66328/consoleFull)**
for PR 15218 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15218
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65856/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15218
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15218
**[Test build #65856 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65856/consoleFull)**
for PR 15218 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15218
**[Test build #65856 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65856/consoleFull)**
for PR 15218 at commit
Github user zhzhan commented on the issue:
https://github.com/apache/spark/pull/15218
@gatorsmile Thanks. #65832 is the latest one which does not have the same
failure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/15218
See
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65832/testReport/org.apache.spark.streaming.kafka010/DirectKafkaStreamSuite/pattern_based_subscription/history/
---
If
Github user zhzhan commented on the issue:
https://github.com/apache/spark/pull/15218
retest please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if
Github user zhzhan commented on the issue:
https://github.com/apache/spark/pull/15218
Failed in DirectKafkaStreamSuite. It should has nothing to do with the
patch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15218
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65831/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15218
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15218
**[Test build #65831 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65831/consoleFull)**
for PR 15218 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15218
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65832/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15218
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15218
**[Test build #65832 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65832/consoleFull)**
for PR 15218 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15218
**[Test build #65832 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65832/consoleFull)**
for PR 15218 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15218
**[Test build #65831 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65831/consoleFull)**
for PR 15218 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15218
**[Test build #65830 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65830/consoleFull)**
for PR 15218 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15218
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15218
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65830/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15218
**[Test build #65830 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65830/consoleFull)**
for PR 15218 at commit
46 matches
Mail list logo
