Github user squito commented on the issue:
https://github.com/apache/spark/pull/15505
@djvulee this is https://issues.apache.org/jira/browse/SPARK-19108. Note
that there is some discussion there about this being a bit harder than what I
originally thought, though I think its still
Github user djvulee commented on the issue:
https://github.com/apache/spark/pull/15505
>I agree with Kay that putting in a smaller change first is better,
assuming it still has the performance gains. That doesn't preclude any further
optimizations that are bigger changes.
Github user witgo commented on the issue:
https://github.com/apache/spark/pull/15505
[SPARK-18890_20170303](https://github.com/witgo/spark/commits/SPARK-18890_20170303)
`s code is older but the test case running time is 5.2 s
---
If your project is set up for it, you can reply to
Github user witgo commented on the issue:
https://github.com/apache/spark/pull/15505
Yes, maybe a multithreaded serialization task code can have a better
performance, let me close the PR
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15505
@kayousterhout I am surprised it is not more, but I agree that the added
complexity for such low returns is not worth it.
---
If your project is set up for it, you can reply to this email and have
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/15505
@witgo I don't think the ~1.5% improvement in runtime merits the added
complexity of this change. I could be convinced to merge this if it simplified
the code or the ability to reason about
Github user witgo commented on the issue:
https://github.com/apache/spark/pull/15505
Do not know which pr causes the run time of this test case to be reduced
from 21.764 s to 9.566 s.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user witgo commented on the issue:
https://github.com/apache/spark/pull/15505
@kayousterhout
Test results have been updated:
| [SPARK-17931](https://github.com/witgo/spark/tree/SPARK-17931)
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/15505
@witgo OK I'll hold off on doing another pass on the code until you have
the test results.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user witgo commented on the issue:
https://github.com/apache/spark/pull/15505
@kayousterhout It takes some time to update the test report.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73721/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #73721 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73721/testReport)**
for PR 15505 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #73721 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73721/testReport)**
for PR 15505 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73694/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #73694 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73694/testReport)**
for PR 15505 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #73694 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73694/testReport)**
for PR 15505 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73678/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #73678 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73678/testReport)**
for PR 15505 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #73678 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73678/testReport)**
for PR 15505 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73520/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #73520 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73520/testReport)**
for PR 15505 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #73520 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73520/testReport)**
for PR 15505 at commit
Github user witgo commented on the issue:
https://github.com/apache/spark/pull/15505
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73468/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #73468 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73468/testReport)**
for PR 15505 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #73468 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73468/testReport)**
for PR 15505 at commit
Github user witgo commented on the issue:
https://github.com/apache/spark/pull/15505
@kayousterhout I think the latest code is ready to merge into the master
branch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/15505
@witgo what's the status of this? I'd like to get this merged and am happy
to take this over if you don't have time to work on it.
---
If your project is set up for it, you can reply to
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72651/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #72651 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72651/testReport)**
for PR 15505 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #72651 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72651/testReport)**
for PR 15505 at commit
Github user witgo commented on the issue:
https://github.com/apache/spark/pull/15505
Okay, this may take some time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/15505
Thanks for adding the unit test but it doesn't quite cover the case Imran
brought up: can you write a test for when the LaunchTasks call includes tasks
from multiple task sets, one with tasks
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72515/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #72515 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72515/testReport)**
for PR 15505 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #72515 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72515/testReport)**
for PR 15505 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72504/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #72504 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72504/testReport)**
for PR 15505 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #72504 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72504/testReport)**
for PR 15505 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71959/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #71959 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71959/testReport)**
for PR 15505 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #71959 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71959/testReport)**
for PR 15505 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71792/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #71792 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71792/testReport)**
for PR 15505 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #71792 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71792/testReport)**
for PR 15505 at commit
Github user witgo commented on the issue:
https://github.com/apache/spark/pull/15505
@squito
My understanding is that the TaskSchedulerImpl class contains many
synchronized statements (synchronized the methods). If a synchronized
statements execution time is very long, it will
Github user squito commented on the issue:
https://github.com/apache/spark/pull/15505
This might be a really dumb question, but please humor me -- can you
explain to me the threads involved, both before and after this change? I must
be missing something, because it seems to me it
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71412/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #71412 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71412/testReport)**
for PR 15505 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #71412 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71412/testReport)**
for PR 15505 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71286/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #71286 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71286/testReport)**
for PR 15505 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #71286 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71286/testReport)**
for PR 15505 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71185/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #71185 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71185/testReport)**
for PR 15505 at commit
Github user squito commented on the issue:
https://github.com/apache/spark/pull/15505
@witgo -- right, I'm saying that we should make local mode work to make
debugging & development as easy as possible, not to maximize performance. That
means we *should* do the extra serialization
Github user witgo commented on the issue:
https://github.com/apache/spark/pull/15505
@squito In the local mode, the performance is relatively less important, we
only guarantee that there will be no performance degradation on it.
---
If your project is set up for it, you can reply to
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15505
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71036/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #71036 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71036/testReport)**
for PR 15505 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15505
**[Test build #71036 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71036/testReport)**
for PR 15505 at commit
73 matches
Mail list logo
