Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/17694
Hi @vundela , thanks for your PR. I was able to reproduce your issue, but
I think your fix here uses too broad of lock, since the shared resource is the
SparkContext pickle registry and needs
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17694
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17694
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77800/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17694
**[Test build #77800 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77800/testReport)**
for PR 17694 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17694
**[Test build #77800 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77800/testReport)**
for PR 17694 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17694
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user vundela commented on the issue:
https://github.com/apache/spark/pull/17694
ping @holdenk, checking if you have some time to review this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user vundela commented on the issue:
https://github.com/apache/spark/pull/17694
Thanks for your time @holdenk.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/17694
Interesting, I don't think I have all of the required context to review,
but I'll try and take a look this weekend (I've got some other things happening
this week already).
---
If your project is
Github user vundela commented on the issue:
https://github.com/apache/spark/pull/17694
cc @holdenk
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/17694
This seems reasonable to me.
There are some typos in the PR description. I think you meant "pickled"
instead of "picked" in a few places.
Using threading.Lock seems okay here from
Github user vundela commented on the issue:
https://github.com/apache/spark/pull/17694
Filed a PR for fixing the issue in spark1.6 branch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user maver1ck commented on the issue:
https://github.com/apache/spark/pull/17694
@vundela
Great.
But I'm planning to migrate to 2.1 as soon as 2.1.1 will be released.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user vundela commented on the issue:
https://github.com/apache/spark/pull/17694
Thanks for testing @maver1ck. I will look into 1.6 and 2.0.2.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user maver1ck commented on the issue:
https://github.com/apache/spark/pull/17694
OK. I did additional tests.
Fix is working only with Spark 2.1.
I tried to apply it on 2.0.2 and that was the reason of my problem.
---
If your project is set up for it, you can reply to
Github user maver1ck commented on the issue:
https://github.com/apache/spark/pull/17694
I checked pyspark.zip of running container and everything is on its place.
So I assume that there is more that one race condition in this code.
I'll try to prepare example of the
Github user vundela commented on the issue:
https://github.com/apache/spark/pull/17694
Hi @maver1ck, Thanks for your time in testing the patch.
I did run the patch with 1000 threads and it works fine.
Can you please check
Github user maver1ck commented on the issue:
https://github.com/apache/spark/pull/17694
The funny thing is this code works for me on 4 threads and throws exception
on 10 threads
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user maver1ck commented on the issue:
https://github.com/apache/spark/pull/17694
I tested your patch in our environment.
Problem still exists.
```
Job aborted due to stage failure: Task 0 in stage 22.0 failed 8 times, most
recent failure: Lost task 0.7 in stage
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17694
Please remove the boilerplate message at the end of your PR description.
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17694
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
21 matches
Mail list logo
