Github user seyfe commented on the issue:
https://github.com/apache/spark/pull/15371
Thanks @zsxwing.
Here is the PR for branch-2.0
https://github.com/apache/spark/pull/15425
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/15371
There are some conflicts with 2.0. @seyfe could you submit a PR for
branch-2.0, please? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/15371
LGTM. Thanks! Merging to master and 2.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/9/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #9 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/9/consoleFull)**
for PR 15371 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #9 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/9/consoleFull)**
for PR 15371 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66632/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66632 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66632/consoleFull)**
for PR 15371 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66632 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66632/consoleFull)**
for PR 15371 at commit
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/15371
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66627 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66627/consoleFull)**
for PR 15371 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66627/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66627 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66627/consoleFull)**
for PR 15371 at commit
Github user seyfe commented on the issue:
https://github.com/apache/spark/pull/15371
@zsxwing , I think that is a good idea. I search it and that is the only
place we use `BlockStatusesAccumulator`. Let me remove it.
---
If your project is set up for it, you can reply to this email
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/15371
@seyfe I think we can remove `BlockStatusesAccumulator` and just use
`private val _updatedBlockStatuses = new CollectionAccumulator[(BlockId,
BlockStatus)]` instead. `BlockStatusesAccumulator`
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66611/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66611 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66611/consoleFull)**
for PR 15371 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66611 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66611/consoleFull)**
for PR 15371 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66584/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66584 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66584/consoleFull)**
for PR 15371 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66584 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66584/consoleFull)**
for PR 15371 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66545/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66545 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66545/consoleFull)**
for PR 15371 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66544/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66544 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66544/consoleFull)**
for PR 15371 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66545 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66545/consoleFull)**
for PR 15371 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66544 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66544/consoleFull)**
for PR 15371 at commit
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/15371
You can find the following errors on Jenkins:
```
[error] * abstract method merge(org.apache.spark.util.AccumulatorV2)Unit
in class org.apache.spark.util.AccumulatorV2 does not have a
Github user seyfe commented on the issue:
https://github.com/apache/spark/pull/15371
@zsxwing , I built it and the test pipelines works fine. So fix is good.
But I don't know how to fix the MiMa tests. Would you mind helping me on this?
---
If your project is set up for it, you can
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66541 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66541/consoleFull)**
for PR 15371 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66541/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66541 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66541/consoleFull)**
for PR 15371 at commit
Github user seyfe commented on the issue:
https://github.com/apache/spark/pull/15371
I don't know if it's related but I found a bug in the last iteration. We
also need to override copy and copyAndReset methods. Otherwise it throws
java.lang.ClassCastException error.
---
If your
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66539 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66539/consoleFull)**
for PR 15371 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66539/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66539 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66539/consoleFull)**
for PR 15371 at commit
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/15371
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user seyfe commented on the issue:
https://github.com/apache/spark/pull/15371
Hi @zsxwing ,
The test failed with below error. I don't think that it's related with my
change. Should we just re-run the test or do you have any suggestion?
```
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66537/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66537 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66537/consoleFull)**
for PR 15371 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66537 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66537/consoleFull)**
for PR 15371 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66535 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66535/consoleFull)**
for PR 15371 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66535/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66534 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66534/consoleFull)**
for PR 15371 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66534/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66535 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66535/consoleFull)**
for PR 15371 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66534 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66534/consoleFull)**
for PR 15371 at commit
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/15371
@seyfe I'm taking my words back. Yea, `BlockStatusesAccumulator.merge` will
be called in driver.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user seyfe commented on the issue:
https://github.com/apache/spark/pull/15371
Hi @srowen , this is the part that I removed extra copy operation. I
changed this line because this conversion is already done by
BlockStatusesAccumulator.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15371
I see the new copy (of course or else this wouldn't help) but where is a
copy removed? I'm probably overlooking it. A `toList` call was moved, which we
discussed above briefly. I still think it's
Github user seyfe commented on the issue:
https://github.com/apache/spark/pull/15371
Hi @srowen,
This PR doesn't introduce any extra data copy operations. It moves the data
copy code from `JsonProtocol:accumValueToJson` method to
`BlockStatusesAccumulator:value` method so
Github user seyfe commented on the issue:
https://github.com/apache/spark/pull/15371
I also want to point out that below is the core part of fix. Rest of the
code changes are side-effects of it.
```
+ // `asScala` accesses the internal values using `java.util.Iterator`
Github user seyfe commented on the issue:
https://github.com/apache/spark/pull/15371
@zsxwing . `JArray(v.asInstanceOf[java.util.List[(BlockId,
BlockStatus)]].asScala.toList.map` this is the line of code which is causing
ConcurrentModificationException. And you can see that from call
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/15371
> @zsxwing. BlockStatusesAccumulator is using synchronizedList which is
_seq for concurrency. But when the value method is executed and the output is
used by asScala.toList it
Github user seyfe commented on the issue:
https://github.com/apache/spark/pull/15371
@zsxwing. `BlockStatusesAccumulator` is using synchronizedList which is
`_seq` for concurrency. But when the `value` method is executed and the output
is used by `asScala.toList` it caused the
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/15371
@seyfe The issue is `TaskInfo.accumulables` is accessed in multiple threads
without synchronization. `TaskMetrics.updatedBlockStatuses` is fine.
---
If your project is set up for it, you can reply
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66473/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66473 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66473/consoleFull)**
for PR 15371 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15371
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15371
**[Test build #66473 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66473/consoleFull)**
for PR 15371 at commit
72 matches
Mail list logo
