Github user ankurdave commented on the pull request:
https://github.com/apache/spark/pull/4136#issuecomment-71286155
@JoshRosen Actually, it seems the test failures still occur, but only when
I add a [unit
test](https://github.com/apache/spark/commit/9b225ac3072de522b40b46aba6df1f1c231f13ef#diff-3ade47bc293ef06e43c25f1ac1f6783bR354)
that sets spark.default.parallelism.
Adding the test causes subsequent tests within the same run to fail with
exceptions like
```
java.io.IOException: org.apache.spark.SparkException: Failed to get
broadcast_0_piece0 of broadcast_0
```
and
```
java.io.IOException: PARSING_ERROR(2)
```
The exception traces always occur in TorrentBroadcast.
It seems like setting spark.default.parallelism is causing some kind of
side effect that corrupts broadcasts in later unit tests, which is strange
since (1) each unit test should have its own SparkContext and therefore its own
temp directory, and (2) I'm only passing spark.default.parallelism to
SparkConf/SparkContext, not setting it as a system property.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
