Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/2871#issuecomment-60710651
@rxin -- I changed it to use a lazy val and the unit test, my local testing
seems to suggest this works. Can you take a look ?
Also @JoshRosen I merged the
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2871#issuecomment-60710753
[Test build #22335 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22335/consoleFull)
for PR 2871 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2871#issuecomment-60040371
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22018/consoleFull)
for PR 2871 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2871#issuecomment-60040376
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2871#discussion_r19197492
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -63,12 +63,22 @@ private[spark] class TorrentBroadcast[T: ClassTag](
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2871#issuecomment-60120473
I spent a second wondering whether this could cause a performance
regression, but I think we're safe. The fetching of the broadcast will still
block the entire task for
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/2871#issuecomment-60121971
Yeah the only performance regression is in reading this volatile boolean if
you call `.value` many times. But I think that should be low (and it should be
insignificant
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2871#issuecomment-59887232
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2871#issuecomment-59887226
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21970/consoleFull)
for PR 2871 at commit
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2871#discussion_r19188778
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -63,12 +63,22 @@ private[spark] class TorrentBroadcast[T: ClassTag](
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/2871#discussion_r19192935
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -63,12 +63,22 @@ private[spark] class TorrentBroadcast[T: ClassTag](
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/2871#issuecomment-60035820
I also added a test case to check if blocks are read on use
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2871#issuecomment-60036043
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22018/consoleFull)
for PR 2871 at commit
GitHub user shivaram opened a pull request:
https://github.com/apache/spark/pull/2871
[WIP] [SPARK-4031] Make torrent broadcast read blocks on use.
This avoids reading broadcast variables when they are referenced in the
closure but not used by the code.
Note: This is a WIP and
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2871#issuecomment-59881787
This seems likely to merge-conflict with my PR #2844, so I'd like to merge
that one first.
---
If your project is set up for it, you can reply to this email and have
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/2871#issuecomment-59881872
@JoshRosen -- yes, that should be fine. I will rebase once #2844 is checked
in
---
If your project is set up for it, you can reply to this email and have your
reply
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2871#issuecomment-59882084
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21970/consoleFull)
for PR 2871 at commit
17 matches
Mail list logo
