Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19130571
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -227,6 +217,7 @@ private object TorrentBroadcast extends Logging {
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19130611
--- Diff:
core/src/test/scala/org/apache/spark/broadcast/BroadcastSuite.scala ---
@@ -84,6 +89,24 @@ class BroadcastSuite extends FunSuite with
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19130957
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -227,6 +217,7 @@ private object TorrentBroadcast extends Logging {
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19131009
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -227,6 +217,7 @@ private object TorrentBroadcast extends Logging {
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2844#issuecomment-59884782
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21974/consoleFull)
for PR 2844 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2844#issuecomment-59890238
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2844#issuecomment-59890235
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21974/consoleFull)
for PR 2844 at commit
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2844#issuecomment-59891101
I've merged this into master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/2844
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2844#issuecomment-59803868
LGTM now, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19118825
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -227,6 +217,7 @@ private object TorrentBroadcast extends Logging {
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19118878
--- Diff:
core/src/test/scala/org/apache/spark/broadcast/BroadcastSuite.scala ---
@@ -17,13 +17,18 @@
package org.apache.spark.broadcast
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19119141
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -104,29 +112,23 @@ private[spark] class TorrentBroadcast[T: ClassTag](
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19119178
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -104,29 +112,23 @@ private[spark] class TorrentBroadcast[T: ClassTag](
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19119195
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -62,6 +59,20 @@ private[spark] class TorrentBroadcast[T: ClassTag](
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19119727
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -104,29 +112,23 @@ private[spark] class TorrentBroadcast[T:
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19119774
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -227,6 +217,7 @@ private object TorrentBroadcast extends Logging {
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19120243
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -104,29 +112,23 @@ private[spark] class TorrentBroadcast[T:
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19121626
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -104,29 +112,23 @@ private[spark] class TorrentBroadcast[T:
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19126060
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -227,6 +217,7 @@ private object TorrentBroadcast extends Logging {
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19126192
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -62,6 +59,20 @@ private[spark] class TorrentBroadcast[T: ClassTag](
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19126189
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -104,29 +112,23 @@ private[spark] class TorrentBroadcast[T:
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2844#issuecomment-59868399
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21953/consoleFull)
for PR 2844 at commit
GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/2844
[SPARK-3958] TorrentBroadcast cleanup / debugging improvements.
This PR makes several changes to TorrentBroadcast in order to make
it easier to reason about, which should help when debugging
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2844#issuecomment-59641699
/cc @rxin for review. I'd like to apply this to `branch-1.1` as well,
since I believe that it's also affected by current TorrentBroadcast bugs.
---
If your project
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2844#issuecomment-59641733
Also, /cc @davies, who helped me to spot the local mode might deblockify
an empty array bug and who's been working on TorrentBroadcast optimizations.
---
If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2844#issuecomment-59641757
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21882/consoleFull)
for PR 2844 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2844#issuecomment-59642634
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21882/consoleFull)
for PR 2844 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2844#issuecomment-59642636
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2844#issuecomment-59643009
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21885/consoleFull)
for PR 2844 at commit
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2844#issuecomment-59643838
It looks like this build is going to fail a ReplSuite test:
```scala
test(broadcast vars) {
// Test that the value that a broadcast var had when it
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2844#issuecomment-59643971
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21885/consoleFull)
for PR 2844 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2844#issuecomment-59643974
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2844#issuecomment-59644064
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21888/consoleFull)
for PR 2844 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2844#issuecomment-59645141
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2844#issuecomment-59645137
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21888/consoleFull)
for PR 2844 at commit
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2844#issuecomment-59653126
This most recent test-failure is another side-effect of removing
TorrentBroadcast's optimizations for local mode:
```
[info] - Unpersisting
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2844#issuecomment-59653640
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21893/consoleFull)
for PR 2844 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2844#issuecomment-59655954
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21893/consoleFull)
for PR 2844 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2844#issuecomment-59655958
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19063222
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -76,23 +87,20 @@ private[spark] class TorrentBroadcast[T: ClassTag](
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19063253
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -62,6 +59,20 @@ private[spark] class TorrentBroadcast[T: ClassTag](
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19063271
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -156,6 +158,7 @@ private[spark] class TorrentBroadcast[T: ClassTag](
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19063287
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -156,6 +158,7 @@ private[spark] class TorrentBroadcast[T: ClassTag](
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19063336
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -76,23 +87,20 @@ private[spark] class TorrentBroadcast[T: ClassTag](
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19063363
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -62,6 +59,20 @@ private[spark] class TorrentBroadcast[T: ClassTag](
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19063455
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -179,43 +183,29 @@ private[spark] class TorrentBroadcast[T: ClassTag](
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2844#discussion_r19066407
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -179,43 +183,29 @@ private[spark] class TorrentBroadcast[T:
48 matches
Mail list logo
