Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-70910247
@andrewor14 @pwendell @tdas How do you feel about committing this patch,
as-is, for 1.2.1? I think it could be a huge support burden reducer /
usability improver for
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3884#discussion_r23342589
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -526,6 +534,7 @@ class SparkContext(config: SparkConf) extends Logging
with
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3884#discussion_r23342851
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -955,6 +993,11 @@ class SparkContext(config: SparkConf) extends Logging
with
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3884#discussion_r23342885
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -969,6 +1012,7 @@ class SparkContext(config: SparkConf) extends Logging
with
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3884#discussion_r23343194
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1199,6 +1260,7 @@ class SparkContext(config: SparkConf) extends Logging
with
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3884#discussion_r23343236
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1458,6 +1522,7 @@ class SparkContext(config: SparkConf) extends Logging
with
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3884#discussion_r23342970
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1002,6 +1047,7 @@ class SparkContext(config: SparkConf) extends Logging
with
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3884#discussion_r23343173
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1146,6 +1206,7 @@ class SparkContext(config: SparkConf) extends Logging
with
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-70943337
[Test build #25928 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25928/consoleFull)
for PR 3884 at commit
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3884#discussion_r23342907
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -992,6 +1036,7 @@ class SparkContext(config: SparkConf) extends Logging
with
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3884#discussion_r23343038
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1068,7 +1120,10 @@ class SparkContext(config: SparkConf) extends
Logging with
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3884#discussion_r23343083
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -,6 +1170,7 @@ class SparkContext(config: SparkConf) extends Logging
with
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-70943192
I audited the uses of `assertNotStopped` and removed a bunch of calls in
methods that sometimes didn't throw exceptions on Spark 1.2.0. Pending
Jenkins, I'm planning
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-70951031
[Test build #25928 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25928/consoleFull)
for PR 3884 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-70951044
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3884#discussion_r23342620
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -550,6 +560,7 @@ class SparkContext(config: SparkConf) extends Logging
with
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3884#discussion_r23342825
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -906,8 +936,10 @@ class SparkContext(config: SparkConf) extends Logging
with
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3884#discussion_r23342756
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -891,14 +913,22 @@ class SparkContext(config: SparkConf) extends Logging
with
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3884#discussion_r23343271
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1466,17 +1531,29 @@ class SparkContext(config: SparkConf) extends
Logging with
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3884#discussion_r23342706
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -891,14 +913,22 @@ class SparkContext(config: SparkConf) extends Logging
with
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3884#discussion_r23343005
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1047,6 +1097,7 @@ class SparkContext(config: SparkConf) extends Logging
with
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3884#discussion_r23343023
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1059,6 +1110,7 @@ class SparkContext(config: SparkConf) extends Logging
with
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-69636824
Alright, I've updated this to use IllegalStateException when methods are
called on a stopped SparkContext. I've also added some more helpful error
messages to PySpark
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-69637150
[Test build #25426 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25426/consoleFull)
for PR 3884 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-69637285
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-69637280
[Test build #25426 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25426/consoleFull)
for PR 3884 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-69664661
[Test build #25433 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25433/consoleFull)
for PR 3884 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-69677921
**[Test build #25433 timed
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25433/consoleFull)**
for PR 3884 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-69677929
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-69684994
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-69684987
[Test build #25441 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25441/consoleFull)
for PR 3884 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-69678415
[Test build #25441 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25441/consoleFull)
for PR 3884 at commit
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-69080412
Any opinions on the `assertNotStopped()` checks here? I'd like to backport
this patch to other branches since I think it's a huge usability improvement.
If there are
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-69081096
Maybe IllegalStateException?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-68935347
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-68936082
[Test build #25114 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25114/consoleFull)
for PR 3884 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-68946403
[Test build #25114 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25114/consoleFull)
for PR 3884 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-68946424
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-68816618
[Test build #25084 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25084/consoleFull)
for PR 3884 at commit
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-68816630
I've added some additional tests to prevent users from calling methods on a
stopped SparkContext, since this usually resulted in confusing
NullPointerExceptions.
---
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-68817012
[Test build #25085 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25085/consoleFull)
for PR 3884 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-68820376
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-68820371
[Test build #25084 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25084/consoleFull)
for PR 3884 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-68820897
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-68820890
[Test build #25085 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25085/consoleFull)
for PR 3884 at commit
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-68646490
Will this work for broadcast variables as well? One thing I often see is
users trying to directly broadcast an RDD without collecting it.
---
If your project is set up
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-68647489
[Test build #25036 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25036/consoleFull)
for PR 3884 at commit
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-68647672
@sryza Good idea; I've added a new check which prevents RDDs from being
directly broadcasted.
I should probably add these checks to PySpark, too. I'm not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-68650048
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-68668004
Haha, the `org.apache.spark.broadcast.BroadcastSuite.Using broadcast after
destroy prints callsite` test actually broadcasts an RDD (which is invalid),
which is what
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3884#discussion_r22447822
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -76,10 +76,22 @@ import org.apache.spark.util.random.{BernoulliSampler,
PoissonSampler,
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3884#discussion_r22447826
--- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala ---
@@ -897,4 +897,23 @@ class RDDSuite extends FunSuite with
SharedSparkContext {
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3884#discussion_r22447845
--- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala ---
@@ -897,4 +897,23 @@ class RDDSuite extends FunSuite with
SharedSparkContext {
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3884#discussion_r22447994
--- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala ---
@@ -897,4 +897,23 @@ class RDDSuite extends FunSuite with
SharedSparkContext {
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3884#discussion_r22448625
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -76,10 +76,22 @@ import org.apache.spark.util.random.{BernoulliSampler,
PoissonSampler,
GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/3884
[SPARK-5063] Useful error messages for nested RDDs and actions inside of
transformations
This patch adds more helpful error messages for invalid programs that
define nested RDDs and performs
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-68584486
[Test build #25005 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25005/consoleFull)
for PR 3884 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-68582760
[Test build #25005 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25005/consoleFull)
for PR 3884 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3884#issuecomment-68584489
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
59 matches
Mail list logo
