[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-30 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47498509 Thanks! I've merged this in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-30 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/143 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-29 Thread willb
Github user willb commented on a diff in the pull request: https://github.com/apache/spark/pull/143#discussion_r14329726 --- Diff: core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala --- @@ -153,6 +153,18 @@ private[spark] object ClosureCleaner extends Logging {

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47458467 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47458471 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47459527 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16243/ --- If your project is set up for it, you can

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47459526 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-29 Thread willb
Github user willb commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47459912 Sorry, I missed FailureSuite. I have a fix but ran out of battery before I could push. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47460336 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47460329 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47462979 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47462980 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16244/ --- If your project

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/143#discussion_r14331097 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala --- @@ -533,7 +533,7 @@ abstract class DStream[T: ClassTag] ( * on

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/143#discussion_r14331096 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala --- @@ -533,7 +533,7 @@ abstract class DStream[T: ClassTag] ( * on

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47492104 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47492111 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47495495 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47495498 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16254/ --- If your project

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/143#discussion_r14319546 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1205,8 +1205,8 @@ class SparkContext(config: SparkConf) extends Logging { *

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/143#discussion_r14319569 --- Diff: core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala --- @@ -153,6 +153,18 @@ private[spark] object ClosureCleaner extends Logging {

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/143#discussion_r14319639 --- Diff: core/src/test/scala/org/apache/spark/serializer/ProactiveClosureSerializationSuite.scala --- @@ -0,0 +1,79 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-27 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47409823 Hi Will: I left some (minor) comments. Thanks for working on this! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-23 Thread willb
Github user willb commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-46860741 Can someone take another look at this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-46479898 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-46479916 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-46484462 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15878/ --- If your project is set up for it, you can

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-46484460 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-18 Thread willb
Github user willb commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-46498345 I just rebased this branch atop master so it could be tested again. I see that it failed under Jenkins. However, I am unable to reproduce the local metrics failure in my

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-18 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-46500308 That test is flaky and being fixed right now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-06-18 Thread willb
Github user willb commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-46500775 Thanks @rxin! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-43133585 Build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-05-15 Thread willb
Github user willb commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-42712135 I'm not able to reproduce the above failure locally (either on OS X or Linux). --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-05-15 Thread willb
GitHub user willb reopened a pull request: https://github.com/apache/spark/pull/143 SPARK-897: preemptively serialize closures These commits cause `ClosureCleaner.clean` to attempt to serialize the cleaned closure with the default closure serializer and throw a `SparkException`

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-43133587 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14991/ --- If your project

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-05-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-42707289 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14851/ --- If your project is set up for it, you can

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-43124951 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-43124937 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-42707287 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-05-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-42703898 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-05-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-42703885 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-05-11 Thread willb
Github user willb commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-42703799 I'd like to reopen this PR, since #189 had to be reverted and is still in flight, but these fixes are independently useful. I've rebased the branch atop the current

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-04-09 Thread willb
Github user willb commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-40035997 This is subsumed by #189. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-03-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-37891896 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-03-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-37894540 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13220/ --- If your project

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-03-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-37897176 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-897: preemptively serialize closures

2014-03-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-37897175 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not