Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-60036532
Thanks Sandy - I've merged this!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/789
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-59053860
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21726/consoleFull)
for PR 789 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-59066603
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21726/consoleFull)
for PR 789 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-59066609
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-58946608
Thanks, this looks good to me now, though there are some merge conflicts.
Mind rebasing it?
---
If your project is set up for it, you can reply to this email and have
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-57589288
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21175/consoleFull)
for PR 789 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-57589292
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-57590497
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-57592234
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-57592605
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21185/consoleFull)
for PR 789 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-57598297
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21185/consoleFull)
for PR 789 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-57598304
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-57572019
I think it's better to do both and explain that there might be problems.
Otherwise users will see this new API and perhaps be surprised that their old
registrator is no
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-57585666
Updated patch allows using both at the same time
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-57585682
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21175/consoleFull)
for PR 789 at commit
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/789#discussion_r18235800
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -140,6 +141,20 @@ class SparkConf(loadDefaults: Boolean) extends
Cloneable with Logging {
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-57370056
Updated the patch to address Patrick's comment
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-57370352
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21059/consoleFull)
for PR 789 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-57378750
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21059/consoleFull)
for PR 789 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-57378756
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-57380469
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-57381144
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21060/consoleFull)
for PR 789 at commit
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-57381401
Hey Sandy, correct me if I'm wrong, but it seems that in this version, if
you call registerKryoClasses but also set a custom registrator, the classes
passed to
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-57390625
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-57390619
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21060/consoleFull)
for PR 789 at commit
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-57395416
That's correct (documented this on the conf page). My thought was that we
could hit strange interactions, for example if the same class is registered
both with a custom
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/789#discussion_r18176335
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -140,6 +141,20 @@ class SparkConf(loadDefaults: Boolean) extends
Cloneable with Logging
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-57213367
Made a comment about naming but overall this is looking good. @mateiz want
to take a pass since you originally suggested this design? IMO it's much nicer
than what we
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/789#discussion_r18182436
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -140,6 +141,20 @@ class SparkConf(loadDefaults: Boolean) extends
Cloneable with Logging {
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-56864247
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/789#discussion_r18051448
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -140,6 +142,20 @@ class SparkConf(loadDefaults: Boolean) extends
Cloneable with
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-56865325
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20814/consoleFull)
for PR 789 at commit
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/789#discussion_r18051738
--- Diff:
core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
@@ -81,21 +80,20 @@ class KryoSerializer(conf: SparkConf)
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/789#discussion_r18051826
--- Diff:
core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
@@ -81,21 +80,20 @@ class KryoSerializer(conf: SparkConf)
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/789#discussion_r18051914
--- Diff: core/src/test/scala/org/apache/spark/SparkConfSuite.scala ---
@@ -133,4 +134,38 @@ class SparkConfSuite extends FunSuite with
LocalSparkContext {
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/789#discussion_r18051969
--- Diff: core/src/test/scala/org/apache/spark/SparkConfSuite.scala ---
@@ -133,4 +134,38 @@ class SparkConfSuite extends FunSuite with
LocalSparkContext {
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/789#discussion_r18052712
--- Diff:
graphx/src/main/scala/org/apache/spark/graphx/GraphKryoRegistrator.scala ---
@@ -30,8 +30,10 @@ import
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/789#discussion_r18052785
--- Diff: graphx/src/main/scala/org/apache/spark/graphx/GraphXUtils.scala
---
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-56867764
Hey @sryza this looks good. Creating a custom class is cumbersome if the
user is just trying things out, and now it becomes 1 extra line to do it. Will
the new API
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/789#discussion_r18053706
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -140,6 +142,20 @@ class SparkConf(loadDefaults: Boolean) extends
Cloneable with Logging {
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-56872052
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20814/consoleFull)
for PR 789 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-56872060
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/789#discussion_r18057942
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -140,6 +142,20 @@ class SparkConf(loadDefaults: Boolean) extends
Cloneable with
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-56880976
Thanks for taking a look @andrewor14. Will post a patch that addresses
your comments and makes sure things work in Java.
---
If your project is set up for it, you can
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-56893739
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20824/consoleFull)
for PR 789 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-56901511
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-56901503
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20824/consoleFull)
for PR 789 at commit
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-56736604
Updated patch adds the APIs discussed. It relies on a new property
spark.kryo.classesToRegister, which registerKryoClasses appends to. The change
also enables users to
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-56737503
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20764/
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-50430005
@sryza are you still looking at coming up with a more modular version of
this? If not, it would be good to close the PR for now.
---
If your project is set up for it, you
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-50431290
@mateiz I posted a couple ideas and was waiting on feedback. Any thoughts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-44369542
Putting this in KryoRegistrator/Serializer makes sense to me.
That's a good point about the modularity.
One way to deal with it would be to add a config
Github user mridulm commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-44400421
This is what we do currently.
Each module has its own registrator - and they are invoked in the order in
which they are registered to our uber registerator (since
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-44239090
BTW one other thing I don't fully like about this is that it's not modular
-- you need to have a single list of all your classes in one place, and you
can't for example
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-43179185
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-43182223
All automated tests passed.
Refer to this link for build results:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15022/
---
If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-4318
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-43179167
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
GitHub user sryza opened a pull request:
https://github.com/apache/spark/pull/789
SPARK-1813. Add a utility to SparkConf that makes using Kryo really easy
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sryza/spark
Github user ash211 commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-43388939
Alternative to adding to `SparkContext`'s API, how about an API along the
lines of:
```
Seq[Class[Any]] classes = ...
Github user mridulm commented on the pull request:
https://github.com/apache/spark/pull/789#issuecomment-43388853
I have elaborated in the jira, but I will repeat it again for clarity:
This is adding an api for a specific case - it assumes single serialization
type
62 matches
Mail list logo
