Repository: spark Updated Branches: refs/heads/master 4d8803048 -> a4f4fbc8f
Include reference to twitter/chill in tuning docs Author: Andrew Ash <[email protected]> Closes #647 from ash211/doc-tuning and squashes the following commits: b87de0a [Andrew Ash] Include reference to twitter/chill in tuning docs Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a4f4fbc8 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a4f4fbc8 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a4f4fbc8 Branch: refs/heads/master Commit: a4f4fbc8fa5886a8c6ee58ee614de0cc6e67dcd7 Parents: 4d88030 Author: Andrew Ash <[email protected]> Authored: Mon Feb 24 21:13:38 2014 -0800 Committer: Reynold Xin <[email protected]> Committed: Mon Feb 24 21:13:38 2014 -0800 ---------------------------------------------------------------------- docs/tuning.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/a4f4fbc8/docs/tuning.md ---------------------------------------------------------------------- diff --git a/docs/tuning.md b/docs/tuning.md index 6b010ae..7047786 100644 --- a/docs/tuning.md +++ b/docs/tuning.md @@ -44,7 +44,10 @@ This setting configures the serializer used for not only shuffling data between nodes but also when serializing RDDs to disk. The only reason Kryo is not the default is because of the custom registration requirement, but we recommend trying it in any network-intensive application. -Finally, to register your classes with Kryo, create a public class that extends +Spark automatically includes Kryo serializers for the many commonly-used core Scala classes covered +in the AllScalaRegistrar from the [Twitter chill](https://github.com/twitter/chill) library. + +To register your own custom classes with Kryo, create a public class that extends [`org.apache.spark.serializer.KryoRegistrator`](api/core/index.html#org.apache.spark.serializer.KryoRegistrator) and set the `spark.kryo.registrator` config property to point to it, as follows: @@ -72,8 +75,8 @@ If your objects are large, you may also need to increase the `spark.kryoserializ config property. The default is 2, but this value needs to be large enough to hold the *largest* object you will serialize. -Finally, if you don't register your classes, Kryo will still work, but it will have to store the -full class name with each object, which is wasteful. +Finally, if you don't register your custom classes, Kryo will still work, but it will have to store +the full class name with each object, which is wasteful. # Memory Tuning
