On 17 Oct 2016, at 10:02, Prasun Ratn <prasun.r...@gmail.com<mailto:prasun.r...@gmail.com>> wrote:
Hi I want to run some Spark applications with some changes in Kryo serializer. Please correct me, but I think I need to recompile spark (instead of just the Spark applications) in order to use the newly built Kryo serializer? I obtained Kryo 3.0.3 source and built it (mvn package install). Next, I took the source code for Spark 2.0.1 and built it (build/mvn -X -DskipTests -Dhadoop.version=2.6.0 clean package) I then compiled the Spark applications. However, I am not seeing my Kryo changes when I run the Spark applications. Kryo versions are very brittle. You'll -need to get an up to date/consistent version of Chill, which is where the transitive dependency on Kryo originates -rebuild spark depending on that chill release if you want hive integration, probably also rebuild Hive to be consistent too; the main reason Spark has its own Hive version is that Kryo version sharing. https://github.com/JoshRosen/hive/commits/release-1.2.1-spark2 Kryo has repackaged their class locations between versions. This lets the versions co-exist, but probably also explains why your apps aren't picking up the diffs. Finally, keep an eye on this github PR https://github.com/twitter/chill/issues/252
