On 17 Oct 2016, at 10:02, Prasun Ratn 
<prasun.r...@gmail.com<mailto:prasun.r...@gmail.com>> wrote:


I want to run some Spark applications with some changes in Kryo serializer.

Please correct me, but I think I need to recompile spark (instead of
just the Spark applications) in order to use the newly built Kryo

I obtained Kryo 3.0.3 source and built it (mvn package install).

Next, I took the source code for Spark 2.0.1 and built it (build/mvn
-X -DskipTests -Dhadoop.version=2.6.0 clean package)

I then compiled the Spark applications.

However, I am not seeing my Kryo changes when I run the Spark applications.

Kryo versions are very brittle.


-need to get an up to date/consistent version of Chill, which is where the 
transitive dependency on Kryo originates
-rebuild spark depending on that chill release

if you want hive integration, probably also rebuild Hive to be consistent too; 
the main reason Spark has its own Hive version is that
Kryo version sharing.


Kryo has repackaged their class locations between versions. This lets the 
versions co-exist, but probably also explains why your apps aren't picking up 
the diffs.

Finally, keep an eye on this github PR


Reply via email to