On 17 Oct 2016, at 10:02, Prasun Ratn
I want to run some Spark applications with some changes in Kryo serializer.
Please correct me, but I think I need to recompile spark (instead of
just the Spark applications) in order to use the newly built Kryo
I obtained Kryo 3.0.3 source and built it (mvn package install).
Next, I took the source code for Spark 2.0.1 and built it (build/mvn
-X -DskipTests -Dhadoop.version=2.6.0 clean package)
I then compiled the Spark applications.
However, I am not seeing my Kryo changes when I run the Spark applications.
Kryo versions are very brittle.
-need to get an up to date/consistent version of Chill, which is where the
transitive dependency on Kryo originates
-rebuild spark depending on that chill release
if you want hive integration, probably also rebuild Hive to be consistent too;
the main reason Spark has its own Hive version is that
Kryo version sharing.
Kryo has repackaged their class locations between versions. This lets the
versions co-exist, but probably also explains why your apps aren't picking up
Finally, keep an eye on this github PR