Spark is currently on a fairly dated version of Kryo 2.x; it's trailing on the fixes in Hive and, as the APIs are incompatible, resulted in that mutant spark-project/hive JAR needed for the Hive 1.2.1 support
But: updating it hasn't been an option, because Spark needs to be in sync with Twitter's Chill library. There's now an offer from Twitter to help coordinate a kryo update across Chill, Scalding and other things they use https://github.com/twitter/chill/pull/230 Given kryo is "The guava jar of serialization", I doubt anyone is jumping up and down wanting this, but it is something to consider. Once hive moves to it, all the hive spark integration is probably going to break again; getting in sync with hive (see SPARK-10793) would reduce the traumaticness of hive updates
