Thanks a lot Steve! On Mon, Oct 17, 2016 at 4:59 PM, Steve Loughran <ste...@hortonworks.com> wrote: > > On 17 Oct 2016, at 10:02, Prasun Ratn <prasun.r...@gmail.com> wrote: > > Hi > > I want to run some Spark applications with some changes in Kryo serializer. > > Please correct me, but I think I need to recompile spark (instead of > just the Spark applications) in order to use the newly built Kryo > serializer? > > I obtained Kryo 3.0.3 source and built it (mvn package install). > > Next, I took the source code for Spark 2.0.1 and built it (build/mvn > -X -DskipTests -Dhadoop.version=2.6.0 clean package) > > I then compiled the Spark applications. > > However, I am not seeing my Kryo changes when I run the Spark applications. > > > Kryo versions are very brittle. > > You'll > > -need to get an up to date/consistent version of Chill, which is where the > transitive dependency on Kryo originates > -rebuild spark depending on that chill release > > if you want hive integration, probably also rebuild Hive to be consistent > too; the main reason Spark has its own Hive version is that > Kryo version sharing. > > https://github.com/JoshRosen/hive/commits/release-1.2.1-spark2 > > Kryo has repackaged their class locations between versions. This lets the > versions co-exist, but probably also explains why your apps aren't picking > up the diffs. > > Finally, keep an eye on this github PR > > https://github.com/twitter/chill/issues/252 >
--------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
