Agree, I think this can / should be fixed with a slightly more conservative version of https://github.com/apache/spark/pull/3938 related to SPARK-5108.
On Sun, Jan 18, 2015 at 3:41 PM, Ted Yu <yuzhih...@gmail.com> wrote: > Please tale a look at SPARK-4048 and SPARK-5108 > > Cheers > > On Sat, Jan 17, 2015 at 10:26 PM, Gil Vernik <g...@il.ibm.com> wrote: > >> Hi, >> >> I took a source code of Spark 1.2.0 and tried to build it together with >> hadoop-openstack.jar ( To allow Spark an access to OpenStack Swift ) >> I used Hadoop 2.6.0. >> >> The build was fine without problems, however in run time, while trying to >> access "swift://" name space i got an exception: >> java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass >> at >> >> org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findDeserializationType(JacksonAnnotationIntrospector.java:524) >> at >> >> org.codehaus.jackson.map.deser.BasicDeserializerFactory.modifyTypeByAnnotation(BasicDeserializerFactory.java:732) >> ...and the long stack trace goes here >> >> Digging into the problem i saw the following: >> Jackson versions 1.9.X are not backward compatible, in particular they >> removed JsonClass annotation. >> Hadoop 2.6.0 uses jackson-asl version 1.9.13, while Spark has reference to >> older version of jackson. >> >> This is the main pom.xml of Spark 1.2.0 : >> >> <dependency> >> <!-- Matches the version of jackson-core-asl pulled in by avro --> >> <groupId>org.codehaus.jackson</groupId> >> <artifactId>jackson-mapper-asl</artifactId> >> <version>1.8.8</version> >> </dependency> >> >> Referencing 1.8.8 version, which is not compatible with Hadoop 2.6.0 . >> If we change version to 1.9.13, than all will work fine and there will be >> no run time exceptions while accessing Swift. The following change will >> solve the problem: >> >> <dependency> >> <!-- Matches the version of jackson-core-asl pulled in by avro --> >> <groupId>org.codehaus.jackson</groupId> >> <artifactId>jackson-mapper-asl</artifactId> >> <version>1.9.13</version> >> </dependency> >> >> I am trying to resolve this somehow so people will not get into this >> issue. >> Is there any particular need in Spark for jackson 1.8.8 and not 1.9.13? >> Can we remove 1.8.8 and put 1.9.13 for Avro? >> It looks to me that all works fine when Spark build with jackson 1.9.13, >> but i am not an expert and not sure what should be tested. >> >> Thanks, >> Gil Vernik. >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
