Hello Andy,
Will our JSON support in Spark SQL help your case? If your JSON files store
one JSON object per line, you can use SQLContext.jsonFile to load it. If
you want to do pre-process these files, once you have an RDD[String] (one
JSON object per String), you can use SQLContext.jsonRDD. In both ways,
Spark SQL will infer the schema and you can query your datasets like
querying tables. The instruction about using jsonFile and jsonRDD can be
found at
https://spark.apache.org/docs/latest/sql-programming-guide.html#json-datasets
.
Thanks,
Yin
On Thu, Sep 18, 2014 at 2:36 PM, Andy Davidson <
a...@santacruzintegration.com> wrote:
> After lots of hacking I figure out how to resolve this problem. This is
> good solution. It severalty cripples jackson but at least for now I am
> unblocked
>
> 1) turn off annotations.
> mapper.configure(Feature.USE_ANNOTATIONS, false);
>
>
> 2) in maven set the jackson dependencies as provided.
>
> <properties>
>
> <jackson.version>1.9.13</jackson.version> <!-- 1.8.8 1.9.13 2.3.4-Final
> -->
>
> </properties>
>
>
> <dependencies>
>
> <dependency>
>
> <groupId>org.codehaus.jackson</groupId>
>
> <artifactId>jackson-core-asl</artifactId>
>
> <scope>provided</scope>
>
> <version>${jackson.version}</version>
>
> </dependency>
>
> <dependency>
>
> <groupId>org.codehaus.jackson</groupId>
>
> <artifactId>jackson-mapper-asl</artifactId>
>
> <scope>provided</scope>
>
> <version>${jackson.version}</version>
>
> </dependency>
>
>
> Hope this helps someone else
>
>
> andy
>
>
> From: Andrew Davidson <a...@santacruzintegration.com>
> Date: Wednesday, September 17, 2014 at 4:28 PM
> To: "user@spark.apache.org" <user@spark.apache.org>
> Subject: spark-1.1.0-bin-hadoop2.4 java.lang.NoClassDefFoundError:
> org/codehaus/jackson/annotate/JsonClass
>
> Hi I am new to spark.
>
> I am trying to write a simple java program that process tweets that where
> collected and stored in a file. I figured the simplest thing to do would be
> to convert the JSON string into a java map. When I submit my jar file I
> keep getting the following error
>
> java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass
>
> For the life of me I can not figure out what the problem is. I am using
> maven shade plugin and check to see that the missing class is my uber jar
>
> Any suggestions would be greatly appreciated.
>
> Andy
>
> P.s. I should mention that I am running everything on my local machine.
>
> <properties>
>
> <jackson.version>1.8.8</jackson.version> <!-- 1.9.13 -->
>
> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
>
> </properties>
>
>
> <dependencies>
>
> <dependency>
>
> <groupId>org.codehaus.jackson</groupId>
>
> <artifactId>jackson-core-asl</artifactId>
>
> <version>${jackson.version}</version>
>
> </dependency>
>
> <dependency>
>
> <groupId>org.codehaus.jackson</groupId>
>
> <artifactId>jackson-mapper-asl</artifactId>
>
> <version>${jackson.version}</version>
>
> </dependency>
>
>
> I am using shade to build an uber jar
>
>
> $jar –tvf target/myUber.jar
>
> …
>
> 580 Wed Sep 17 16:17:36 PDT 2014
> org/codehaus/jackson/annotate/JsonClass.class
> …
>
>
> 14/09/17 16:13:24 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID
> 0)
>
> java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass
>
> at
> org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findDeserializationType(JacksonAnnotationIntrospector.java:524)
>
> at
> org.codehaus.jackson.map.deser.BasicDeserializerFactory.modifyTypeByAnnotation(BasicDeserializerFactory.java:732)
>
> at
> org.codehaus.jackson.map.deser.BasicDeserializerFactory.createMapDeserializer(BasicDeserializerFactory.java:337)
>
> at
> org.codehaus.jackson.map.deser.StdDeserializerProvider._createDeserializer(StdDeserializerProvider.java:377)
>
> at
> org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdDeserializerProvider.java:307)
>
> at
> org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueDeserializer(StdDeserializerProvider.java:287)
>
> at
> org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer(StdDeserializerProvider.java:136)
>
> at
> org.codehaus.jackson.map.deser.StdDeserializerProvider.findTypedValueDeserializer(StdDeserializerProvider.java:157)
>
> at
> org.codehaus.jackson.map.ObjectMapper._findRootDeserializer(ObjectMapper.java:2468)
>
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2402)
>
> at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1609)
>
> at com.santacruzintegration.spark.JSONHelper.toJsonMap(JSONHelper.java:40)
>
>
>
> public class JSONHelper {
>
> public static final ObjectMapper mapper = new ObjectMapper();
>
> static {
>
> mapper.configure(Feature.FAIL_ON_UNKNOWN_PROPERTIES, false);
>
> //mapper.configure(Feature.USE_ANNOTATIONS, false);
>
> }
>
> …
>
> public static Map<String, String> toJsonMap(String json) {
>
> System.err.println("AEDWIP toJsonMap: " + json);
>
> try {
>
> TypeReference<HashMap<String, String>> typeRef = new
> TypeReference<HashMap<String, String>>() {
>
> };
>
> //return mapper.readValue(json, new
> TypeReference<HashMap<String,String>>(){});
>
> return mapper.readValue(json, typeRef);
>
> //return mapper.convertValue(json, typeRef);
>
>
>
> // HashMap<String,String> ret =
>
> // new ObjectMapper().readValue(json, HashMap.class);
>
> } catch (Exception e) {
>
> throw new IOError(e);
>
> }
>
> }
>
>
>
>
>
>
