Hi Kevin, How did you try to use the Scala module? Spark has this code when setting up the ObjectMapper used to generate the output:
mapper.registerModule(com.fasterxml.jackson.module.scala.DefaultScalaModule) As for supporting direct serialization to Java objects, I don't think that was the goal of the API. The Scala API classes are public mostly so that API compatibility checks are performed against them. If you don't mind the duplication, you could write your own Java POJOs that mirror the Scala API, and use them to deserialize the JSON. On Tue, Sep 8, 2015 at 12:46 PM, Kevin Chen <kc...@palantir.com> wrote: > Hello Spark Devs, > > I am trying to use the new Spark API json endpoints at /api/v1/[path] > (added in SPARK-3454). > > In order to minimize maintenance on our end, I would like to use > Retrofit/Jackson to parse the json directly into the Scala classes in > org/apache/spark/status/api/v1/api.scala (ApplicationInfo, > ApplicationAttemptInfo, etc…). However, Jackson does not seem to know how to > handle Scala Seqs, and will throw an error when trying to parse the > attempts: Seq[ApplicationAttemptInfo] field of ApplicationInfo. Our codebase > is in Java. > > My questions are: > > Do you have any recommendations on how to easily deserialize Scala objects > from json? For example, do you have any current usage examples of SPARK-3454 > with Java? > Alternatively, are you committed to the json formats of /api/v1/path? I > would guess so, because of the ‘v1’, but wanted to confirm. If so, I could > deserialize the json into instances of my own Java classes instead, without > worrying about changing the class structure later due to changes in the > Spark API. > > Some further information: > > The error I am getting with Jackson when trying to deserialize the json into > ApplicationInfo is Caused by: > com.fasterxml.jackson.databind.JsonMappingException: Can not construct > instance of scala.collection.Seq, problem: abstract types either need to be > mapped to concrete types, have custom deserializer, or be instantiated with > additional type information > I tried using Jackson’s DefaultScalaModule, which seems to have support for > Scala Seqs, but got no luck. > Deserialization works if the Scala class does not have any Seq fields, and > works if the fields are Java Lists instead of Seqs. > > Thanks very much for your help! > Kevin Chen > -- Marcelo --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
