Hello Spark Devs, I am trying to use the new Spark API json endpoints at /api/v1/[path] (added in SPARK-3454).
In order to minimize maintenance on our end, I would like to use Retrofit/Jackson to parse the json directly into the Scala classes in org/apache/spark/status/api/v1/api.scala (ApplicationInfo, ApplicationAttemptInfo, etc…). However, Jackson does not seem to know how to handle Scala Seqs, and will throw an error when trying to parse the attempts: Seq[ApplicationAttemptInfo] field of ApplicationInfo. Our codebase is in Java. My questions are: 1. Do you have any recommendations on how to easily deserialize Scala objects from json? For example, do you have any current usage examples of SPARK-3454 with Java? 2. Alternatively, are you committed to the json formats of /api/v1/path? I would guess so, because of the ‘v1’, but wanted to confirm. If so, I could deserialize the json into instances of my own Java classes instead, without worrying about changing the class structure later due to changes in the Spark API. Some further information: * The error I am getting with Jackson when trying to deserialize the json into ApplicationInfo is Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not construct instance of scala.collection.Seq, problem: abstract types either need to be mapped to concrete types, have custom deserializer, or be instantiated with additional type information * I tried using Jackson’s DefaultScalaModule, which seems to have support for Scala Seqs, but got no luck. * Deserialization works if the Scala class does not have any Seq fields, and works if the fields are Java Lists instead of Seqs. Thanks very much for your help! Kevin Chen
smime.p7s
Description: S/MIME cryptographic signature
