hello,
i hope this is the right place for this question.
i'm currently experimenting and comparing flink/stratosphere and apache
spark.
my goal is to analyse large json-files of twitter-data and now i'm
looking for a way to parse the json-tuples in a map-function and put in
a dataset.
for this i'm using the flink scala api and json4s.
but in flink the problem is to parse the json-file.
val words = cleaned.map { ( line => parse(line) }
Error Message is:
Error analyzing UDT org.json4s.JValue: Subtype
org.json4s.JsonAST.JInt - Field num: BigInt - Unsupported type BigInt
Subtype
org.json4s.JsonAST.JArray - Field arr:
List[org.json4s.JsonAST.JValue] - Subtype org.json4s.JsonAST.JInt -
Field num: BigInt - Unsupported type BigInt Subtype
org.json4s.JsonAST.JArray - Field arr:
List[org.json4s.JsonAST.JValue] - Subtype org.json4s.JsonAST.JDecimal -
Field num: BigDecimal - Unsupported type
BigDecimal Subtype org.json4s.JsonAST.JDecimal - Field num:
BigDecimal - Unsupported type BigDecimal Subtype
org.json4s.JsonAST.JObject - Field obj:
List[(String, org.json4s.JsonAST.JValue)] - Field _2:
org.json4s.JsonAST.JValue - Subtype org.json4s.JsonAST.JInt - Field num:
BigInt - Unsupported type BigInt
Subtype org.json4s.JsonAST.JObject - Field obj: List[(String,
org.json4s.JsonAST.JValue)] - Field _2: org.json4s.JsonAST.JValue - Subtype
org.json4s.JsonAST.JDecimal - Field num: BigDecimal - Unsupported
type BigDecimal
in spark i found a way based on
https://gist.github.com/cotdp/b471cfff183b59d65ae1
val user_interest = lines.map(line => {parse(line)})
.map(json => {implicit lazy val formats =
org.json4s.DefaultFormats
val name = (json \
"name").extract[String]
val location_x = (json \ "location"
\ "x").extract[Double]
val location_y = (json \ "location"
\ "y").extract[Double]
val likes = (json \
"likes").extract[Seq[String]].map(_.toLowerCase()).mkString(";")
( UserInterest(name, location_x,
location_y, likes) )
})
this works fine in spark, but is it possible to do the same with flink?
kind regards,
norman
