[
https://issues.apache.org/jira/browse/SPARK-19434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-19434.
-------------------------------
Resolution: Duplicate
There's not a great reason to separate this into another JIRA
> Dataframe/Dataset unserialization failing with Map
> --------------------------------------------------
>
> Key: SPARK-19434
> URL: https://issues.apache.org/jira/browse/SPARK-19434
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.2, 2.1.0
> Reporter: Loic Descotte
>
> If I serialize a type containing a scala Map into a dataframe, it will fail
> to unserialize it unless I flag the type as a collection.Map in the case
> class. Else it will make a mismatch between collection.Map (trait) and
> collection.immutable.Map (default implementation).
> {code}
> case class Person(name: String, details: Map[String, String])
> val peopleSeq = Seq(
> Person("bob", Map("age" -> "30", "address" -> "blabla")),
> Person("john", Map("age" -> "25", "address" -> "blabla"))
> )
> val peopleDS = peopleSeq.toDS
> peopleDS.collect should contain theSameElementsAs peopleSeq
> // failed to compile: org.codehaus.commons.compiler.CompileException:
> File 'generated.java',
> // Line 127, Column 40: No applicable constructor/method found for actual
> parameters "java.lang.String, scala.collection.Map";
> // candidates are: "Person(java.lang.String,
> scala.collection.immutable.Map)"
> {code}
> With this trick it is working :
> {code}
> case class PersonWithForcedMapType(name: String, details:
> collection.Map[String, String])
> val peopleSeq = Seq(
> PersonWithForcedMapType("bob", Map("age" -> "30", "address" ->
> "blabla")),
> PersonWithForcedMapType("john", Map("age" -> "25", "address" ->
> "blabla"))
> )
> val peopleDS = peopleSeq.toDS
> peopleDS.collect should contain theSameElementsAs peopleSeq //OK
> {code}
> It is the same if I don't use datasets API but dataframes instead :
> {code}
> val peopleSeq = Seq(
> Person("bob", Map("age" -> "30", "address" -> "blabla")),
> Person("john", Map("age" -> "25", "address" -> "blabla"))
> )
> val peopleDF = peopleSeq.toDF
> val peopleResult = peopleDF.map { row =>
> val name = row.getAs[String](0)
> val details = row.getAs[Map[String, String]](1)
> Person(name, details)
> }
> peopleResult.collect should contain theSameElementsAs peopleSeq
> // failed to compile: org.codehaus.commons.compiler.CompileException:
> File 'generated.java',
> // Line 127, Column 40: No applicable constructor/method found for actual
> parameters "java.lang.String, scala.collection.Map";
> // candidates are: "Person(java.lang.String,
> scala.collection.immutable.Map)"
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
