[
https://issues.apache.org/jira/browse/SPARK-16792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15735476#comment-15735476
]
Michal Šenkýř commented on SPARK-16792:
---------------------------------------
Found a simpler example:
{code:none}
case class X(l: List[String]) // works for Seq
spark.createDataset(Seq(List("A"))).map(X).show
{code}
Exception thrown:
{code:none}
Caused by: java.lang.ClassCastException:
scala.collection.mutable.WrappedArray$ofRef cannot be cast to
scala.collection.immutable.List
at X$.apply(<console>:11)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
Source)
at
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:370)
{code}
> Dataset containing a Case Class with a List type causes a CompileException
> (converting sequence to list)
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-16792
> URL: https://issues.apache.org/jira/browse/SPARK-16792
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: Jamie Hutton
> Priority: Critical
>
> The issue occurs when we run a .map over a dataset containing Case Class with
> a List in it. A self contained test case is below:
> case class TestCC(key: Int, letters: List[String]) //List causes the issue -
> a Seq/Array works fine
> /*simple test data*/
> val ds1 = sc.makeRDD(Seq(
> (List("D")),
> (List("S","H")),
> (List("F","H")),
> (List("D","L","L"))
> )).map(x=>(x.length,x)).toDF("key","letters").as[TestCC]
> //This will fail
> val test1=ds1.map{_.key}
> test1.show
> Error:
> Caused by: org.codehaus.commons.compiler.CompileException: File
> 'generated.java', Line 72, Column 70: No applicable constructor/method found
> for actual parameters "int, scala.collection.Seq"; candidates are:
> "TestCC(int, scala.collection.immutable.List)"
> It seems to be internally converting the List to a sequence, then it cant
> convert it back...
> If you change the List[String] to Seq[String] or Array[String] the issue
> doesnt appear
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
