[
https://issues.apache.org/jira/browse/SPARK-13155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15132039#comment-15132039
]
Jia Li commented on SPARK-13155:
--------------------------------
[~cloud_fan] I was able to reproduce this bug on an older code base. But after
I refreshed to the latest code just now, it is now throwing this exception
below. Is it already fixed?
scala> Seq(("a", Seq(null, new Integer(1)))).toDS().as[(String,
Array[Int])].collect()
org.apache.spark.sql.AnalysisException: cannot resolve 'cast(_2 as array<int>)'
due to data type mismatch: cannot cast ArrayType(IntegerType,true) to
ArrayType(IntegerType,false);
at
org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:65)
> add runtime null check when convert catalyst array to external array
> --------------------------------------------------------------------
>
> Key: SPARK-13155
> URL: https://issues.apache.org/jira/browse/SPARK-13155
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Wenchen Fan
>
> {code}
> scala> Seq(("a", Seq(null, new Integer(1)))).toDS().as[(String,
> Array[Int])].collect()
> res5: Array[(String, Array[Int])] = Array((a,Array(0, 1)))
> {code}
> This is wrong, we should throw exception for this case
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
