Github user lokm01 commented on the issue:
https://github.com/apache/spark/pull/21215
Hey @maropu,
So we've encountered a number of issues with casting:
1. Casting an empty array to an array of primitive types caused an
exception on 2.2.1, but works on 2.3.0+ so that's sorted
2. We're still facing an issue on 2.3.0 when we try to cast an empty array
to an array of complex types. See the following example:
`
case class Outer(a: List[Inner])
case class Inner(b: Int, c: String)
object App4 extends App {
val spark =
SparkSession.builder().appName("").master("local[*]").getOrCreate()
import spark.implicits._
import org.apache.spark.sql.functions._
val df = spark.createDataFrame(Seq[Outer]())
val r = spark.range(100).select(array().cast(df.schema("a").dataType))
r.printSchema()
r.show
}
`
This code produces
> Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot
resolve 'array()' due to data type mismatch: cannot cast array<string> to
array<struct<b:int,c:string>>;;
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
