Max Kießling created SPARK-23610:
------------------------------------
Summary: Cast of ArrayType of NullType to ArrayType of nullable
material type does not work
Key: SPARK-23610
URL: https://issues.apache.org/jira/browse/SPARK-23610
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.2.0
Reporter: Max Kießling
Given a DataFrame that contains a column with _ArrayType of NullType_
casting this column into ArrayType of any material nullable type (e.g.
_ArrayType(LongType, true)_ ) should be possible.
{code}
it("can cast arrays of null type into arrays of nullable material types") {
val inputData = Seq(
Row(Array())
).asJava
val schema = StructType(Seq(
StructField("list", ArrayType(NullType, true), false)
))
val data = caps.sparkSession.createDataFrame(inputData, schema)
data.withColumn("longList",data.col("list").cast(ArrayType(LongType,
true))).show
}
{code}
This test fails with the message:
{noformat}
NullType (of class org.apache.spark.sql.types.NullType$)
scala.MatchError: NullType (of class org.apache.spark.sql.types.NullType$)
at org.apache.spark.sql.catalyst.expressions.Cast.castToLong(Cast.scala:310)
at
org.apache.spark.sql.catalyst.expressions.Cast.org$apache$spark$sql$catalyst$expressions$Cast$$cast(Cast.scala:516)
at org.apache.spark.sql.catalyst.expressions.Cast.castArray(Cast.scala:455)
at
org.apache.spark.sql.catalyst.expressions.Cast.org$apache$spark$sql$catalyst$expressions$Cast$$cast(Cast.scala:519)
at
org.apache.spark.sql.catalyst.expressions.Cast.cast$lzycompute(Cast.scala:531)
at org.apache.spark.sql.catalyst.expressions.Cast.cast(Cast.scala:531)
at org.apache.spark.sql.catalyst.expressions.Cast.nullSafeEval(Cast.scala:533)
at
org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:327)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
