Niek Bartholomeus created SPARK-17809:
-----------------------------------------
Summary: scala.MatchError: BooleanType when casting a struct
Key: SPARK-17809
URL: https://issues.apache.org/jira/browse/SPARK-17809
Project: Spark
Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Niek Bartholomeus
I have a Dataframe with a struct and I need to rename some fields to lower case
before saving it to cassandra.
It turns out that it's not possible to cast a boolean field of a struct to
another boolean field in the renamed struct:
{quote}
case class ClassWithBoolean(flag: Boolean)
case class Parent(cwb: ClassWithBoolean)
val structCwb: DataType = StructType(Seq(
StructField("flag", BooleanType, true)
))
Seq(Parent(ClassWithBoolean(true)))
.toDF
.withColumn("cwb", $"cwb".cast(structCwb))
.collect
scala.MatchError: BooleanType (of class org.apache.spark.sql.types.BooleanType$)
{quote}
A workaround is to temporarily cast the field to an Integer and back:
{quote}
val structCwbTmp: DataType = StructType(Seq(
StructField("flag", IntegerType, true)
))
Seq(Parent(ClassWithBoolean(true)))
.toDF
.withColumn("cwb", $"cwb".cast(structCwbTmp))
.withColumn("cwb", $"cwb".cast(structCwb))
.collect
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
