Takuya Ueshin created SPARK-4293: ------------------------------------ Summary: Make Cast be able to handle complex types. Key: SPARK-4293 URL: https://issues.apache.org/jira/browse/SPARK-4293 Project: Spark Issue Type: Improvement Components: SQL Reporter: Takuya Ueshin
Inserting data of type including {{ArrayType.containsNull == false}} or {{MapType.valueContainsNull == false}} or {{StructType.fields.exists(_.nullable == false)}} into Hive table will fail because {{Cast}} inserted by {{HiveMetastoreCatalog.PreInsertionCasts}} rule of {{Analyzer}} can't handle these types correctly. Complex type cast rule proposal: * Cast for non-complex types should be able to cast the same as before. * Cast for {{ArrayType}} can evaluate if ** Element type can cast ** Nullability rule doesn't break * Cast for {{MapType}} can evaluate if ** Key type can cast ** Nullability for casted key type is {{false}} ** Value type can cast ** Nullability rule for value type doesn't break * Cast for {{StructType}} can evaluate if ** The field size is the same ** Each field can cast ** Nullability rule for each field doesn't break * The nested structure should be the same. Nullability rule: * If the casted type is {{nullable == true}}, the target nullability should be {{true}} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org