Takuya Ueshin created SPARK-4293:
------------------------------------
Summary: Make Cast be able to handle complex types.
Key: SPARK-4293
URL: https://issues.apache.org/jira/browse/SPARK-4293
Project: Spark
Issue Type: Improvement
Components: SQL
Reporter: Takuya Ueshin
Inserting data of type including {{ArrayType.containsNull == false}} or
{{MapType.valueContainsNull == false}} or {{StructType.fields.exists(_.nullable
== false)}} into Hive table will fail because {{Cast}} inserted by
{{HiveMetastoreCatalog.PreInsertionCasts}} rule of {{Analyzer}} can't handle
these types correctly.
Complex type cast rule proposal:
* Cast for non-complex types should be able to cast the same as before.
* Cast for {{ArrayType}} can evaluate if
** Element type can cast
** Nullability rule doesn't break
* Cast for {{MapType}} can evaluate if
** Key type can cast
** Nullability for casted key type is {{false}}
** Value type can cast
** Nullability rule for value type doesn't break
* Cast for {{StructType}} can evaluate if
** The field size is the same
** Each field can cast
** Nullability rule for each field doesn't break
* The nested structure should be the same.
Nullability rule:
* If the casted type is {{nullable == true}}, the target nullability should be
{{true}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
