GitHub user ueshin opened a pull request:
https://github.com/apache/spark/pull/3150
[SPARK-4293][SQL] Make Cast be able to handle complex types.
Inserting data of type including `ArrayType.containsNull == false` or
`MapType.valueContainsNull == false` or `StructType.fields.exists(_.nullable ==
false)` into Hive table will fail because `Cast` inserted by
`HiveMetastoreCatalog.PreInsertionCasts` rule of `Analyzer` can't handle these
types correctly.
Complex type cast rule proposal:
- Cast for non-complex types should be able to cast the same as before.
- Cast for `ArrayType` can evaluate if
- Element type can cast
- Nullability rule doesn't break
- Cast for `MapType` can evaluate if
- Key type can cast
- Nullability for casted key type is `false`
- Value type can cast
- Nullability rule for value type doesn't break
- Cast for `StructType` can evaluate if
- The field size is the same
- Each field can cast
- Nullability rule for each field doesn't break
- The nested structure should be the same.
Nullability rule:
- If the casted type is `nullable == true`, the target nullability should
be `true`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ueshin/apache-spark issues/SPARK-4293
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3150.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3150
----
commit 4f71bb8e4fa83f160f0f131c9b60bca911acef99
Author: Takuya UESHIN <[email protected]>
Date: 2014-11-07T05:13:11Z
Make Cast be able to handle complex types.
commit 287f410329edf375c8d6142ea2400aa75537da5f
Author: Takuya UESHIN <[email protected]>
Date: 2014-11-07T05:13:38Z
Add tests to insert data of types ArrayType / MapType / StructType with
nullability is false into Hive table.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
