GitHub user ueshin opened a pull request:
https://github.com/apache/spark/pull/2032
[WIP][SPARK-3036][SPARK-3037][SQL] Add MapType/ArrayType containing null
value support to Parquet.
JIRA:
- https://issues.apache.org/jira/browse/SPARK-3036
- https://issues.apache.org/jira/browse/SPARK-3037
Currently this uses the following Parquet schema for `MapType` when
`valueContainsNull` is `true`:
```
message root {
optional group a (MAP) {
repeated group map (MAP_KEY_VALUE) {
required int32 key;
optional int32 value;
}
}
}
```
for `ArrayType` when `containsNull` is `true`:
```
message root {
optional group a (LIST) {
repeated group bag {
optional int32 array;
}
}
}
```
We have to think about compatibilities with older version of Spark or Hive
or others I mentioned in the JIRA issues.
Notice:
This PR is based on #1963 and #1889.
Please check them first.
/cc @marmbrus, @yhuai
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ueshin/apache-spark issues/SPARK-3036_3037
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2032.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2032
----
commit 8e38b539557c6e9694159a709810ed3526716625
Author: Takuya UESHIN <[email protected]>
Date: 2014-08-19T08:39:57Z
Merge branch 'issues/SPARK-3063' into issues/SPARK-3036_3037
Squashed commit of the following:
commit 3ba41f2c6336acda124c3363dd82eec5de4ec5ed
Merge: 4d7bae2 cd0720c
Author: Takuya UESHIN <[email protected]>
Date: Tue Aug 19 16:41:05 2014 +0900
Merge branch 'master' into issues/SPARK-3063
commit 4d7bae2911ba824fb9f981882669857d0f4e6753
Merge: 9321379 c77f406
Author: Takuya UESHIN <[email protected]>
Date: Mon Aug 18 14:45:25 2014 +0900
Merge branch 'master' into issues/SPARK-3063
commit 93213793ec8c19156e5a2867e833d12b6b3af84a
Merge: d8a900a cc36487
Author: Takuya UESHIN <[email protected]>
Date: Sat Aug 16 09:14:04 2014 +0900
Merge branch 'master' into issues/SPARK-3063
commit d8a900ad1408ea3662d2e43b2df24db485cd28e5
Author: Takuya UESHIN <[email protected]>
Date: Fri Aug 15 15:48:52 2014 +0900
Make ExistingRdd.convertToCatalyst be able to convert Map value.
commit 62989de35649f6e0aca0428f38d0cce0861c6268
Author: Takuya UESHIN <[email protected]>
Date: 2014-08-19T08:40:36Z
Merge branch 'issues/SPARK-2969' into issues/SPARK-3036_3037
Squashed commit of the following:
commit 24f1c5c4b483ab61adc937e0818b1fa144a4981d
Author: Takuya UESHIN <[email protected]>
Date: Tue Aug 12 19:41:10 2014 +0900
Change the default value of ArrayType.containsNull to true in Python
API.
commit 79f5b65c519299991db23baa6561400e9d76a599
Author: Takuya UESHIN <[email protected]>
Date: Tue Aug 12 19:40:39 2014 +0900
Change the default value of ArrayType.containsNull to true in Java API.
commit 7cd1a7aae693539543eac7d9cc1f7efc57b89d0d
Author: Takuya UESHIN <[email protected]>
Date: Tue Aug 12 17:10:03 2014 +0900
Fix json test failures.
commit 2cfb862c1d9161023c587a1a5b3d9294a4125b62
Author: Takuya UESHIN <[email protected]>
Date: Tue Aug 12 15:13:15 2014 +0900
Change the default value of ArrayType.containsNull to true.
commit 2f38e612f4d4606117e0b97914b660d070b53f01
Author: Takuya UESHIN <[email protected]>
Date: Tue Aug 12 15:06:39 2014 +0900
Revert the default value of MapTypes.valueContainsNull.
commit 9fa02f5952ff729ef7de2d86125b1efc77313ce0
Author: Takuya UESHIN <[email protected]>
Date: Mon Aug 11 23:34:42 2014 +0900
Fix a test failure.
commit 1a9a96bf752dc33d19cb3654e05bc35c28b9ae08
Author: Takuya UESHIN <[email protected]>
Date: Mon Aug 11 19:46:24 2014 +0900
Modify ScalaReflection to handle ArrayType.containsNull and
MapType.valueContainsNull.
commit 013c2ca51439210f1dfb8e03f604be3bf235d892
Author: Takuya UESHIN <[email protected]>
Date: 2014-08-15T05:51:02Z
Add MapType containing null value support to Parquet.
commit 4e8e9e769d45b662bfa2eba8a67b386f70831da6
Author: Takuya UESHIN <[email protected]>
Date: 2014-08-15T10:12:11Z
Add ArrayType containing null value support to Parquet.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
