GitHub user viirya opened a pull request:
https://github.com/apache/spark/pull/14045
[SPARK-16362][SQL] Support ArrayType and StructType in vectorization
Parquet reader
## What changes were proposed in this pull request?
Vectorization parquet reader now doesn't support nested types such as
ArrayType, MapType and StructType. We should support it. This patch is to add
ArrayType and StructType first.
## How was this patch tested?
Existing unit tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/viirya/spark-1 parquet-vectorized-column9
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14045.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14045
----
commit 38be47eed7987ac46fea9a17eda2400c704a1df3
Author: Liang-Chi Hsieh <[email protected]>
Date: 2016-06-22T14:05:11Z
Support ArrayType and StructType in vectorization parquet reader.
commit aa11162b1671ca5013dadf72a9c7274faec40008
Author: Liang-Chi Hsieh <[email protected]>
Date: 2016-07-04T09:38:41Z
Merge remote-tracking branch 'upstream/master' into
parquet-vectorized-column9
Conflicts:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVector.java
commit d5e5a605bf8c6b0bcd510f56dac583dd30629859
Author: Liang-Chi Hsieh <[email protected]>
Date: 2016-07-04T09:43:39Z
Remove commented code.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
