GitHub user marmbrus opened a pull request:
https://github.com/apache/spark/pull/3269
[WIP][SPARK-4413][SQL] Parquet support through datasource API
**This is a initial draft intended only for early feedback**
Goals:
- Support for accessing parquet using SQL but not requiring Hive
- Support for folder based partitioning with automatic discovery of
available partitions
- Caching of file metadata
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/marmbrus/spark newParquet
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3269.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3269
----
commit 77e845ca7726ffee2d6f8e33ea56ec005dde3874
Author: Michael Armbrust <[email protected]>
Date: 2014-11-14T20:00:08Z
[SPARK-4394][SQL] Data Sources API Improvements
This PR adds two features to the data sources API:
- Support for pushing down `IN` filters
- The ability for relations to optionally provide information about their
`sizeInBytes`.
Author: Michael Armbrust <[email protected]>
Closes #3260 from marmbrus/sourcesImprovements and squashes the following
commits:
9a5e171 [Michael Armbrust] Use method instead of configuration directly
99c0e6b [Michael Armbrust] Add support for sizeInBytes.
416f167 [Michael Armbrust] Support for IN in data sources API.
2a04ab3 [Michael Armbrust] Simplify implementation of InSet.
commit 9bd3c859729887eb0787cf5943ea15f868c4766d
Author: Michael Armbrust <[email protected]>
Date: 2014-11-14T20:36:26Z
First draft.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
