GitHub user mateiz opened a pull request:
https://github.com/apache/spark/pull/2005
[SPARK-3091] [SQL] Add support for caching metadata on Parquet files
For larger Parquet files, reading the file footers (which is done in
parallel on up to 5 threads) and HDFS block locations (which is serial) can
take multiple seconds. We can add an option to cache this data within
FilteringParquetInputFormat. Unfortunately ParquetInputFormat only caches
footers within each instance of ParquetInputFormat, not across them.
Note: this PR leaves this turned off by default for 1.1, but I believe it's
safe to turn it on after. The keys in the hash maps are FileStatus objects that
include a modification time, so this will work fine if files are modified. The
location cache could become invalid if files have moved within HDFS, but that's
rare so I just made it invalidate entries every 15 minutes.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mateiz/spark parquet-cache
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2005.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2005
----
commit 4094df611b88a14dda2d023e9d2d9f56225a40be
Author: Matei Zaharia <[email protected]>
Date: 2014-08-17T21:33:10Z
First attempt at caching Parquet footers
commit 453bd2181c25c78e5a32c6ceb2bafa662510191f
Author: Matei Zaharia <[email protected]>
Date: 2014-08-17T22:20:15Z
Bug fix
commit 8fb56cef7f5f8a03409c9645490dc7d9c27dbb44
Author: Matei Zaharia <[email protected]>
Date: 2014-08-17T22:24:51Z
Cache file block locations too
commit 22072b0583b14f1696fd99ec97b05bfff609972c
Author: Matei Zaharia <[email protected]>
Date: 2014-08-17T23:34:35Z
Use Guava caches and add a config option for caching metadata
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
