[
https://issues.apache.org/jira/browse/DRILL-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952326#comment-14952326
]
ASF GitHub Bot commented on DRILL-3918:
---------------------------------------
GitHub user amansinha100 opened a pull request:
https://github.com/apache/drill/pull/196
DRILL-3918: During expansion save the metadata for future use.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/amansinha100/incubator-drill
refresh_metadata_bugs_1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/196.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #196
----
commit 2f261d97f4f9eb8f13479a82b1e39028484fc49a
Author: Aman Sinha <[email protected]>
Date: 2015-10-11T15:02:30Z
DRILL-3918: During expansion save the metadata for future use.
----
> Avoid extra loading of the metadata cache file
> ----------------------------------------------
>
> Key: DRILL-3918
> URL: https://issues.apache.org/jira/browse/DRILL-3918
> Project: Apache Drill
> Issue Type: Bug
> Components: Metadata
> Reporter: Aman Sinha
>
> The metadata cache file is currently being deserialized and read twice: once
> during {{ParquetFormatPlugin.expandSelection()}} that happens as part of the
> creation of DynamicDrillTable and once during ParquetGroupScan. This was
> also pointed out by [~sphillips] in DRILL-3901. We should avoid doing the
> read twice.
> The performance issue is getting exposed more now because of the fix for
> DRILL-3917 which fixed the behavior of expandSelection() by reading the
> metadata cache file through the correct interface (it was previously erroring
> out and not spending any time in the expansion). This fix is needed for
> correct functionality. However, performance numbers show a slowdown of
> about 2.7x for the 400K files test using caching. In my view, this
> performance comparison is not very meaningful because of the prior bug.
> This JIRA is to specifically targeting the extra load of the metadata cache
> file. There are other opportunities for improvement (for instance reading
> from the metadata cache is single threaded whereas reading from parquet files
> gets parallelized. That should be a separate JIRA).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
