[
https://issues.apache.org/jira/browse/DRILL-4857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15433770#comment-15433770
]
ASF GitHub Bot commented on DRILL-4857:
---------------------------------------
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/575#discussion_r75964268
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java
---
@@ -173,8 +173,18 @@ public ParquetGroupScan( //
final FileSelection fileSelection = expandIfNecessary(selection);
this.entries = Lists.newArrayList();
- for (String fileName : fileSelection.getFiles()) {
- entries.add(new ReadEntryWithPath(fileName));
+ if (fileSelection.getMetaContext() != null &&
+ (fileSelection.getMetaContext().wasPruningStarted() &&
+ ! fileSelection.getMetaContext().wasPruned())) {
+ // if pruning was attempted and nothing was pruned, initialize the
entries with just
+ // the selection root instead of the fully expanded list to reduce
overhead. The fully
+ // expanded list is already stored as part of the fileSet.
+ // TODO: at some point we should examine whether the list of entries
is absolutely needed.
--- End diff --
Maybe we log a JIRA for this TODO, so that we keep track of this issue?
Seems it makes sense to either use the list of entries, or fileSet, but not
both.
> When no partition pruning occurs with metadata caching there's a performance
> regression
> ---------------------------------------------------------------------------------------
>
> Key: DRILL-4857
> URL: https://issues.apache.org/jira/browse/DRILL-4857
> Project: Apache Drill
> Issue Type: Bug
> Components: Metadata, Query Planning & Optimization
> Affects Versions: 1.7.0
> Reporter: Aman Sinha
> Assignee: Aman Sinha
> Fix For: 1.8.0
>
>
> After DRILL-4530, we see the (expected) performance improvements in planning
> time with metadata cache for cases where partition pruning got applied.
> However, in cases where it did not get applied and for sufficiently large
> number of files (tested with up to 400K files), there's performance
> regression. Part of this was addressed by DRILL-4846. This JIRA is to
> track some remaining fixes to address the regression.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
