[
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366929#comment-15366929
]
ASF GitHub Bot commented on DRILL-4530:
---------------------------------------
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/519#discussion_r70002787
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/partition/PruneScanRule.java
---
@@ -269,13 +283,54 @@ protected void doOnMatch(RelOptRuleCall call, Filter
filterRel, Project projectR
int recordCount = 0;
int qualifiedCount = 0;
- // Inner loop: within each batch iterate over the
PartitionLocations
- for(PartitionLocation part: partitions){
- if(!output.getAccessor().isNull(recordCount) &&
output.getAccessor().get(recordCount) == 1){
- newPartitions.add(part);
- qualifiedCount++;
+ if (checkForSingle &&
+ partitions.get(0).isCompositePartition() /* apply single
partition check only for composite partitions */) {
+ // Inner loop: within each batch iterate over the
PartitionLocations
+ for (PartitionLocation part : partitions) {
--- End diff --
@jinfengni I started doing the refactoring to move this particular
optimization into a partition descriptor specific class. However, I was having
to propagate several internal states to the doSinglePartOpt() method...such as
partition map, the referenced dirs bitset , the value vector array and others.
Also, since this optimization occurs inside an outer loop, it is not
straightforward without doing a broader restructuring of the
PruneScanRule.doOnMatch() method. Would you be ok to defer this refactoring to
an enhancement JIRA ?
> Improve metadata cache performance for queries with single partition
> ---------------------------------------------------------------------
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
> Affects Versions: 1.6.0
> Reporter: Aman Sinha
> Assignee: Aman Sinha
> Fix For: Future
>
>
> Consider two types of queries which are run with Parquet metadata caching:
> {noformat}
> query 1:
> SELECT col FROM `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2
> elapsed time is 9 sec even though both are accessing the same amount of data.
> The user expectation is that they should perform roughly the same. The main
> difference comes from reading the bigger metadata cache file at the root
> level 'A' for query2 and then applying the partitioning filter. query1 reads
> a much smaller metadata cache file at the subdirectory level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
