[
https://issues.apache.org/jira/browse/DRILL-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aman Sinha updated DRILL-3901:
------------------------------
Attachment: 0001-DRILL-3901-Don-t-do-early-expansion-of-directory-in-.patch
I've attached a minimal fix for the non-metadata-cache issue. This brings the
performance with 100K files back to the ballpark of the Sept 26 run.
The with-metadata-cache performance does not change with this fix and my
thought at this point is this is ok for 1.2 release because of the following
reasons:
- When the timing WithCache was measured for the master branch on
Sept 26 directory based partition pruning was not working in
conjunction with metadata cache.
- In order to fix this bug, DRILL-3788 and subsequently DRILL-3887 does
an expansion of the directory using the cache file and this adds to the
elapsed time. It is possible that we may be able to optimize this in a
subsequent release.
> Performance regression with doing Explain of COUNT(*) over 100K files
> ---------------------------------------------------------------------
>
> Key: DRILL-3901
> URL: https://issues.apache.org/jira/browse/DRILL-3901
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.1.0
> Reporter: Aman Sinha
> Assignee: Mehant Baid
> Attachments:
> 0001-DRILL-3901-Don-t-do-early-expansion-of-directory-in-.patch
>
>
> We are seeing a performance regression when doing an Explain of SELECT
> COUNT(*) over 100K files in a flat directory (no subdirectories) on latest
> master branch compared to a run that was done on Sept 26. Some initial
> details (I will have more later):
> {code}
> master branch on Sept 26
> No metadata cache: 71.452 secs
> With metadata cache: 15.804 secs
> Latest master branch
> No metadata cache: 110 secs
> With metadata cache: 32 secs
> {code}
> So, both cases show regression.
> [~mehant] and I took an initial look at this and it appears we might be doing
> the directory expansion twice.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
