Dechang Gu created DRILL-4827:
---------------------------------
Summary: Checking modification time of directories takes too long,
needs to be improved
Key: DRILL-4827
URL: https://issues.apache.org/jira/browse/DRILL-4827
Project: Apache Drill
Issue Type: Improvement
Components: Functions - Drill
Affects Versions: 1.8.0
Environment: RHEL 6
Reporter: Dechang Gu
This is tracking bug for metadata cache performance for directory checking.
When evaluating the fix for Drill-4530, we run the following two queries on 50K
parquet files in a 3-layer directory hierarchy:
Query1: explain plan for select * from
dfs.`/tpchMetaParquet/tpch100_dir_partitioned_50000files/lineitem` where
dir0=2006 and dir1=12 and dir2=15;
Query2: explain plan for select * from
dfs.`/tpchMetaParquet/tpch100_dir_partitioned_50000files/lineitem/2006/12/15`;
Query1 takes 3.254 secs. Query2 0.505 secs.
Drillbit.log shows that for Query1, 2.5 secs spent after metadata cache was
read and before partition pruning:
2016-08-02 15:43:43,051 ucs-node7.perf.lab
[285edddf-b1f3-cd74-e826-84cb91ebc6e1:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query id
285edddf-b1f3-cd74-e826-84cb91ebc6e1: explain plan for select * from
dfs.`/tpchMetaParquet/tpch100_dir_partitioned_50000files/lineitem` where
dir0=2006 and dir1=12 and dir2=15
2016-08-02 15:43:43,193 ucs-node7.perf.lab
[285edddf-b1f3-cd74-e826-84cb91ebc6e1:foreman] INFO
o.a.d.exec.store.parquet.Metadata - Took 6 ms to read directories from
directory cache file
2016-08-02 15:43:45,745 ucs-node7.perf.lab
[285edddf-b1f3-cd74-e826-84cb91ebc6e1:foreman] INFO
o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning
class:
org.apache.drill.exec.planner.logical.partition.PruneScanRule$DirPruneScanFilterOnScanRule
Further investigation shows that the 2.5 secs was for checking modification
time of directories, which is proportional to the number of directories to be
checked.
Looks like this can be improved by only checking the top level directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
