John Omernik created DRILL-4379:
-----------------------------------
Summary: Unexpected Table Behavior with only one subdirectory vs.
Many
Key: DRILL-4379
URL: https://issues.apache.org/jira/browse/DRILL-4379
Project: Apache Drill
Issue Type: Bug
Components: Query Planning & Optimization
Affects Versions: 1.4.0
Reporter: John Omernik
A common practice is to use directories below a main directory as a
partitioning device. Say you have a table named "myawesomedata" and you get
data into that table every day, it would be valuable to create the main
directory, then subdirectories per day to help optimize queries running against
only certain days of data.
/myawesomedata/
/myawesomedata/2016-02-01
/myawesomedata/2016-02-02
/myawesomedata/2016-02-03
/myawesomedata/2016-02-04
I have identified a condition that if there is ONLY one subdirectory, queries
do not return results as expected by a user.
Example:
In the above, if I run a query of
select count(1) from `myawesomedata`;
I get accurate results of the count in all subdirectories
If I run:
select count(1) from `myawesomedata` where dir0 = '2016-02-01';
I get accurate results of the count of only the subdirectory 2016-02-01
However, if I delete subdirectories 2016-02-02, 2016-02-03, and 2016-02-04 and
am left with:
/myawesomedata/
/myawesomedata/2016-02-01
Then if I run
select count(1) from `myawesomedata`;
It returns the accurate count (which is just that of the 2016-02-01 directory).
However, if I run
select count(1) from `myawesomedata` where dir0 = '2016-02-01';
It takes much longer (15 seconds vs instant on the other queries) and returns
no results. Even though this is the same query as above that worked with 2 or
more subdirectories. Basically, when there is only one subdirectory, a query
asking for only that directory does not work in the same way as when there are
more subdirectories. This is an unexpected user experience and something I
believe could cause user frustration and unexpected results from Drill usage on
data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
