Paul Rogers commented on DRILL-6223:

[~sachouche], schema changes in Drill are complex and we've all had the fun of 
thinking we understand them, only to find out that there are implicit rules in 
Drill that complicate things.

There are two cases to consider here.

First, a {{SELECT a, b, c}} (explicit) projection list. In this case, the 
*reader* is supposed to project only the requested columns (and fill in nulls 
for missing columns.) This is pretty simple for top-level column, but gets very 
tricky for nested columns.

It is important to be aware (for any future readers) that Drill does allow us 
to specify nested columns: {{SELECT m.a, m.b}}. But this does not do what you 
think, it does not return a map {{m}} with only columns {{a}} and {{b}}. 
Instead, it (possibly helpfully) projects the columns to the top level as 
{{EXPR$1}} and {{EXPR$2}}. There is no effective way to project a map as a map, 
and decide which columns. It is all or nothing.

In the explicit case, there is nothing for the Project operator to do, the 
reader will have done all the needed work to produce a schema that matches the 
project list.

Here, we must be aware of the fatal flaw in the "schema-free" idea as Drill 
implements it. We mentioned missing columns. Drill readers assign the type of 
{{Nullable INT}} to those. Suppose the query is {{SELECT a, b FROM `someday`}} 
in which file "p.parquet" has both columns (as {{VARCHAR}}) but file 
{{q.parquet}} has only {{a}}. Then, the reader for {{q}} will add a column 
{{b}} of type {{Nullable INT}}. This will cause an schema change exception 

This is a known, fundamental problem. Would be great if {{q}} could say, "Hey, 
I don't have column {{b}}. So, I'll give you a vector of unspecified type, you 
supply the type later." But, today, Drill does not work that way.

OK, that's the first case. Now onto the second: wildcard: {{SELECT *}}. In this 
case, the reader must return all columns from each file. Back to our two files: 
{{p}} returns {{(a, b)}}, {{q}} returns just {{(a)}}. What does the user expect?

Your change says the user expects only {{(a))}} (the change identifies {{b}} as 
dangling.) But, is that correct?

Historically Drill says that we expect schema evolution. In that environment, 
we want both columns, but we want the "dangling" columns to be filled with 
nulls where they are missing. That is, the proper result is {{(a, b)}}, with 
the {{b}} columns filled with nulls for records that came from {{q}} (the file 
without {{b}}).

The trick, of course, is that {{a}} may have a non-nullable type. (This is the 
issue that [~vitalii] was working on with DRILL-5970 a while back.)

Overall, I don't think we want to remove columns that are "dangling"; the user 
could never query files that have evolved if we do. Instead, we need to find a 
way to include the "dangling" columns, filling in missing values.

Now, why can I write so much about this? Much of the reader-side logic for this 
is implemented in the "batch size" set of changes that is slowly merging into 
master. The question here is touched up on in the 
(This work did not attempt to address downstream behavior, that is an open 
issue. But, it does try to do "schema smoothing" so that, if we read file {{p}} 
and {{q}} in the same reader, the reader will fill in the missing column {{b}} 
for file {[q}} using the type from file {{p}}. Obviously, this trick only works 
if the files are read in the same reader, in the order {{p}}, {{q}}.)

Rather than do this as a quiet bug fix, I suggest we write up a design for how 
we actually want Drill to behave. The only real way to get rational behavior is 
to define a schema up front so that all readers can coordinate, or the 
downstream operators know that they need to fill in columns and how to do so. 
That is a big task; one that has been put off for years, but the underlying 
conflict and ambiguities have not gone away.

For Parquet (only), we have a possible solution. The planner checks all the 
files (I believe, I think that is why planning can be slow. Let the planner 
figure out the union of all the queries, using information from, say, file 
{{p}} in our example to know what column to fill in for file {{q}}. Pass this 
information into the physical plan. Let the readers use it to "do the right 
thing." The work mentioned earlier anticipates this solution: it provides a 
schema hint mechanism to fill in missing columns. Right now, it only looks at 
the very faint hints obtained from the project list, but is intended to use 
full schema info.

> Drill fails on Schema changes 
> ------------------------------
>                 Key: DRILL-6223
>                 URL: https://issues.apache.org/jira/browse/DRILL-6223
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Execution - Relational Operators
>    Affects Versions: 1.10.0, 1.12.0
>            Reporter: salim achouche
>            Assignee: salim achouche
>            Priority: Major
>             Fix For: 1.14.0
> Drill Query Failing when selecting all columns from a Complex Nested Data 
> File (Parquet) Set). There are differences in Schema among the files:
>  * The Parquet files exhibit differences both at the first level and within 
> nested data types
>  * A select * will not cause an exception but using a limit clause will
>  * Note also this issue seems to happen only when multiple Drillbit minor 
> fragments are involved (concurrency higher than one)

This message was sent by Atlassian JIRA

Reply via email to