Inline On Sun, Apr 8, 2018 at 12:36 PM, Aman Sinha <amansi...@apache.org> wrote:
> On Sun, Apr 8, 2018 at 10:57 AM, Ted Dunning <ted.dunn...@gmail.com> > wrote: > > > [cast pushdown is cheaper than you might think] > > It is true that the amount of work done by the planner would be about the > same as when > determining projection pushdowns into the scan. In my mind I was > contrasting with the > pure DDL based approach with an explicitly specified schema (such as with > a 'CREATE EXTERNAL TABLE ...' or with per query hints as Paul mentioned). > Actually, I think that these are equivalent in many ways. The only important difference is when the constraints on type are expressed. With DDL, it is expressed ahead of time. With cast-pushdown, it is done at the last moment in the query. With DDL, we traditionally assume that the reader has already taken the information into account before the query starts, but nothing really says that it has to. The information expressed in either case is essentially identical, however, and the reader should take heed. We are just allowing late typing if we add cast pushdown. ... keeping in mind that the same column may be > referenced in multiple ways: e.g CAST(a as varchar(10)), CAST(a as > varchar(20)) > in the same query/view. In such cases, we would want to either not do the > pushdown or determine the highest common datatype and push that down. > I disagree. It is the job of the reader to return both variants if it accepts the pushdown or return the raw data and not do any casting. This is really quite similar to cases where the column is retrieved twice. If the reader can do something clever by reading the data once with one case and then modifying the column after reading, that's great, but we can't really assume that it will. > > All of this, though, does not preclude the real need for the 'source of > truth' of the schema for the cases where data has been already explored and > curated. We do want to have a solution for that core issue. > Why is the answer different from a view with casts (that might be pushed down)?
