paul-rogers opened a new pull request #1683: DRILL-6952: Host complaint text reader on the row set framework URL: https://github.com/apache/drill/pull/1683 This PR hosts the "compliant" text reader on the new row-set-based scan framework. The version here is termed "V3". The existing "compliant" (AKA "new") text reader is termed "V2", with the "old" text reader assumed as "V1." (Likely no one has used the "old" "V1" version in years.) The scan framework is new and this will be the first commit that allows it to be used in a query outside of a unit test. Because of this, V3 is disabled by default. To enable it: ``` alter session set `exec.storage.enable_v3_text_reader` = true; ``` V3 introduces some fixes compared with V2: * When using the reader with headers, and rows omit some columns, V2 crashes but V3 works. * When using a SELECT * query with a partitioned file, the partition columns appear after the data columns. (In V2, since Drill 1.12, they appeared before data columns.) With partition columns after the data columns, positions don't shift if another partition level is added. * File metadata (AKA "implicit") columns are of type required VARCHAR in V3 (where nullable VARCHAR in V2.) * V3 fixes DRILL-7083: if a query explicitly requests "dir3" say, but the data only has two levels of partition, V2 provide columns as Nullable INT while V3 provides them as nullable Varchar (same as "valid" partition columns.) * V3 fixes DRILL-7082: if a partitioned table is read by multiple scans, V2 may provide multiple schemas depending on partition depth. V3 provides a consistent schema. The fix for this is to compute maximum depth in the planner, not the scan operator (since the planner sees all files, the scan operator sees only a subset.) The same fix could be applied to other "Easy" readers, even in V2. * Does *not* fix DRILL-7080: Drill seems to really not want a query of form "SELECT *, dir0 FROM...". The project operator creates a second "dir00" column. * And, of course, V2 will limit maximum vector size to 16 MB. The ongoing resource management project can add one additional config setting to set the overall batch size limit. Unit tests demonstrate that V2 and V3 work identically, except for intentional changes such as those above. To test DRILL-7082, it was useful to force Drill to parallelize a query even if it has two small files. (Drill normally parallelizes only when its guess of row count is above some minimum.) To do this: ``` alter session set `exec.storage.min_width` = 2; ``` The above asks to create at least two minor fragments (if there are at least two files to read.)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
