Thanks much to Igor for listing the bugs associated with the type functions.
A bit of research (listed in DRILL-6362) shows how other engines (such as SqlLite and Postgres) implement the typeof() function. As a result, I've issued a PR [1] with three changes: * When typeof() is used with the UNION type, it reports the type of the value, not the type of the column. if the value is NULL, then there is no type and typeof() returns "NULL". This is in line with the way Snowflake implements the function. (DRILL-6362). * When typeof() is used with a column other than UNION, it always returns the type name, even if the column value is null. (DRILL-6362). * sqlTypeOf() returns the data type for a repeated type instead of "ARRAY". sqlTypeOf() returns "ARRAY" for the LIST type since the LIST type seems to be designed to be a repeated UNION. (DRILL-7499.) * drillTypeOf() returns the type of column. like typeof(). However, if the column is UNION or DICT, then drillTypeOf() returns the static type (UNION or DICT), not the value type. As it turns out, we do have comprehensive unit tests for these functions in TestTypeFns.java. The reason that we did not catch the change above was that is is actually hard to set up a query that uses REPEATED, UNION or LIST types -- which are exactly where we had problems. So, I added tests for these cases, and for the UNION type. As it turned out, when I added the UNION test case, I ran into a nasty codegen bug: DRILL-7502. Basically, this code could never have worked for UNION, but UNION is the type where the typeof() function is most needed. Was able to find a fix. (As an aside, whoever wrote the expression code generation stuff was wicked smart: that stuff is hugely complex, with many complex cases. Very impressive.) Thanks, - Paul [1] https://github.com/apache/drill/pull/1945 On Friday, December 27, 2019, 2:48:40 AM PST, Igor Guzenko <[email protected]> wrote: Hello Paul, While working on complex types I also was unable to find any document with expected results of the *typeOf functions. Since you've implemented them, could you please describe them in some way? Or it would be even better to extend test coverage in TestTypeFns.java so nobody could accidentally break the functionality. Related Jira tickets: https://issues.apache.org/jira/browse/DRILL-6360 https://issues.apache.org/jira/browse/DRILL-6362 https://issues.apache.org/jira/browse/DRILL-6377 https://issues.apache.org/jira/browse/DRILL-5189 Thanks, Igor On Fri, Dec 27, 2019 at 11:18 AM Arina Yelchiyeva < [email protected]> wrote: > Hi Paul, > > thanks for verifying the release. > Regression for the release is considered something that worked in previous > version, in our case in 1.16 and stopped working in the released, in our > case case 1.17. Vova has checked both sqltypeof issues, they behave the > same in 1.16 so from the release perspective, they are not regressions but > bugs. Thus they won’t have sinked the last release candidate. By the way, > one of the issues actually was a deliberate change in 1.16, I have left > comment in Jira. > > Regarding minor release, since both issues are not regressions, there is > no point of doing it. Though if they were, I would still say we should not > prepare minor release unless found bugs are something really serious that > prevents Drill from starting. Mostly because release preparation takes time > not only for the release manager but for the PMCs and committers to verify > and cast votes. > > Actually, it a good time to think when we want to release 1.18 and find > volunteer to be a release manager. > > Kind regards, > Arina > > > On 27 Dec 2019, at 04:48, Paul Rogers <[email protected]> wrote: > > > > Hi Charles, > > > > Good question! We have a number of big changes queued up. To do a minor > release, we'd want to branch off of the Drill 1.17 release, which is extra > work. > > > > > > Let's see if anyone finds other issues we'd like to fix so we can see if > the effort would be worthwhile. > > > > > > Thanks, > > - Paul > > > > > > > > On Thursday, December 26, 2019, 6:24:27 PM PST, Charles Givre < > [email protected]> wrote: > > > > Paul, > > Do you think it's worth fixing these and releasing a minor release in a > month or so with this and any other minor bug fixes? > > -- C > > > >> On Dec 26, 2019, at 9:17 PM, Paul Rogers <[email protected]> > wrote: > >> > >> Hi All, > >> > >> I'm late to the party (was distracted by a certain holiday). Downloaded > the artifacts and ran though the examples in the first several sections of > Chapter 8 of the Learning Apache Drill book. > >> > >> It turns out we have a regression in the sqlTypeOf() function. See > DRILL-7499 and DRILL-7501. > >> > >> These are minor as they only affect users of the functions. But, the > errors cause the results in Drill 1.17 to regress from those in Drill 1.14 > (when the book was written.) > >> > >> Otherwise, everything seemed to work fine. > >> > >> Thanks, > >> - Paul > >> > >> > >> > >> On Sunday, December 22, 2019, 9:00:52 AM PST, Volodymyr Vysotskyi < > [email protected]> wrote: > >> > >> Hi all, > >> > >> I'd like to propose the third release candidate (RC2) of Apache Drill, > >> version 1.17.0. > >> > >> Changes since the previous release candidate: fixed show-stopper > DRILL-7494 > >> <https://issues.apache.org/jira/browse/DRILL-7494>. > >> > >> The release candidate covers a total of 205 resolved JIRAs [1]. Thanks > to > >> everyone who contributed to this release. > >> > >> The tarball artifacts are hosted at [2] and the maven artifacts are > hosted > >> at [3]. > >> > >> This release candidate is based on > >> commit 2eb6bbe0501cb6553106e63dc1f2810ff10ae375 located at [4]. > >> > >> Please download and try out the release. > >> > >> The vote ends at 5 PM UTC (9 AM PDT, 7 PM EET, 10:30 PM IST), December > 25, > >> 2019 > >> > >> [ ] +1 > >> [ ] +0 > >> [ ] -1 > >> > >> Here's my vote: +1 > >> > >> [1] > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820&version=12344870 > >> [2] http://home.apache.org/~volodymyr/drill/releases/1.17.0/rc2/ > >> [3] > https://repository.apache.org/content/repositories/orgapachedrill-1077/ > >> [4] https://github.com/vvysotskyi/drill/commits/drill-1.17.0 > >> > >> Kind regards, > >> Volodymyr Vysotskyi >
