[
https://issues.apache.org/jira/browse/DRILL-7502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006249#comment-17006249
]
ASF GitHub Bot commented on DRILL-7502:
---------------------------------------
paul-rogers commented on pull request #1945: DRILL-7502: Invalid codegen for
typeof() with UNION
URL: https://github.com/apache/drill/pull/1945
The gist of the problem is that codegen for a `UNION` using a `FieldReader`
for a `@Param` was handled incorrectly. We had code to handle complex and
repeated types specially. Turned out we just needed to add `UNION` type to that
same logic. Since the fix turned out to be simple (after much sleuthing), I
suspect this may have been a regression and this fix simply puts things back
the way the original author intended.
Also fixes DRILL-6362: typeof() reports NULL for primitive columns with a
NULL value.
typeof() is meant to return "NULL" if a UNION has a NULL value, but the
column type when known, such as for non-UNION columns.
Also fixes DRILL-7499: sqltypeof() function with an array returns "ARRAY",
not type. This was due to treating REPEATED like LIST.
Includes cleanup in files visited during the work.
### Documentation
User-visible behaviour changes after this fix:
* `typeof()` returns the value type of a specific column value. This is the
Drill minor type for most types. For types other than `UNION`, if the value is
`NULL`, `typeof() `still reports the column type (since the NULL has a type.)
But for a `UNION` column, `typeof()` returns the type of that specific column
value, or "NULL" if the column is null. (Since, for UNION, NULL has no type.)
* `sqlTypeOf()` works the same except for arrays. Previously (sometime
between Drill 1.15 and Drill 1.17), `sqlTypeOf()` returned "ARRAY" for a
repeated column. After this fix, it works as in Drill 1.14: `sqlTypeOf()`
returns the type name. Thus a repeated INT returns "INT", not "ARRAY". (The
`LIST` type still returns "ARRAY", however.)
* `drillTypeOf()` returns the actual minor type of a column, even for
`UNION` and `DICT` columns. Thus, if a column is `UNION`, `typeOf()` returns
the value type, but `drillTypeOf()` returns "UNION."
### Tests
Added two new tests to `TestTypeFns`, one which uses the JSON reader to
verify `UNION` type behaviour (the test which uncovered this bug.) Reran all
unit tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Incorrect/invalid codegen for typeof() with UNION
> -------------------------------------------------
>
> Key: DRILL-7502
> URL: https://issues.apache.org/jira/browse/DRILL-7502
> Project: Apache Drill
> Issue Type: Bug
> Reporter: Paul Rogers
> Priority: Major
>
> The {{typeof()}} function is defined as follows:
> {code:java}
> @FunctionTemplate(names = {"typeOf"},
> scope = FunctionTemplate.FunctionScope.SIMPLE,
> nulls = NullHandling.INTERNAL)
> public static class GetType implements DrillSimpleFunc {
> @Param
> FieldReader input;
> @Output
> VarCharHolder out;
> @Inject
> DrillBuf buf;
> @Override
> public void setup() {}
> @Override
> public void eval() {
> String typeName = input.getTypeString();
> byte[] type = typeName.getBytes();
> buf = buf.reallocIfNeeded(type.length);
> buf.setBytes(0, type);
> out.buffer = buf;
> out.start = 0;
> out.end = type.length;
> }
> }
> {code}
> Note that the {{input}} field is defined as {{FieldReader}} which has a
> method called {{getTypeString()}}. As a result, the code works fine in all
> existing tests in {{TestTypeFns}}.
> I tried to add a function to use {{typeof()}} on a column of type {{UNION}}.
> When I did, the query failed with a compile error in generated code:
> {noformat}
> SYSTEM ERROR: CompileException: Line 42, Column 43:
> A method named "getTypeString" is not declared in any enclosing class nor
> any supertype, nor through a static import
> {noformat}
> The stack trace shows the generated code; Note that the type of {{input}}
> changes from a reader to a holder, causing code to be invalid:
> {code:java}
> public class ProjectorGen0 {
> DrillBuf work0;
> UnionVector vv1;
> VarCharVector vv6;
> DrillBuf work9;
> VarCharVector vv11;
> DrillBuf work14;
> VarCharVector vv16;
> public void doEval(int inIndex, int outIndex)
> throws SchemaChangeException
> {
> {
> UnionHolder out4 = new UnionHolder();
> {
> out4 .isSet = vv1 .getAccessor().isSet((inIndex));
> if (out4 .isSet == 1) {
> vv1 .getAccessor().get((inIndex), out4);
> }
> }
> //---- start of eval portion of typeOf function. ----//
> VarCharHolder out5 = new VarCharHolder();
> {
> final VarCharHolder out = new VarCharHolder();
> UnionHolder input = out4;
> DrillBuf buf = work0;
> UnionFunctions$GetType_eval:
> {
> String typeName = input.getTypeString();
> byte[] type = typeName.getBytes();
> buf = buf.reallocIfNeeded(type.length);
> buf.setBytes(0, type);
> out.buffer = buf;
> out.start = 0;
> out.end = type.length;
> }
> {code}
> By contrast, here is the generated code for one of the existing
> {{TestTypeFns}} tests where things work:
> {code:java}
> public class ProjectorGen0
> extends ProjectorTemplate
> {
> DrillBuf work0;
> NullableBigIntVector vv1;
> VarCharVector vv7;
> public ProjectorGen0() {
> try {
> __DRILL_INIT__();
> } catch (SchemaChangeException e) {
> throw new UnsupportedOperationException(e);
> }
> }
> public void doEval(int inIndex, int outIndex)
> throws SchemaChangeException
> {
> {
> ..
> //---- start of eval portion of typeOf function. ----//
> VarCharHolder out6 = new VarCharHolder();
> {
> final VarCharHolder out = new VarCharHolder();
> FieldReader input = new NullableIntHolderReaderImpl(out5);
> DrillBuf buf = work0;
> UnionFunctions$GetType_eval:
> {
> String typeName = input.getTypeString();
> byte[] type = typeName.getBytes();
> buf = buf.reallocIfNeeded(type.length);
> buf.setBytes(0, type);
> out.buffer = buf;
> out.start = 0;
> out.end = type.length;
> }
> work0 = buf;
> out6 .start = out.start;
> out6 .end = out.end;
> out6 .buffer = out.buffer;
> }
> //---- end of eval portion of typeOf function. ----//
> {code}
> Notice that the {{input}} variable is of type {{FieldReader}} as expected.
> Queries that work:
> {code:java}
> String sql = "SELECT typeof(CAST(a AS " + castType + ")) FROM (VALUES
> (1)) AS T(a)";
> sql = "SELECT typeof(CAST(a AS " + castType + ")) FROM
> cp.`functions/null.json`";
> String sql = "SELECT typeof(" + expr + ") FROM (VALUES (" + value + "))
> AS T(a)";
> {code}
> Query that fails:
> {code:java}
> String sql ="SELECT typeof(a) AS t, modeof(a) as m, drilltypeof(a) AS
> dt\n" +
> "FROM cp.`jsoninput/union/c.json`";
> {code}
> The queries that work all include either a CAST or constant values. The query
> that fails works with data read from a file. Also, the queries that work use
> scalar types, the query that fails uses the UNION type.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
