Hi Jacques, did not know about that, have one similar and one different result after setting union types on
## enabling works ok ## 0: jdbc:drill:zk=local> ALTER SESSION SET `exec.enable_union_type` = true; +-------+----------------------------------+ | ok | summary | +-------+----------------------------------+ | true | exec.enable_union_type updated. | +-------+----------------------------------+ 1 row selected (0.157 seconds) ## now let's try query over two rows that are the same ## got same error as before 0: jdbc:drill:zk=local> select case when is_map(t.user_info.`user`) then 'map' else 'string' end from dfs.`/Users/jos/Downloads/testcase2.json` t ; Error: SYSTEM ERROR: SchemaChangeException: Failure while trying to materialize incoming schema. Errors: Error in expression at index -1. Error: Missing function implementation: [is_map(MAP-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--. Error in expression at index -1. Error: Failure composing If Expression. All conditions must return a boolean type. Condition was of Type NULL.. Full expression: --UNKNOWN EXPRESSION--.. Fragment 0:0 [Error Id: 7c1b4dd4-8485-4429-a082-d936f5b3b95a on 10.19.220.63:31010] (state=,code=0) ## now lets try the query over different row types, ## this time we get an exception - I will cut and past full stack trace at end 0: jdbc:drill:zk=local> select case when is_map(t.user_info.`user`) then 'map' else 'string' end from dfs.`/Users/jos/Downloads/testcase.json` t ; Error: SYSTEM ERROR: NullPointerException Fragment 0:0 [Error Id: 9daa2496-d774-47b6-b786-014aac9abe59 on 10.19.220.63:31010] (state=,code=0) [Error Id: 9daa2496-d774-47b6-b786-014aac9abe59 on 10.19.220.63:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534) ~[drill-common-1.3.0.jar:1.3.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:321) [drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:184) [drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:290) [drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.3.0.jar:1.3.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_51] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_51] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] Caused by: java.lang.NullPointerException: null at org.apache.drill.exec.vector.complex.UnionVector.getFieldIdIfMatches(UnionVector.java:729) ~[drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.vector.complex.FieldIdUtil.getFieldIdIfMatches(FieldIdUtil.java:95) ~[drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.vector.complex.AbstractContainerVector.getFieldIdIfMatches(AbstractContainerVector.java:114) ~[drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.record.SimpleVectorWrapper.getFieldIdIfMatches(SimpleVectorWrapper.java:146) ~[drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.record.VectorContainer.getValueVectorId(VectorContainer.java:252) ~[drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.physical.impl.ScanBatch.getValueVectorId(ScanBatch.java:307) ~[drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.expr.ExpressionTreeMaterializer$MaterializeVisitor.visitSchemaPath(ExpressionTreeMaterializer.java:628) ~[drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.expr.ExpressionTreeMaterializer$MaterializeVisitor.visitSchemaPath(ExpressionTreeMaterializer.java:217) ~[drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.common.expression.SchemaPath.accept(SchemaPath.java:152) ~[drill-common-1.3.0.jar:1.3.0] at org.apache.drill.exec.expr.ExpressionTreeMaterializer$MaterializeVisitor.visitFunctionCall(ExpressionTreeMaterializer.java:274) ~[drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.expr.ExpressionTreeMaterializer$MaterializeVisitor.visitFunctionCall(ExpressionTreeMaterializer.java:217) ~[drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.common.expression.FunctionCall.accept(FunctionCall.java:60) ~[drill-common-1.3.0.jar:1.3.0] at org.apache.drill.exec.expr.ExpressionTreeMaterializer$MaterializeVisitor.visitIfExpression(ExpressionTreeMaterializer.java:494) ~[drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.expr.ExpressionTreeMaterializer$MaterializeVisitor.visitIfExpression(ExpressionTreeMaterializer.java:217) ~[drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.common.expression.IfExpression.accept(IfExpression.java:64) ~[drill-common-1.3.0.jar:1.3.0] at org.apache.drill.exec.expr.ExpressionTreeMaterializer.materialize(ExpressionTreeMaterializer.java:120) ~[drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:386) ~[drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:78) ~[drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:131) ~[drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:156) ~[drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) ~[drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:80) ~[drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) ~[drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:256) ~[drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:250) ~[drill-java-exec-1.3.0.jar:1.3.0] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_51] at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_51] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) ~[hadoop-common-2.7.1.jar:na] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:250) [drill-java-exec-1.3.0.jar:1.3.0] ... 4 common frames omitted 2015-12-01 10:36:15,231 [CONTROL-rpc-event-queue] WARN o.a.drill.exec.work.foreman.Foreman - Dropping request to move to COMPLETED state as query is already at FAILED state (which is terminal). 2015-12-01 10:36:15,232 [CONTROL-rpc-event-queue] WARN o.a.d.e.w.b.ControlMessageHandler - Dropping request to cancel fragment. 29a2175f-d3f2-caf9-2b51-12754264abe9:0:0 does not exist. 2015-12-01 10:36:15,234 [USER-rpc-event-queue] INFO o.a.d.j.i.DrillResultSetImpl$ResultsListener - [#7] Query failed: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: NullPointerException Fragment 0:0 [Error Id: 9daa2496-d774-47b6-b786-014aac9abe59 on 10.19.220.63:31010] at org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:118) [drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:112) [drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:47) [drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:32) [drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:69) [drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.rpc.RpcBus$RequestEvent.run(RpcBus.java:400) [drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.common.SerializedExecutor$RunnableProcessor.run(SerializedExecutor.java:105) [drill-common-1.3.0.jar:1.3.0] at org.apache.drill.exec.rpc.RpcBus$SameExecutor.execute(RpcBus.java:264) [drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.common.SerializedExecutor.execute(SerializedExecutor.java:142) [drill-common-1.3.0.jar:1.3.0] at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:298) [drill-java-exec-1.3.0.jar:1.3.0] at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:269) [drill-java-exec-1.3.0.jar:1.3.0] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) [netty-codec-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254) [netty-handler-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) [netty-codec-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-transport-4.0.27.Final.jar:4.0.27.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] _____________ john o schneider [email protected] 408-203-7891 On Tue, Dec 1, 2015 at 10:01 AM, Jacques Nadeau <[email protected]> wrote: > Did you enable the union type? You'll need to do that (for now) as > Heterogeneous type support is currently an experimental feature. > > ALTER SESSION SET `exec.enable_union_type` = true; > > See here: > > > https://drill.apache.org/docs/json-data-model/#experimental-feature:-heterogeneous-types > > -- > Jacques Nadeau > CTO and Co-Founder, Dremio > > On Tue, Dec 1, 2015 at 9:57 AM, John Schneider <[email protected]> > wrote: > > > Hi All, > > > > I'm trying to use case statements to manage a heterogeneous stream of > json > > objects as > > shown in the example from > > https://drill.apache.org/blog/2015/11/23/drill-1.3-released/ > > but I'm not getting any love yet. drill 1.1 -> 1.3 is chock full of > > goodness and case statements will help me with the last real hurdles I > have > > using drill with my logs. > > Would you please review the tests I created below and tell me if I'm just > > missing something obvious? > > > > Thanks > > /jos > > > > ## first test, two lines, one with a field that's a string and second > > field is a map > > ## first lets just select all records, I expect this to barf since there > > are two schemas > > : jdbc:drill:zk=local> select * from > > dfs.`/Users/jos/work/drill/casetest.json` t ; > > Error: DATA_READ ERROR: Error parsing JSON - You tried to start when you > > are using a ValueWriter of type NullableVarCharWriterImpl. > > > > File /Users/jos/work/drill/casetest.json > > Record 2 > > Fragment 0:0 > > > > [Error Id: 1385aea5-68cb-4775-ae17-fad6b4901ea6 on 10.0.1.9:31010] > > (state=,code=0) > > > > ## now lets use a case statement to sort out the schemas, I don't expect > > this to > > ## barf but barf it does, seems like this should have worked, what am I > > missing > > > > 0: jdbc:drill:zk=local> select case when is_map(t.user_info.`user`) then > > 'map' else 'string' end from dfs.`/Users/jos/work/drill/casetest.json` t > ; > > Error: DATA_READ ERROR: Error parsing JSON - You tried to start when you > > are using a ValueWriter of type NullableVarCharWriterImpl. > > > > File /Users/jos/Downloads/2015-11-30-bad-3.json > > Record 2 > > Fragment 0:0 > > > > [Error Id: 872a5347-93dd-49ae-a55c-e861b807b4a6 on 10.0.1.9:31010] > > (state=,code=0) > > 0: jdbc:drill:zk=local> > > > > > > ## data I used is this > > ## casetest.json has two lines in it > > > > > > > {"level":"EVENT","time":1448844983160,"user_info":{"session":"9OOLJ8HEGEQ0sTCVSXsK9ddJWVpFM5wM","user":" > > [email protected]"}} > > > > > {"level":"EVENT","time":1448844983160,"user_info":{"session":"9OOLJ8HEGEQ0sTCVSXsK9ddJWVpFM5wM","user":{"id":" > > [email protected] > > ","roles":null,"isNotadmins":true,"iscoders":true}}} > > > > > > ## now lets see if any case will work on any structure > > ## new test file with same line in it twice > > ## select * works as expected > > 0: jdbc:drill:zk=local> select * from > > dfs.`/Users/jos/work/drill/testcase2.json` t ; > > +-------+------+-----------+ > > | level | time | user_info | > > +-------+------+-----------+ > > | EVENT | 1448844983160 | > > {"session":"9OOLJ8HEGEQ0sTCVSXsK9ddJWVpFM5wM","user":{"id":" > > [email protected]","isNotadmins":true,"iscoders":true}} | > > | EVENT | 1448844983160 | > > {"session":"9OOLJ8HEGEQ0sTCVSXsK9ddJWVpFM5wM","user":{"id":" > > [email protected]","isNotadmins":true,"iscoders":true}} | > > +-------+------+-----------+ > > 2 rows selected (1.701 seconds) > > > > ## now lets try to use the line in a case statement > > ## it doesn't work, but we get different more puzzling errors this time > > 0: jdbc:drill:zk=local> select case when is_map(t.user_info.`user`) then > > 'map' else 'string' end from dfs.`/Users/jos/work/drill/testcase2.json` > t > > ; > > Error: SYSTEM ERROR: SchemaChangeException: Failure while trying to > > materialize incoming schema. Errors: > > > > Error in expression at index -1. Error: Missing function implementation: > > [is_map(MAP-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--. > > Error in expression at index -1. Error: Failure composing If Expression. > > All conditions must return a boolean type. Condition was of Type NULL.. > > Full expression: --UNKNOWN EXPRESSION--.. > > > > Fragment 0:0 > > > > [Error Id: c3a7f989-4d93-48c0-9a16-a38dd195314c on 10.19.220.63:31010] > > (state=,code=0) > > 0: jdbc:drill:zk=local> > > > > > > ## data I used is this test > > ## casetest2.json has two lines in it > > > > > {"level":"EVENT","time":1448844983160,"user_info":{"session":"9OOLJ8HEGEQ0sTCVSXsK9ddJWVpFM5wM","user":{"id":" > > [email protected] > > ","roles":null,"isNotadmins":true,"iscoders":true}}} > > > > > {"level":"EVENT","time":1448844983160,"user_info":{"session":"9OOLJ8HEGEQ0sTCVSXsK9ddJWVpFM5wM","user":{"id":" > > [email protected] > > ","roles":null,"isNotadmins":true,"iscoders":true}}} > > > > > > > > > > > > _____________ > > > > john o schneider > > [email protected] > > 408-203-7891 > > >
