Steven, can you look at this? -- Jacques Nadeau CTO and Co-Founder, Dremio
On Wed, Dec 2, 2015 at 10:05 AM, John Schneider <[email protected]> wrote: > All, Is there a link to Jira where I can log this error? > > _____________ > > john o schneider > [email protected] > 408-203-7891 > > > On Tue, Dec 1, 2015 at 10:43 AM, John Schneider <[email protected]> > wrote: > > > Hi Jacques, did not know about that, have one similar and one different > > result after setting union types on > > > > ## enabling works ok > > ## > > 0: jdbc:drill:zk=local> ALTER SESSION SET `exec.enable_union_type` = > true; > > +-------+----------------------------------+ > > | ok | summary | > > +-------+----------------------------------+ > > | true | exec.enable_union_type updated. | > > +-------+----------------------------------+ > > 1 row selected (0.157 seconds) > > > > ## now let's try query over two rows that are the same > > ## got same error as before > > 0: jdbc:drill:zk=local> select case when is_map(t.user_info.`user`) then > > 'map' else 'string' end from dfs.`/Users/jos/Downloads/testcase2.json` > t ; > > Error: SYSTEM ERROR: SchemaChangeException: Failure while trying to > > materialize incoming schema. Errors: > > > > Error in expression at index -1. Error: Missing function implementation: > > [is_map(MAP-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--. > > Error in expression at index -1. Error: Failure composing If Expression. > > All conditions must return a boolean type. Condition was of Type NULL.. > > Full expression: --UNKNOWN EXPRESSION--.. > > > > Fragment 0:0 > > > > [Error Id: 7c1b4dd4-8485-4429-a082-d936f5b3b95a on 10.19.220.63:31010] > > (state=,code=0) > > > > ## now lets try the query over different row types, > > ## this time we get an exception - I will cut and past full stack trace > at > > end > > > > 0: jdbc:drill:zk=local> select case when is_map(t.user_info.`user`) then > > 'map' else 'string' end from dfs.`/Users/jos/Downloads/testcase.json` t ; > > Error: SYSTEM ERROR: NullPointerException > > > > Fragment 0:0 > > > > [Error Id: 9daa2496-d774-47b6-b786-014aac9abe59 on 10.19.220.63:31010] > > (state=,code=0) > > > > > > [Error Id: 9daa2496-d774-47b6-b786-014aac9abe59 on 10.19.220.63:31010] > > at > > > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534) > > ~[drill-common-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:321) > > [drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:184) > > [drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:290) > > [drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > > [drill-common-1.3.0.jar:1.3.0] > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > [na:1.8.0_51] > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > [na:1.8.0_51] > > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] > > Caused by: java.lang.NullPointerException: null > > at > > > org.apache.drill.exec.vector.complex.UnionVector.getFieldIdIfMatches(UnionVector.java:729) > > ~[drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.vector.complex.FieldIdUtil.getFieldIdIfMatches(FieldIdUtil.java:95) > > ~[drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.vector.complex.AbstractContainerVector.getFieldIdIfMatches(AbstractContainerVector.java:114) > > ~[drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.record.SimpleVectorWrapper.getFieldIdIfMatches(SimpleVectorWrapper.java:146) > > ~[drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.record.VectorContainer.getValueVectorId(VectorContainer.java:252) > > ~[drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.physical.impl.ScanBatch.getValueVectorId(ScanBatch.java:307) > > ~[drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.expr.ExpressionTreeMaterializer$MaterializeVisitor.visitSchemaPath(ExpressionTreeMaterializer.java:628) > > ~[drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.expr.ExpressionTreeMaterializer$MaterializeVisitor.visitSchemaPath(ExpressionTreeMaterializer.java:217) > > ~[drill-java-exec-1.3.0.jar:1.3.0] > > at > > org.apache.drill.common.expression.SchemaPath.accept(SchemaPath.java:152) > > ~[drill-common-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.expr.ExpressionTreeMaterializer$MaterializeVisitor.visitFunctionCall(ExpressionTreeMaterializer.java:274) > > ~[drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.expr.ExpressionTreeMaterializer$MaterializeVisitor.visitFunctionCall(ExpressionTreeMaterializer.java:217) > > ~[drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.common.expression.FunctionCall.accept(FunctionCall.java:60) > > ~[drill-common-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.expr.ExpressionTreeMaterializer$MaterializeVisitor.visitIfExpression(ExpressionTreeMaterializer.java:494) > > ~[drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.expr.ExpressionTreeMaterializer$MaterializeVisitor.visitIfExpression(ExpressionTreeMaterializer.java:217) > > ~[drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.common.expression.IfExpression.accept(IfExpression.java:64) > > ~[drill-common-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.expr.ExpressionTreeMaterializer.materialize(ExpressionTreeMaterializer.java:120) > > ~[drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:386) > > ~[drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:78) > > ~[drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:131) > > ~[drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:156) > > ~[drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) > > ~[drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:80) > > ~[drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) > > ~[drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:256) > > ~[drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:250) > > ~[drill-java-exec-1.3.0.jar:1.3.0] > > at java.security.AccessController.doPrivileged(Native Method) > > ~[na:1.8.0_51] > > at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_51] > > at > > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > > ~[hadoop-common-2.7.1.jar:na] > > at > > > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:250) > > [drill-java-exec-1.3.0.jar:1.3.0] > > ... 4 common frames omitted > > 2015-12-01 10:36:15,231 [CONTROL-rpc-event-queue] WARN > > o.a.drill.exec.work.foreman.Foreman - Dropping request to move to > > COMPLETED state as query is already at FAILED state (which is terminal). > > 2015-12-01 10:36:15,232 [CONTROL-rpc-event-queue] WARN > > o.a.d.e.w.b.ControlMessageHandler - Dropping request to cancel fragment. > > 29a2175f-d3f2-caf9-2b51-12754264abe9:0:0 does not exist. > > 2015-12-01 10:36:15,234 [USER-rpc-event-queue] INFO > > o.a.d.j.i.DrillResultSetImpl$ResultsListener - [#7] Query failed: > > org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: > > NullPointerException > > > > Fragment 0:0 > > > > [Error Id: 9daa2496-d774-47b6-b786-014aac9abe59 on 10.19.220.63:31010] > > at > > > org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:118) > > [drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:112) > > [drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:47) > > [drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:32) > > [drill-java-exec-1.3.0.jar:1.3.0] > > at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:69) > > [drill-java-exec-1.3.0.jar:1.3.0] > > at org.apache.drill.exec.rpc.RpcBus$RequestEvent.run(RpcBus.java:400) > > [drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.common.SerializedExecutor$RunnableProcessor.run(SerializedExecutor.java:105) > > [drill-common-1.3.0.jar:1.3.0] > > at org.apache.drill.exec.rpc.RpcBus$SameExecutor.execute(RpcBus.java:264) > > [drill-java-exec-1.3.0.jar:1.3.0] > > at > > > org.apache.drill.common.SerializedExecutor.execute(SerializedExecutor.java:142) > > [drill-common-1.3.0.jar:1.3.0] > > at > org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:298) > > [drill-java-exec-1.3.0.jar:1.3.0] > > at > org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:269) > > [drill-java-exec-1.3.0.jar:1.3.0] > > at > > > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) > > [netty-codec-4.0.27.Final.jar:4.0.27.Final] > > at > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > at > > > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > at > > > io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254) > > [netty-handler-4.0.27.Final.jar:4.0.27.Final] > > at > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > at > > > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > at > > > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) > > [netty-codec-4.0.27.Final.jar:4.0.27.Final] > > at > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > at > > > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > at > > > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) > > [netty-codec-4.0.27.Final.jar:4.0.27.Final] > > at > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > at > > > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > at > > > io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > at > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > at > > > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > at > > > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > at > > > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > at > > > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > at > > > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > at > > > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > > [netty-transport-4.0.27.Final.jar:4.0.27.Final] > > at > > > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > > [netty-common-4.0.27.Final.jar:4.0.27.Final] > > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] > > > > > > _____________ > > > > john o schneider > > [email protected] > > 408-203-7891 > > > > > > On Tue, Dec 1, 2015 at 10:01 AM, Jacques Nadeau <[email protected]> > > wrote: > > > >> Did you enable the union type? You'll need to do that (for now) as > >> Heterogeneous type support is currently an experimental feature. > >> > >> ALTER SESSION SET `exec.enable_union_type` = true; > >> > >> See here: > >> > >> > >> > https://drill.apache.org/docs/json-data-model/#experimental-feature:-heterogeneous-types > >> > >> -- > >> Jacques Nadeau > >> CTO and Co-Founder, Dremio > >> > >> On Tue, Dec 1, 2015 at 9:57 AM, John Schneider <[email protected]> > >> wrote: > >> > >> > Hi All, > >> > > >> > I'm trying to use case statements to manage a heterogeneous stream of > >> json > >> > objects as > >> > shown in the example from > >> > https://drill.apache.org/blog/2015/11/23/drill-1.3-released/ > >> > but I'm not getting any love yet. drill 1.1 -> 1.3 is chock full of > >> > goodness and case statements will help me with the last real hurdles I > >> have > >> > using drill with my logs. > >> > Would you please review the tests I created below and tell me if I'm > >> just > >> > missing something obvious? > >> > > >> > Thanks > >> > /jos > >> > > >> > ## first test, two lines, one with a field that's a string and second > >> > field is a map > >> > ## first lets just select all records, I expect this to barf since > there > >> > are two schemas > >> > : jdbc:drill:zk=local> select * from > >> > dfs.`/Users/jos/work/drill/casetest.json` t ; > >> > Error: DATA_READ ERROR: Error parsing JSON - You tried to start when > you > >> > are using a ValueWriter of type NullableVarCharWriterImpl. > >> > > >> > File /Users/jos/work/drill/casetest.json > >> > Record 2 > >> > Fragment 0:0 > >> > > >> > [Error Id: 1385aea5-68cb-4775-ae17-fad6b4901ea6 on 10.0.1.9:31010] > >> > (state=,code=0) > >> > > >> > ## now lets use a case statement to sort out the schemas, I don't > expect > >> > this to > >> > ## barf but barf it does, seems like this should have worked, what am > I > >> > missing > >> > > >> > 0: jdbc:drill:zk=local> select case when is_map(t.user_info.`user`) > then > >> > 'map' else 'string' end from dfs.`/Users/jos/work/drill/casetest.json` > >> t ; > >> > Error: DATA_READ ERROR: Error parsing JSON - You tried to start when > you > >> > are using a ValueWriter of type NullableVarCharWriterImpl. > >> > > >> > File /Users/jos/Downloads/2015-11-30-bad-3.json > >> > Record 2 > >> > Fragment 0:0 > >> > > >> > [Error Id: 872a5347-93dd-49ae-a55c-e861b807b4a6 on 10.0.1.9:31010] > >> > (state=,code=0) > >> > 0: jdbc:drill:zk=local> > >> > > >> > > >> > ## data I used is this > >> > ## casetest.json has two lines in it > >> > > >> > > >> > > >> > {"level":"EVENT","time":1448844983160,"user_info":{"session":"9OOLJ8HEGEQ0sTCVSXsK9ddJWVpFM5wM","user":" > >> > [email protected]"}} > >> > > >> > > >> > {"level":"EVENT","time":1448844983160,"user_info":{"session":"9OOLJ8HEGEQ0sTCVSXsK9ddJWVpFM5wM","user":{"id":" > >> > [email protected] > >> > ","roles":null,"isNotadmins":true,"iscoders":true}}} > >> > > >> > > >> > ## now lets see if any case will work on any structure > >> > ## new test file with same line in it twice > >> > ## select * works as expected > >> > 0: jdbc:drill:zk=local> select * from > >> > dfs.`/Users/jos/work/drill/testcase2.json` t ; > >> > +-------+------+-----------+ > >> > | level | time | user_info | > >> > +-------+------+-----------+ > >> > | EVENT | 1448844983160 | > >> > {"session":"9OOLJ8HEGEQ0sTCVSXsK9ddJWVpFM5wM","user":{"id":" > >> > [email protected]","isNotadmins":true,"iscoders":true}} | > >> > | EVENT | 1448844983160 | > >> > {"session":"9OOLJ8HEGEQ0sTCVSXsK9ddJWVpFM5wM","user":{"id":" > >> > [email protected]","isNotadmins":true,"iscoders":true}} | > >> > +-------+------+-----------+ > >> > 2 rows selected (1.701 seconds) > >> > > >> > ## now lets try to use the line in a case statement > >> > ## it doesn't work, but we get different more puzzling errors this > time > >> > 0: jdbc:drill:zk=local> select case when is_map(t.user_info.`user`) > then > >> > 'map' else 'string' end from > >> dfs.`/Users/jos/work/drill/testcase2.json` t > >> > ; > >> > Error: SYSTEM ERROR: SchemaChangeException: Failure while trying to > >> > materialize incoming schema. Errors: > >> > > >> > Error in expression at index -1. Error: Missing function > >> implementation: > >> > [is_map(MAP-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--. > >> > Error in expression at index -1. Error: Failure composing If > >> Expression. > >> > All conditions must return a boolean type. Condition was of Type > NULL.. > >> > Full expression: --UNKNOWN EXPRESSION--.. > >> > > >> > Fragment 0:0 > >> > > >> > [Error Id: c3a7f989-4d93-48c0-9a16-a38dd195314c on 10.19.220.63:31010 > ] > >> > (state=,code=0) > >> > 0: jdbc:drill:zk=local> > >> > > >> > > >> > ## data I used is this test > >> > ## casetest2.json has two lines in it > >> > > >> > > >> > {"level":"EVENT","time":1448844983160,"user_info":{"session":"9OOLJ8HEGEQ0sTCVSXsK9ddJWVpFM5wM","user":{"id":" > >> > [email protected] > >> > ","roles":null,"isNotadmins":true,"iscoders":true}}} > >> > > >> > > >> > {"level":"EVENT","time":1448844983160,"user_info":{"session":"9OOLJ8HEGEQ0sTCVSXsK9ddJWVpFM5wM","user":{"id":" > >> > [email protected] > >> > ","roles":null,"isNotadmins":true,"iscoders":true}}} > >> > > >> > > >> > > >> > > >> > > >> > _____________ > >> > > >> > john o schneider > >> > [email protected] > >> > 408-203-7891 > >> > > >> > > > > >
