Indeed, it is weird practice at best. The real index is an RTree index on FacebookMessages.sender-location, which is nullable, and the code path to create that index just happens to trigger this.
On Tue, Jun 9, 2015 at 11:35 PM, Mike Carey <[email protected]> wrote: > This sounds like a bad practice (doing any mods on the MD node - we should > view the call as passing in immutable data). > But I'm not sure which index is being referred to here - what index on > what dataset is the "actual index" below? > > > > On 6/9/15 5:11 PM, Ian Maxon wrote: > >> So I think I might have a fix for this: >> https://asterix-gerrit.ics.uci.edu/#/c/283/1 >> >> This bug is really insidious. What I think is happening is that in >> AqlTranslator, the types are computed and given to the MetadataManager as >> a >> new Datatype, to insert into the metadata node via RMI on MetadataNode. >> However MetadataNode modifies the object it is given before it inserts it >> into the actual index. That is where the names of the nullable types get >> generated, and so the MetadataManager doesn't get to see those changes >> unless MetadataManager and MetadataNode are actually sharing the same >> object. So if it is stale, that object gets inserted into the cache, and >> when the index is created, it sees the stale type information rather than >> the true value that's stored in the index. >> >> My fix is just to read it again before inserting into the cache. I'm not >> really sure thats the right thing to do though. And if there are any other >> instances of this type (caller expecting changes on remote side to >> reflect), the same sort of behavior can happen. >> >> - Ian >> >> On Tue, Jun 9, 2015 at 10:12 AM, Ian Maxon <[email protected]> wrote: >> >> My quack theory is that it is due to the RMI somehow, or that the RMI >>> reveals it. It isn't specific to managix. It happens as long as the CC >>> and >>> NC reside in different JVMs. >>> >>> - Ian >>> >>> On Tue, Jun 9, 2015 at 10:03 AM, Ildar Absalyamov < >>> [email protected]> wrote: >>> >>> As Ian mentioned test case already exists (which I thought was good >>>> enough to prevent regressions like this one), but for some reason it’s >>>> not >>>> triggering the error, whereas if the same ddl\query is executed on a >>>> local >>>> cluster, which was prepared by managix, the error will trigger. >>>> >>>> On Jun 9, 2015, at 08:07, Mike Carey <[email protected]> wrote: >>>>> >>>>> Ildar, can you try to nail this one down and also add a test case if >>>>> you >>>>> can create one once this is making more sense...? >>>>> On Jun 8, 2015 11:58 PM, "Ildar Absalyamov" < >>>>> [email protected] >>>>> >>>>> wrote: >>>>> >>>>> Interesting, I guess step 1 here would be to investigate what’s >>>>>> >>>>> different >>>> >>>>> in test framework\AsterixHyracksIntegrationUntil setup and regular >>>>>> >>>>> cluster >>>> >>>>> configuration, created by managix. >>>>>> >>>>>> On Jun 8, 2015, at 23:37, Ian Maxon <[email protected]> wrote: >>>>>>> >>>>>>> I was playing around with the latest snapshot version, trying to >>>>>>> >>>>>> create a >>>> >>>>> fresher version of my demo docker container, and so I ran through the >>>>>>> ADM/AQL 101. However I couldn't get it to work, which was really >>>>>>> >>>>>> surprising >>>>>> >>>>>>> (and distressing). The error appears after trying to create an RTree >>>>>>> >>>>>> index >>>>>> >>>>>>> on >>>>>>> sender-location: >>>>>>> >>>>>>> >>>>>>> SEVERE: java.lang.NullPointerException >>>>>>> >>>>>>>> edu.uci.ics.asterix.metadata.MetadataException: >>>>>>>> java.lang.NullPointerException >>>>>>>> at >>>>>>>> >>>>>>>> >>>> edu.uci.ics.asterix.metadata.MetadataNode.addIndex(MetadataNode.java:229) >>>> >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>>> at >>>>>>>> >>>>>>>> >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> >>>>> at java.lang.reflect.Method.invoke(Method.java:497) >>>>>>>> at >>>>>>>> >>>>>>> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323) >>>> >>>>> at sun.rmi.transport.Transport$1.run(Transport.java:200) >>>>>>>> at sun.rmi.transport.Transport$1.run(Transport.java:197) >>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>> at sun.rmi.transport.Transport.serviceCall(Transport.java:196) >>>>>>>> at >>>>>>>> >>>>>>> >>>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$97(TCPTransport.java:683) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$2/1502895914.run(Unknown >>>> >>>>> Source) >>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>> at >>>>>>>> >>>>>>>> >>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>> >>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>> at >>>>>>>> >>>>>>>> >>>> sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253) >>>> >>>>> at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162) >>>>>>>> at >>>>>>>> >>>>>>>> >>>> java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:194) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:148) >>>> >>>>> at com.sun.proxy.$Proxy11.addIndex(Unknown Source) >>>>>>>> at >>>>>>>> >>>>>>>> >>>> edu.uci.ics.asterix.metadata.MetadataManager.addIndex(MetadataManager.java:417) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> edu.uci.ics.asterix.aql.translator.AqlTranslator.handleCreateIndexStatement(AqlTranslator.java:920) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> edu.uci.ics.asterix.aql.translator.AqlTranslator.compileAndExecute(AqlTranslator.java:261) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> edu.uci.ics.asterix.api.http.servlet.APIServlet.doPost(APIServlet.java:97) >>>> >>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) >>>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) >>>>>>>> at >>>>>>>> >>>>>>> >>>>>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:546) >>>>>> >>>>>>> at >>>>>>>> >>>>>>>> >>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:483) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:970) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:411) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:904) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:110) >>>> >>>>> at org.eclipse.jetty.server.Server.handle(Server.java:347) >>>>>>>> at >>>>>>>> >>>>>>>> >>>> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:439) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:924) >>>> >>>>> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:781) >>>>>>>> at >>>>>>>> >>>>>>> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:220) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:43) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:545) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:43) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) >>>> >>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>> Caused by: java.lang.NullPointerException >>>>>>>> at java.io.DataOutputStream.writeUTF(DataOutputStream.java:347) >>>>>>>> at java.io.DataOutputStream.writeUTF(DataOutputStream.java:323) >>>>>>>> at >>>>>>>> >>>>>>>> >>>> edu.uci.ics.hyracks.dataflow.common.data.marshalling.UTF8StringSerializerDeserializer.serialize(UTF8StringSerializerDeserializer.java:44) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> edu.uci.ics.asterix.dataflow.data.nontagged.serde.AStringSerializerDeserializer.serialize(AStringSerializerDeserializer.java:47) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> edu.uci.ics.asterix.dataflow.data.nontagged.serde.AStringSerializerDeserializer.serialize(AStringSerializerDeserializer.java:26) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> edu.uci.ics.asterix.formats.nontagged.AqlSerializerDeserializerProvider$1.serialize(AqlSerializerDeserializerProvider.java:208) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> edu.uci.ics.asterix.formats.nontagged.AqlSerializerDeserializerProvider$1.serialize(AqlSerializerDeserializerProvider.java:189) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> edu.uci.ics.asterix.metadata.entitytupletranslators.IndexTupleTranslator.getTupleFromMetadataEntity(IndexTupleTranslator.java:260) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> edu.uci.ics.asterix.metadata.MetadataNode.addIndex(MetadataNode.java:224) >>>> >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>>> at >>>>>>>> >>>>>>>> >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> >>>>> at java.lang.reflect.Method.invoke(Method.java:497) >>>>>>>> at >>>>>>>> >>>>>>> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323) >>>> >>>>> at sun.rmi.transport.Transport$1.run(Transport.java:200) >>>>>>>> at sun.rmi.transport.Transport$1.run(Transport.java:197) >>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>> at sun.rmi.transport.Transport.serviceCall(Transport.java:196) >>>>>>>> at >>>>>>>> >>>>>>> >>>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$97(TCPTransport.java:683) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$2/1502895914.run(Unknown >>>> >>>>> Source) >>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>> at >>>>>>>> >>>>>>>> >>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>> >>>>> at >>>>>>>> >>>>>>>> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>> >>>>> ... 1 more >>>>>>>> >>>>>>> >>>>>>> I started going back, trying this on older and older revisions until >>>>>>> >>>>>> it >>>> >>>>> stopped happening. It looks like this somehow has been lurking in the >>>>>>> Open/nested indexing code for a long time. The error does not >>>>>>> >>>>>> reproduce >>>> >>>>> in >>>>>> >>>>>>> the testing framework, for some reason I don't understand. >>>>>>> >>>>>>> The source of the bug seems to be that the RTree index is on a type >>>>>>> of >>>>>>> UNION(sender-location, null). The name of the type is not set like it >>>>>>> should be, it is null. Then in IndexTupleTranslator, the 9th field of >>>>>>> >>>>>> the >>>> >>>>> new metadata record is written with the name of the type- which is >>>>>>> >>>>>> null, >>>> >>>>> so >>>>>> >>>>>>> everything explodes. >>>>>>> >>>>>>> What is especially puzzling- is that this doesn't reproduce in >>>>>>> AsterixHyracksIntegrationUtil. There, the name of the type is >>>>>>> "Field_sender-location_in_FacebookMessageType". This type is created >>>>>>> >>>>>> too >>>> >>>>> in >>>>>> >>>>>>> the deployed version, however it doesn't seem to be picked up. >>>>>>> >>>>>>> I will keep digging on this, but I imagine someone more familiar with >>>>>>> >>>>>> the >>>> >>>>> Open/Nested indexing patch might have an idea about why this is >>>>>>> >>>>>> happening. >>>>>> >>>>>>> To reproduce just try deploying the snapshot version from the website >>>>>>> normally on your machine, and run through ADM/AQL 101. >>>>>>> >>>>>>> - Ian >>>>>>> >>>>>> Best regards, >>>>>> Ildar >>>>>> >>>>>> >>>>>> Best regards, >>>> Ildar >>>> >>>> >>>> >
