Nelson, I can try to setup a simple pipeline with one AE which will add 20 views and than test serialization. Not sure if I get to it today. If not this will have to wait till Monday next week. I've already mentioned this before, don't operate on a CAS once it leaves an AE. The contract is CAS-In CAS-out. A CAS instance can only be operated on by one AE at a time.
-jerry On Wed, Feb 15, 2017 at 11:06 AM, Marshall Schor <[email protected]> wrote: > On 2/15/2017 9:51 AM, Jaroslaw Cwiklik wrote: > > Not exactly sure how to debug this. > > a small-ish test case we could run would enable debugging... > > > The UIMA-AS does not touch contents of > > a CAS directly. Are there any other errors in the log besides NPE? The > > UIMA-AS uses uima-sdk to serialize CASes. Since you are getting null from > > getView(N), this view must have been deleted somehow. > > > > -jerry > > > > On Mon, Feb 13, 2017 at 11:43 AM, nelson rivera < > [email protected]> > > wrote: > > > >> I was able to check your email just today. The agregegate is async, > >> but only process one input CAS at the same time,default numberOfCASes. > >> I read your possible explanation but i have no idea that another > >> thread can modificate the cas, because the last annotator's execution > >> is correct and only missing that the framework uima-as serializes the > >> cas. > >> > >> This is the configuration of deploy of the aggregate: > >> > >> <?xml version="1.0" encoding="UTF-8"?> > >> <analysisEngineDeploymentDescription > >> xmlns="http://uima.apache.org/resourceSpecifier";> > >> > >> <name>XClusterAnalyzerAE Deploy Descriptor</name> > >> <description>Deploys XClusterAnalyzerAE</description> > >> > >> <deployment protocol="jms" provider="activemq"> > >> > >> <service> > >> <inputQueue endpoint="XClusterAnalyzerAggregate" > >> brokerURL="${defaultBrokerURL}"/> > >> <topDescriptor> > >> <import location="./XClusterAnalyzerAggregate.xml"/> > >> </topDescriptor> > >> <!-- remoteReplyQueueScaleout for remote delegate--> > >> <analysisEngine inputQueueScaleout="2" > >> internalReplyQueueScaleout="3"> > >> <delegates> > >> <analysisEngine key="FileSystemMultiplerCas"> > >> <casMultiplier poolSize="10"/> > >> </analysisEngine> > >> <analysisEngine key="XFileFormatDetector"> > >> <scaleout numberOfInstances="2"/> > >> <asyncAggregateErrorConfiguration> > >> <processCasErrors maxRetries="0" > >> continueOnRetryFailure="true"/> > >> </asyncAggregateErrorConfiguration> > >> </analysisEngine> > >> <analysisEngine key="XDataFileExtractor"> > >> <scaleout numberOfInstances="2"/> > >> <asyncAggregateErrorConfiguration> > >> <processCasErrors maxRetries="0" > >> continueOnRetryFailure="true"/> > >> </asyncAggregateErrorConfiguration> > >> </analysisEngine> > >> <remoteAnalysisEngine key="XLanguageDetector"> > >> <inputQueue endpoint="XLanguageDetector" > >> brokerURL="${defaultBrokerURL}"/> > >> <serializer method="xmi"/> > >> <asyncAggregateErrorConfiguration> > >> <processCasErrors maxRetries="0" > >> continueOnRetryFailure="true"/> > >> </asyncAggregateErrorConfiguration> > >> </remoteAnalysisEngine> > >> <analysisEngine key="XTokenizer"> > >> <scaleout numberOfInstances="2"/> > >> <asyncAggregateErrorConfiguration> > >> <processCasErrors maxRetries="0" > >> continueOnRetryFailure="true"/> > >> </asyncAggregateErrorConfiguration> > >> </analysisEngine> > >> <analysisEngine key="XBoTModeler"> > >> <scaleout numberOfInstances="3"/> > >> <asyncAggregateErrorConfiguration> > >> <processCasErrors maxRetries="0" > >> continueOnRetryFailure="true"/> > >> </asyncAggregateErrorConfiguration> > >> </analysisEngine> > >> <analysisEngine key="MergerInViewCasMultipler"> > >> <casMultiplier poolSize="1"/> > >> </analysisEngine> > >> <analysisEngine key="XClusterAnalyzer"> > >> <scaleout numberOfInstances="1"/> > >> <asyncAggregateErrorConfiguration> > >> <processCasErrors maxRetries="0" > >> continueOnRetryFailure="true"/> > >> </asyncAggregateErrorConfiguration> > >> </analysisEngine> > >> </delegates> > >> </analysisEngine> > >> </service> > >> </deployment> > >> > >> </analysisEngineDeploymentDescription> > >> > >> 2017-02-10 16:43 GMT-05:00, Jaroslaw Cwiklik <[email protected]>: > >>> Just a bit more evidence. The caller of the gerSofaAddr() > >>> > >>> public void writeViewsCommons() throws Exception { > >>> // Get indexes for each SofaFS in the CAS > >>> int numViews = cas.getBaseSofaCount(); > >>> > >>> for (int sofaNum = 1; sofaNum <= numViews; sofaNum++) { > >>> FSIndexRepositoryImpl loopIR = (FSIndexRepositoryImpl) > >>> cas.getBaseCAS().getSofaIndexRepository(sofaNum); > >>> final int sofaAddr = getSofaAddr(sofaNum); > >>> > >>> Not an expert of this code, but it smells like another thread is > >> changing a > >>> CAS which is being serialized. > >>> > >>> -jerry > >>> > >>> On Fri, Feb 10, 2017 at 4:31 PM, Jaroslaw Cwiklik <[email protected]> > >> wrote: > >>>> Is this a primitive (single-threaded) aggregate or async > >>>> (multi-threaded)? > >>>> If async, try to simplify and run primitive aggregate with scaleout=1. > >>>> > >>>> The CAS does not seem to be null in this case. The caller of the > >>>> getSerializedCas() > >>>> checks for null. > >>>> > >>>> The code dies here: > >>>> Caused by: java.lang.NullPointerException > >>>> at org.apache.uima.cas.impl.CasSerializerSupport$ > CasDocSerializ > >>>> er.getSofaAddr(CasSerializerSupport.java:454) > >>>> > >>>> public int getSofaAddr(int sofaNum) { > >>>> if (sofaNum != 1 || cas.isInitialSofaCreated()) { //skip if > >> initial > >>>> view && no Sofa yet > >>>> // all > >>>> non-initial-views must have a sofa > >>>> * return ((CASImpl)cas.getView(sofaNum)).getSofaRef();* > >>>> } > >>>> return 0; > >>>> } > >>>> > >>>> Looks to me that getView(sofaNum) is returning null. Is it possible > that > >>>> two threads are operating on the same CAS maybe? One removing a view > >>>> while > >>>> another trying to serialize. Have no idea what else could it be. > >>>> > >>>> -jerry > >>>> > >>>> > >>>> > >>>> On Fri, Feb 10, 2017 at 8:45 AM, nelson rivera < > >> [email protected]> > >>>> wrote: > >>>> > >>>>> Hi, The first thing I did was these tests,i made a simple test case > >>>>> that create a Cas with 17 views and then serialize using > >>>>> XmiCasSerializer.serialize(newJCas.getCas(), fis) and serializes > >>>>> correctly. > >>>>> Also i made other test, initialize the same AE but of local way with > >>>>> UIMA API and process the same input documents and the processing is > >>>>> correct and then serialize the CAS, without problem. > >>>>> > >>>>> The error is with AE deployed in uima-as and consuming it. > >>>>> > >>>>> 2017-02-09 17:30 GMT-05:00, Marshall Schor <[email protected]>: > >>>>>> one thing that would help track this down is a small isolated test > >>>>>> case. > >>>>>> > >>>>>> Do you think uima-as is needed? I'm wondering if a simple test case > >>>>> which > >>>>>> generated 17 views and then tried to serialize would show the > >>>>>> failure... > >>>>>> > >>>>>> If you could supply a small test case that showed the failure so we > >>>>> could > >>>>>> reproduce it, that would enable a rapid resolution. > >>>>>> > >>>>>> -Marshall > >>>>>> > >>>>>> > >>>>>> On 2/9/2017 3:58 PM, Marshall Schor wrote: > >>>>>>> The line throwing the null pointer exception is : > >>>>>>> > >>>>>>> cas.getView(sofaNum).getSofaRef() > >>>>>>> > >>>>>>> So the NPE is either the cas is null, or the getView(sofaNum) is > >>>>> returning > >>>>>>> null. > >>>>>>> > >>>>>>> I'm not sure what the best way is to debug this... > >>>>>>> > >>>>>>> -Marshall > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On 2/9/2017 12:42 PM, nelson rivera wrote: > >>>>>>>> I have a aggregate service uima-as, at the end of aggregate the > cas > >>>>>>>> to > >>>>>>>> return is composed of as many views as the number of input files, > >>>>>>>> each > >>>>>>>> view with annotations of processing. > >>>>>>>> With a number of input documents less than 15 the processing is > >>>>>>>> successful always, > >>>>>>>> but if the number of documents is greater than 15, i get a > >>>>>>>> NullPointerException at the aggregate service trying to serialize > >>>>>>>> the > >>>>>>>> cas, not in the processing of AE aggregate. > >>>>>>>> the logs of aggregate service: > >>>>>>>> > >>>>>>>> 11:51:38.815 - 42: > >>>>>>>> cu.datys.xinetica.uima.core.MergerInViewCasMultipler. > hasNext(285): > >>>>>>>> INFO: HasNext false > >>>>>>>> 11:51:38.875 - 44: > >>>>>>>> org.apache.uima.uimacpp.UimacppAnalysisComponent.log(396): INFO: > : > >>>>>>>> XClusterAnalyzer::process --- OK > >>>>>>>> 11:51:39.145 - 45: > >>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineContro > >>>>> ller_impl.replyToClient: > >>>>>>>> WARNING: Service: XClusterAnalyzerAggregate Runtime Exception > >>>>>>>> 11:51:39.145 - 45: > >>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineContro > >>>>> ller_impl.replyToClient: > >>>>>>>> WARNING: > >>>>>>>> org.apache.uima.aae.error.AsynchAEException: > >>>>>>>> org.apache.uima.UIMARuntimeException > >>>>>>>> at > >>>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.getSer > >>>>> ializedCas(JmsOutputChannel.java:1265) > >>>>>>>> at > >>>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendRe > >>>>> ply(JmsOutputChannel.java:800) > >>>>>>>> at > >>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineContro > >>>>> ller_impl.sendReplyToRemoteClient(AggregateAnalysisEngineCon > >>>>> troller_impl.java:2173) > >>>>>>>> at > >>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineContro > >>>>> ller_impl.replyToClient(AggregateAnalysisEngineControl > >> ler_impl.java:2342) > >>>>>>>> at > >>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineContro > >>>>> ller_impl.finalStep(AggregateAnalysisEngineControl > ler_impl.java:1862) > >>>>>>>> at > >>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineContro > >>>>> ller_impl.executeFlowStep(AggregateAnalysisEngineController_ > >>>>> impl.java:2489) > >>>>>>>> at > >>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineContro > >>>>> ller_impl.process(AggregateAnalysisEngineController_impl.java:1271) > >>>>>>>> at > >>>>>>>> org.apache.uima.aae.handler.HandlerBase.invokeProcess(Handle > >>>>> rBase.java:118) > >>>>>>>> at > >>>>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.can > >>>>> celTimerAndProcess(ProcessResponseHandler.java:117) > >>>>>>>> at > >>>>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.han > >>>>> dleProcessResponseWithCASReference(ProcessResponseHandler.java:485) > >>>>>>>> at > >>>>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.han > >>>>> dle(ProcessResponseHandler.java:767) > >>>>>>>> at > >>>>>>>> org.apache.uima.aae.handler.HandlerBase.delegate(HandlerBase > >>>>> .java:149) > >>>>>>>> at > >>>>>>>> org.apache.uima.aae.handler.input.ProcessRequestHandler_impl > >>>>> .handle(ProcessRequestHandler_impl.java:1113) > >>>>>>>> at > >>>>>>>> org.apache.uima.aae.spi.transport.vm.UimaVmMessageListener.o > >>>>> nMessage(UimaVmMessageListener.java:107) > >>>>>>>> at > >>>>>>>> org.apache.uima.aae.spi.transport.vm.UimaVmMessageDispatcher > >>>>> $1.run(UimaVmMessageDispatcher.java:70) > >>>>>>>> at > >>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool > >>>>> Executor.java:1145) > >>>>>>>> at > >>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo > >>>>> lExecutor.java:615) > >>>>>>>> at > >>>>>>>> org.apache.uima.aae.UimaAsThreadFactory$1.run(UimaAsThreadFa > >>>>> ctory.java:132) > >>>>>>>> at java.lang.Thread.run(Thread.java:745) > >>>>>>>> Caused by: org.apache.uima.UIMARuntimeException > >>>>>>>> at > >>>>>>>> org.apache.uima.cas.impl.XmiCasSerializer.serialize(XmiCasSe > >>>>> rializer.java:420) > >>>>>>>> at > >>>>>>>> org.apache.uima.cas.impl.XmiCasSerializer.serialize(XmiCasSe > >>>>> rializer.java:385) > >>>>>>>> at > >>>>>>>> org.apache.uima.aae.UimaSerializer.serializeCasToXmi(UimaSer > >>>>> ializer.java:145) > >>>>>>>> at > >>>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.serial > >>>>> izeCAS(JmsOutputChannel.java:251) > >>>>>>>> at > >>>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.getSer > >>>>> ializedCas(JmsOutputChannel.java:1250) > >>>>>>>> ... 18 more > >>>>>>>> Caused by: java.lang.NullPointerException > >>>>>>>> at > >>>>>>>> org.apache.uima.cas.impl.CasSerializerSupport$CasDocSerializ > >>>>> er.getSofaAddr(CasSerializerSupport.java:454) > >>>>>>>> at > >>>>>>>> org.apache.uima.cas.impl.CasSerializerSupport$CasDocSerializ > >>>>> er.writeViewsCommons(CasSerializerSupport.java:465) > >>>>>>>> at > >>>>>>>> org.apache.uima.cas.impl.XmiCasSerializer$XmiDocSerializer. > >>>>> writeViews(XmiCasSerializer.java:572) > >>>>>>>> at > >>>>>>>> org.apache.uima.cas.impl.CasSerializerSupport$CasDocSerializ > >>>>> er.serialize(CasSerializerSupport.java:441) > >>>>>>>> at > >>>>>>>> org.apache.uima.cas.impl.XmiCasSerializer.serialize(XmiCasSe > >>>>> rializer.java:415) > >>>>>>>> ... 22 more > >>>>>>>> > >>>>>> > >>>> > >
