The sax error is because the CR is putting characters in the CAS that cannot be serialized to XML. The JobDriver sends an XML serialized CAS to the JPs.
I believe that all_in_one may not do the XML serialization, so that specific error would not be found by all_in_one. It could be useful to configure the UIMA log configuration file to log at FINE in order to see exactly what component was last running in all_in_one. Information on changing the UIMA log configuration is at: https://uima.apache.org/downloads/releaseDocs/2.1.0-incubating/docs/html/tutorials_and_users_guides/tutorials_and_users_guides.html#ugr.tug.aae.logging Eddie On Tue, Nov 17, 2015 at 6:07 PM, Yi-Wen Liu <[email protected]> wrote: > Hi, > > I run both, remotely and locally, and no errors from logs. > 2015.11.17 14:47:17 Tue Received > 2015.11.17 14:47:17 Tue WaitingForResources > 2015.11.17 14:47:22 Tue Initializing > 2015.11.17 14:47:22 Tue Running > 2015.11.17 14:48:18 Tue Completing > 2015.11.17 14:48:18 Tue Completed > 2015.11.17 14:48:18 Tue completion type: ProgramExit > 2015.11.17 14:48:18 Tue rationale: code=1 > > However I didn't get any output. > If run without all in one there's output, it is quite confusing.. > > Thanks, > Yi-Wen > > On Tue, Nov 17, 2015 at 3:04 PM, Lou DeGenaro <[email protected]> > wrote: > > > Yi-Wen, > > > > Oh, I see now you ran all-in-one local. But that had errors or ran to > > completion successfully? > > > > Lou. > > > > On Tue, Nov 17, 2015 at 5:29 PM, Lou DeGenaro <[email protected]> > > wrote: > > > > > Yi-Wen, > > > > > > With respect to the ducc.log, I still saw that the DUCC Job was Running > > > as far as the JD was concerned (as evidenced by the JD's log file). > > > There may be some latency in reflecting this fact back to the > > Orchestrator > > > daemon from the JD. Note that the Orchestrator writes the ducc.log > > > file. The ducc.log not showing Running could be the subject of a Jira, > > > but is not of concern with respect to getting your Job to actually run. > > > > > > I'm not sure I understand your statements regarding All-In-One testing. > > > You ran your Job successfully without any error using All-In-One, or > that > > > had some issue too? Did you run All-In-One locally or remotely? > > > > > > Lou. > > > > > > On Tue, Nov 17, 2015 at 5:18 PM, Yi-Wen Liu <[email protected]> wrote: > > > > > >> Hi, > > >> > > >> The ducc.log didn't show running: > > >> 2015.11.16 23:29:26 Mon Received > > >> 2015.11.16 23:29:26 Mon WaitingForDriver > > >> 2015.11.16 23:29:54 Mon WaitingForResources > > >> 2015.11.16 23:30:04 Mon Assigned > > >> 2015.11.16 23:30:14 Mon Initializing > > >> 2015.11.16 23:34:48 Mon Completing > > >> 2015.11.16 23:35:18 Mon Completed > > >> 2015.11.16 23:35:18 Mon completion type: CanceledByDriver > > >> 2015.11.16 23:35:18 Mon rationale: unspecified > > >> > > >> And I didn't find error in all_in_one local: > > >> > > >> [1] Loading model: > > >> > > >> [1] ... > > >> > > >> [1] Loading model: > > >> > > >> [1] . > > >> > > >> [1] Loading configuration. > > >> > > >> [1] Loading feature templates. > > >> > > >> [1] Loading lexica. > > >> > > >> [1] Loading model: > > >> > > >> [1] ................................ > > >> > > >> [1] Loading model: > > >> > > >> [1] ............................. > > >> > > >> [1] 17 Nov 2015 13:49:33 INFO Chunker - Chunker model file: > > >> org/apache/ctakes/chunker/models/chunker-model.zip > > >> > > >> [1] 17 Nov 2015 13:49:35 INFO JdbcConnectionResourceImpl - Connection > > >> established to: > > >> jdbc:hsqldb:res:org/apache/ctakes/dictionary/lookup/umls2011ab/umls > > >> > > >> [1] 17 Nov 2015 13:49:35 INFO JdbcConnectionResourceImpl - Connection > > >> established to: > > >> jdbc:hsqldb:res:org/apache/ctakes/dictionary/lookup/rxnorm-hsqldb/umls > > >> > > >> [1] 17 Nov 2015 13:49:35 INFO JdbcConnectionResourceImpl - Connection > > >> established to: > > >> > > >> > > > jdbc:hsqldb:res:org/apache/ctakes/dictionary/lookup/orange_book_hsqldb/umls > > >> > > >> [1] 17 Nov 2015 13:49:35 INFO UmlsDictionaryLookupAnnotator - Parsing > > >> descriptor: > > >> > > >> > > > /Users/ducc/src/apache-ctakes-3.2.2/resources/org/apache/ctakes/dictionary/lookup/LookupDesc_Db.xml > > >> > > >> [1] 17 Nov 2015 13:49:35 INFO FirstTokenPermLookupInitializerImpl - > > >> Exclusion tagset loaded: [dt, to, rp, ls, pos, md, vbd, vbg, vb, ex, > > vbp, > > >> vbn, pdt, vbz, wp, wrb, in, wps, pp$, prp$, wdt, prp, pp, cc, cd] > > >> > > >> [1] 17 Nov 2015 13:49:35 INFO FirstTokenPermLookupInitializerImpl - > > >> Exclusion tagset loaded: [to, dt, rp, ex, vbp, ls, vbn, pdt, wp, vbz, > > wrb, > > >> in, pos, wps, md, wdt, pp$, vbd, vb, vbg, pp, cc, cd] > > >> > > >> [1] 17 Nov 2015 13:49:35 INFO UmlsDictionaryLookupAnnotator - Using > > >> ctakes.umlsaddr: https://uts-ws.nlm.nih.gov/restful/isValidUMLSUser: > > >> yiwenliu > > >> > > >> [1] 17 Nov 2015 13:49:39 INFO LvgCmdApiResourceImpl - Loading NLM > Norm > > >> and > > >> Lvg with config file = > > >> > > >> > > > /Users/ducc/src/apache-ctakes-3.2.2/resources/org/apache/ctakes/lvg/data/config/lvg.properties > > >> > > >> [1] 17 Nov 2015 13:49:39 INFO LvgCmdApiResourceImpl - config file > > >> absolute path = > > >> > > >> > > > /Users/ducc/src/apache-ctakes-3.2.2/resources/org/apache/ctakes/lvg/data/config/lvg.properties > > >> > > >> [1] 17 Nov 2015 13:49:39 INFO LvgCmdApiResourceImpl - cwd = > > >> /Users/ducc/src > > >> > > >> [1] 17 Nov 2015 13:49:39 INFO LvgCmdApiResourceImpl - cd > > >> /Users/ducc/src/apache-ctakes-3.2.2/resources/org/apache/ctakes/lvg/ > > >> > > >> [1] 17 Nov 2015 13:49:40 INFO LvgCmdApiResourceImpl - cd > > /Users/ducc/src > > >> > > >> It looks like it suddenly canceled by driver and no more output on > > screen. > > >> Another log file shows connection refused so I checked driver logs, > and > > >> found the three errors in first email. > > >> > > >> Is that error org.xml.sax.SAXParseException; Trying to serialize > non-XML > > >> 1.0 character caused by invalid space and characters in Xml, or it's > > more > > >> like syntax error of CR Xml? > > >> > > >> Thanks, > > >> Yi-Wen > > >> > > >> On Tue, Nov 17, 2015 at 1:36 PM, Lou DeGenaro <[email protected] > > > > >> wrote: > > >> > > >> > Hi Yi-Wen, > > >> > > > >> > Does your CR function correctly outside of DUCC? Have you tired > > >> debugging > > >> > using DUCC's All-In-One feature as suggested previously by Eddie? > > >> > > > >> > Lou. > > >> > > > >> > On Tue, Nov 17, 2015 at 3:54 PM, Yi-Wen Liu <[email protected]> > wrote: > > >> > > > >> > > Hi, > > >> > > > > >> > > I tried to remove redundant white spaces but it doesn't work, and > I > > >> > didn't > > >> > > add any other special characters in CR. > > >> > > > > >> > > Is that error caused by invalid space and characters in Xml, or > it's > > >> more > > >> > > like syntax error of CR Xml? > > >> > > > > >> > > > > >> > > > > >> > > Thanks, > > >> > > Yi-Wen > > >> > > > > >> > > > On Nov 17, 2015, at 5:37 AM, Lou DeGenaro < > [email protected] > > > > > >> > > wrote: > > >> > > > > > >> > > > Yi-Wen, > > >> > > > > > >> > > > The DUCC Job first visits the Initialization state. This is > when > > >> the > > >> > Job > > >> > > > Driver (JD) initializes your UIMA CollecitonReader (CR). The CR > > that > > >> > you > > >> > > > specified in your DUCC Job submission is constructed and its > > >> getTotal() > > >> > > > method is invoked to determine the number of work items. But I > > >> think > > >> > > your > > >> > > > Job made it past the Initialization state and into the Running > > >> state. > > >> > > > Looking at the log snippits that you posted, I see: > > >> > > > > > >> > > > 16 Nov 2015 23:34:38,353 INFO JobDriver - T[18] advanceJdState > > >> > > > current=Initializing request=Active result=Active > > >> > > > > > >> > > > This means the JD has exited its Initializing state and is now > > >> Active > > >> > and > > >> > > > is fetching work items. Further evidence is the entry: > > >> > > > > > >> > > > 16 Nov 2015 23:34:39,066 ERROR CasManager - T[17] getMetaCas > > >> > > > > > >> > > > What do the contents of the file ducc.log in your Job's log > > >> directory > > >> > > > show? You should see "Initializing" followed by "Running". > > >> > > > > > >> > > > It looks like the root cause of your problem is that your CR's > > >> > getNext() > > >> > > is > > >> > > > trying to produce work items for the JD but is failing: > > >> > > > > > >> > > > org.xml.sax.SAXParseException; Trying to serialize non-XML 1.0 > > >> > > > character: , 0x0 > > >> > > > at > > >> > > org.apache.uima.util.XMLSerializer$CharacterValidatingContentHand > > >> > > > ler.checkForInvalidXmlChars(XMLSerializer.java:254) > > >> > > > > > >> > > > I recommend determining why your CR is broken, fixing, and > > >> > re-submitting > > >> > > > your Job to DUCC. > > >> > > > > > >> > > > Lou. > > >> > > > > > >> > > >> On Tue, Nov 17, 2015 at 2:46 AM, Yi-Wen Liu <[email protected]> > > >> wrote: > > >> > > >> > > >> > > >> Hello, > > >> > > >> > > >> > > >> I am still stuck in the following error of DUCC and job always > > got > > >> > > canceled > > >> > > >> by driver: > > >> > > >> > > >> > > >> 16 Nov 2015 23:34:38,353 INFO JobDriver - T[18] advanceJdState > > >> > > >> current=Initializing request=Active result=Active > > >> > > >> 16 Nov 2015 23:34:39,066 ERROR ProxyLogger - T[17] loggify > > >> > > >> org.xml.sax.SAXParseException; Trying to serialize non-XML 1.0 > > >> > > >> character: , 0x0 > > >> > > >> at > > >> > > >> > > >> > > > > >> > > > >> > > > org.apache.uima.util.XMLSerializer$CharacterValidatingContentHandler.checkForInvalidXmlChars(XMLSerializer.java:254) > > >> > > >> > > >> > > >> 16 Nov 2015 23:34:39,066 ERROR CasManager - T[17] getMetaCas > > >> > > >> > org.apache.uima.ducc.container.common.classloader.ProxyException > > >> > > >> at > > >> > > >> > > >> > > > > >> > > > >> > > > org.apache.uima.ducc.container.jd.classload.ProxyJobDriverCollectionReader.getMetaCas(ProxyJobDriverCollectionReader.java:127) > > >> > > >> at > > >> > > >> > > >> > > > > >> > > > >> > > > org.apache.uima.ducc.container.jd.cas.CasManager.getMetaCas(CasManager.java:65) > > >> > > >> at > > >> > > >> > > >> > > > > >> > > > >> > > > org.apache.uima.ducc.container.jd.fsm.wi.ActionGet.engage(ActionGet.java:118) > > >> > > >> at > > >> > > >> > > >> org.apache.uima.ducc.container.common.fsm.Fsm.transition(Fsm.java:172) > > >> > > >> > > >> > > >> 16 Nov 2015 23:34:39,068 ERROR ActionGet - T[17] engage > > >> > > >> org.apache.uima.ducc.container.jd.JobDriverException > > >> > > >> at > > >> > > >> > > >> > > > > >> > > > >> > > > org.apache.uima.ducc.container.jd.cas.CasManager.getMetaCas(CasManager.java:74) > > >> > > >> at > > >> > > >> > > >> > > > > >> > > > >> > > > org.apache.uima.ducc.container.jd.fsm.wi.ActionGet.engage(ActionGet.java:118) > > >> > > >> > > >> > > >> > > >> > > >> However, when I looked into output directory, there were output > > >> (much > > >> > > >> omitted from below): > > >> > > >> > > >> > > >> <?xml version="1.0" encoding="UTF-8"?><CAS version="2"> > > >> > > >> <uima.cas.Sofa _indexed="0" _id="1" sofaNum="1" > > >> > sofaID="_InitialView" > > >> > > >> mimeType="text" sofaString=" Dr. > > >> > > >> Nutritious Medical Nutrition Therapy for > > >> > > >> Hyperlipidemia Referral from: Julie Tester, > > RD, > > >> LD, > > >> > > >> CNSD Phone contact: (555) 555-1212 Height: > 144 > > cm > > >> > > >> Current Weight: 45 kg Date of current weight: 02-29-2001 > > Admit > > >> > > Weight: > > >> > > >> 53 kg BMI: 18 kg/m2 Diet: General Daily > > Calorie > > >> > > needs > > >> > > >> (kcals): 1500 calories, assessed as HB + 20% for > > >> > > activity. Daily > > >> > > >> Protein needs: 40 grams, assessed as 1.0 g/kg. Pt has > > >> been > > >> > on > > >> > > a > > >> > > >> 3-day calorie count and has had an average intake of 1100 > > calories. > > >> > She > > >> > > >> was instructed to drink 2-3 cans of liquid supplement to help > > >> promote > > >> > > >> weight gain. She agrees with the plan and has my number for > > >> further > > >> > > >> assessment. May want a Resting Metabolic Rate as well. She > takes > > an > > >> > > aspirin > > >> > > >> a day for knee pain. "/> > > >> > > >> <org.apache.ctakes.typesystem.type.structured.DocumentID > > >> > _indexed="1" > > >> > > >> _id="8" documentID="testpatient_plaintext_1.txt"/> > > >> > > >> <uima.tcas.DocumentAnnotation _indexed="1" _id="10" > > >> _ref_sofa="1" > > >> > > >> begin="0" end="735" language="x-unspecified"/> > > >> > > >> <org.apache.ctakes.typesystem.type.textspan.Segment > > _indexed="1" > > >> > > >> _id="15" _ref_sofa="1" begin="0" end="735" > id="SIMPLE_SEGMENT"/> > > >> > > >> <org.apache.ctakes.typesystem.type.textspan.Sentence > > >> _indexed="1" > > >> > > >> _id="21" _ref_sofa="1" begin="2" end="16" sentenceNumber="0"/> > > >> > > >> .. > > >> > > >> .. > > >> > > >> .. > > >> > > >> <org.apache.ctakes.typesystem.type.refsem.UmlsConcept > > _id="2975" > > >> > > >> codingScheme="SNOMED" code="103733002" oid="103733002#SNOMED" > > >> > > score="0.0" > > >> > > >> disambiguated="false" cui="C0087111" tui="T061"/> > > >> > > >> <uima.cas.FSArray _id="3005" size="4"> > > >> > > >> <i>2975</i> > > >> > > >> <i>2985</i> > > >> > > >> <i>2965</i> > > >> > > >> <i>2995</i> > > >> > > >> </uima.cas.FSArray> > > >> > > >> <org.apache.ctakes.typesystem.type.refsem.UmlsConcept > > _id="2924" > > >> > > >> codingScheme="SNOMED" code="243121000" oid="243121000#SNOMED" > > >> > > score="0.0" > > >> > > >> disambiguated="false" cui="C0418981" tui="T061"/> > > >> > > >> <uima.cas.FSArray _id="2934" size="1"> > > >> > > >> <i>2924</i> > > >> > > >> </uima.cas.FSArray> > > >> > > >> <org.apache.ctakes.typesystem.type.refsem.UmlsConcept > > _id="2870" > > >> > > >> codingScheme="RXNORM" code="1191" oid="1191#RXNORM" score="0.0" > > >> > > >> disambiguated="false" cui="C0004057" tui="T121"/> > > >> > > >> <org.apache.ctakes.typesystem.type.refsem.UmlsConcept > > _id="2880" > > >> > > >> codingScheme="RXNORM" code="1191" oid="1191#RXNORM" score="0.0" > > >> > > >> disambiguated="false" cui="C0004057" tui="T109"/> > > >> > > >> <uima.cas.FSArray _id="2890" size="2"> > > >> > > >> <i>2880</i> > > >> > > >> <i>2870</i> > > >> > > >> </uima.cas.FSArray> > > >> > > >> </CAS> > > >> > > >> > > >> > > >> > > >> > > >> I am not sure what's going on here, the job wasn't in running > > stage > > >> > but > > >> > > has > > >> > > >> output... > > >> > > >> It always got canceled by driver and showed run fails after > > >> > > initializing, > > >> > > >> so I didn't expect there would have any output file. > > >> > > >> And I also doubt whether the content in output file is > > >> correct....any > > >> > > >> suggestion is appreciated! > > >> > > >> > > >> > > >> Thanks, > > >> > > >> Yi-Wen > > >> > > >> > > >> > > > > >> > > > >> > > > > > > > > >
