Hi Bhagwat. You wrote: > Now coming back to our original question for which this thread has been > initiated. > I was trying to get the context out of clinical notes. for ex. If any > procedure is happened or not on patient. > ... > By following this clues, I am nowhere. > Could you please provide information on which pipeline I should have to run > to achieve this goal, and how?
Your snippet of my previous message is incomplete. The sentence that I wrote following what you copied was: --- In those files each "add" statement is followed by an annotator class name. Those were your clues. Those annotator classes provide the information that you were asking about. Here are the contents of the AttributeClearTkSubPipe.piper: ========================================== // Commands and parameters to create a default entity attributes processing sub-pipeline. This is not a full pipeline. // Add the Dependency parser for use by cleartk addDescription ClearNLPDependencyParserAE // Add the Semantic Role Labeler parser for use by cleartk addLogged ClearNLPSemanticRoleLabelerAE // Add the cleartk package for cleartk class lookups package org.apache.ctakes.assertion.medfacts.cleartk // Add the cleartk attribute engines, logging for them as a block add StartFinishLogger LOGGER_NAME=CleartkAnalysisEngine LOGGER_TASK="Assigning Attributes" IS_START=true addDescription PolarityCleartkAnalysisEngine addDescription UncertaintyCleartkAnalysisEngine addDescription HistoryCleartkAnalysisEngine addDescription ConditionalCleartkAnalysisEngine addDescription GenericCleartkAnalysisEngine addDescription SubjectCleartkAnalysisEngine add StartFinishLogger LOGGER_NAME=CleartkAnalysisEngine LOGGER_TASK="Assigning Attributes" ========================================== The comments in the file provide hints, and the names of the annotator classes should give an even greater indication of function. "PolarityCleartkAnalysisEngine" assigns polarity (negation), "UncertaintyCleartkAnalysisEngine" assigns uncertainty, etc. It is far from exhaustive, but you should always browse the ctakes wiki before posting to the dev list. The page on Assertion: https://cwiki.apache.org/confluence/display/CTAKES/cTAKES+4.0+-+Assertion If you visit the page on the default clinical pipeline you will see that it extracts negation, uncertainty and subject (right at the top, plus a diagram). https://cwiki.apache.org/confluence/display/CTAKES/Default+Clinical+Pipeline Near the bottom of that wiki page is stated "... runClinicalPipeline runs the piper file DefaultFastPipeline.piper ...", and "piper file" is a link. https://cwiki.apache.org/confluence/display/CTAKES/Piper+Files Near the bottom of the piper files wiki page is a diagram of the DefaultFastPipeline.piper that clearly indicates that it uses the AttributeClearTkSubPipe.piper, and it even lists the contents (as above) of that sub-pipeline. So, I am really confused as to what information you really want. You actually started this topic with: > I have seen the results of user installation of cTakes , the output is pretty > interesting. > Can anybody point to the source code of the same? Gandi pointed you towards the ctakes source repository. You apparently got the source code for ctakes, and then stated: > Now I am looking for the source-code that gives this result for a clinical > note. Tim pointed you to Apache UIMA and the different modules that make up the default clinical pipeline. I pointed you toward the wiki page for some additional information. You then stated: > I have run ctakes-example project that internally executes > ClinicalPipelineFactory.getDefaultPipeline() method. > This has given results like POS tag, Polarity, etc. > Now, I am more interested in finding Procedure, Medication, Drug, etc. > Could you please point me to the code file ... At this point Gandhi pointed to some helpful code. Then later you asked about temporal links because you thought that you needed them to provide certain information. > I think TLINKS values on web API maybe helpful to get the required result, > not sure though. Please confirm > I am not able to find the code to generate TLINKS, could you please help me > locate the code. Tim (correctly) stated that you might be more interested in event/entity attributes such as negation and pointed you towards modules that make up the default pipeline. Then, in your last message you ask: > Could you please provide information on which pipeline I should have to run > to achieve this goal, and how? Since we seem to have gone back and forth on this, I think that it might help everybody trying to help you if you could start over and clearly state: 1) What information you are trying to obtain from a note. An example sentence would help. 2) What you have already done to successfully or unsuccessfully obtain the information. 3) What you do understand at this point in time. 4) What you think is missing from your understanding. 5) What you really need from us. A workflow? A pipeline? Pointers toward existing code? Pointers toward datatypes? A deeper explanation of existing code (pick 1 class)? Help writing new ctakes code of your own? Thank you, Sean -----Original Message----- From: Bhagwat Posane [mailto:bhagwat.pos...@gmail.com] Sent: Monday, November 27, 2017 12:49 AM To: dev@ctakes.apache.org Subject: Re: Interpret context out of clinical note [EXTERNAL] Hello Sean, Thank you for the inputs, Now coming back to our original question for which this thread has been initiated. I was trying to get the context out of clinical notes. for ex. If any procedure is happened or not on patient. To achieve this you mentioned some points : You can visit the ctakes-assertion-res module for some clues. > Navigate to ctakes-assertion-res src/main/resources/ > org/apache/ctakes/assertion/pipeline/ > There are two .piper files there. Each one covers a different way to > get the information that you mentioned. By following this clues, I am nowhere. Could you please provide information on which pipeline I should have to run to achieve this goal, and how? On 24 November 2017 at 21:24, Finan, Sean <sean.fi...@childrens.harvard.edu> wrote: > Hi Bhagwat, > > Ok, a couple of things are problematic. > > 1. the DefaultFastPipeline already contains attribute detection using > the AttributeClearTkSubPipe.piper. That sub- pipeline adds the > ClearNLPSemanticRoleLabelerAE. So does the AssertionSubPipe.piper. > The pipeline cannot have two instances of that annotator. The fix for > this is to just add the AssertionEngines that are at the bottom of the > AssertionSubPipe.piper. > > 2. The assertion engines don't work. They will spit out some log > messages about not being usable until ctakes-76 is implemented. Then > you will get a NullPointerException and ctakes will happily crash. > > James Masanz made the last comment on ctakes-76: > https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org > _jira_browse_CTAKES-2D76&d=DwIBaQ&c=qS4goWBT7poplM69zy_3xhKwEW14JZMSdi > oCoppxeFU&r=fs67GvlGZstTpyIisCYNYmQCP6r0bcpKGd4f7d4gTao&m=wGmqUbPS3-WJ > RSYmkNbjZ1e-OfMJTOVk6wbzOElQUrE&s=ZyPD1MbOImp08cCdSo0HeecCXAqUatZEZmhA > W-cJVtc&e= > which includes the information: > "when this issue gets resolved (the jars are available from maven > central) then ctakes-assertion could be updated again to use" > > So, what to do? > Well, my advice would be to just skip the Assertion engines. The > default ClearTk attribute engines will produce the same types of event > and entity attributes and they are already in the default pipeline. > Is there a reason that you wanted to use the older Assertion engines? > Was it just listed in (unfortunately) outdated documentation? > > Sean > > > -----Original Message----- > From: Bhagwat Posane [mailto:bhagwat.pos...@gmail.com] > Sent: Friday, November 24, 2017 8:58 AM > To: dev@ctakes.apache.org > Subject: Re: Interpret context out of clinical note [EXTERNAL] > > Hello Sean, > > As you mentioned in previous email The annotators in the Assertion > piper require several other annotators to run first, so to run > AssertionSubPipe.piper I have added DefaultFastPipeline and > AssertionDefaults piper file before that. > > for reference below text is from my piper file: > > ************************************************************ > ********************* > reader FileTreeReader InputDirectory=D:\Bhagwat\Notes load > org/apache/ctakes/clinical/pipeline/DefaultFastPipeline.piper > load org/apache/ctakes/examples/pipeline/AssertionDefaults.piper > load AssertionSubPipe.piper > > // Temporal (event, time, dtr, tlink) > load TemporalSubPipe.piper > > // Html output > set OutputDirectory=D:\Bhagwat\HTMLoutput > add pretty.html.HtmlTextWriter > > //XMI output > //writeXmis D:\Bhagwat\XMI > ************************************************************ > ********************* > > As I run this piper file the command addLogged > ClearNLPSemanticRoleLabelerAE It is showing started processing.... > and never finished. when I debugged the code I found it is going in > infinite loop for one of the sentence from clinical note. > > The code which goes in Infinite loop is in DEPNode.class file > ************************************************************ > ************************************************************ > ******************** > public boolean isDescendentOf(DEPNode node) { DEPNode head = > getHead(); while (head != null) { if (head == node) return true; head > = head.getHead(); } return false; } > ************************************************************ > ************************************************************ > ******************** > if (head == node) this condition never becomes true ! > > > I have added clinical notes for your reference,for the highlighted > line the ClearNLPSemanticRoleLablerAE goes in infinite loop. > > ************************************************************ > ************************************************************ > ********************* > Note 1: > The patient is XX years old with gross hematuria beginning ten days ago. > He has pain over the bladder. > He smoked until XXXX. His country of origin is XXXX. He has been > here for two years. > Ultrasound is planned on 3/26/2015 shows a mass in the bladder 3.4 x > 3.8 x > 3.1 cm. > > > > Note 2: > This is a 67-year-old patient followed for erectile dysfunction and benign > prostatic hyperplasia. Prostate symptoms consist of slow stream and NX > X 2. TUMT explained to the patient. > Urinalysis showed 2+ sugar before and trace of blood. No retention, > burning or blood. He is up perhaps once at night to void depending > upon his fluid > intake. PSA was .3 on 1/12. > The patient use to take AndroGel squirt three a day. T was 483 in Nov. > 2011. T was 531 on 1/13 by internist. He has been off the Androgel. > Testosterone is rechecked. He may choose to use IM testosterone at home > self administered. He also uses TriMix for ED. > He has diminished number and frequency of erections. Duration is years. > Modifying factors include Diabetes. Severity is 8 of 10. He no logner > uses Dialis. TriMix refilled. > Patient has lost 40 lbs on Victosa administered by his endocrinologist. > > ************************************************************ > ************************************************************ > ********************* > > The Interesting thing is when I run only the highlighted sentence as > a note it executes successfully, it means to get into infinite loop > the highlighted sentence should have at least one sentence with it. > Could you please provide your thoughts on it. > > Sorry for lengthy explanation, but I thought it is necessary. > > > > On 24 November 2017 at 01:22, Finan, Sean <Sean.Finan@childrens.harvard. > edu> > wrote: > > > Hi Bhagwat, > > > > Thanks for the heads up on the "file:" issue. I will check to see > > if the usage has changed and fix the piper. > > > > Sean > > > > -----Original Message----- > > From: Bhagwat Posane [mailto:bhagwat.pos...@gmail.com] > > Sent: Thursday, November 23, 2017 9:17 AM > > To: dev@ctakes.apache.org > > Subject: Re: Interpret context out of clinical note [EXTERNAL] > > > > Thanks James, It worked using cast to Map. ! > > > > Sean, > > In AssertionDefaults.piper you set the file like: > > > > set > > assertionModelResource=file:org/apache/ctakes/assertion/models/i2b2. > > mo > > del > > > > As I run this piper file it throws file not found exception, it > > resolved for me by removing "file:" from command, I suppose this > > needs > to be fixed. > > > > > > On 22 November 2017 at 23:21, Finan, Sean <Sean.Finan@childrens.harvard. > > edu> > > wrote: > > > > > Wow, that is really strange. > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__uima.apache.o > > > rg > > > _d > > > _uimafit-2Dcurrent_api_org_&d=DwIBaQ&c=qS4goWBT7poplM69zy_3xhKwEW1 > > > 4J > > > ZM > > > SdioCoppxeFU&r=fs67GvlGZstTpyIisCYNYmQCP6r0bcpKGd4f7d4gTao&m=1A9as > > > f1 > > > ga > > > bEGz-5fQfVfXbp2BcZEJaZIiJ4zSk09P1M&s=jo0yN7BJTt98IHerLXLK4aQvmSOAn > > > 90 > > > pU > > > lzHg00ynFI&e= > > > apache/uima/fit/util/JCasUtil.html > > > > > > > > > > > > -----Original Message----- > > > From: James Masanz [mailto:masanz.ja...@gmail.com] > > > Sent: Wednesday, November 22, 2017 12:50 PM > > > To: dev@ctakes.apache.org > > > Subject: Re: Interpret context out of clinical note [EXTERNAL] > > > > > > If I use Eclipse, at build time I also get the error > > > > > > method putAll(Map<? extends Annotation,? extends > > > List<IdentifiedAnnotation>>) in the type Map<Annotation,List< > > > IdentifiedAnnotation>> > > > is not applicable for the arguments (Map) > > > > > > I haven't looked into why. > > > > > > A quick workaround is to add a cast to Map > > > > > > allCovering.putAll( (Map)JCasUtil.indexCovering( jCas, covered, > > > covering ) ); > > > > > > > > > > > > > > > On Wed, Nov 22, 2017 at 8:40 AM, Finan, Sean < > > > sean.fi...@childrens.harvard.edu> wrote: > > > > > > > Hi Bhagwat, > > > > > > > > I have no idea what is going on with your system. > > > > > > > > > The method putAll(Map<? extends Annotation,? extends > > > > > List<IdentifiedAnnotation>>) in the type > > > > > Map<Annotation,List<IdentifiedAnnotation>> is not applicable > > > > > for the arguments (Map) > > > > > > > > That doesn't make any sense to me. That is exactly what putAll > > > > is for ... Unless your uimafit does not return a typed map? In > > > > which case it is an old version? I am not sure how that would > happen. > > > > > > > > > When I run AssertionSubPipe.piper I am getting below error: > > > > The .piper files should all have line comments in them. The > > > > *SubPipe pipers should all have something like "This is not a > > > > complete pipeline" at the top. This means that it cannot be run > > > > alone. The annotators in the Assertion piper require several > > > > other annotators to run first. In addition to the default > > > > clinical piper I think that there are some piper files in > > > > ctakes-example-res that you can look at to > > > start. > > > > > > > > >, I don't find any variables value in understandable format > > > > >like (JCas > > > > jcas) object. How would I understand JCas object values? > > > > > > > > ctakes is built upon the Apache UIMA framework. > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__uima.apache > > > > .o > > > > rg > > > > _& > > > > d=DwIBaQ&c=qS4goWBT7poplM69zy_3xhKwEW14JZMSdioCoppxeFU&r=fs67Gvl > > > > GZ > > > > st > > > > Tp > > > > yIisCYNYmQCP6r0bcpKGd4f7d4gTao&m=rVKcbkbznwk6V-PLUy56sA4hGgN0F4q > > > > rL > > > > N5 Ik 6FIDnM&s=ebIFQbW0D4VLpTWv9a4YQqy_G30S_hDeDEbhCTesFuk&e= > > > > > > > > This should probably be on our wiki ... Anyway, the best way to > > > > learn about how ctakes pipelines are put together and how data > > > > is passed is to read some of the intro documentation for uima. > > > > > > > > The Javadoc for jcas is here: > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__uima.apache > > > > .o > > > > rg > > > > _d > > > > _&d=DwIBaQ&c=qS4goWBT7poplM69zy_3xhKwEW14JZMSdioCoppxeFU&r=fs67G > > > > vl > > > > GZ > > > > st > > > > TpyIisCYNYmQCP6r0bcpKGd4f7d4gTao&m=rVKcbkbznwk6V-PLUy56sA4hGgN0F > > > > 4q > > > > rL > > > > N5 Ik6FIDnM&s=I_7BH2bDGR48vI03bE1dirKGCgq33PKB9vXN2VDrMA8&e= > > > > uimaj-current/apidocs/org/apache/uima/jcas/JCas.html > > > > > > > > Sean > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: Bhagwat Posane [mailto:bhagwat.pos...@gmail.com] > > > > Sent: Wednesday, November 22, 2017 3:38 AM > > > > To: dev@ctakes.apache.org > > > > Subject: Re: Interpret context out of clinical note [EXTERNAL] > > > > > > > > Hello Sean, > > > > > > > > Thank you for the inputs. I have gone through both the piper > > > > files which you have mentioned. > > > > > > > > When I run AssertionSubPipe.piper I am getting below error: > > > > > > > > //////////////////////////////////////////////////////////// > > > > //////////////////////////////////////////////////////////// > > > > //////////////////////////////////////////// > > > > > > > > 22 Nov 2017 12:55:31 WARN SingleDocumentProcessor - This class > > > > cannot be used until CTAKES-76 is implemented. > > > > 22 Nov 2017 12:55:31 WARN SingleDocumentProcessor - This class > > > > cannot be used until CTAKES-76 is implemented. > > > > Nov 22, 2017 12:55:31 PM > > > > org.apache.uima.analysis_engine.impl.PrimitiveAnalysisEngine_imp > > > > l > > > > callAnalysisComponentProcess(430) > > > > SEVERE: Exception occurred > > > > org.apache.uima.analysis_engine.AnalysisEngineProcessException: > > > > Annotator processing failed. > > > > at org.apache.uima.analysis_engine.impl. > PrimitiveAnalysisEngine_impl. > > > > callAnalysisComponentProcess(PrimitiveAnalysisEngine_impl.java:4 > > > > 12 > > > > ) at > > > > org.apache.uima.analysis_engine.impl.PrimitiveAnalysisEngine_impl. > > > > processAndOutputNewCASes(PrimitiveAnalysisEngine_impl.java:314) > > > > at org.apache.uima.analysis_engine.asb.impl.ASB_impl$ > > > AggregateCasIterator. > > > > processUntilNextOutputCas(ASB_impl.java:570) > > > > at org.apache.uima.analysis_engine.asb.impl.ASB_impl$ > > > > AggregateCasIterator.<init>(ASB_impl.java:412) > > > > at org.apache.uima.analysis_engine.asb.impl.ASB_impl. > > > > process(ASB_impl.java:344) > > > > at org.apache.uima.analysis_engine.impl. > AggregateAnalysisEngine_impl. > > > > processAndOutputNewCASes(AggregateAnalysisEngine_impl.java:265) > > > > at org.apache.uima.analysis_engine.asb.impl.ASB_impl$ > > > AggregateCasIterator. > > > > processUntilNextOutputCas(ASB_impl.java:570) > > > > at org.apache.uima.analysis_engine.asb.impl.ASB_impl$ > > > > AggregateCasIterator.<init>(ASB_impl.java:412) > > > > at org.apache.uima.analysis_engine.asb.impl.ASB_impl. > > > > process(ASB_impl.java:344) > > > > at org.apache.uima.analysis_engine.impl. > AggregateAnalysisEngine_impl. > > > > processAndOutputNewCASes(AggregateAnalysisEngine_impl.java:265) > > > > at > > > > org.apache.uima.analysis_engine.impl.AnalysisEngineImplBase.proc > > > > es > > > > s( > > > > AnalysisEngineImplBase.java:269) at > > > > org.apache.uima.fit.pipeline.SimplePipeline.runPipeline( > > > > SimplePipeline.java:150) > > > > at org.apache.ctakes.core.pipeline.PipelineBuilder.run( > > > > PipelineBuilder.java:348) > > > > at org.apache.ctakes.core.pipeline.PiperFileRunner.run( > > > > PiperFileRunner.java:72) > > > > at org.apache.ctakes.core.pipeline.PiperFileRunner.main( > > > > PiperFileRunner.java:30) > > > > Caused by: java.lang.NullPointerException at > > org.apache.ctakes.assertion. > > > > medfacts.AssertionAnalysisEngineFit.process( > > > > AssertionAnalysisEngineFit.java:145) > > > > at > > > > org.apache.uima.analysis_component.JCasAnnotator_ImplBase.proces > > > > s( > > > > JCasAnnotator_ImplBase.java:48) > > > > at org.apache.uima.analysis_engine.impl. > PrimitiveAnalysisEngine_impl. > > > > callAnalysisComponentProcess(PrimitiveAnalysisEngine_impl.java:3 > > > > 96 > > > > ) > > > > ... 14 more > > > > > > > > //////////////////////////////////////////////////////////// > > > > //////////////////////////////////////////////////////////// > > > > //////////////////////////////////////////////////////////// > > > > /////////////////////// > > > > > > > > The error is causing due to the command : add > > > > AssertionAnalysisEngineFit I have comment out this command and > > > > ran it, the html output doesn't have any difference with previous > > > > output. > > > > Could you please help me to resolve this error. > > > > > > > > > > > > On 21 November 2017 at 18:43, Finan, Sean > > <Sean.Finan@childrens.harvard. > > > > edu> > > > > wrote: > > > > > > > > > Hi Bhagwat, > > > > > > > > > > You can visit the ctakes-assertion-res module for some clues. > > > > > > > > > > Navigate to ctakes-assertion-res src/main/resources/ > > > > > org/apache/ctakes/assertion/pipeline/ > > > > > > > > > > There are two .piper files there. Each one covers a different > > > > > way to get the information that Tim mentioned. In those files > each "add" > > > > > statement is followed by an annotator class name. In IntelliJ > > > > > you can press CTRL-N and enter the class name to view the code. > > > > > There are also a few comments in the piper files that may help. > > > > > > > > > > Sean > > > > > > > > > > -----Original Message----- > > > > > From: Bhagwat Posane [mailto:bhagwat.pos...@gmail.com] > > > > > Sent: Tuesday, November 21, 2017 6:21 AM > > > > > To: dev@ctakes.apache.org > > > > > Subject: Interpret context out of clinical note [EXTERNAL] > > > > > > > > > > Hello Sean,Tim, et al, > > > > > > > > > > I would like to Interpret context out of clinical note, e.g. > > > > > interpretation of the procedure, lab tests for a patient. For > > > > > this, I suppose TLINK statements would be of much help. > > > > > > > > > > Ex. for the note- "Cystoscopy is planned on 3 / 26 / 2018." > > > > > I get TLINK as > > > > > Cystoscopy contains 3 / 26 / 2018, > > > > > 3 / 26 / 2018 contains Cystoscopy > > > > > 3 / 26 / 2018 contains planned we can make out that > > > > > Cystoscopy is not yet done looking at "planned"(in this case), > > > > > > > > > > Could you please let me know if I am on right path and also > > > > > help me understand if any other approach to do this. > > > > > > > > > > > > > > > @Tim, > > > > > Couple of days back you mentioned about the context understanding : > > > > > > > > > > "if any procedure is happened or not." > > > > > > I think you need a formal definition of this to understand > > > > > > what you're trying to do. > > > > > > As a guess, you want it to be in the past, related to the > > > > > > patient (not a family member), and not negated or > > > > > > hypothetical or uncertain, > > > > etc.? > > > > > > This would require multiple components, but most of them > > > > > > should run in the temporal pipeline. > > > > > > > > > > > > > > > > Tim > > > > > > > > > > > > > > > Could you please provide some insight on this, as what are the > > > > > multiple components are required and how to use them to > > > > > achieve this > > > > goal. > > > > > > > > > > -- > > > > > Thanks, > > > > > Bhagwat Posane > > > > > > > > > > > > > > > > > > > > > -- > > > > Thanks, > > > > Bhagwat Posane > > > > > > > > > > > > > > > -- > > Thanks, > > Bhagwat Posane > > > > > > -- > Thanks, > Bhagwat Posane > -- Thanks, Bhagwat Posane
