Hi Pei, Great job! I just tested it out and the status annotator with the historyOf feature works like a charm. Thank you. Next, I will test out the sectionizer. And yes, that is the objective to support clinical research using plaintext notes. You have a wonderful Thanksgiving! Regards, Paula From: pei.c...@childrens.harvard.edu To: user@ctakes.apache.org Subject: RE: Question pertaining to HistoryOf and SegmentID features Date: Wed, 27 Nov 2013 14:42:31 +0000
Paula, It’s already fixed in trunk J; feel free to give it a test. https://issues.apache.org/jira/browse/CTAKES-274 Thanks for pointing that out- would you like to try out the sectionizer? Just curious on the use case for the sections. Is it clinical research on top of plaintext notes? --Pei From: digital paula [mailto:cybersat...@hotmail.com] Sent: Wednesday, November 27, 2013 12:46 AM To: user@ctakes.apache.org Subject: RE: Question pertaining to HistoryOf and SegmentID features Pei, Thanks so much for the super prompt response. A good reminder to us all. When it's not working as expected then try a clean/rebuild to fix it. After the clean/rebuild it's working as expected. By the way, as for your side note on whether it's worth updating the StatusAnnotator to populate historyOf. If the Assertion module is going to be released soon with historyOf feature then I don't think it matters with updating Status Annotator historyOf feature since the Assertion component really is a replacement for the Status Annotator. However, if it's a minor effort to make the Status Annotator populate historyOf, why not. I'd like to fix it as my first code contribution to the open source community. :-) Regards, Paula Date: Tue, 26 Nov 2013 23:04:05 -0500 Subject: Re: Question pertaining to HistoryOf and SegmentID features From: chen...@apache.org To: user@ctakes.apache.org Paula, Could you attach the desc xml and the output xml? Also, are you using trunk? Strange, i'm seeing uncertainty being set to 1/history: <..DiseaseDisorderMention _indexed="1" _id="299" _ref_sofa="6" begin="11" end="19" id="5" _ref_ontologyConceptArr="293" typeID="2" segmentID="SIMPLE_SEGMENT" discoveryTechnique="1" confidence="0.0" polarity="0" uncertainty="1" conditional="false" generic="false" historyOf="0" _ref_Segment="20"/> Side note- Not sure if its worth updating the StatusAnnotator to populate the hisoryOf attribute now (obviously those types were not avail previously.) On Tue, Nov 26, 2013 at 10:37 PM, Pei Chen <chen...@apache.org> wrote: That annotator will set the IdentifiedAnnotation.*uncertainty()* with one of the following values: HISTORY_STATUS = 1; FAMILY_HISTORY_STATUS = 2; PROBABLE_STATUS = 3; On Tue, Nov 26, 2013 at 10:32 PM, digital paula <cybersat...@hotmail.com> wrote: Hi Pei, Thanks for the prompt response. Okay, I uncommented out StatusAnnotator in the AggregatePlaintextPipeline config/descriptor and tried again with this text. "history of diabetes. hx diabetes. prior diabetes. " Uncertainty and HistoryOf features remained 0.....doesn't look like anything happened. Note that I'm validating using the CVD debugger tool. I did read something on that the Status and Assertion have overlapping functionality so I commented out Assertion and still the same thing. I looked at the StatusIndicatorFSM.java file, text with history "hx" and "history" should be annotated so I'm not sure why it's not working. Am I missing some config? Regards, Paula From: pei.c...@childrens.harvard.edu To: user@ctakes.apache.org Subject: RE: Question pertaining to HistoryOf and SegmentID features Date: Tue, 26 Nov 2013 20:46:12 +0000 Paula, If you would like to use the old status annotator for history in the interim while the new assertion module models are added. One can just uncomment the below from your AggregatePlaintextPipeline config/descriptor: <!-- <node>StatusAnnotator</node> --> That annotator will set the IdentifiedAnnotation.uncertainty() with one of the following values: HISTORY_STATUS = 1; FAMILY_HISTORY_STATUS = 2; PROBABLE_STATUS = 3; --Pei From: digital paula [mailto:cybersat...@hotmail.com] Sent: Tuesday, November 26, 2013 10:00 AM To: user@ctakes.apache.org Subject: RE: Question pertaining to HistoryOf and SegmentID features Hi Tim, I remember you, you helped me out a while back! I can't tell you how much that BagOfAnnotations.java file has been a Godsend! Thanks again for posting it last month. As for historyOf feature, I don't so much mind performance so if you have something that works for the historyOf with just only a performance issue, I'll still take it. I do appreciate your prompt response on the historyOf feature which saved me from stepping through the code which I've done a few times....the time spent stepping through is equivalent to watching a feature film and there's a lot going on in the code that I don't understand yet. As for sementID's with the rule-based sectionizer/segmenter, I'd like to help out with testing. It appears that the first step is manual configuration per your email. Pei, when you have a moment I'd like to start with explanation of how it works then we can move forward from there. Thanks. Regards, Paula Date: Tue, 26 Nov 2013 09:24:20 -0500 From: timothy.mil...@childrens.harvard.edu To: user@ctakes.apache.org Subject: Re: Question pertaining to HistoryOf and SegmentID features Hi Paula, The historyOf module is currently under development, that explains those files you're seeing. As of right now we are not satisfied enough with the performance to release it publicly. It is marked 0 because it is defined as an integer field I believe, so it can't be assigned null. As for the segment IDs in your other question, the default segmenter just makes the whole document one segment because we don't yet have a module robust enough to run on notes from any institution. We have been hoping to get one developed but nothing has materialized so far. There is a rule-based sectionizer/segmenter in the cTAKES distribution but it requires some manual configuration and is not very widely tested. If you are interested in helping test it out that would be great. Pei would be the best person to explain to you how it works, we'll see if he's able to chime in. (Ah, I just checked and saw he already pointed you to it. If you want to try it we can help you get it working.) Thanks Tim On 11/25/2013 11:13 PM, digital paula wrote: Matt, sorry but I sent the email by accident and didn't finish my thought. Since there's this file (historyOf.psv) in the Assertion folder it gives the impression that Assertion can be configured to account for historyOf. Though I tried some of the words such as "had diabetes" or "prior diabetes" and value still 0 for historyOf feature. I think it is configurable because of the fact the default value for historyOf is 0 as opposed to <NULL> . Maybe I should step through the code and see where and when it's assigning 0. Regards, Paula From: cybersat...@hotmail.com To: user@ctakes.apache.org Subject: RE: Question pertaining to HistoryOf and SegmentID features Date: Mon, 25 Nov 2013 23:09:33 -0500 Hi Matt, it's me again. Are you sure that historyOf feature can't be configured using Assertion. I was looking around the Assertion folder and came across a file called historyOf.psv. What is this? It contains the following: has been|historyOf|historyOf had|historyOf|historyOf hx of recent|historyOf|historyOf hx of|historyOf|historyOf hx|historyOf|historyOf recent|historyOf|historyOf s/p|historyOf|historyOf status post|historyOf|historyOf post|historyOf|historyOf previous|historyOf|historyOf prior|historyOf|historyOf From: mco...@mitre.org To: user@ctakes.apache.org Subject: Re: Question pertaining to HistoryOf and SegmentID features Date: Tue, 26 Nov 2013 02:50:59 +0000 The ctakes assertion module does not have an option to set the historyOf attribute. The assertion module only sets subject, polarity, uncertainty, conditional, and generic. Matt From: digital paula <cybersat...@hotmail.com> Reply-To: "user@ctakes.apache.org" <user@ctakes.apache.org> Date: Monday, 2013-11-25 15:13 To: "user@ctakes.apache.org" <user@ctakes.apache.org> Subject: RE: Question pertaining to HistoryOf and SegmentID features
