Hi Clayton, I'm glad that you got it working. Though I stated that I would, I haven't yet checked the fidelity of trunk. Urgent data request one day, "must have" writing the next ... and I still live with the delusion that I left academia to have free time ...
I have never used ytex or weka, so I'm unfamiliar with all things .arff . Could it be that the ytex .arff exporter needs to change consumed cTakes annotation classes (>3.1)? I have a custom CasConsumer that saves text spans and Cuis to file in a simple list, and that is what I used for the performance analysis of the lookup module. For our other projects here in Beantown we have other various outputs that fit the job at hand: text flat files, xml files, sql database tables, knot-encoded lace doilies, etc. I'm sure that none of the above helps you, but I felt obliged to provide some kind of answer to your question. Sean > -----Original Message----- > From: clayclay...@gmail.com [mailto:clayclay...@gmail.com] On Behalf Of > Clayton Turner > Sent: Wednesday, August 13, 2014 12:25 PM > To: dev@ctakes.apache.org > Subject: Re: v_snomed_fword_lookup view > > Okay, I believe I have ctakes dictionary fast working now. Something I'm > curious > about, though, is how you extract the data in order to conduct analysis. > > I've, in the past, been using the SparseDataExporterImpl from ytex in order to > create a .arff file for use in weka, but the ctakes pipeline I'm using > doesn't seem > to be compatible with this ytex exporting as I'm not getting any cuis in my > arff > file. > > I'm using the aggregate plain text umls processor analysis engine from ctakes > and then using the dbconsumer analysis engine from ytex (for storing into the > database with regard to analysis batch). > > Any tips for exporting or some simple issue I'm missing? > > Thanks, > Clayton > > > On Mon, Aug 11, 2014 at 2:09 PM, Harpreet Khanduja <hsk5...@rit.edu> > wrote: > > > Yes, absolutely and > > no problem at all. > > > > Regards, > > Harpreet > > > > > > On Mon, Aug 11, 2014 at 1:16 PM, Finan, Sean < > > sean.fi...@childrens.harvard.edu> wrote: > > > > > Thanks Harpreet, > > > That is definitely necessary to build! > > > > > > Those lines should already be in the pom, but commented out. I > > > think > > that > > > some version/branching issues may have arisen at some point wrt this > > module > > > ... > > > > > > If somebody beats me to it then cheers, otherwise I will try to > > > check out tonight and get all the bits in place. > > > > > > Sean > > > > > > > -----Original Message----- > > > > From: Harpreet Khanduja [mailto:hsk5...@rit.edu] > > > > Sent: Monday, August 11, 2014 1:12 PM > > > > To: dev@ctakes.apache.org > > > > Subject: Re: v_snomed_fword_lookup view > > > > > > > > Hello Clayton, > > > > I do not know about ytex, but I did switch from > > > > dictionary-lookup to > > > dictionary- > > > > lookup-fast. > > > > I update my ctakes-dictionary-lookup-fast project using maven. > > > > I think I used Team- Update and switched to the latest revision > > > available and > > > > then > > > > I downloaded new 3.2 resources from the for umls. and then I > > > > added > > > these > > > > resources to my > > > > ctakes-dictionary-lookup-fast resources folder and also the > > > > classpath > > > in ctakes- > > > > clinical-pipeline. > > > > > > > > Then I changed the pom.xml file which belongs to the whole ctakes > > > project and > > > > added <dependency> <groupId>org.apache.ctakes</groupId> > > > > <artifactId>ctakes-dictionary-lookup-res</artifactId> > > > > <version>${ctakes.version}</version> > > > > </dependency> > > > > <dependency> > > > > <groupId>org.apache.ctakes</groupId> > > > > <artifactId>ctakes-dictionary-lookup-fast</artifactId> > > > > <version>${ctakes.version}</version> > > > > </dependency> > > > > > > > > > > > > these two dependencies to the file. > > > > > > > > > > > > After this, I also added the dependency > > > > <dependency> > > > > <groupId>org.apache.ctakes</groupId> > > > > <artifactId>ctakes-dictionary-lookup-fast</artifactId> > > > > </dependency> > > > > > > > > to the pom.xml of ctakes-clinical-pipeline. > > > > > > > > And then add the resources folder in ctakes-clinical-pipeline > > > > using > > > build path > > > > configuration under "add class" option. > > > > > > > > After this it should work. > > > > > > > > > > > > Regards, > > > > Harpreet > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Aug 11, 2014 at 12:44 PM, Clayton Turner > > > > <caturn...@g.cofc.edu > > > > > > > wrote: > > > > > > > > > I still get the same error with the ctakes3.2 branch. Any > > suggestions? > > > > > > > > > > > > > > > On Mon, Aug 11, 2014 at 12:06 PM, Clayton Turner > > > > > <caturn...@g.cofc.edu> > > > > > wrote: > > > > > > > > > > > I'm going to do a clean install through the repo rather than > > > > > > the binaries and see if that fixes my issue because I think I > > > > > > just read a past post saying the lookup2 folders exist there. > > > > > > > > > > > > > > > > > > On Mon, Aug 11, 2014 at 11:52 AM, Clayton Turner > > > > > > <caturn...@g.cofc.edu> > > > > > > wrote: > > > > > > > > > > > >> When navigating to > > > > > >> ctakes-dictionary-lookup-fast\desc\analysis_engine > > > > > >> there are 2 files, assumedly analysis engines. > > > > > >> > > > > > >> SnomedLookupAnnotator.xml and SnomedOvLookupAnnotator.xml > > > > > >> > > > > > >> If I pick either, I put in my UMLS information but receive an > > error > > > > > >> when trying to run the CPE: > > > > > >> > > > > > >> Initialization of CAS Processor with name > > "SnomedOvLookupAnnotator" > > > > > >> failed. > > > > > >> CausedBy: > org.apache.uima.resource.ResourceConfigurationException: > > > > > >> Initialization of CAS processor with name > > "SnomedOvLookupAnnotator" > > > > > >> failed. > > > > > >> CausedBy: > > org.apache.uima.resource.ResourceInitializationException: > > > > > Error > > > > > >> initializing "org.apache.uima.resource.impl.DataResource_impl" > > from > > > > > >> descriptor file:..............SnomedLookupAnnotator.xml > > > > > >> CausedBy: > > org.apache.uima.resource.ResourceInitializationException: > > > > > Could > > > > > >> not > > > > > >> access the resource data at > > > > > >> > > > > > >> > > > > > > > file:org\apache\ctakes\dictionary\lookup2\Snomed2011ab_ctakesTui\cTake > > > > > sSnomed.xml > > > > > >> > > > > > >> Now, I don't even have a "lookup2" folder and, subsequently > > > > > >> the > > Tui > > > > > >> folder and cTakesSnomed.xml file. This seems to be the > > > > > >> problem, > > but > > > > > >> I'm > > > > > not > > > > > >> sure where these files are supposed to be grabbed from. > > > > > >> > > > > > >> > > > > > >> On Mon, Aug 11, 2014 at 11:47 AM, Clayton Turner > > > > > >> <caturn...@g.cofc.edu> > > > > > >> wrote: > > > > > >> > > > > > >>> Hi again: > > > > > >>> > > > > > >>> How exactly do you switch to using the cTakes > > > dictionary-lookup-fast. > > > > > Do > > > > > >>> I need to go in and alter xml files or is it as simple as > > > > > >>> adding > > a > > > > > certain > > > > > >>> item to the list of analysis engines? > > > > > >>> > > > > > >>> > > > > > >>> On Fri, Aug 8, 2014 at 3:48 PM, Finan, Sean < > > > > > >>> sean.fi...@childrens.harvard.edu> wrote: > > > > > >>> > > > > > >>>> Hi Clayton, > > > > > >>>> > > > > > >>>> I don't know how the ytex dictionary lookup works, so I'm > > > > > >>>> afraid that > > > > > I > > > > > >>>> can't help you with an answer. Maybe Vijay is the best > > > > > >>>> person > > to > > > > > >>>> do > > > > > this. > > > > > >>>> If you aren't tied to ytex you could try the new cTakes > > > > > >>>> dictionary-lookup-fast. I tested "Patient came in with a > > > > > >>>> malar > > > rash" > > > > > and > > > > > >>>> it found "malar" and "malar rash". > > > > > >>>> > > > > > >>>> Vijay, > > > > > >>>> > > > > > >>>> At some point the lookup-fast module will be the default > > > > > >>>> for the > > > > > cTakes > > > > > >>>> clinical pipeline. In order to synchronize the ytex lookup > > > > > >>>> with > > > > > cTakes, > > > > > >>>> would you like to eventually work together on reusing the > > > > > >>>> same code > > > > > for > > > > > >>>> ytex? I have no idea what ytex does, but I know the ins > > > > > >>>> and > > outs > > > > > >>>> of > > > > > the > > > > > >>>> cdl-fast module. > > > > > >>>> > > > > > >>>> Sean > > > > > >>>> > > > > > >>>> > -----Original Message----- > > > > > >>>> > From: clayclay...@gmail.com > > > > > >>>> > [mailto:clayclay...@gmail.com] On > > > > > Behalf > > > > > >>>> Of > > > > > >>>> > Clayton Turner > > > > > >>>> > Sent: Friday, August 08, 2014 2:08 PM > > > > > >>>> > To: dev@ctakes.apache.org > > > > > >>>> > Subject: v_snomed_fword_lookup view > > > > > >>>> > > > > > > >>>> > Hi Everyone: > > > > > >>>> > > > > > > >>>> > I have a question about how the v_snomed_fword_lookup > > > > > >>>> > view works > > > > > when > > > > > >>>> > running the CPE. > > > > > >>>> > > > > > > >>>> > So my understanding of the view is that it is a view > > > > > >>>> > comprised of > > > > > the > > > > > >>>> > ytex.umls_aui_fword table, the umls.mrconso table and > > > > > >>>> > bits/pieces > > > > > from > > > > > >>>> > other umls tables. > > > > > >>>> > > > > > > >>>> > I feel like this is not completely correct or my idea of > > > > > >>>> > how the > > > > > join > > > > > >>>> to > > > > > >>>> > create the view works is off. For example, let's say I > > > > > >>>> > want > > the > > > > > >>>> > CPE > > > > > >>>> to find > > > > > >>>> > "malar ____" (e.g. malar rash) as a concept in the > > annotations. > > > > > >>>> > It > > > > > >>>> never > > > > > >>>> > happens after running my CPE descriptor and I cannot find > > > > > >>>> > it > > in > > > > > >>>> > my v_snomed_fword_lookup view. > > > > > >>>> > > > > > > >>>> > select count(*) from umls_aui_fword where fword='malar'; > > yields > > > > > >>>> > 34 > > > > > >>>> results > > > > > >>>> > > > > > > >>>> > select count(*) from umls.mrconso where str='malar'; > > > > > >>>> > yields 3 > > > > > results. > > > > > >>>> > > > > > > >>>> > So clearly these two tables know what the cui and > > > > > >>>> > context(s) are for > > > > > >>>> malar > > > > > >>>> > ____. Yet, whenever I run a gold standard set of notes > > > > > >>>> > through the > > > > > >>>> CPE, > > > > > >>>> > malar is constantly flagged as just a word token and the > > > > > >>>> > concept is > > > > > >>>> never > > > > > >>>> > grabbed. This is recurrent for lots of other concepts, as > > well, > > > > > >>>> > I > > > > > just > > > > > >>>> > wanted to use an example to illustrate my issue. > > > > > >>>> > > > > > > >>>> > Some troubleshooting I already went through: > > > > > >>>> > 1) Reinstalled ytex and umls database objects > > > > > >>>> > 2) Reinstalled a second time after redownloading umls > > > > > >>>> > through metamorphosys, ensuring that snomed vocabularies > > > > > >>>> > were included (also checked file sizes and noticed a big > > > > > >>>> > difference so I > > know > > > > > >>>> > those vocabularies ARE included > > > > > >>>> > > > > > > >>>> > Anyone got any ideas as to what the issue could be? > > > > > >>>> > > > > > > >>>> > Thank you, > > > > > >>>> > Clayton Turner > > > > > >>>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> -- > > > > > >>> -- > > > > > >>> Clayton Turner > > > > > >>> email: caturn...@g.cofc.edu > > > > > >>> phone: (843)-424-3784 > > > > > >>> web: claytonturner.blogspot.com > > > > > >>> > > > > > >>> > > > > > > > ---------------------------------------------------------------------- > > > > > --------------------------- > > > > > >>> “When scientifically investigating the natural world, the > > > > > >>> only thing worse than a blind believer is a seeing denier.” > > > > > >>> - Neil deGrasse Tyson > > > > > >>> > > > > > >> > > > > > >> > > > > > >> > > > > > >> -- > > > > > >> -- > > > > > >> Clayton Turner > > > > > >> email: caturn...@g.cofc.edu > > > > > >> phone: (843)-424-3784 > > > > > >> web: claytonturner.blogspot.com > > > > > >> > > > > > >> > > > > > > > ---------------------------------------------------------------------- > > > > > --------------------------- > > > > > >> “When scientifically investigating the natural world, the > > > > > >> only thing worse than a blind believer is a seeing denier.” > > > > > >> - Neil deGrasse Tyson > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > -- > > > > > > Clayton Turner > > > > > > email: caturn...@g.cofc.edu > > > > > > phone: (843)-424-3784 > > > > > > web: claytonturner.blogspot.com > > > > > > > > > > > > > > > > > > > ---------------------------------------------------------------------- > > > > > --------------------------- > > > > > > “When scientifically investigating the natural world, the only > > thing > > > > > worse > > > > > > than a blind believer is a seeing denier.” > > > > > > - Neil deGrasse Tyson > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > -- > > > > > Clayton Turner > > > > > email: caturn...@g.cofc.edu > > > > > phone: (843)-424-3784 > > > > > web: claytonturner.blogspot.com > > > > > > > > > > > > ---------------------------------------------------------------------- > > > > > --------------------------- “When scientifically investigating > > > > > the natural world, the only thing worse than a blind believer is > > > > > a seeing denier.” > > > > > - Neil deGrasse Tyson > > > > > > > > > > > > > > -- > -- > Clayton Turner > email: caturn...@g.cofc.edu > phone: (843)-424-3784 > web: claytonturner.blogspot.com > ------------------------------------------------------------------------------------------------- > “When scientifically investigating the natural world, the only thing worse > than a > blind believer is a seeing denier.” > - Neil deGrasse Tyson
