That's an excellent suggestion! In a Spark implementation, build the
pipeline outside of the map function and pass the aed in as an input.
Then just ensure the jcas object persists between each mapping
iteration and leverage the reset method.
On Fri, Jul 28, 2017 at 11:11 AM, Abramowitsch, Peter
<pabramowit...@hearst.com> wrote:
> About your second question with UMLS, You can build the pipeline
> initially and it will verify the license info, then just reuse the
> pipeline on each call.
>
>
>
> On 7/25/17, 4:53 PM, "Michael Trepanier" <m...@metistream.com> wrote:
>
>>Hi,
>>
>>I am currently leveraging cTAKES inside of Apache Spark and have
>>written a function that takes in a single clinical note as as string
>>and does the following:
>>
>>1) Sets the UMLS system properties.
>>2) Instantiates JCAS object.
>>3) Runs the default pipeline
>>4) (Not shown below) Grabs the annotations and places them in a JSON
>>object for each note.
>>
>> def generateAnnotations(paragraph:String): String = {
>> System.setProperty("ctakes.umlsuser", "MY_UMLS_USERNAME")
>> System.setProperty("ctakes.umlspw", "MY_UMLS_PASSWORD")
>>
>> var jcas =
>>JCasFactory.createJCas("org.apache.ctakes.typesystem.types.TypeSystem")
>> var aed = ClinicalPipelineFactory.getDefaultPipeline()
>> jcas.setDocumentText(paragraph)
>> SimplePipeline.runPipeline(jcas, aed)
>> ...
>>
>>This function is being implemented as a UDF which is applied to a
>>Spark Dataframe with clinical notes in each row. I have two
>>implementation questions that follow:
>>
>>1) When cTAKES is being applied iteratively to clinical notes, is it
>>necessary to instantiate a new JCAS object for every annotation? Or
>>can the same JCAS object be utilized over and over with the document
>>text being changed?
>>2) For each application of this function, the
>>UmlsDictionaryLookupAnnotator has to connect to UMLS using the
>>provided UMLS information. This Is there any way to instead perform
>>this step locally? Ie. ingest UMLS and place it in either HDFS or just
>>mount it somewhere on each node? I'm worried about spamming the UMLS
>>server in this step, and about how long this seems to take.
>>
>>Thanks,
>>
>>Mike
>>
>>
>>--
>>
>>Mike Trepanier| Big Data Engineer | MetiStream, Inc. |
>>m...@metistream.com | 845 - 270 - 3129 (m) | www.metistream.com
>
--
Mike Trepanier| Big Data Engineer | MetiStream, Inc. |
m...@metistream.com | 845 - 270 - 3129 (m) | www.metistream.com
