Tim, Thanks for your quick response! Probably unsurprisingly, I’ll have to do some googling to learn how to check those things. If you could point me in the right direction, that’d be great!
Thanks again, Leah From: "Miller, Timothy" <timothy.mil...@childrens.harvard.edu> Reply-To: "user@ctakes.apache.org" <user@ctakes.apache.org> Date: Tuesday, January 29, 2019 at 12:14 PM To: "user@ctakes.apache.org" <user@ctakes.apache.org> Subject: Re: Processing large batches of files in cTAKES [EXTERNAL] I am able to process that number of files in a reasonable amount of time (maybe an hour) on an average desktop. Luckily, debugging your setup should be much easier than doing a scaleout. A few possibilities: * You are running the old (slow) dictionary instead of the new fast one * Your document has extremely long sentences * Your VM is _extremely_ resource constrained and is thrashing constantly Do you know how to check these things? Tim -----Original Message----- From: "Baas,Leah" <leah.b...@sanfordhealth.org<mailto:%22Baas,leah%22%20%3cleah.b...@sanfordhealth.org%3e>> Reply-to: <user@ctakes.apache.org> To: user@ctakes.apache.org <user@ctakes.apache.org<mailto:%22u...@ctakes.apache.org%22%20%3cu...@ctakes.apache.org%3e>> Subject: Processing large batches of files in cTAKES [EXTERNAL] Date: Tue, 29 Jan 2019 17:58:48 +0000 Hi all, I would like to process a batch of 13,414 files (avg file size 6.2 KB) using the default clinical pipeline. I am new to cTAKES and computer programming, and I’m looking for guidance on how to process these files with maximum time/CPU efficiency. I am currently running my program on an Ubuntu VM with 3 CPUs. It takes me 28 seconds (real time) to process one 6.0 KB file. I’m reading up on parallel processing strategies, but would be grateful for any suggestions, tips, etc. that you might have! Thanks, Leah ----------------------------------------------------------------------- Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain privileged and confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
