A slightly different (but related question):
I've been playing around with this type of computation. We are loading
data into a DB. We have a small Linux cluster (15 multi-core nodes at the
moment) that we have scaled up to run 224+ instances of our pipeline. I've
noticed for most of our calculations, it's really Oracle that is holding us
back.
In some instances, a 'computation' is classified as one 'medium to long'
data pull from Oracle, a bunch of analysis, and then 'small to large'
insertions of results. I've dabbled in placing static DB connections and
mutexes through the code to guarantee that the instances on a machine only
access the DB one at a time, but are free to run analysis simultaneously
otherwise.
I have also toyed with the idea of locks that allow N number of connections
(instead of only the mutual exclusion one at a time) so that I can
increase the connections to a point, but not overload the system.
Has anyone tried anything like this? Or is anyone else at least running a
similar hardware set-up? It would be great to compare note.
Thanks,
Steve
At 04:09 AM 11/12/2007, Marshall Schor wrote:
This may not be quite precise enough. Your Annotators will be
instantiated multiple times, so that a single *instance* of an
annotator will not be run on multiple threads at once. So - if you have
non-static fields in your annotator, they do not need to be accessed
with threading in mind. But if you make use of "static" fields, there is
only one instance of these, so access to them must be thread-safe.
If your *application* (not your annotator) is multi-threaded, it will
need to be thread-safe. You can find relevant information about this in
the tutorial and reference docs for UIMA (search for "thread").
-Marshall
Michael Baessler wrote:
> Benjamin Sznajder wrote:
>> Hi all,
>>
>> I am interested in using multi-threading in UIMA.
>> My aim is that the flow runs several annotators in parallel.
>> One of my annotators is not thread-safe. My question is, then,
>> Does the UIMA parallelism ( setting "MultipleDeployment=true") requires
>> that annotators on which this flag is set, are thread-safe?
>>
>> Regards,
>> Benjamin.
>>
>>
> Yes, your annotator have to be thread-safe when you want to run them
> multi-threaded.
>
> -- Michael
>
>
