It is useful for parsing PDFs on a multi-processor machine. Also, if a
sub-entity does an outbound I/O call to a database, a file, or another
SOLR (SOLR-1499).
Anything where the pipeline time outweighs disk i/o time.
Threading happens on a per-document level- there is no concurrent
access inside a document pipeline.
There is a bug which causes Entityprocessor that look up attributes to
throw an exception. This make Tika unusable inside a thread. Two other
EPs also won't work, but I did not test them.
https://issues.apache.org/jira/browse/SOLR-2186
On Mon, Nov 1, 2010 at 10:43 AM, Dyer, James james.d...@ingrambook.com wrote:
Mark,
I have the same question so I did a little research on this. Not a complete
answer but here is what I've found:
- threads was aded with SOLR-1352
(https://issues.apache.org/jira/browse/SOLR-1352).
- Also see
http://www.lucidimagination.com/search/document/a9b26ade46466ee/queries_regarding_a_paralleldataimporthandler
for background info.
- Only available in 3.x and trunk. Committed on 1/12/2010 by Noble Paul (who
surely can tell you more accurate info than I can).
- Seems like when using, each thread will call nextRow on your root entity
datasource in parallel.
- Not sure this will help with child entities (ie. I had hoped I could get it
to build child caches in parallel but I don't think this is the case).
- A doc comment on ThreadedEntityProcessorWrapper indicates this will help
speed up running transformers becauses they'd be in parallel. This would
make sense if maybe your database can only pull back so fast, but then you
have an intensive transformer. Maybe adding a thread would make your
processing no slower than the db...
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-Original Message-
From: markwaddle [mailto:m...@markwaddle.com]
Sent: Tuesday, October 26, 2010 2:25 PM
To: solr-user@lucene.apache.org
Subject: How does DIH multithreading work?
I understand that the thread count is specified on root entities only. Does
it spawn multiple threads per root entity? Or multiple threads per
descendant entity? Can someone give an example of how you would make a
database query in an entity with 4 threads that would select 1 row per
thread?
Thanks,
Mark
--
View this message in context:
http://lucene.472066.n3.nabble.com/How-does-DIH-multithreading-work-tp1776111p1776111.html
Sent from the Solr - User mailing list archive at Nabble.com.
--
Lance Norskog
goks...@gmail.com
