[
https://issues.apache.org/jira/browse/SOLR-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028363#comment-14028363
]
Ted Sullivan commented on SOLR-3585:
------------------------------------
Good discussion Dmitry and David - I agree with your comments. As I said in my
initial comment, I came at this thread while looking for ways to speed up the
DataImportHandler. Even though this is not directly related to the
client-server issue that you are discussing, it does factor in because with
DIH, Solr is pulling data from an outside source. With a SolrJ push, you do
have the opportunity to load Solr by taking advantage of the multi-threaded
capability built into the REST/RequestHandler. However, with DIH you don't have
this option (except by running multiple DIH handlers which seems hacky to me) -
so in this case the Solr side is the only place where you can add
multithreading. As I said, my client is doing both (multiple DIH handlers and
using the patch) and they both help.
To Mikhail's comment that with DIH it would be better to refactored to support
concurrency - yes I agree - that would be good, but it was tried before, was
found to be problematic and was removed in Solr 4.0. I see that you
contributed a patch to fix multithreading in 3.6 Mikhail - so I would not want
to challenge your expertise here :) My take on it is that a DIH process can
get fairly complicated with nested event handlers and possibly transformers
doing strange things so maybe there were just too many dependencies that are
not or can't be guaranteed to be thread-safe. Whatever the reason, I find your
idea to move this problem downstream to the update processor chain to be a
good, clean solution. It makes threading configurable and pluggable which both
sound like good things to me.
> processing updates in multiple threads
> --------------------------------------
>
> Key: SOLR-3585
> URL: https://issues.apache.org/jira/browse/SOLR-3585
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0-ALPHA, 5.0
> Reporter: Mikhail Khludnev
> Attachments: SOLR-3585.patch, SOLR-3585.patch, multithreadupd.patch,
> report.tar.gz
>
>
> Hello,
> I'd like to contribute update processor which forks many threads which
> concurrently process the stream of commands. It may be beneficial for users
> who streams many docs through single request.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
