[
https://issues.apache.org/jira/browse/SOLR-6760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14707457#comment-14707457
]
Gregory Chanan commented on SOLR-6760:
--------------------------------------
The naming here conflicts a bit with what I was trying to accomplish with
SOLR-7789, so I'm asking here for suggestions on how to resolve. In SOLR-7789
I am trying to accomplish the following:
#1 Add another OverseerMessageHandler (OverseerConfigSetMessageHandler) to
handle ConfigSet-related operations.
#2 From the perspective of the non-overseer (i.e. the ConfigSetsHandler), it
looks like the operations are written to a separate queue from the collection
queue, i.e. getOverseerConfigSetQueue()
#3 Since the ConfigSet operations are most likely rare and fast, it made sense
to just use the existing collections queue "under the covers" and handle the
dispatch separately. The naming here breaks the illusion of #2, i.e. if I
return an OverseerCollectionQueue it's pretty obvious to the non-overseer
what's going on.
So, here's my plan:
Short term: rename OverseerCollectionQueue to something more
generic...DistributedTaskQueue? DistributedAsyncAwareQueue? There's nothing
in there that is actually collection specific (which is why it works for the
ConfigSet operations)
Longer term: I have some more suggestions for the queue interface in
SOLR-7789. For example, on the insertion side the queue should be ZKNodeProps
based rather than byte [] based so we can return different queue types that
understand the semantics of the operations being inserted (hard to do that with
a byte []). In particular, I want to prefix all operation names to the
ConfigSetQueue with "configsets:" automatic to simplify the dispatching to the
correct OverseerMessageHandler. The ConfigSetsHandler needs to do this now (so
sort of breaks the illusion of #2) because of the interface. There's probably
a lot more room to break things up for client vs processing side as well --
i.e. why does the CollectionsHandler / ConfigSetsHandler get access to an
object that lets it remove something from the queue?
> New optimized DistributedQueue implementation for overseer
> ----------------------------------------------------------
>
> Key: SOLR-6760
> URL: https://issues.apache.org/jira/browse/SOLR-6760
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
> Reporter: Noble Paul
> Assignee: Shalin Shekhar Mangar
> Fix For: Trunk, 5.4
>
> Attachments: SOLR-6760-branch_5x.patch, SOLR-6760.patch,
> SOLR-6760.patch, SOLR-6760.patch, SOLR-6760.patch, deadlock.patch
>
>
> Currently the DQ works as follows
> * read all items in the directory
> * sort them all
> * take the head and return it and discard everything else
> * rinse and repeat
> This works well when we have only a handful of items in the Queue. If the
> items in the queue is much larger (in tens of thousands) , this is
> counterproductive
> As the overseer queue is a multiple producers + single consumer queue, We can
> read them all in bulk and before processing each item , just do a
> zk.exists(itemname) and if all is well we don't need to do the fetch all +
> sort thing again
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
