[
https://issues.apache.org/jira/browse/SOLR-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14490623#comment-14490623
]
Damien Kamerman commented on SOLR-7361:
---------------------------------------
Regarding loading the cores in the background, I've made a few tweaks to work
with thousands of cores (See SOLR-7191):
1. Load cores in fixed threadPool. Cores are registered in background threads,
so no reason to load all cores at once!
2. Register cores in corename order with a fixed 128 threadPool. This is to not
flood the DistributedQueue.
3. Publish an entire node as 'down' (4.10 branch)
4. Cache ConfigSetService.createIndexSchema() in cloud mode.
So, my questions are:
What threadPool size will be used to load the cores?
What order will cores be loaded in?
Will cores be registered as they are loaded, or offloaded to another threadPool?
My initial thought was to register as a live node after cores are loaded.
> Main Jetty thread blocked by core loading delays HTTP listener from binding
> if core loading is slow
> ---------------------------------------------------------------------------------------------------
>
> Key: SOLR-7361
> URL: https://issues.apache.org/jira/browse/SOLR-7361
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Reporter: Timothy Potter
> Assignee: Timothy Potter
>
> During server startup, the CoreContainer uses an ExecutorService to load
> cores in multiple back-ground threads but then blocks until cores are loaded,
> see: CoreContainer#load around line 290 on trunk (invokeAll). From the
> JavaDoc on that method, we have:
> {quote}
> Executes the given tasks, returning a list of Futures holding their status
> and results when all complete. Future.isDone() is true for each element of
> the returned list.
> {quote}
> In other words, this is a blocking call.
> This delays the Jetty HTTP listener from binding and accepting requests until
> all cores are loaded. Do we need to block the main thread?
> Also, prior to this happening, the node is registered as a live node in ZK,
> which makes it a candidate for receiving requests from the Overseer, such as
> to service a create collection request. The problem of course is that the
> node listed in /live_nodes isn't accepting requests yet. So we either need to
> unblock the main thread during server loading or maybe wait longer before we
> register as a live node ... not sure which is the better way forward?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
