[
https://issues.apache.org/jira/browse/SOLR-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123353#comment-13123353
]
Jamie Johnson commented on SOLR-2765:
-------------------------------------
{quote}
I don't follow this at all. Why would servers need to know about other servers?
{quote}
The searcher servers would need to be aware of the other searcher servers
wouldn't they? Assuming that a query could come into any one of those servers,
they would all need to be aware of any other server which was acting as a
searcher, right?
{quote}
This wouldn't be quite n^2 in any case. It could be order n for each change,
but the constant factor would be quite small for reasonable clusters. The cost
would be n^2/2 as the cluster came up, but even for 1000 nodes, this would
clear in less than a minute which is much less time than it would take to
actually bring the servers up.
{quote}
Right, each change would get replicated to the n servers watching that node.
The total number of watchers is 1 for each node for each server, so n^2 total
watchers. As you say though the number of watch events triggered should be
reasonable as a stable cluster will not have a significant amount of
shards/slices going up and down during operation. This seems to be an argument
for not going with the summarized approach. The amount of information pulled
in this case would be much less compared to a summarized file as each searcher
would need to pull the entire state instead of just 1 change.
{quote}
I still don't think that there would be any need for all servers to know about
this. The clients need to know, but not the servers.
If you mean that a searcher serves as a query proxy between the clients and the
servers, then you would require a notification to each search for each change.
If you have k searchers and n nodes, bringing up the cluster would require k n
/ 2 notifications. For 100 proxies and 1000 search nodes, this is a few seconds
of ZK work. Again, this is much less than the time required to bring up so many
nodes.
{quote}
You're right that all servers don't need to be aware of the cloud state, but
the servers which are acting as searchers do (are there any other roles that
also need to know?), I don't see any difference between them and any other
client since they'll need to know about the other searchers to distribute the
query.
{quote}
If you put server status into a single file, far fewer notifications would
actually be sent because as the notifications are delayed, the watchers get
delayed in being reset so you have natural quenching while still staying very
nearly up to date.
{quote}
I see your point, to do something like this though we'd need to have some
leader which was responsible for maintaining this state. Wouldn't this still
result in a large amount of data being pulled by the searchers for a very small
change (i.e. 1 additional searcher being added). I am guessing that your of
the opinion that it would be negligible since the number of changes should be
small so it would not pull that information very frequently (maybe just on
startup)? I'm not sure if that would be an issue or not, so I'm open for
input. The loggly guys ran into an issue with pulling the entire state but
that may have been because the current implementation on trunk polls the state
every 5 seconds for updates instead of using watchers.
{quote}
Regarding putting the information about the collections available into the live
nodes, I think that would be inefficient for the clients compared to putting it
into a summarized file. For commanding the nodes, it is very bad practice to
mix command and status files in ZK.
{quote}
Understood, I'll have to look at the leader task to see where it is at because
that would be a blocker to this if we were to attempt to do some summarized
file.
{quote}
I am a Zookeeper brother, btw. (PMC member and all that)
{quote}
Good to know, I am just a wanderer in this world so never sure who worships
where ;)
> Shard/Node states
> -----------------
>
> Key: SOLR-2765
> URL: https://issues.apache.org/jira/browse/SOLR-2765
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud, update
> Reporter: Yonik Seeley
> Fix For: 4.0
>
> Attachments: incremental_update.patch, shard-roles.patch
>
>
> Need state for shards that indicate they are recovering, active/enabled, or
> disabled.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
