[
https://issues.apache.org/jira/browse/SOLR-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123607#comment-13123607
]
Ted Dunning edited comment on SOLR-2765 at 10/9/11 3:55 AM:
------------------------------------------------------------
{quote}
the idea being if node after node comes up and triggers the watch, if there is
already a state update scheduled (5 seconds after the watch is triggered by
default), a new update is not asked for - eg if nodes are firing up by the
hundreds, the state is still only updated once every 5 seconds.
{quote}
Something like this is a great approach to metering the number of watch
notifications that need to be sent by the ZK server. I would prefer to change
the strategy slightly so that an update is scheduled immediately if no update
has happened in the last 5 seconds, but is not scheduled more often than about
every 5 seconds. The update can then unconditionally set the requisite watches.
This means that you get immediate response to the occasional change, but never
require more than n/5 updates per second where n is the number of clients. You
can even make the delay be something like n/log_2 ( n )/20 for n >= 2 where n
is the number of clients. This gives fast response for small clusters with
notification rates that grow slowly for large clusters. At 1000 clients, you
get 200 notifications per second worst case and at 8000 clients you get 500
notifications per second. In return for this very moderate transaction growth
rate, it takes longer to register that a new cluster has appeared. If many
nodes register in rapid succession, it takes 16 seconds for 8000 clients to all
get the news, 5 seconds for 1000 clients and just less than a second for 100
clients.
My guess is that 100 clients is a very large number.
If counting clients is a pain, the constant value of 5 seconds delay is pretty
good.
was (Author: tdunning):
{quote}
the idea being if node after node comes up and triggers the watch, if there is
already a state update scheduled (5 seconds after the watch is triggered by
default), a new update is not asked for - eg if nodes are firing up by the
hundreds, the state is still only updated once every 5 seconds.
{quote}
Something like this is a great approach to metering the number of watch
notifications that need to be sent by the ZK server. I would prefer to change
the strategy slightly so that an update is scheduled immediately if no update
has happened in the last 5 seconds, but is not scheduled more often than about
every 5 seconds. The update can then unconditionally set the requisite watches.
This means that you get immediate response to the occasional change, but never
require more than n/5 updates per second where n is the number of clients. You
can even make the delay be something like n/log_2(n)/20 for n >= 2 where n is
the number of clients. This gives fast response for small clusters with
notification rates that grow slowly for large clusters. At 1000 clients, you
get 200 notifications per second worst case and at 8000 clients you get 500
notifications per second. In return for this very moderate transaction growth
rate, it takes longer to register that a new cluster has appeared. If many
nodes register in rapid succession, it takes 16 seconds for 8000 clients to all
get the news, 5 seconds for 1000 clients and just less than a second for 100
clients.
My guess is that 100 clients is a very large number.
If counting clients is a pain, the constant value of 5 seconds delay is pretty
good.
> Shard/Node states
> -----------------
>
> Key: SOLR-2765
> URL: https://issues.apache.org/jira/browse/SOLR-2765
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud, update
> Reporter: Yonik Seeley
> Fix For: 4.0
>
> Attachments: incremental_update.patch, scheduled_executors.patch,
> shard-roles.patch
>
>
> Need state for shards that indicate they are recovering, active/enabled, or
> disabled.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
