On 1/10/2013 2:09 AM, Shahar Davidson wrote:
As for your first question, the core info needs to be gathered upon every
search request because cores are created dynamically.
When a user initiates a search request, the system must be aware of all
available cores in order to execute distributed search on _all_ relevant cores.
(the user must get reliable and most up to date data)
The reason that 800ms seems a lot to me is because the overall execution time
takes about 2500ms and a large part of it is due to the STATUS request.
The "minimal interval" concept is a good idea and indeed we've considered it,
yet it poses a slight problem when building a RT system which needs to return to most up
to date data.
I am just trying to understand if there's some other way to hasten the STATUS
reply (for example, by asking the STATUS request to return just certain core
attributes, such as name, instead of collecting everything)
Are there a *huge* number of SolrJ clients in the wild, or is it
something like a server farm where you are in control of everything? If
it's the latter, what I think I would do is have an asynchronous thread
that periodically (every few seconds) updates the client's view of what
cores exist. When a query is made, it will use that information,
speeding up your queries by 800 milliseconds and ensuring that new cores
will not have long delays before they become searchable. If you have a
huge number of clients in the wild, it would still be possible, but
ensuring that those clients get updated might be hard.
If you also delete cores as well as add them, that complicates things.
You'd have to have the clients be smart enough to exclude the last core
on the list (by whatever sorting mechanism you require), and you'd have
to wait long enough (30 seconds, maybe?) before *actually* deleting the
last core to be sure that no clients are accessing it.
Or you could use SolrCloud, as Per suggested, but with 4.1, not the
released 4.0. SolrCloud manages your cores for you automatically.
You'd probably be using a slightly customized SolrCloud, including the
custom hashing capability added by SOLR-2592. I don't know what other
customizations you might need.
Thanks,
Shawn
