[
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14259180#comment-14259180
]
Shalin Shekhar Mangar commented on SOLR-6810:
---------------------------------------------
{quote}
When a different searcher is used (because of a commit) the ordinals could
refer to different docs.
But this seems to lead to acceptable behavior (unlike using internal docids
which leads to catastrophic types of fails).
You may get a different doc than expected in the second phase, but it will
still be highly ranked.
The failure modes (if you can call them that) when the index changes seem to be
relatively equivalent to using external IDs.
{quote}
I think the only failure mode that gets worse is the one with duplicate ids
across shards. When we merged with ids in the first phase, a dup would be
discarded but we would (usually) have enough ids to return the requested number
of rows. In this new algorithm, we will either:
# return less rows than requested (which can be very confusing considering that
numFound will likely be larger than rows), or
# fetch more docs than necessary in the second phase (just in case there are
dups)
# introduce a third request to fetch more docs as and when necessary
But the advantages of this algorithm outweigh this annoyance.
bq. So if it's actually true that the behavior is comparable to the current
strategy when the index changes between phases, we should consider changing
implementations (as opposed to keeping the old implementation and making it
configurable).
Perhaps this is too big a change to do without some field testing?
bq. Doesn't work well with non-standard sorts? (like a diversifying sort?)
Yes that'd need some special handling. Maybe something in tandem with
LUCENE-6066 can work.
bq. Query re-execution when index changes (minor impact except for very high
frequency commits?)
Worth benchmarking though.
bq. Everything I'm thinking of so far leads me to believe the new strategy
should be the default.
+1
> Faster searching limited but high rows across many shards all with many hits
> ----------------------------------------------------------------------------
>
> Key: SOLR-6810
> URL: https://issues.apache.org/jira/browse/SOLR-6810
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Per Steffensen
> Assignee: Shalin Shekhar Mangar
> Labels: distributed_search, performance
> Attachments: branch_5x_rev1642874.patch, branch_5x_rev1642874.patch,
> branch_5x_rev1645549.patch
>
>
> Searching "limited but high rows across many shards all with many hits" is
> slow
> E.g.
> * Query from outside client: q=something&rows=1000
> * Resulting in sub-requests to each shard something a-la this
> ** 1) q=something&rows=1000&fl=id,score
> ** 2) Request the full documents with ids in the global-top-1000 found among
> the top-1000 from each shard
> What does the subject mean
> * "limited but high rows" means 1000 in the example above
> * "many shards" means 200-1000 in our case
> * "all with many hits" means that each of the shards have a significant
> number of hits on the query
> The problem grows on all three factors above
> Doing such a query on our system takes between 5 min to 1 hour - depending on
> a lot of things. It ought to be much faster, so lets make it.
> Profiling show that the problem is that it takes lots of time to access the
> store to get id’s for (up to) 1000 docs (value of rows parameter) per shard.
> Having 1000 shards its up to 1 mio ids that has to be fetched. There is
> really no good reason to ever read information from store for more than the
> overall top-1000 documents, that has to be returned to the client.
> For further detail see mail-thread "Slow searching limited but high rows
> across many shards all with high hits" started 13/11-2014 on
> dev@lucene.apache.org
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
