[
https://issues.apache.org/jira/browse/SOLR-13494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joel Bernstein updated SOLR-13494:
----------------------------------
Description:
Currently the *random* Streaming Expression performs a conventional distributed
search. This involves retrieving the top N docs from each shard and then
selecting the top N from all the shards in the aggregator node. This technique
eventually bogs down as the number of shards goes up and/or N goes up.
Selecting distributed random samples does not actually require this behavior.
Instead you can select N/numShards from each shard and simply return all
results. This technique will actually get faster as more shards are added
instead of slowing down.
This ticket will allow the random Streaming Expression to use the strategy
above when N reaches a certain threshold (ie 10000).
The *DeepRandomStream* class will implement the deep random sampling behavior.
The random Streaming Expression will switch between the RandomStream and
DeepRandomStream depending on N.
was:
Currently the *random* Streaming Expression performs a conventional distributed
search. This involves retrieving the top N docs from each shard and then
selecting the top N from all the shards in the aggregator node. This technique
eventually bogs down as the number of shards goes up and/or N goes up.
Selecting distributed random samples does not actually require this behavior.
Instead you can select N/numShards from each shard and simply return all
results. This technique will actually get faster as more shards are added
instead of slowing down.
This ticket will allow the random Streaming Expression to use the strategy
above when N reaches a certain threshold (ie 10000).
The *DeepRandomStream* class will implement the deep random sampling behavior.
> Add DeepRandomStream implementation
> -----------------------------------
>
> Key: SOLR-13494
> URL: https://issues.apache.org/jira/browse/SOLR-13494
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: streaming expressions
> Reporter: Joel Bernstein
> Assignee: Joel Bernstein
> Priority: Major
>
> Currently the *random* Streaming Expression performs a conventional
> distributed search. This involves retrieving the top N docs from each shard
> and then selecting the top N from all the shards in the aggregator node. This
> technique eventually bogs down as the number of shards goes up and/or N goes
> up.
> Selecting distributed random samples does not actually require this behavior.
> Instead you can select N/numShards from each shard and simply return all
> results. This technique will actually get faster as more shards are added
> instead of slowing down.
> This ticket will allow the random Streaming Expression to use the strategy
> above when N reaches a certain threshold (ie 10000).
> The *DeepRandomStream* class will implement the deep random sampling behavior.
> The random Streaming Expression will switch between the RandomStream and
> DeepRandomStream depending on N.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
