Re: clarification regarding shard splitting and composite IDs

2015-02-05 Thread Dan Davis
Thanks, Anshum - I should never have posted so late.It is true that different users will have different word frequencies, but an application exploiting that for better relevancy would be going far for the relevancy of individual user's results. On Thu, Feb 5, 2015 at 12:41 AM, Anshum Gupta

Re: clarification regarding shard splitting and composite IDs

2015-02-04 Thread Gili Nachum
Hi, I'm also interested. When using composite the ID, the _route_ information is not kept on the document itself, so to me it looks like it's not possible as the split API https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-api3 doesn't have a relevant parameter to

Re: clarification regarding shard splitting and composite IDs

2015-02-04 Thread Anshum Gupta
In one line, shard splitting doesn't cater to depend on the routing mechanism but just the hash range so you could have documents for the same prefix split up. Here's an overview of routing in SolrCloud: * Happens based on a hash value * The hash is calculated using the multiple parts of the

Re: clarification regarding shard splitting and composite IDs

2015-02-04 Thread Gili Nachum
Alright. So shard splitting and composite routing plays nicely together. Thank you Anshum. On Wed, Feb 4, 2015 at 11:24 AM, Anshum Gupta ans...@anshumgupta.net wrote: In one line, shard splitting doesn't cater to depend on the routing mechanism but just the hash range so you could have

Re: clarification regarding shard splitting and composite IDs

2015-02-04 Thread Anshum Gupta
Solr 5.0 has support for distributed IDF. Also, users having the same IDF is orthogonal to the original question. In general, the Doc Freq. is only per-shard. If for some reason, a single user has documents split across shards, the IDF used would be different for docs on different shards. On

Re: clarification regarding shard splitting and composite IDs

2015-02-04 Thread Dan Davis
Doesn't relevancy for that assume that the IDF and TF for user1 and user2 are not too different?SolrCloud still doesn't use a distributed IDF, correct? On Wed, Feb 4, 2015 at 7:05 PM, Gili Nachum gilinac...@gmail.com wrote: Alright. So shard splitting and composite routing plays nicely