Re: When is too many fields in qf is too many?

2015-05-31 Thread Tomasz Borek
White [mailto:swhite4...@gmail.com] Sent: Thursday, May 28, 2015 5:59 PM To: solr-user@lucene.apache.org Subject: Re: When is too many fields in qf is too many? Hi Folks, First, thanks for taking the time to read and reply to this subject, it is much appreciated, I have yet to come up

RE: When is too many fields in qf is too many?

2015-05-29 Thread Reitzel, Charles
...@gmail.com] Sent: Thursday, May 28, 2015 5:59 PM To: solr-user@lucene.apache.org Subject: Re: When is too many fields in qf is too many? Hi Folks, First, thanks for taking the time to read and reply to this subject, it is much appreciated, I have yet to come up with a final solution

Re: When is too many fields in qf is too many?

2015-05-28 Thread Jack Krupansky
the use case you describe. Just a thought ... -Original Message- From: Steven White [mailto:swhite4...@gmail.com] Sent: Tuesday, May 26, 2015 4:48 PM To: solr-user@lucene.apache.org Subject: Re: When is too many fields in qf is too many? Thanks Doug. I might

Re: When is too many fields in qf is too many?

2015-05-28 Thread Steven White
is too many fields in qf is too many? Thanks Doug. I might have to take you on the hangout offer. Let me refine the requirement further and if I still see the need, I will let you know. Steve On Tue, May 26, 2015 at 2:01 PM, Doug Turnbull dturnb

RE: When is too many fields in qf is too many?

2015-05-28 Thread Reitzel, Charles
@lucene.apache.org Subject: Re: When is too many fields in qf is too many? Gotta agree with Jack here. This is an insane number of fields, query performance on any significant corpus will be fraught etc. The very first thing I'd look at is having that many fields. You have 3,500 different fields

Re: When is too many fields in qf is too many?

2015-05-28 Thread Steven White
...@gmail.com] Sent: Tuesday, May 26, 2015 4:48 PM To: solr-user@lucene.apache.org Subject: Re: When is too many fields in qf is too many? Thanks Doug. I might have to take you on the hangout offer. Let me refine the requirement further and if I still see the need, I will let you know. Steve

Re: When is too many fields in qf is too many?

2015-05-28 Thread Jack Krupansky
Message- From: Steven White [mailto:swhite4...@gmail.com] Sent: Tuesday, May 26, 2015 4:48 PM To: solr-user@lucene.apache.org Subject: Re: When is too many fields in qf is too many? Thanks Doug. I might have to take you on the hangout offer. Let me refine the requirement further

Re: When is too many fields in qf is too many?

2015-05-28 Thread Erick Erickson
, but it seems like a reasonable approach given the use case you describe. Just a thought ... -Original Message- From: Steven White [mailto:swhite4...@gmail.com] Sent: Tuesday, May 26, 2015 4:48 PM To: solr-user@lucene.apache.org Subject: Re: When is too many fields in qf is too

RE: When is too many fields in qf is too many?

2015-05-27 Thread Reitzel, Charles
if this is feasible for you, but it seems like a reasonable approach given the use case you describe. Just a thought ... -Original Message- From: Steven White [mailto:swhite4...@gmail.com] Sent: Tuesday, May 26, 2015 4:48 PM To: solr-user@lucene.apache.org Subject: Re: When is too many

Re: When is too many fields in qf is too many?

2015-05-26 Thread Doug Turnbull
How you have tie is fine. Setting tie to 1 might give you reasonable results. You could easily still have scores that are just always an order of magnitude or two higher, but try it out! BTW Anything you put in teh URL can also be put into a request handler. If you ever just want to have a 15

Re: When is too many fields in qf is too many?

2015-05-26 Thread Steven White
Thanks Doug. I might have to take you on the hangout offer. Let me refine the requirement further and if I still see the need, I will let you know. Steve On Tue, May 26, 2015 at 2:01 PM, Doug Turnbull dturnb...@opensourceconnections.com wrote: How you have tie is fine. Setting tie to 1

Re: When is too many fields in qf is too many?

2015-05-26 Thread Steven White
Hi Doug, I'm back to this topic. Unfortunately, due to my DB structer, and business need, I will not be able to search against a single field (i.e.: using copyField). Thus, I have to use list of fields via qf. Given this, I see you said above to use tie=1.0 will that, more or less, address

When is too many fields in qf is too many?

2015-05-20 Thread Steven White
Hi everyone, My solution requires that users in group-A can only search against a set of fields-A and users in group-B can only search against a set of fields-B, etc. There can be several groups, as many as 100 even more. To meet this need, I build my search by passing in the list of fields via

Re: When is too many fields in qf is too many?

2015-05-20 Thread Steven White
Also, is this 1500 fields that are always populated, or are there really a larger number of different record types, each with a relatively small number of fields populated in a particular document? Answer: This is a large number of different record types, each with a relatively small number of

Re: When is too many fields in qf is too many?

2015-05-20 Thread Steven White
Thanks Shawn. I have already switched to using POST because I need to send a long list of data in qf. My question isn't about POST / GET, it's about Solr and Lucene having to deal with such long list of fields. Here is the text of my question reposted: Given the above, beside the fact that a

Re: When is too many fields in qf is too many?

2015-05-20 Thread Shawn Heisey
On 5/20/2015 9:24 AM, Steven White wrote: I have already switched to using POST because I need to send a long list of data in qf. My question isn't about POST / GET, it's about Solr and Lucene having to deal with such long list of fields. Here is the text of my question reposted: Given

Re: When is too many fields in qf is too many?

2015-05-20 Thread Doug Turnbull
Yeah a copyField into one could be a good space/time tradeoff. It can be more manageable to use an all field for both relevancy and performance, if you can handle the duplication of data. You could set tie=1.0, which effectively sums all the matches instead of picking the best match. You'll still

Re: When is too many fields in qf is too many?

2015-05-20 Thread Steven White
Hi Doug, Your blog write up on relevancy is very interesting, I didn't know this. Looks like I have to go back to my drawing board and figure out an alternative solution: somehow get those group-based-fields data into a single field using copyField. Thanks Steve On Wed, May 20, 2015 at 11:17

Re: When is too many fields in qf is too many?

2015-05-20 Thread Steven White
Thanks for calling out maxBooleanClauses. The current default of 1024 has not caused me any issues (so far) in my testing. However, you probably saw Doug Tumbull's reply, it looks like my relevance will suffer. Steve On Wed, May 20, 2015 at 11:42 AM, Shawn Heisey apa...@elyograg.org wrote:

Re: When is too many fields in qf is too many?

2015-05-20 Thread Jack Krupansky
The uf parameter is used to specify which fields a user may query against - the qf parameter specifies the set of fields that an unfielded query term must be queried against. The user is free to specify fielded query terms, like field1:term1 OR field2:term2. So, which use case are you really

Re: When is too many fields in qf is too many?

2015-05-20 Thread Doug Turnbull
Steven, I'd be concerned about your relevance with that many qf fields. Dismax takes a winner takes all point of view to search. Field scores can vary by an order of magnitude (or even two) despite the attempts of query normalization. You can read more here

Re: When is too many fields in qf is too many?

2015-05-20 Thread Shawn Heisey
On 5/20/2015 6:27 AM, Steven White wrote: My solution requires that users in group-A can only search against a set of fields-A and users in group-B can only search against a set of fields-B, etc. There can be several groups, as many as 100 even more. To meet this need, I build my search by