Steven, What does being your hero entails, beside a salute? :-)
Approach 1: Tinker with your-app - Solr relationship. Approach 2: Gauge what's really used and limit the customization. Approach 3: Offer what's wanted (might be different than what you're trying to achieve). In your write-up I'm unsure on what is being demanded of Solr, but I assume you're after searchability no matter the view / ... / field combination. Every single field can be searchable and I assume you're looking for a way to provide just that - search on every field is to be possible if user wants it. Ad 3: with "customize-all" apps folks usually quickly find patterns where they have what they want and they don't use other patterns unless they really need to. Offer relevant search for their favourite patterns and much weaker search for other patterns. Your current approach may be over-engineering. You may be trying to answer a problem product folks posed you, while real need lies elsewhere. So, I'm kinda asking you question the problem, to find out your work won't be in vain, or not as good for end-user as they might have a different problem (like: which field is which in my so-and-so customized view nr 57 which changed again this month). Ad 2: find out which fields are most important for searches and offer these to Solr. The real usage is usually much less than capability offered - so if you have a view with 200 fields, I doubt folks even want to query all 200, but perhaps only 5 matter. Find a way to know those 5 (via user prefs per view perhaps or default view config) and search only them. Ties nicely with #3 as folks most likely don't even WANT to query all fields. Humans like limits: we don't want too much elements on screen, we like simple UIs, we don't want to input too long search query and we often don't want too many choices. Ad 1. EXPERIMENT. First, create a way for you to manage your configs automatically and have them in version control, you'll need fast generation and even faster revert / regeneration when something is NOT OK. Set up more than one way to achieve your search-them-all-as-user-pleases approach and test and compare them. Your case is quite unique (3,5k qf fields anyone? changing collections monthly) and I don't think without experimenting you will get good results, you need to compare number of options. Can you offer Solr servers per user group? Do you have similarities in views for user groups - even if informal? Like, 30% of your user base uses only 20% of all views that you have? Then it makes sense to have dedicated Solr for those 20% of all views. You'll need routing here and rules per user groups in your app. How many customizations you have and how can you use that? Are there any patterns in customizing views that you can predict / observe / use? Kinda synthesis of all approaches, but with your customization level I don't think one Solr for all cases will be of any use, even if you do manage to have it by some tinkering with settings. As I kinda looked at the problem not in terms of Solr settings, this is somewhat off-topic, so if you wish to ask something it might be better off the group, unless others want the thread to continue here out of curiosity how it ends. pozdrawiam, LAFK > -----Original Message----- > From: Steven White [mailto:swhite4...@gmail.com] > Sent: Thursday, May 28, 2015 5:59 PM > To: solr-user@lucene.apache.org > Subject: Re: When is too many fields in "qf" is too many? > > Hi Folks, > > First, thanks for taking the time to read and reply to this subject, it is > much appreciated, I have yet to come up with a final solution that > optimizes Solr. To give you more context, let me give you the big picture > of how the application and the database is structured for which I'm trying > to enable Solr search on. > > Application: Has the concept of "views". A view contains one or more > object types. An object type may exist in any view. An object type has > one or more field groups. A field group has a set of fields. A field > group can be used with any object type of any view. Notice how field > groups are free standing, that they can be "linked" to an object type of > any view? > > Here is a diagram of the above: > > FieldGroup-#1 == Field-1, Field-2, Field-5, etc. > FieldGroup-#2 == Field-1, Field-5, Field-6, Field-7, Field-8, etc. > FieldGroup-#3 == Field-2, Field-5, Field-8, etc. > > View-#1 == ObjType-#2 (using FieldGroup-#1 & #3) + ObjType-#4 (using > FieldGroup-#1) + ObjType-#5 (using FieldGroup-#1, #2, #3, etc). > > View-#2 == ObjType-#1 (using FieldGroup-#3, #15, #16, #19, etc.) + > ObjType-#4 (using FieldGroup-#1, #4, #19, etc.) + etc. > > View-#3 == ObjType-#1 (using FieldGroup-#1, & #8) + etc. > > Do you see where this is heading? To make it even a bit more interesting, > ObjType-#4 (which is in view-#1 and #2 per the above) which in both views, > it uses FieldGroup-#1, in one view it can be configured to have its own > fields off FieldGroup-#1. > > With the above setting, a user is assigned a view and can be moved around > views but cannot be in multiple views at the same time. Based on which > view that user is in, that user will see different fields of ObjType-#1 > (the example I gave for FieldGroup-#1) or even not see an object type that > he was able to see in another view. > > If I have not lost you with the above, you can see that per view, there > can be may fields. To make it even yet more interesting, a field in > FieldGroup-#1 may have the exact same name as a field in another > FieldGroup and the two could be of different type (one is date, the other > is string type). Thus when I build my Solr doc object (and create list of > Solr > fields) those fields must be prefixed with the FieldGroup name otherwise I > could end up overwriting the type of another field. > > Are you still with me? :-) > > Now you see how a view can end up with many fields (over 3500 in my case), > but a doc I post to Solr for indexing will have on average 50 fields, worse > case maybe 200 fields. This is fine, and it is not my issue but I want to > call it out to get it out of our way. > > Another thing I need to mention is this (in case it is not clear from the > above). Users create and edit records in the DB by an instance of > ObjType-#N. Those object types that are created do NOT belong to a view, > in fact they do NOT have any view concept in them. They simply have the > concept of what fields the user can see / edit based on which view that > user is in. In effect, in the DB, we have instances of object types data. > > One last thing I should point out is that views, and field groups are > dynamic. This month, View-#3 may have ObjType-#1, but next month it may > not or a new object type may be added to it. > > Still with me? If so, you are my hero!! :-) > > So, I setup my Solr schema.xml to include all fields off each field group > that exists in the database like so: > > <field name="FieldGroup-1.Headline" type="text" multiValued="true" > indexed="true" stored="false" required="false"/> > <field name="FieldGroup-1.Summary" type="text" multiValued="true" > indexed="true" stored="false" required="false"/> > <field name="FieldGroup-1. ... ... ... ... /> > <field name="FieldGroup-2.Headline" type="text" multiValued="true" > indexed="true" stored="false" required="false"/> > <field name="FieldGroup-2.Summary" type="text" multiValued="true" > indexed="true" stored="false" required="false"/> > <field name="FieldGroup-2.Date" type="text" multiValued="true" > indexed="true" stored="false" required="false"/> > <field name="FieldGroup-2. ... ... ... ... /> > <field name="FieldGroup-3. ... ... ... ... /> > <field name="FieldGroup-4. ... ... ... ... /> > > You got the idea. Each record of an object type I index contains ALL the > fields off that that object type REGARDLESS which view that object type is > set to be in (remember, all that views does is let you configure the list > of fields visible / accessible in that view). > > Next, in Solr I created request handlers per view. The request handler > utilizes "qf" to list all fields that are viewable for that view. When a > user logs into the application, I know which view that user is in so I > issue a search request against that view in effect the search is against > the list of fields of that view. > > Why not create a per view pseudo Solr field and copyField into it the > fields data and than use that single field as the "qf" vs. 100's of filed? > Two reasons: > > 1) Like I said above, views are dynamic. On a monthly basic, a object > types or even field groups can be added / removed from a view. If I was > using copyField it means I have to reindex my entire database to reflect a > view change even when the actual data has not changed. > > 2) My Solr index size will now be larger. I have to create a pseudo Solr > field to copyField to it for each view in my database. > > I have also considered creating multiple cores per view, but that still > doesn't solve the above two issues, requiring reindex and increasing the > index size. > > Now that you see what my backend application is like, let me know if you > have any ideas on how you would solve this puzzle. > > And if you have read this all the way to the end, I solute you!! > > Steve > > > On Thu, May 28, 2015 at 4:23 PM, Jack Krupansky <jack.krupan...@gmail.com> > wrote: > > > I would reconsider the strategy of mashing so many different record > > types into one Solr collection. Sure, you get some advantage from > > denormalizing data, but if the downside cost gets too high, it may not > > make so much sense. > > > > I'd consider a collection per record type, or at least group similar > > record types, and then query as many collections - in parallel - as > > needed for a given user. That should also assure that a query for a > > given record type should be much faster as well. > > > > Surely you should be able to examine the query in the app and > > determine what record types it might apply to. > > > > When in doubt, make your schema as clean and simple as possible. > > Simplicity over complexity. > > > > > > -- Jack Krupansky > > > > On Thu, May 28, 2015 at 12:06 PM, Erick Erickson > > <erickerick...@gmail.com> > > wrote: > > > > > Gotta agree with Jack here. This is an insane number of fields, > > > query performance on any significant corpus will be "fraught" etc. > > > The very first thing I'd look at is having that many fields. You > > > have 3,500 different fields! Whatever the motivation for having that > > > many fields is the place I'd start..... > > > > > > Best, > > > Erick > > > > > > On Thu, May 28, 2015 at 5:50 AM, Jack Krupansky > > > <jack.krupan...@gmail.com> wrote: > > > > This does not even pass a basic smell test for reasonability of > > matching > > > > the capabilities of Solr and the needs of your application. I'd > > > > like to hear from others, but I personally would be -1 on this > > > > approach to > > > misusing > > > > qf. I'd simply say that you need to go back to the drawing board, > > > > and > > > that > > > > your primary focus should be on working with your application > > > > product manager to revise your application requirements to more > > > > closely match > > the > > > > capabilities of Solr. > > > > > > > > To put it simply, if you have more than a dozen fields in qf, > > > > you're probably doing something wrong. In this case horribly wrong. > > > > > > > > Focus on designing your app to exploit the capabilities of Solr, > > > > not to misuse them. > > > > > > > > In short, to answer the original question, more than a couple > > > > dozen > > > fields > > > > in qf is indeed too many. More than a dozen raises a yellow flag > > > > for > > me. > > > > > > > > > > > > -- Jack Krupansky > > > > > > > > On Thu, May 28, 2015 at 8:13 AM, Steven White > > > > <swhite4...@gmail.com> > > > wrote: > > > > > > > >> Hi Charles, > > > >> > > > >> That is what I have done. At the moment, I have 22 request > > > >> handlers, > > > some > > > >> have 3490 field items in "qf" (that's the most and the qf line > > > >> spans > > > over > > > >> 95,000 characters in solrconfig.xml file) and the least one has > > > >> 1341 fields. I'm working on seeing if I can use copyField to > > > >> copy the data > > > of > > > >> that view's field into a single pseudo-view-field and use that > > > >> pseudo > > > field > > > >> for "qf" of that view's request handler. The I still have > > > >> outstanding > > > with > > > >> using copyField in this way is that it could lead to a complete > > > re-indexing > > > >> of all the data in that view when a field is adding / removing > > > >> from > > that > > > >> view. > > > >> > > > >> Thanks > > > >> > > > >> Steve > > > >> > > > >> On Wed, May 27, 2015 at 6:02 PM, Reitzel, Charles < > > > >> charles.reit...@tiaa-cref.org> wrote: > > > >> > > > >> > One request handler per view? > > > >> > > > > >> > I think if you are able to make the actual view in use for the > > current > > > >> > request a single value (vs. all views that the user could use > > > >> > over > > > time), > > > >> > it would keep the qf list down to a manageable size (e.g. > > > >> > specified > > > >> within > > > >> > the request handler XML). Not sure if this is feasible for you, > > > but it > > > >> > seems like a reasonable approach given the use case you describe. > > > >> > > > > >> > Just a thought ... > > > >> > > > > >> > -----Original Message----- > > > >> > From: Steven White [mailto:swhite4...@gmail.com] > > > >> > Sent: Tuesday, May 26, 2015 4:48 PM > > > >> > To: solr-user@lucene.apache.org > > > >> > Subject: Re: When is too many fields in "qf" is too many? > > > >> > > > > >> > Thanks Doug. I might have to take you on the hangout offer. > > > >> > Let me refine the requirement further and if I still see the > > > >> > need, I will > > let > > > >> you > > > >> > know. > > > >> > > > > >> > Steve > > > >> > > > > >> > On Tue, May 26, 2015 at 2:01 PM, Doug Turnbull < > > > >> > dturnb...@opensourceconnections.com> wrote: > > > >> > > > > >> > > How you have tie is fine. Setting tie to 1 might give you > > reasonable > > > >> > > results. You could easily still have scores that are just > > > >> > > always > > an > > > >> > > order of magnitude or two higher, but try it out! > > > >> > > > > > >> > > BTW Anything you put in teh URL can also be put into a > > > >> > > request > > > handler. > > > >> > > > > > >> > > If you ever just want to have a 15 minute conversation via > > hangout, > > > >> > > happy to chat with you :) Might be fun to think through your > > > >> > > prob > > > >> > together. > > > >> > > > > > >> > > -Doug > > > >> > > > > > >> > > On Tue, May 26, 2015 at 1:42 PM, Steven White < > > swhite4...@gmail.com > > > > > > > >> > > wrote: > > > >> > > > > > >> > > > Hi Doug, > > > >> > > > > > > >> > > > I'm back to this topic. Unfortunately, due to my DB > > > >> > > > structer, > > and > > > >> > > business > > > >> > > > need, I will not be able to search against a single field > (i.e.: > > > >> > > > using copyField). Thus, I have to use list of fields via > "qf". > > > >> > > > Given this, I see you said above to use "tie=1.0" will > > > >> > > > that, > > more > > > or > > > >> > > > less, address this scoring issue? Should "tie=1.0" be set > > > >> > > > on > > the > > > >> > request handler like so: > > > >> > > > > > > >> > > > <requestHandler name="/select" class="solr.SearchHandler"> > > > >> > > > <lst name="defaults"> > > > >> > > > <str name="echoParams">explicit</str> > > > >> > > > <int name="rows">20</int> > > > >> > > > <str name="defType">edismax</str> > > > >> > > > <str name="qf">F1 F2 F3 F4 ... ... ...</str> > > > >> > > > <float name="tie">1.0</float> > > > >> > > > <str name="fl">_UNIQUE_FIELD_,score</str> > > > >> > > > <str name="wt">xml</str> > > > >> > > > <str name="indent">true</str> > > > >> > > > </lst> > > > >> > > > </requestHandler> > > > >> > > > > > > >> > > > Or must "tie" be passed as part of the URL? > > > >> > > > > > > >> > > > Thanks > > > >> > > > > > > >> > > > Steve > > > >> > > > > > > >> > > > > > > >> > > > On Wed, May 20, 2015 at 2:58 PM, Doug Turnbull < > > > >> > > > dturnb...@opensourceconnections.com> wrote: > > > >> > > > > > > >> > > > > Yeah a copyField into one could be a good space/time > tradeoff. > > > It > > > >> > > > > can > > > >> > > be > > > >> > > > > more manageable to use an all field for both relevancy > > > >> > > > > and performance, > > > >> > > > if > > > >> > > > > you can handle the duplication of data. > > > >> > > > > > > > >> > > > > You could set tie=1.0, which effectively sums all the > > > >> > > > > matches instead > > > >> > > of > > > >> > > > > picking the best match. You'll still have cases where one > > > field's > > > >> > > > > score might just happen to be far off of another, and > > > >> > > > > thus dominating the summation. But something easy to try > > > >> > > > > if you > > want > > > to > > > >> > > > > keep playing with dismax. > > > >> > > > > > > > >> > > > > -Doug > > > >> > > > > > > > >> > > > > On Wed, May 20, 2015 at 2:56 PM, Steven White > > > >> > > > > <swhite4...@gmail.com> > > > >> > > > > wrote: > > > >> > > > > > > > >> > > > > > Hi Doug, > > > >> > > > > > > > > >> > > > > > Your blog write up on relevancy is very interesting, I > > didn't > > > >> > > > > > know > > > >> > > > this. > > > >> > > > > > Looks like I have to go back to my drawing board and > > > >> > > > > > figure > > > out > > > >> > > > > > an alternative solution: somehow get those > > group-based-fields > > > >> > > > > > data into > > > >> > > a > > > >> > > > > > single field using copyField. > > > >> > > > > > > > > >> > > > > > Thanks > > > >> > > > > > > > > >> > > > > > Steve > > > >> > > > > > > > > >> > > > > > On Wed, May 20, 2015 at 11:17 AM, Doug Turnbull < > > > >> > > > > > dturnb...@opensourceconnections.com> wrote: > > > >> > > > > > > > > >> > > > > > > Steven, > > > >> > > > > > > > > > >> > > > > > > I'd be concerned about your relevance with that many > > > >> > > > > > > qf > > > fields. > > > >> > > > Dismax > > > >> > > > > > > takes a "winner takes all" point of view to search. > > > >> > > > > > > Field scores > > > >> > > can > > > >> > > > > vary > > > >> > > > > > > by an order of magnitude (or even two) despite the > > attempts > > > of > > > >> > > query > > > >> > > > > > > normalization. You can read more here > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > http://opensourceconnections.com/blog/2013/07/02/getting-dissed-by-d > > > is > > > >> > > > > > max-why-your-incorrect-assumptions-about-dismax-are-hurting-search-r > > > el > > > >> > > evancy/ > > > >> > > > > > > > > > >> > > > > > > I'm about to win the "blashphemer" merit badge, but > > > >> > > > > > > ad-hoc > > > >> > > all-field > > > >> > > > > like > > > >> > > > > > > searching over many fields is actually a good use > > > >> > > > > > > case for > > > >> > > > > > Elasticsearch's > > > >> > > > > > > cross field queries. > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > https://www.elastic.co/guide/en/elasticsearch/guide/master/_cross_fi > > > el > > > >> > > ds_queries.html > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > http://opensourceconnections.com/blog/2015/03/19/elasticsearch-cross > > > -f > > > >> > > ield-search-is-a-lie/ > > > >> > > > > > > > > > >> > > > > > > It wouldn't be hard (and actually a great feature for > > > >> > > > > > > the > > > >> > > > > > > project) > > > >> > > to > > > >> > > > > get > > > >> > > > > > > the Lucene query associated with cross field search > > > >> > > > > > > into > > > Solr. > > > >> > > > > > > You > > > >> > > > > could > > > >> > > > > > > easily write a plugin to integrate it into a query > parser: > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > https://github.com/elastic/elasticsearch/blob/master/src/main/java/o > > > rg > > > >> > > /apache/lucene/queries/BlendedTermQuery.java > > > >> > > > > > > > > > >> > > > > > > Hope that helps > > > >> > > > > > > -Doug > > > >> > > > > > > -- > > > >> > > > > > > *Doug Turnbull **| *Search Relevance Consultant | > > OpenSource > > > >> > > > > Connections, > > > >> > > > > > > LLC | 240.476.9983 | > > > >> > > > > > > http://www.opensourceconnections.com > > > >> > > > > > > Author: Relevant Search <http://manning.com/turnbull> > > from > > > >> > > > > > > Manning Publications This e-mail and all contents, > > including > > > >> > > > > > > attachments, is considered > > > >> > > to > > > >> > > > > be > > > >> > > > > > > Company Confidential unless explicitly stated > > > >> > > > > > > otherwise, regardless of whether attachments are marked > as such. > > > >> > > > > > > On Wed, May 20, 2015 at 8:27 AM, Steven White < > > > >> > > swhite4...@gmail.com> > > > >> > > > > > > wrote: > > > >> > > > > > > > > > >> > > > > > > > Hi everyone, > > > >> > > > > > > > > > > >> > > > > > > > My solution requires that users in group-A can only > > search > > > >> > > against > > > >> > > > a > > > >> > > > > > set > > > >> > > > > > > of > > > >> > > > > > > > fields-A and users in group-B can only search > > > >> > > > > > > > against a > > > set > > > >> > > > > > > > of > > > >> > > > > > fields-B, > > > >> > > > > > > > etc. There can be several groups, as many as 100 > > > >> > > > > > > > even > > > more. > > > >> > > > > > > > To > > > >> > > > meet > > > >> > > > > > > this > > > >> > > > > > > > need, I build my search by passing in the list of > > > >> > > > > > > > fields > > > via > > > >> > > "qf". > > > >> > > > > > What > > > >> > > > > > > > goes into "qf" can be large: as many as 1500 fields > > > >> > > > > > > > and > > > each > > > >> > > field > > > >> > > > > name > > > >> > > > > > > > averages 15 characters long, in effect the data > > > >> > > > > > > > passed > > via > > > >> "qf" > > > >> > > > will > > > >> > > > > be > > > >> > > > > > > > over 20K characters. > > > >> > > > > > > > > > > >> > > > > > > > Given the above, beside the fact that a search for > > "apple" > > > >> > > > > translating > > > >> > > > > > > to a > > > >> > > > > > > > 20K characters passing over the network, what else > > within > > > >> > > > > > > > Solr > > > >> > > and > > > >> > > > > > > Lucene I > > > >> > > > > > > > should be worried about if any? Will I hit some > > > >> > > > > > > > kind > > of a > > > >> > limit? > > > >> > > > > Will > > > >> > > > > > > > each search now require more CPU cycles? Memory? > Etc. > > > >> > > > > > > > > > > >> > > > > > > > If the network traffic becomes an issue, my > > > >> > > > > > > > alternative solution > > > >> > > is > > > >> > > > > to > > > >> > > > > > > > create a /select handler for each group and in that > > > handler > > > >> > > > > > > > list > > > >> > > > the > > > >> > > > > > > fields > > > >> > > > > > > > under "qf". > > > >> > > > > > > > > > > >> > > > > > > > I have considered creating pseudo-fields for each > > > >> > > > > > > > group > > > and > > > >> > > > > > > > then > > > >> > > > use > > > >> > > > > > > > copyField into that group. During search, I than > > > >> > > > > > > > can > > "qf" > > > >> > > against > > > >> > > > > that > > > >> > > > > > > one > > > >> > > > > > > > field. Unfortunately, this is not ideal for my > > > >> > > > > > > > solution because > > > >> > > > the > > > >> > > > > > > fields > > > >> > > > > > > > that go into each group dynamically change (at > > > >> > > > > > > > least > > once > > > a > > > >> > > month) > > > >> > > > > and > > > >> > > > > > > when > > > >> > > > > > > > they do change, I have to re-index everything (this > > > >> > > > > > > > I > > have > > > >> > > > > > > > to > > > >> > > > avoid) > > > >> > > > > to > > > >> > > > > > > > sync that group-field. > > > >> > > > > > > > > > > >> > > > > > > > I'm using "qf" with edismax and my Solr version is > 5.1. > > > >> > > > > > > > > > > >> > > > > > > > Thanks > > > >> > > > > > > > > > > >> > > > > > > > Steve > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > -- > > > >> > > > > *Doug Turnbull **| *Search Relevance Consultant | > > > >> > > > > OpenSource > > > >> > > Connections, > > > >> > > > > LLC | 240.476.9983 | http://www.opensourceconnections.com > > > >> > > > > Author: Relevant Search <http://manning.com/turnbull> > > > >> > > > > from > > > Manning > > > >> > > > > Publications This e-mail and all contents, including > > > attachments, > > > >> > > > > is considered to > > > >> > > be > > > >> > > > > Company Confidential unless explicitly stated otherwise, > > > >> > > > > regardless of whether attachments are marked as such. > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > -- > > > >> > > *Doug Turnbull **| *Search Relevance Consultant | OpenSource > > > >> > > Connections, LLC | 240.476.9983 | > > > http://www.opensourceconnections.com > > > >> > > Author: Relevant Search <http://manning.com/turnbull> from > > Manning > > > >> > > Publications This e-mail and all contents, including > > > >> > > attachments, > > is > > > >> > > considered to be Company Confidential unless explicitly > > > >> > > stated otherwise, regardless of whether attachments are marked > as such. > > > >> > > > > > >> > > > > >> > > > > ******************************************************************** > > > ***** > > > >> > This e-mail may contain confidential or privileged information. > > > >> > If you are not the intended recipient, please notify the sender > > > >> > immediately and then delete it. > > > >> > > > > >> > TIAA-CREF > > > >> > > > > ******************************************************************** > > > ***** > > > >> > > > > >> > > > > > > > ************************************************************************* > This e-mail may contain confidential or privileged information. > If you are not the intended recipient, please notify the sender > immediately and then delete it. > > TIAA-CREF > ************************************************************************* >
