On 3/6/2018 10:16 AM, Moncif Aidi wrote:
I am using Solr to power faceting features for our  application.

I know that SOLR can do free text search but what is the best practice for
faceting on common terms inside SOLR text fields?

Based on everything below, there might be a little bit of confusion about exactly what faceting can offer you.  It is an enormously powerful feature, and generally has impressive performance.  But there are limitations, and sometimes performance is not what people expect.

As your other reply mentioned, configuring docValues on a field is recommended for performance and other reasons with faceting.  But when you're dealing with a field set up for full-text search, that recommendation generally has to be ignored, because you can't configure docValues on a field using the TextField class.

For example, we have a large blob of text (a description of a property)
which contains useful text to facet on like 'city', 'formation', 'year',
'school', 'skill', ... dozens more like these.

When you have a "large blob of text" there are generally two choices for the information in a facet.

One is the entirety of the blob, which usually means that every single document has a unique value, and in that case, facets are pretty much useless, and will have terrible performance.  It's useless because all of the entries in the facet are probably going to have "1" for the count, because only one document has each value.

The other is the individual terms (usually words) in the text.  This is also generally useless for facets, and usually has terrible performance.  Knowing that there are 100 million documents that have "the" in the field somewhere is not very useful.

One obvious solution is to pre-process the data, parse the text, and create
the facets for each of these key phrases with a boolean yes/no value.

I'd ideally like to automate this, so I imagine the SOLR free text search
engine might allow this? e.g. Can I use the free text search engine to
remove stop words and collect counts of common phrases which we can then
present to the user?

And now you've mentioned that what you want is *phrases*. How do you suggest Solr obtain this information?  There are no filters included with Solr that can figure out that one section of a few words is NOT a phrase that people will be interested in, but another IS.

To get document counts that include a phrase, you have to have something that can extract phrases from the big blob of text and add them to another field, usually of type "string" -- using class StrField.  This probably has to happen in your indexing pipeline, not in Solr.

Then when you facet on that field, Solr will count the documents for each value and give you that information.


Reply via email to