Doug Turnbull created SOLR-16894:
------------------------------------
Summary: Configurable doc freq: Allow StatsCache instances to be
ResourceLoaderAware
Key: SOLR-16894
URL: https://issues.apache.org/jira/browse/SOLR-16894
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Reporter: Doug Turnbull
I had been working on a plugin to allow the document frequency stats to be
controlled by the user. This has precedent in other search engines where
another corpus is more representative of a terms true document frequency /
significance. Specifically, [Vespa lets you pass significance at query
time.|[https://docs.vespa.ai/en/reference/query-language-reference.html#significance]].
This doesn't just apply to how doc freq is represented, but the entire set of
stats from total term freq, etc.
This is a common painpoint in test corpuses where you have a smaller sample of
the documents than the global corpus. It was a frequency bugabear at Shopify,
and now at my current employer, for doing relevance testing. It's also a
problem whenever you have a corpus that may include some "outliers" that
actually aren't outliers in the sense of natural language.
I had made some progress ([here|http://example.com]), however I noticed only
certain types of classes can be ResourceLoaderAware in order to read
configuration. Specifically I see this error running my tests:
{code:java}
./gradlew --stacktrace --info test
{code}
{code:java}
org.apache.solr.common.SolrException: Invalid 'Aware' object:
manual.idf.stats.ManagedStatsCache@5c19c030 --
org.apache.lucene.util.ResourceLoaderAware must be an instance of:
[org.apache.lucene.analysis.CharFilterFactory]
[org.apache.lucene.analysis.TokenFilterFactory]
[org.apache.lucene.analysis.TokenizerFactory]
[org.apache.solr.search.QParserPlugin] [org.apache.solr.schema.FieldType]{code}
Can I propose we add the StatsCache to the list of allowed ResourceLoaderAware
objects?
Some alternatives I've thought about:
* I probably can do some ugly hacks to work around this, but I'd rather do the
"right thing"
* I'd prefer not to create a separate fieldtype that changes how the stats are
managed. For one, in my specific case, I don't want to have to have a radically
different test config compared to my setup. This is still "text" with texty
like configurability
** Second I like the ability with the stats cache to "fall back" to an
internal stat if one is missing.
* Pass at query time - this is a more radical change similar to what it would
take to make BM25 params configurable at query time
* It's possible I could create a Similarity to change doc freq, however it
too, would not be ResourceLoaderAware apparently.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
