I've been looking at the slow queries our Solr installation is
receiving. They are dominated by queries with a simple q parameter
(often *:* for all docs) and a VERY complicated fq parameter. The
filter query is built by going through a set of rules for the user and
putting together each rule's query clause separated by OR -- we can't
easily break it into multiple filters.
In addition to causing queries themselves to run slowly, this causes
large autowarm times for our filterCache -- my filterCache autowarmCount
is tiny (4), but it sometimes takes 30 seconds to warm.
I've seen a number of requests here for the ability to have multiple fq
parameters ORed together. This is probably possible, but in the
interests of compatibility between versions, very impractical. What if
a new parameter was introduced? It could be named fqi, for filter query
intersection. To figure out the final bitset for multiple fq and fqi
parameters, it would use this kind of logic:
fq AND fq AND fq AND (fqi OR fqi OR fqi)
This would let us break our filters into manageable pieces that can
efficiently populate the filterCache, and they would autowarm quickly.
Is the filter design in Solr separated cleanly enough to make this at
all reasonable? I'm not a Java developer, so I'd have a tough time
implementing it myself. When I have a free moment I will take a look at
the code anyway. I'm trying to teach myself Java.
Thanks,
Shawn
