David Spencer wrote:
2 files attached, SubstringQuery (which you'll use) and SubstringTermEnum ( used by the former to be
consistent w/ other Query code).
I find this kind of query useful to have and think that the query parser should allow it in spite of the perception
of this being slow, however I think the debate is the "user centric view" (say mine, allow substring queries)
vs the "protect the engines performance" view which says not to allow expensive queries.
I think the argument is more complex.
Thanks for elaboration, a few notes below.
One issue is cost of execution: very slow queries can be used to implement a denial-of-service attack. Maybe that's an overstatement, but in a web server setting, once a few of slow searches are running, no others may complete. When folks hit "Stop" in their browser the server does not stop processing the query. If they hit "Reload" then another new search is started. So these can be very problematic.
I guess this is defendable right? The search engine could allow only one query at a time per session.
This is real. Lots of folks have deployed Lucene with large indexes and then found that their server randomly crashes. Closer scrutiny shows that they were permitting operators that are too slow for their combination of index size and query traffic. The BooleanQuery.TooManyClauses exception was added to address this, but it can still be too late, if the problem is caused before the query is built, e.g., while enumerating all terms.
Yeah, this is tricky, as really the TooManyClauses is not necessarily the cure, and every query can't defend itself against expense as these heuristics (e.g. max # of clauses) are not always right (depends on speed of system/disk, size of index, etc).
It's too bad the Java VM doesn't have any kind of execution time quotas for threads or something like that as that seems like in the ideal world to get to the root of the issue. Going back in time I believe the Telescript VM
from General Magic...
A releated issue is that users (and even most developers) don't understand the relative costs of different query operators.
Sure.
I'm coming at this from the point of view of the smaller personal/small enterprise search server, where
you "know" you need a certain kind of possibly expensive query and don't want your hands tied.
Some things are fast, others are surprisingly slow. That's not a great user experience, and triggers problems like those described above.
Reminds me of a human factors paper I saw in some ACM pub years ago (um, prob the 80's). Said more or less what you're saying, that humans prefer a constant response time even if a varying response time has a lower average. Example is, say, for a compiler, if it always takes 10sec to compile a file they users are happy, but
if it takes 1sec 99% of the time and a minute the other 1% then users are less happy. What was most interesting
about the paper was they suggested something like this:
goal = 10; // 10 sec t1 = time(); // time in seconds search(); // execute search, don't display results dt = time() - t1; // elapsed if ( elapsed < goal) sleep( goal- elapsed); // the shocker display_results()
i.e. artifically delay(!) if less than some threshold, thus making the user happier (!) to get a more constant response time.
People think that the rare slow cases are network problems or something, and hit "Reload".
I have no problem with including slow operators with Lucene, but they should be well documented as such, at least for developers. Perhaps we should make a pass through the existing Query classes, in particular those which expand into other queries, and add some performance notes, so that folks don't blindly start using things which may bite them. By default I think it would be safest if the QueryParser only permitted operators which are efficient. Folks can then, at their own risk, enable other operators.
In summary, removing operators can be user-centric, if it removes unpredictablity. And the reason for protecting engine performance is not miserly, it's to guarantee availablility. And finally, an issue dear to me, a predicatble search engine results in fewer spurious bug reports, saving developer time for real bugs.
Great, thx.. -Dave
Doug
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
