jenkins-bot has submitted this change and it was merged.
Change subject: Create wrapper query that makes delegate safer
......................................................................
Create wrapper query that makes delegate safer
This creates a wrapper query that statically inspects the query that it wraps
and removes slow constructs. In this commit its only going to handle large
phrase queries or queries that contain too many phrase queries.
Change-Id: I9560105c31d429df24d13232222740acba9b8e9a
---
M README.md
A docs/safer.md
M pom.xml
M src/main/java/org/wikimedia/search/extra/ExtraPlugin.java
M src/main/java/org/wikimedia/search/extra/regex/SourceRegexFilterParser.java
A src/main/java/org/wikimedia/search/extra/safer/ActionModuleParser.java
A
src/main/java/org/wikimedia/search/extra/safer/DefaultNoopSafeifierActions.java
A
src/main/java/org/wikimedia/search/extra/safer/DefaultQueryExplodingSafeifierActions.java
A src/main/java/org/wikimedia/search/extra/safer/Safeifier.java
A src/main/java/org/wikimedia/search/extra/safer/SaferQueryBuilder.java
A src/main/java/org/wikimedia/search/extra/safer/SaferQueryParser.java
A src/main/java/org/wikimedia/search/extra/safer/UnknownQueryException.java
A src/main/java/org/wikimedia/search/extra/safer/phrase/PhraseQueryAdapter.java
A
src/main/java/org/wikimedia/search/extra/safer/phrase/PhraseTooLargeAction.java
A
src/main/java/org/wikimedia/search/extra/safer/phrase/PhraseTooLargeActionModule.java
A
src/main/java/org/wikimedia/search/extra/safer/phrase/PhraseTooLargeActionModuleParser.java
A
src/main/java/org/wikimedia/search/extra/safer/phrase/TooManyPhraseTermsException.java
A src/test/java/org/wikimedia/search/extra/AbstractPluginIntegrationTest.java
M src/test/java/org/wikimedia/search/extra/regex/SourceRegexFilterTest.java
A src/test/java/org/wikimedia/search/extra/safer/SafeifierNoopQueriesTest.java
A
src/test/java/org/wikimedia/search/extra/safer/SafeifierQueryExplodingTest.java
A src/test/java/org/wikimedia/search/extra/safer/SafeifierTest.java
A src/test/java/org/wikimedia/search/extra/safer/SaferQueryStringTest.java
23 files changed, 1,498 insertions(+), 36 deletions(-)
Approvals:
BearND: Looks good to me, approved
jenkins-bot: Verified
diff --git a/README.md b/README.md
index 66f45bf..3d23d32 100644
--- a/README.md
+++ b/README.md
@@ -10,10 +10,14 @@
all documents.
Queries:
+* [safer](docs/safer.md) - Wraps other queries and analyzes them for
+potentially expensive constructs. Expensive constructs either cause errors to
+be sent back to the user or are degraded into cheaper, less precise constructs.
| Extra Queries and Filters Plugin | ElasticSearch |
|----------------------------------|-----------------|
-| 0.0.1 -> master | 1.3.2 -> master |
+| master | 1.3.4 -> 1.3.X |
+| 0.0.1 -> 0.0.2 | 1.3.2 -> 1.3.3 |
Install it like so:
```bash
diff --git a/docs/safer.md b/docs/safer.md
new file mode 100644
index 0000000..ba0785c
--- /dev/null
+++ b/docs/safer.md
@@ -0,0 +1,83 @@
+safer_query_string
+==================
+
+The ```safer``` query wraps other queries and analyzes them for potentially
+expensive constructs. Expensive constructs either cause errors to be sent back
+to the user or are degraded into cheaper, less precise constructs.
+
+Note that this adds some negligible overhead as queries are blown appart,
+inspected, and rewritten. Quick and dirty performance testing (on my laptop)
+puts this in the range of 3ish nanoseconds for a 10 clause boolean query
+containing 10 term phrase queries.
+
+Another important note:
+Filters are not currently processed by ```safer``` so a ```filtered``` query
+containing a ```query``` filter containing a phrase query would be silently
+ignored.
+
+Options
+-------
+
+```safer``` supports only the following options:
+* ```query``` The query to wrap. Required.
+* ```error_on_unknown``` Should an error be thrown back to the user if the we
+ encounter a query that we don't understand? Defaults to true.
+* ```phrase``` Configuration for handling phrase queries with too many clauses.
+ It can contain:
+ * ```max_terms_per_query``` The maximum number of terms a phrase query
+ must have before it trips the ```phrase_too_large_action```. Defaults
+ to ```max_terms_in_all_phrase_queries```.
+ * ```max_terms_in_all_queries``` The maximum number of terms allowed across
+ all phrase queries. Defaults to 64.
+ * ```phrase_too_large_action``` What to do if we hit a phrase with more
than
+ ```max_terms_per_query``` terms or the query contains more than
+ ```max_terms_in_all_queries``` phrase terms. Defaults to
+ ```error```. Values can be:
+ * ```error``` Send an error back to the user.
+ * ```convert_to_term_queries``` Convert the phrase query into a
+ ```bool``` query containing term queries. These are much much more
+ efficient to execute.
+ * ```convert_to_match_none_query``` Convert the phrase query into a
+ query that matches no documents.
+ * ```convert_to_match_all_query``` Convert the phrase query into a
+ query that matches all documents.
+
+Note on phrases:
+ If the ```phrase_too_large_action``` is tripped by
+```max_terms_in_all_phrase_queries``` then the action only applies to the
+phrase that pushed the count over the limit. The other phrases are not
+modified.
+
+Example
+-------
+```bash
+curl -XPOST localhost:9200/wiki/_search -d'{
+ "query": {
+ "safer": {
+ "query": {
+ "query_string": {
+ "query": "\"I am a long long long long long long long long
long phrase query\"",
+ "default_field": "text"
+ }
+ },
+ "phrase": {
+ "max_terms_per_query": 6
+ }
+ }
+ }
+}'
+```
+
+Default-ness
+------------
+Elasticsearch doesn't allow plugins to create wrap all queries so it wouldn't
+be possible to wrap ```safer``` around all queries by default. It also
+probably would be the wrong thing to do from a feature standpoint as well
+because:
+# It'd add extra overhead for simple queries that are known safe like term
+and match queries.
+# You'd just get the default configuration. While the default configuration is
+pretty good, its probably worth thinking about.
+# It'd be a breaking change to Elasticsearch. Stuff that worked before
+installing the plugin could fail afterwords. That's just too surprising for a
+plugin.
diff --git a/pom.xml b/pom.xml
index b562a82..29cf2e0 100644
--- a/pom.xml
+++ b/pom.xml
@@ -43,8 +43,8 @@
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
- <elasticsearch.version>1.3.2</elasticsearch.version>
- <lucene.version>4.9.0</lucene.version>
+ <elasticsearch.version>1.3.4</elasticsearch.version>
+ <lucene.version>4.9.1</lucene.version>
</properties>
<build>
diff --git a/src/main/java/org/wikimedia/search/extra/ExtraPlugin.java
b/src/main/java/org/wikimedia/search/extra/ExtraPlugin.java
index f0a5b92..8f628f6 100644
--- a/src/main/java/org/wikimedia/search/extra/ExtraPlugin.java
+++ b/src/main/java/org/wikimedia/search/extra/ExtraPlugin.java
@@ -1,8 +1,19 @@
package org.wikimedia.search.extra;
+import java.util.Collection;
+
+import org.elasticsearch.common.collect.ImmutableList;
+import org.elasticsearch.common.inject.AbstractModule;
+import org.elasticsearch.common.inject.Module;
+import org.elasticsearch.common.inject.multibindings.Multibinder;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.index.query.QueryParser;
import org.elasticsearch.indices.query.IndicesQueriesModule;
import org.elasticsearch.plugins.AbstractPlugin;
import org.wikimedia.search.extra.regex.SourceRegexFilterParser;
+import org.wikimedia.search.extra.safer.ActionModuleParser;
+import org.wikimedia.search.extra.safer.SaferQueryParser;
+import
org.wikimedia.search.extra.safer.phrase.PhraseTooLargeActionModuleParser;
/**
* Setup the Elasticsearch plugin.
@@ -21,7 +32,26 @@
/**
* Register our parsers.
*/
+ @SuppressWarnings("unchecked")
public void onModule(IndicesQueriesModule module) {
module.addFilter(new SourceRegexFilterParser());
+ module.addQuery((Class<QueryParser>) (Class<?>)SaferQueryParser.class);
+ }
+
+ @Override
+ public Collection<Class<? extends Module>> modules() {
+ return ImmutableList.<Class<? extends
Module>>of(SafeifierActionsModule.class);
+ }
+
+ public static class SafeifierActionsModule extends AbstractModule {
+ public SafeifierActionsModule(Settings settings) {
+ }
+
+ @Override
+ @SuppressWarnings("rawtypes")
+ protected void configure() {
+ Multibinder<ActionModuleParser> moduleParsers =
Multibinder.newSetBinder(binder(), ActionModuleParser.class);
+
moduleParsers.addBinding().to(PhraseTooLargeActionModuleParser.class).asEagerSingleton();
+ }
}
}
diff --git
a/src/main/java/org/wikimedia/search/extra/regex/SourceRegexFilterParser.java
b/src/main/java/org/wikimedia/search/extra/regex/SourceRegexFilterParser.java
index 76f77d5..d832112 100644
---
a/src/main/java/org/wikimedia/search/extra/regex/SourceRegexFilterParser.java
+++
b/src/main/java/org/wikimedia/search/extra/regex/SourceRegexFilterParser.java
@@ -51,44 +51,72 @@
if (token == XContentParser.Token.FIELD_NAME) {
currentFieldName = parser.currentName();
} else if (token.isValue()) {
- if ("regex".equals(currentFieldName)) {
+ switch (currentFieldName) {
+ case "regex":
regex = parser.text();
- } else if ("field".equals(currentFieldName)) {
+ break;
+ case "field":
fieldPath = parser.text();
- } else if ("load_from_source".equals(currentFieldName) ||
"loadFromSource".equals(currentFieldName)) {
+ break;
+ case "load_from_source":
+ case "loadFromSource":
if (parser.booleanValue()) {
loader = FieldValues.loadFromSource();
} else {
loader = FieldValues.loadFromStoredField();
}
- } else if ("ngram_field".equals(currentFieldName) ||
"ngramField".equals(currentFieldName)) {
+ break;
+ case "ngram_field":
+ case "ngramField":
ngramFieldPath = parser.text();
- } else if ("gram_size".equals(currentFieldName) ||
"gramSize".equals(currentFieldName)) {
+ break;
+ case "gram_size":
+ case "gramSize":
gramSize = parser.intValue();
- } else if ("max_expand".equals(currentFieldName) ||
"maxExpand".equals(currentFieldName)) {
+ break;
+ case "max_expand":
+ case "maxExpand":
maxExpand = parser.intValue();
- } else if ("max_states_traced".equals(currentFieldName) ||
"maxStatesTraced".equals(currentFieldName)) {
+ break;
+ case "max_states_traced":
+ case "maxStatesTraced":
maxStatesTraced = parser.intValue();
- } else if ("max_inspect".equals(currentFieldName) ||
"maxInspect".equals(currentFieldName)) {
+ break;
+ case "max_inspect":
+ case "maxInspect":
maxInspect = parser.intValue();
- } else if ("max_determinized_states".equals(currentFieldName)
|| "maxDeterminizedStates".equals(currentFieldName)) {
+ break;
+ case "max_determinized_states":
+ case "maxDeterminizedStates":
maxDeterminizedStates = parser.intValue();
- } else if ("max_ngrams_extracted".equals(currentFieldName) ||
"maxNgramsExtracted".equals(currentFieldName) ||
- "maxNGramsExtracted".equals(currentFieldName)) {
+ break;
+ case "max_ngrams_extracted":
+ case "maxNgramsExtracted":
+ case "maxNGramsExtracted":
maxNgramsExtracted = parser.intValue();
- } else if ("case_sensitive".equals(currentFieldName) ||
"caseSensitive".equals(currentFieldName)) {
+ break;
+ case "case_sensitive":
+ case "caseSensitive":
caseSensitive = parser.booleanValue();
- } else if ("locale".equals(currentFieldName)) {
+ break;
+ case "locale":
locale = LocaleUtils.parse(parser.text());
- } else if ("reject_unaccelerated".equals(currentFieldName) ||
"rejectUnaccelerated".equals(currentFieldName)) {
+ break;
+ case "reject_unaccelerated":
+ case "rejectUnaccelerated":
rejectUnaccelerated = parser.booleanValue();
- } else if ("_cache".equals(currentFieldName)) {
+ break;
+ case "_cache":
cache = parser.booleanValue();
- } else if ("_name".equals(currentFieldName)) {
+ break;
+ case "_name":
filterName = parser.text();
- } else if ("_cache_key".equals(currentFieldName) ||
"_cacheKey".equals(currentFieldName)) {
+ break;
+ case "_cache_key":
+ case "_cacheKey":
cacheKey = new CacheKeyFilter.Key(parser.text());
- } else {
+ break;
+ default:
throw new QueryParsingException(parseContext.index(),
"[source-regex] filter does not support [" + currentFieldName
+ "]");
}
diff --git
a/src/main/java/org/wikimedia/search/extra/safer/ActionModuleParser.java
b/src/main/java/org/wikimedia/search/extra/safer/ActionModuleParser.java
new file mode 100644
index 0000000..ed48e83
--- /dev/null
+++ b/src/main/java/org/wikimedia/search/extra/safer/ActionModuleParser.java
@@ -0,0 +1,18 @@
+package org.wikimedia.search.extra.safer;
+
+import java.io.IOException;
+
+import org.elasticsearch.index.query.QueryParseContext;
+import org.wikimedia.search.extra.safer.Safeifier.ActionModule;
+
+public interface ActionModuleParser<T extends ActionModule> {
+ /**
+ * @return the name of the safeifier action module
+ */
+ String moduleName();
+
+ /**
+ * Parse the safeifier action module.
+ */
+ T parse(QueryParseContext parseContext) throws IOException;
+}
diff --git
a/src/main/java/org/wikimedia/search/extra/safer/DefaultNoopSafeifierActions.java
b/src/main/java/org/wikimedia/search/extra/safer/DefaultNoopSafeifierActions.java
new file mode 100644
index 0000000..b7ac9f5
--- /dev/null
+++
b/src/main/java/org/wikimedia/search/extra/safer/DefaultNoopSafeifierActions.java
@@ -0,0 +1,70 @@
+package org.wikimedia.search.extra.safer;
+
+import org.apache.lucene.queries.CommonTermsQuery;
+import org.apache.lucene.queries.ExtendedCommonTermsQuery;
+import org.apache.lucene.search.FuzzyQuery;
+import org.apache.lucene.search.MultiPhraseQuery;
+import org.apache.lucene.search.PhraseQuery;
+import org.apache.lucene.search.PrefixQuery;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.RegexpQuery;
+import org.apache.lucene.search.TermQuery;
+import org.apache.lucene.search.WildcardQuery;
+import org.apache.lucene.search.payloads.PayloadNearQuery;
+import org.apache.lucene.search.payloads.PayloadTermQuery;
+import org.apache.lucene.search.spans.FieldMaskingSpanQuery;
+import org.apache.lucene.search.spans.SpanFirstQuery;
+import org.apache.lucene.search.spans.SpanMultiTermQueryWrapper;
+import org.apache.lucene.search.spans.SpanNearPayloadCheckQuery;
+import org.apache.lucene.search.spans.SpanNearQuery;
+import org.apache.lucene.search.spans.SpanNotQuery;
+import org.apache.lucene.search.spans.SpanOrQuery;
+import org.apache.lucene.search.spans.SpanPayloadCheckQuery;
+import org.apache.lucene.search.spans.SpanPositionRangeQuery;
+import org.apache.lucene.search.spans.SpanTermQuery;
+import org.elasticsearch.common.lucene.all.AllTermQuery;
+import org.elasticsearch.common.lucene.search.MultiPhrasePrefixQuery;
+import org.elasticsearch.common.lucene.search.XConstantScoreQuery;
+import org.wikimedia.search.extra.safer.Safeifier.Action;
+
+public class DefaultNoopSafeifierActions {
+ public static void register(Safeifier safeifier) {
+ safeifier.register(TermQuery.class, NOOP);
+ safeifier.register(FuzzyQuery.class, NOOP);
+ safeifier.register(RegexpQuery.class, NOOP);
+ safeifier.register(WildcardQuery.class, NOOP);
+ safeifier.register(PrefixQuery.class, NOOP);
+ safeifier.register(CommonTermsQuery.class, NOOP);
+ safeifier.register(ExtendedCommonTermsQuery.class, NOOP);
+ // XConstantScoreQuery only contains filters and we don't safeify them
right now
+ safeifier.register(XConstantScoreQuery.class, NOOP);
+
+ safeifier.register(PhraseQuery.class, NOOP);
+ safeifier.register(MultiPhraseQuery.class, NOOP);
+ safeifier.register(MultiPhrasePrefixQuery.class, NOOP);
+
+ // Span queries are quite expensive but they can't contain phrase
queries.
+ // TODO optionally count span queries as phrase queries? They have
similar performance characteristics.
+ safeifier.register(SpanTermQuery.class, NOOP);
+ safeifier.register(SpanMultiTermQueryWrapper.class, NOOP);
+ safeifier.register(SpanNearQuery.class, NOOP);
+ safeifier.register(PayloadNearQuery.class, NOOP);
+ safeifier.register(SpanNotQuery.class, NOOP);
+ safeifier.register(SpanOrQuery.class, NOOP);
+ safeifier.register(SpanNearPayloadCheckQuery.class, NOOP);
+ safeifier.register(SpanPayloadCheckQuery.class, NOOP);
+ safeifier.register(SpanPositionRangeQuery.class, NOOP);
+ safeifier.register(AllTermQuery.class, NOOP);
+ safeifier.register(PayloadTermQuery.class, NOOP);
+
+ safeifier.register(SpanFirstQuery.class, NOOP);
+ safeifier.register(FieldMaskingSpanQuery.class, NOOP);
+ }
+
+ private static Action<Query, Query> NOOP = new Action<Query, Query>() {
+ @Override
+ public Query apply(Safeifier safeifier, Query q) {
+ return q;
+ }
+ };
+}
diff --git
a/src/main/java/org/wikimedia/search/extra/safer/DefaultQueryExplodingSafeifierActions.java
b/src/main/java/org/wikimedia/search/extra/safer/DefaultQueryExplodingSafeifierActions.java
new file mode 100644
index 0000000..bd274d2
--- /dev/null
+++
b/src/main/java/org/wikimedia/search/extra/safer/DefaultQueryExplodingSafeifierActions.java
@@ -0,0 +1,80 @@
+package org.wikimedia.search.extra.safer;
+
+import org.apache.lucene.search.BooleanClause;
+import org.apache.lucene.search.BooleanQuery;
+import org.apache.lucene.search.ConstantScoreQuery;
+import org.apache.lucene.search.DisjunctionMaxQuery;
+import org.apache.lucene.search.FilteredQuery;
+import org.apache.lucene.search.Query;
+import org.elasticsearch.common.lucene.search.XFilteredQuery;
+import org.wikimedia.search.extra.safer.Safeifier.Action;
+
+public class DefaultQueryExplodingSafeifierActions {
+ public static void register(Safeifier safeifier) {
+ safeifier.register(BooleanQuery.class, BOOLEAN_QUERY_ACTION);
+ safeifier.register(DisjunctionMaxQuery.class,
DISJUNCTION_MAX_QUERY_ACTION);
+ safeifier.register(FilteredQuery.class, FILTERED_QUERY_ACTION);
+ safeifier.register(XFilteredQuery.class, XFILTERED_QUERY_ACTION);
+ safeifier.register(ConstantScoreQuery.class,
CONSTANT_SCORE_QUERY_ACTION);
+ }
+
+ private static final Action<BooleanQuery, BooleanQuery>
BOOLEAN_QUERY_ACTION = new Action<BooleanQuery, BooleanQuery>() {
+ @Override
+ public BooleanQuery apply(Safeifier safeifier, BooleanQuery bq) {
+ BooleanQuery replaced = new BooleanQuery();
+ replaced.setBoost(bq.getBoost());
+ for (BooleanClause clause : bq.getClauses()) {
+ replaced.add(safeifier.safeify(clause.getQuery()),
clause.getOccur());
+ }
+ return replaced;
+ }
+ };
+ private static final Action<DisjunctionMaxQuery, DisjunctionMaxQuery>
DISJUNCTION_MAX_QUERY_ACTION = new Action<DisjunctionMaxQuery,
DisjunctionMaxQuery>() {
+ @Override
+ public DisjunctionMaxQuery apply(Safeifier safeifier,
DisjunctionMaxQuery dmq) {
+ DisjunctionMaxQuery replaced = new
DisjunctionMaxQuery(dmq.getTieBreakerMultiplier());
+ replaced.setBoost(dmq.getBoost());
+ for (Query disjunct : dmq.getDisjuncts()) {
+ replaced.add(safeifier.safeify(disjunct));
+ }
+ return replaced;
+ }
+ };
+ private static final Action<FilteredQuery, XFilteredQuery>
FILTERED_QUERY_ACTION = new Action<FilteredQuery, XFilteredQuery>() {
+ @Override
+ public XFilteredQuery apply(Safeifier safeifier, FilteredQuery fq) {
+ // FilterQuery is unsafe and banned from Elasticsearch so we
+ // just convert....
+ XFilteredQuery newQuery = new
XFilteredQuery(safeifier.safeify(fq.getQuery()), fq.getFilter(),
fq.getFilterStrategy());
+ // TODO safeify filters
+ newQuery.setBoost(fq.getBoost());
+ return newQuery;
+ }
+ };
+ private static final Action<XFilteredQuery, XFilteredQuery>
XFILTERED_QUERY_ACTION = new Action<XFilteredQuery, XFilteredQuery>() {
+ @Override
+ public XFilteredQuery apply(Safeifier safeifier, XFilteredQuery fq) {
+ XFilteredQuery newQuery = new
XFilteredQuery(safeifier.safeify(fq.getQuery()), fq.getFilter(),
+ /*
+ * WOW! I can't actually read the old filter strategy so we just
+ * have guess.... fq.getFilterStrategy() doesn't exist
+ */
+ XFilteredQuery.CUSTOM_FILTER_STRATEGY);
+ // TODO safeify filters
+ newQuery.setBoost(fq.getBoost());
+ return newQuery;
+ }
+ };
+ private static final Action<ConstantScoreQuery, ConstantScoreQuery>
CONSTANT_SCORE_QUERY_ACTION = new Action<ConstantScoreQuery,
ConstantScoreQuery>() {
+ @Override
+ public ConstantScoreQuery apply(Safeifier safeifier,
ConstantScoreQuery csq) {
+ // TODO safeify filters
+ if (csq.getQuery() != null) {
+ ConstantScoreQuery newQuery = new
ConstantScoreQuery(safeifier.safeify(csq.getQuery()));
+ newQuery.setBoost(csq.getBoost());
+ return newQuery;
+ }
+ return csq;
+ }
+ };
+}
diff --git a/src/main/java/org/wikimedia/search/extra/safer/Safeifier.java
b/src/main/java/org/wikimedia/search/extra/safer/Safeifier.java
new file mode 100644
index 0000000..0261843
--- /dev/null
+++ b/src/main/java/org/wikimedia/search/extra/safer/Safeifier.java
@@ -0,0 +1,62 @@
+package org.wikimedia.search.extra.safer;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.lucene.search.Query;
+
+/**
+ * Single use class to make a query safe. External users should build it with a
+ * config and then call safeify once. Internally it recursively calls safeify
as
+ * it analyzes the query.
+ */
+public class Safeifier {
+ /**
+ * Module containing actions to take to safeify queries of certain types.
+ */
+ public interface ActionModule {
+ void register(Safeifier safeifier);
+ }
+
+ public interface Action<I extends Query, O extends Query> {
+ O apply(Safeifier safeifier, I q);
+ }
+
+ private final Map<Class<?>, Action<Query, Query>> registry = new
HashMap<>();
+ private final boolean errorOnUnknownQueryType;
+
+ public Safeifier(boolean errorOnUnknownQueryType, Iterable<ActionModule>
modules) {
+ this.errorOnUnknownQueryType = errorOnUnknownQueryType;
+ DefaultNoopSafeifierActions.register(this);
+ DefaultQueryExplodingSafeifierActions.register(this);
+ for (ActionModule module: modules) {
+ module.register(this);
+ }
+ }
+
+ public Safeifier(boolean errorOnUnknownQueryType, ActionModule... modules)
{
+ this(errorOnUnknownQueryType, Arrays.asList(modules));
+ }
+
+ public Safeifier(ActionModule... modules) {
+ this(true, modules);
+ }
+
+ @SuppressWarnings("unchecked")
+ public void register(Class<? extends Query> queryClass, Action<? extends
Query, ? extends Query> handler) {
+ registry.put(queryClass, (Action<Query, Query>) handler);
+ }
+
+ public Query safeify(Query q) {
+ Action<Query, Query> action = registry.get(q.getClass());
+ if (action == null) {
+ if (errorOnUnknownQueryType) {
+ throw new UnknownQueryException(q);
+ } else {
+ return q;
+ }
+ }
+ return action.apply(this, q);
+ }
+}
diff --git
a/src/main/java/org/wikimedia/search/extra/safer/SaferQueryBuilder.java
b/src/main/java/org/wikimedia/search/extra/safer/SaferQueryBuilder.java
new file mode 100644
index 0000000..493be9c
--- /dev/null
+++ b/src/main/java/org/wikimedia/search/extra/safer/SaferQueryBuilder.java
@@ -0,0 +1,52 @@
+package org.wikimedia.search.extra.safer;
+
+import java.io.IOException;
+import java.util.Locale;
+
+import org.elasticsearch.common.xcontent.XContentBuilder;
+import org.elasticsearch.index.query.BaseQueryBuilder;
+import org.elasticsearch.index.query.QueryBuilder;
+import org.wikimedia.search.extra.safer.phrase.PhraseTooLargeAction;
+
+public class SaferQueryBuilder extends BaseQueryBuilder {
+ private final QueryBuilder delegate;
+ private Integer maxTermsPerPhraseQuery;
+ private Integer maxTermsInAllPhraseQueries;
+ private PhraseTooLargeAction phraseTooLargeAction;
+ public SaferQueryBuilder(QueryBuilder delegate) {
+ this.delegate = delegate;
+ }
+
+ public SaferQueryBuilder maxTermsPerPhraseQuery(int
maxTermsPerPhraseQuery) {
+ this.maxTermsPerPhraseQuery = maxTermsPerPhraseQuery;
+ return this;
+ }
+
+ public SaferQueryBuilder maxTermsInAllPhraseQueries(int
maxTermsInAllPhraseQueries) {
+ this.maxTermsInAllPhraseQueries = maxTermsInAllPhraseQueries;
+ return this;
+ }
+
+ public SaferQueryBuilder phraseTooLargeAction(PhraseTooLargeAction
phraseTooLargeAction) {
+ this.phraseTooLargeAction = phraseTooLargeAction;
+ return this;
+ }
+
+ @Override
+ protected void doXContent(XContentBuilder builder, Params params) throws
IOException {
+ builder.startObject("safer");
+ builder.rawField("query",
delegate.buildAsBytes(builder.contentType()));
+ builder.startObject("phrase");
+ if (maxTermsPerPhraseQuery != null) {
+ builder.field("max_terms_per_query", maxTermsPerPhraseQuery);
+ }
+ if (maxTermsInAllPhraseQueries != null) {
+ builder.field("max_terms_in_all_queries",
maxTermsInAllPhraseQueries);
+ }
+ if (phraseTooLargeAction != null) {
+ builder.field("phrase_too_large_action",
phraseTooLargeAction.toString().toLowerCase(Locale.ROOT));
+ }
+ builder.endObject();
+ builder.endObject();
+ }
+}
diff --git
a/src/main/java/org/wikimedia/search/extra/safer/SaferQueryParser.java
b/src/main/java/org/wikimedia/search/extra/safer/SaferQueryParser.java
new file mode 100644
index 0000000..19b3cfe
--- /dev/null
+++ b/src/main/java/org/wikimedia/search/extra/safer/SaferQueryParser.java
@@ -0,0 +1,81 @@
+package org.wikimedia.search.extra.safer;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.lucene.search.Query;
+import org.elasticsearch.common.collect.ImmutableMap;
+import org.elasticsearch.common.inject.Inject;
+import org.elasticsearch.common.xcontent.XContentParser;
+import org.elasticsearch.common.xcontent.XContentParser.Token;
+import org.elasticsearch.index.query.QueryParseContext;
+import org.elasticsearch.index.query.QueryParser;
+import org.elasticsearch.index.query.QueryParsingException;
+import org.wikimedia.search.extra.safer.Safeifier.ActionModule;
+
+public class SaferQueryParser implements QueryParser {
+ private final Map<String, ActionModuleParser<?>> moduleParsers;
+
+ @Inject
+ public SaferQueryParser(@SuppressWarnings("rawtypes")
Set<ActionModuleParser> parsers) {
+ ImmutableMap.Builder<String, ActionModuleParser<?>> moduleParsers =
ImmutableMap.builder();
+ for(ActionModuleParser<?> parser: parsers) {
+ moduleParsers.put(parser.moduleName(), parser);
+ }
+ this.moduleParsers = moduleParsers.build();
+ }
+
+ @Override
+ public String[] names() {
+ return new String[] { "safer" };
+ }
+
+ @Override
+ public Query parse(QueryParseContext parseContext) throws IOException,
QueryParsingException {
+ boolean errorOnUnknownQueryType = true;
+ List<ActionModule> actionModules = new ArrayList<ActionModule>();
+ Query delegate = null;
+
+ XContentParser parser = parseContext.parser();
+ String currentFieldName = null;
+ XContentParser.Token token;
+ while ((token = parser.nextToken()) !=
XContentParser.Token.END_OBJECT) {
+ if (token == XContentParser.Token.FIELD_NAME) {
+ currentFieldName = parser.currentName();
+ } else if (token == Token.START_OBJECT) {
+ switch (currentFieldName) {
+ case "query":
+ if (delegate != null) {
+ throw new QueryParsingException(parseContext.index(),
"[safer] Can only wrap a single query ([" + currentFieldName
+ + "]) is the second query.");
+ }
+ delegate = parseContext.parseInnerQuery();
+ break;
+ default:
+ ActionModuleParser<?> moduleParser =
moduleParsers.get(currentFieldName);
+ if (moduleParser == null) {
+ throw new QueryParsingException(parseContext.index(),
"[safer] query does not support the object [" + currentFieldName + "]");
+ }
+ actionModules.add(moduleParser.parse(parseContext));
+ }
+ } else if (token.isValue()) {
+ switch (currentFieldName) {
+ case "error_on_unknown":
+ case "errorOnUnknown":
+ errorOnUnknownQueryType = parser.booleanValue();
+ break;
+ default:
+ throw new QueryParsingException(parseContext.index(),
"[safer] query does not support the field [" + currentFieldName + "]");
+ }
+ }
+ }
+
+ if (delegate == null) {
+ throw new QueryParsingException(parseContext.index(), "[safer]
requires a query");
+ }
+ return new Safeifier(errorOnUnknownQueryType,
actionModules).safeify(delegate);
+ }
+}
\ No newline at end of file
diff --git
a/src/main/java/org/wikimedia/search/extra/safer/UnknownQueryException.java
b/src/main/java/org/wikimedia/search/extra/safer/UnknownQueryException.java
new file mode 100644
index 0000000..fc8df80
--- /dev/null
+++ b/src/main/java/org/wikimedia/search/extra/safer/UnknownQueryException.java
@@ -0,0 +1,18 @@
+package org.wikimedia.search.extra.safer;
+
+import java.util.Locale;
+
+import org.apache.lucene.search.Query;
+
+/**
+ * Thrown when safer query encounters a query it doesn't know how to analyze.
+ * Its possible to configure safer query to ignore queries it doesn't
understand
+ * in which case this exception isn't thrown.
+ */
+public class UnknownQueryException extends RuntimeException {
+ private static final long serialVersionUID = 216120442826582111L;
+
+ public UnknownQueryException(Query q) {
+ super(String.format(Locale.ROOT, "Encountered an unknown query (%s=%s)
so can't ensue safety.", q.getClass(), q));
+ }
+}
diff --git
a/src/main/java/org/wikimedia/search/extra/safer/phrase/PhraseQueryAdapter.java
b/src/main/java/org/wikimedia/search/extra/safer/phrase/PhraseQueryAdapter.java
new file mode 100644
index 0000000..311f5ea
--- /dev/null
+++
b/src/main/java/org/wikimedia/search/extra/safer/phrase/PhraseQueryAdapter.java
@@ -0,0 +1,164 @@
+package org.wikimedia.search.extra.safer.phrase;
+
+import java.util.List;
+
+import org.apache.lucene.index.Term;
+import org.apache.lucene.search.BooleanClause;
+import org.apache.lucene.search.BooleanQuery;
+import org.apache.lucene.search.MultiPhraseQuery;
+import org.apache.lucene.search.PhraseQuery;
+import org.apache.lucene.search.PrefixQuery;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TermQuery;
+import org.elasticsearch.common.lucene.search.MultiPhrasePrefixQuery;
+
+/**
+ * Adapts PhraseQuery like queries to return what safer_query_string needs from
+ * them.
+ */
+public abstract class PhraseQueryAdapter {
+ /**
+ * The number of positions in the phrase query. Client code expects this to
+ * be fast so don't go allocating memory on every call. Capisce?
+ *
+ * @return the number of positions in the phrase query
+ */
+ public abstract int terms();
+ /**
+ * @return the original query unwrapped
+ */
+ public abstract Query unwrap();
+ /**
+ * @return the wrapper query as term queries
+ */
+ public abstract Query convertToTermQueries();
+
+ public static PhraseQueryAdapter adapt(PhraseQuery pq) {
+ return new PhraseQueryAdapterForPhraseQuery(pq);
+ }
+
+ public static PhraseQueryAdapter adapt(MultiPhraseQuery pq) {
+ return new PhraseQueryAdapterForMultiPhraseQuery(pq);
+ }
+
+ public static PhraseQueryAdapter adapt(MultiPhrasePrefixQuery pq) {
+ return new PhraseQueryAdapterForMultiPhrasePrefixQuery(pq);
+ }
+
+ private static final class PhraseQueryAdapterForPhraseQuery extends
PhraseQueryAdapter {
+ private final PhraseQuery pq;
+ private final int terms;
+
+ private PhraseQueryAdapterForPhraseQuery(PhraseQuery pq) {
+ this.pq = pq;
+ terms = pq.getTerms().length;
+ }
+
+ @Override
+ public int terms() {
+ return terms;
+ }
+
+ @Override
+ public Query unwrap() {
+ return pq;
+ }
+
+ @Override
+ public Query convertToTermQueries() {
+ BooleanQuery bq = new BooleanQuery();
+ bq.setBoost(pq.getBoost());
+ for (Term term : pq.getTerms()) {
+ bq.add(new TermQuery(term), BooleanClause.Occur.MUST);
+ }
+ return bq;
+ }
+ }
+
+ private static final class PhraseQueryAdapterForMultiPhraseQuery extends
PhraseQueryAdapter {
+ private final MultiPhraseQuery pq;
+ private final int totalTerms;
+
+ private PhraseQueryAdapterForMultiPhraseQuery(MultiPhraseQuery pq) {
+ this.pq = pq;
+ int total = 0;
+ for (Term[] terms : pq.getTermArrays()) {
+ total += terms.length;
+ }
+ totalTerms = total;
+ }
+
+ @Override
+ public int terms() {
+ return totalTerms;
+ }
+
+ @Override
+ public Query unwrap() {
+ return pq;
+ }
+
+ @Override
+ public Query convertToTermQueries() {
+ BooleanQuery bq = new BooleanQuery();
+ bq.setBoost(pq.getBoost());
+ for (Term[] terms : pq.getTermArrays()) {
+ BooleanQuery inner = new BooleanQuery();
+ for (Term term: terms) {
+ inner.add(new TermQuery(term), BooleanClause.Occur.SHOULD);
+ }
+ bq.add(inner, BooleanClause.Occur.MUST);
+ }
+ return bq;
+ }
+ }
+
+ private static final class PhraseQueryAdapterForMultiPhrasePrefixQuery
extends PhraseQueryAdapter {
+ private final MultiPhrasePrefixQuery pq;
+ private final int totalTerms;
+
+ private
PhraseQueryAdapterForMultiPhrasePrefixQuery(MultiPhrasePrefixQuery pq) {
+ this.pq = pq;
+ // Calculate total terms the same way that MultiPhraseQuery does.
+ // This is a lie because the final term could explode into a ton
+ // more terms but we're trying to do this pre-evaluation so we
can't
+ // figure out how many.
+ int total = 0;
+ for (Term[] terms : pq.getTermArrays()) {
+ total += terms.length;
+ }
+ totalTerms = total;
+ }
+
+ @Override
+ public int terms() {
+ return totalTerms;
+ }
+
+ @Override
+ public Query unwrap() {
+ return pq;
+ }
+
+ @Override
+ public Query convertToTermQueries() {
+ BooleanQuery bq = new BooleanQuery();
+ bq.setBoost(pq.getBoost());
+ List<Term[]> termArrays = pq.getTermArrays();
+ int prefixPosition = termArrays.size() - 1;
+ for (int current = 0; current < prefixPosition; current++) {
+ BooleanQuery inner = new BooleanQuery();
+ for (Term term: termArrays.get(current)) {
+ inner.add(new TermQuery(term), BooleanClause.Occur.SHOULD);
+ }
+ bq.add(inner, BooleanClause.Occur.MUST);
+ }
+ BooleanQuery inner = new BooleanQuery();
+ for (Term term: termArrays.get(prefixPosition)) {
+ inner.add(new PrefixQuery(term), BooleanClause.Occur.SHOULD);
+ }
+ bq.add(inner, BooleanClause.Occur.MUST);
+ return bq;
+ }
+ }
+}
diff --git
a/src/main/java/org/wikimedia/search/extra/safer/phrase/PhraseTooLargeAction.java
b/src/main/java/org/wikimedia/search/extra/safer/phrase/PhraseTooLargeAction.java
new file mode 100644
index 0000000..eab3ec1
--- /dev/null
+++
b/src/main/java/org/wikimedia/search/extra/safer/phrase/PhraseTooLargeAction.java
@@ -0,0 +1,69 @@
+package org.wikimedia.search.extra.safer.phrase;
+
+import java.util.Locale;
+
+import org.apache.lucene.search.MatchAllDocsQuery;
+import org.apache.lucene.search.Query;
+import org.elasticsearch.ElasticsearchIllegalArgumentException;
+import org.elasticsearch.common.logging.ESLogger;
+import org.elasticsearch.common.logging.Loggers;
+import org.elasticsearch.common.lucene.search.MatchNoDocsQuery;
+
+/**
+ * Detects large phrase queries or queries containing many phrase queries and
rejects or degrades them.
+ */
+public enum PhraseTooLargeAction {
+ ERROR {
+ @Override
+ public Query perform(PhraseQueryAdapter pq, int
maximumAllowedPositions) {
+ throw new TooManyPhraseTermsException(String.format(Locale.ROOT,
"Query has %s terms but only %s are allowed", pq.terms(),
+ maximumAllowedPositions));
+ }
+ },
+ CONVERT_TO_TERM_QUERIES {
+ @Override
+ public Query perform(PhraseQueryAdapter pq, int
maximumAllowedPositions) {
+ Query q = pq.convertToTermQueries();
+ logger.debug("Converted phrase query with {} terms to {}",
pq.terms(), q);
+ return q;
+ }
+ },
+ CONVERT_TO_MATCH_NONE_QUERY {
+ @Override
+ public Query perform(PhraseQueryAdapter pq, int
maximumAllowedPositions) {
+ logger.debug("Converted phrase query with {} terms to
MatchNoDocsQuery", pq.terms());
+ Query q = new MatchNoDocsQuery();
+ q.setBoost(pq.unwrap().getBoost());
+ return q;
+ }
+ },
+ CONVERT_TO_MATCH_ALL_QUERY {
+ @Override
+ public Query perform(PhraseQueryAdapter pq, int
maximumAllowedPositions) {
+ logger.debug("Converted phrase query with {} terms to
MatchNoDocsQuery", pq.terms());
+ Query q = new MatchAllDocsQuery();
+ q.setBoost(pq.unwrap().getBoost());
+ return q;
+ }
+ },
+ ;
+ protected static final ESLogger logger =
Loggers.getLogger(PhraseTooLargeAction.class);
+
+ public abstract Query perform(PhraseQueryAdapter pq, int
maximumAllowedPositions);
+
+ public static PhraseTooLargeAction parse(String text) {
+ if ("error".equals(text)) {
+ return ERROR;
+ }
+ if ("convert_to_term_queries".equals(text) ||
"convertToTermQueries".equals(text)) {
+ return CONVERT_TO_TERM_QUERIES;
+ }
+ if ("convert_to_match_none_query".equals(text) ||
"convertToMatchNoneQuery".equals(text)) {
+ return CONVERT_TO_MATCH_NONE_QUERY;
+ }
+ if ("convert_to_match_all_query".equals(text) ||
"convertToMatchAllQuery".equals(text)) {
+ return CONVERT_TO_MATCH_ALL_QUERY;
+ }
+ throw new
ElasticsearchIllegalArgumentException(String.format(Locale.ROOT, "Invalid
PhraseBreakUpMode: %s", text));
+ }
+}
diff --git
a/src/main/java/org/wikimedia/search/extra/safer/phrase/PhraseTooLargeActionModule.java
b/src/main/java/org/wikimedia/search/extra/safer/phrase/PhraseTooLargeActionModule.java
new file mode 100644
index 0000000..f1a904a
--- /dev/null
+++
b/src/main/java/org/wikimedia/search/extra/safer/phrase/PhraseTooLargeActionModule.java
@@ -0,0 +1,109 @@
+package org.wikimedia.search.extra.safer.phrase;
+
+import static
org.wikimedia.search.extra.safer.phrase.PhraseTooLargeAction.ERROR;
+
+import java.util.Locale;
+
+import org.apache.lucene.search.MultiPhraseQuery;
+import org.apache.lucene.search.PhraseQuery;
+import org.apache.lucene.search.Query;
+import org.elasticsearch.common.base.Function;
+import org.elasticsearch.common.lucene.search.MultiPhrasePrefixQuery;
+import org.wikimedia.search.extra.safer.Safeifier;
+import org.wikimedia.search.extra.safer.Safeifier.Action;
+import org.wikimedia.search.extra.safer.Safeifier.ActionModule;
+
+/**
+ * Safeifier action module for detecting and degrading queries with too many
+ * phrase terms. Not for reuse at all, holds yummy yummy state.
+ */
+public class PhraseTooLargeActionModule implements ActionModule {
+ private int maxTermsPerQuery = -1;
+ private int maxTermsInAllQueries = 64;
+ private PhraseTooLargeAction phraseTooLargeAction = ERROR;
+ private int phraseTermsSoFar = 0;
+
+ /**
+ * @param maxTermsPerQuery maximum terms per phrase query or -1 to
+ * default to the same value as maxTermsInAllPhraseQueries.
+ * Default is -1.
+ * @return this for chaining
+ */
+ public PhraseTooLargeActionModule maxTermsPerQuery(int maxTermsPerQuery) {
+ this.maxTermsPerQuery = maxTermsPerQuery;
+ return this;
+ }
+
+ /**
+ * @return maximum number of terms allowed per phrase query
+ */
+ public int maxTermsPerQuery() {
+ if (maxTermsPerQuery == -1) {
+ return maxTermsInAllQueries;
+ }
+ return maxTermsPerQuery;
+ }
+
+ /**
+ * @param maxTermsInAllQueries maximum terms in all phrase
+ * queries. Default is 64.
+ * @return this for chaining
+ */
+ public PhraseTooLargeActionModule maxTermsInAllQueries(int
maxTermsInAllQueries) {
+ this.maxTermsInAllQueries = maxTermsInAllQueries;
+ return this;
+ }
+
+ /**
+ * @param maxTermsInAllPhraseQueries action taken when a phrase is too
+ * large. Defaults to ERROR.
+ * @return this for chaining
+ */
+ public PhraseTooLargeActionModule
phraseTooLargeAction(PhraseTooLargeAction phraseTooLargeAction) {
+ this.phraseTooLargeAction = phraseTooLargeAction;
+ return this;
+ }
+
+ public void register(Safeifier registry) {
+ registry.register(PhraseQuery.class, new Action<PhraseQuery, Query>() {
+ @Override
+ public Query apply(Safeifier safeifier, PhraseQuery pq) {
+ return common.apply(PhraseQueryAdapter.adapt(pq));
+ }
+ });
+ registry.register(MultiPhraseQuery.class, new Action<MultiPhraseQuery,
Query>() {
+ @Override
+ public Query apply(Safeifier safeifier, MultiPhraseQuery pq) {
+ return common.apply(PhraseQueryAdapter.adapt(pq));
+ }
+ });
+ registry.register(MultiPhrasePrefixQuery.class, new
Action<MultiPhrasePrefixQuery, Query>() {
+ @Override
+ public Query apply(Safeifier safeifier, MultiPhrasePrefixQuery pq)
{
+ return common.apply(PhraseQueryAdapter.adapt(pq));
+ }
+ });
+ }
+
+ Function<PhraseQueryAdapter, Query> common = new
Function<PhraseQueryAdapter, Query>() {
+ @Override
+ public Query apply(PhraseQueryAdapter pq) {
+ if (pq.terms() > maxTermsPerQuery()) {
+ return phraseTooLargeAction.perform(pq, maxTermsPerQuery());
+ }
+ if (phraseTermsSoFar + pq.terms() > maxTermsInAllQueries) {
+ try {
+ // Delegate but transform any exceptions to reflect that
+ // this is for the global number of terms.
+ return phraseTooLargeAction.perform(pq, phraseTermsSoFar -
maxTermsInAllQueries);
+ } catch (TooManyPhraseTermsException e) {
+ throw new
TooManyPhraseTermsException(String.format(Locale.ROOT,
+ "Query has %s total terms but only %s total terms
are allowed", pq.terms(), maxTermsInAllQueries), e);
+ }
+ }
+ phraseTermsSoFar += pq.terms();
+ return pq.unwrap();
+ }
+
+ };
+}
diff --git
a/src/main/java/org/wikimedia/search/extra/safer/phrase/PhraseTooLargeActionModuleParser.java
b/src/main/java/org/wikimedia/search/extra/safer/phrase/PhraseTooLargeActionModuleParser.java
new file mode 100644
index 0000000..55aa28d
--- /dev/null
+++
b/src/main/java/org/wikimedia/search/extra/safer/phrase/PhraseTooLargeActionModuleParser.java
@@ -0,0 +1,49 @@
+package org.wikimedia.search.extra.safer.phrase;
+
+import java.io.IOException;
+
+import org.elasticsearch.common.xcontent.XContentParser;
+import org.elasticsearch.index.query.QueryParseContext;
+import org.elasticsearch.index.query.QueryParsingException;
+import org.wikimedia.search.extra.safer.ActionModuleParser;
+
+public class PhraseTooLargeActionModuleParser implements
ActionModuleParser<PhraseTooLargeActionModule> {
+ @Override
+ public String moduleName() {
+ return "phrase";
+ }
+
+ @Override
+ public PhraseTooLargeActionModule parse(QueryParseContext parseContext)
throws IOException {
+ PhraseTooLargeActionModule module = new PhraseTooLargeActionModule();
+ XContentParser parser = parseContext.parser();
+ String currentFieldName = null;
+ XContentParser.Token token;
+ while ((token = parser.nextToken()) !=
XContentParser.Token.END_OBJECT) {
+ if (token == XContentParser.Token.FIELD_NAME) {
+ currentFieldName = parser.currentName();
+ } else if (token.isValue()) {
+ switch (currentFieldName) {
+ case "max_terms_per_query":
+ case "maxTermsPerQuery":
+ module.maxTermsPerQuery(parser.intValue());
+ break;
+ case "max_terms_in_all_queries":
+ case "maxTermsInAllQueries":
+ module.maxTermsInAllQueries(parser.intValue());
+ break;
+ case "phrase_too_large_action":
+ case "phraseTooLargeAction":
+
module.phraseTooLargeAction(PhraseTooLargeAction.parse(parser.text()));
+ break;
+ default:
+ throw new QueryParsingException(parseContext.index(),
"[safer][phrase] query does not support the field ["
+ + currentFieldName + "]");
+ }
+ } else {
+ throw new QueryParsingException(parseContext.index(),
"[safer][phrase] only supports values, not objects.");
+ }
+ }
+ return module;
+ }
+}
diff --git
a/src/main/java/org/wikimedia/search/extra/safer/phrase/TooManyPhraseTermsException.java
b/src/main/java/org/wikimedia/search/extra/safer/phrase/TooManyPhraseTermsException.java
new file mode 100644
index 0000000..cea369b
--- /dev/null
+++
b/src/main/java/org/wikimedia/search/extra/safer/phrase/TooManyPhraseTermsException.java
@@ -0,0 +1,16 @@
+package org.wikimedia.search.extra.safer.phrase;
+
+/**
+ * Thrown when a query contains too many terms.
+ */
+public class TooManyPhraseTermsException extends RuntimeException {
+ private static final long serialVersionUID = -7347696985098154791L;
+
+ public TooManyPhraseTermsException(String message, Throwable cause) {
+ super(message, cause);
+ }
+
+ public TooManyPhraseTermsException(String message) {
+ super(message);
+ }
+}
diff --git
a/src/test/java/org/wikimedia/search/extra/AbstractPluginIntegrationTest.java
b/src/test/java/org/wikimedia/search/extra/AbstractPluginIntegrationTest.java
new file mode 100644
index 0000000..e67418e
--- /dev/null
+++
b/src/test/java/org/wikimedia/search/extra/AbstractPluginIntegrationTest.java
@@ -0,0 +1,18 @@
+package org.wikimedia.search.extra;
+
+import org.elasticsearch.common.settings.ImmutableSettings;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.plugins.PluginsService;
+import org.elasticsearch.test.ElasticsearchIntegrationTest;
+
[email protected](scope =
ElasticsearchIntegrationTest.Scope.SUITE, transportClientRatio = 0.0)
+public class AbstractPluginIntegrationTest extends
ElasticsearchIntegrationTest {
+ /**
+ * Enable plugin loading.
+ */
+ @Override
+ protected Settings nodeSettings(int nodeOrdinal) {
+ return ImmutableSettings.builder().put(super.nodeSettings(nodeOrdinal))
+ .put("plugins." + PluginsService.LOAD_PLUGIN_FROM_CLASSPATH,
true).build();
+ }
+}
diff --git
a/src/test/java/org/wikimedia/search/extra/regex/SourceRegexFilterTest.java
b/src/test/java/org/wikimedia/search/extra/regex/SourceRegexFilterTest.java
index ad9c48a..01b1d33 100644
--- a/src/test/java/org/wikimedia/search/extra/regex/SourceRegexFilterTest.java
+++ b/src/test/java/org/wikimedia/search/extra/regex/SourceRegexFilterTest.java
@@ -17,15 +17,12 @@
import org.elasticsearch.action.index.IndexRequestBuilder;
import org.elasticsearch.action.search.SearchRequestBuilder;
import org.elasticsearch.action.search.SearchResponse;
-import org.elasticsearch.common.settings.ImmutableSettings;
-import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.xcontent.XContentBuilder;
-import org.elasticsearch.plugins.PluginsService;
import org.elasticsearch.rest.RestStatus;
-import org.elasticsearch.test.ElasticsearchIntegrationTest;
import org.junit.Test;
[email protected](scope =
ElasticsearchIntegrationTest.Scope.SUITE, transportClientRatio = 0.0)
-public class SourceRegexFilterTest extends ElasticsearchIntegrationTest {
+import org.wikimedia.search.extra.AbstractPluginIntegrationTest;
+
+public class SourceRegexFilterTest extends AbstractPluginIntegrationTest {
@Test
public void basicUnacceleratedRegex() throws InterruptedException,
ExecutionException, IOException {
setup();
@@ -335,14 +332,5 @@
settings.field("min_gram", size);
settings.field("max_gram", size);
settings.endObject();
- }
-
- /**
- * Enable plugin loading.
- */
- @Override
- protected Settings nodeSettings(int nodeOrdinal) {
- return ImmutableSettings.builder().put(super.nodeSettings(nodeOrdinal))
- .put("plugins." + PluginsService.LOAD_PLUGIN_FROM_CLASSPATH,
true).build();
}
}
diff --git
a/src/test/java/org/wikimedia/search/extra/safer/SafeifierNoopQueriesTest.java
b/src/test/java/org/wikimedia/search/extra/safer/SafeifierNoopQueriesTest.java
new file mode 100644
index 0000000..80f20ee
--- /dev/null
+++
b/src/test/java/org/wikimedia/search/extra/safer/SafeifierNoopQueriesTest.java
@@ -0,0 +1,76 @@
+package org.wikimedia.search.extra.safer;
+import static org.wikimedia.search.extra.safer.SafeifierTest.mpq;
+import static
org.wikimedia.search.extra.safer.SafeifierTest.multiPhrasePrefixQuery;
+import static org.wikimedia.search.extra.safer.SafeifierTest.pq;
+
+import java.util.Collections;
+
+import org.apache.lucene.index.Term;
+import org.apache.lucene.queries.CommonTermsQuery;
+import org.apache.lucene.queries.ExtendedCommonTermsQuery;
+import org.apache.lucene.queries.TermFilter;
+import org.apache.lucene.search.BooleanClause.Occur;
+import org.apache.lucene.search.FuzzyQuery;
+import org.apache.lucene.search.PrefixQuery;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.RegexpQuery;
+import org.apache.lucene.search.TermQuery;
+import org.apache.lucene.search.WildcardQuery;
+import org.apache.lucene.search.payloads.MaxPayloadFunction;
+import org.apache.lucene.search.payloads.PayloadNearQuery;
+import org.apache.lucene.search.payloads.PayloadTermQuery;
+import org.apache.lucene.search.spans.FieldMaskingSpanQuery;
+import org.apache.lucene.search.spans.SpanFirstQuery;
+import org.apache.lucene.search.spans.SpanMultiTermQueryWrapper;
+import org.apache.lucene.search.spans.SpanNearPayloadCheckQuery;
+import org.apache.lucene.search.spans.SpanNearQuery;
+import org.apache.lucene.search.spans.SpanNotQuery;
+import org.apache.lucene.search.spans.SpanOrQuery;
+import org.apache.lucene.search.spans.SpanPayloadCheckQuery;
+import org.apache.lucene.search.spans.SpanPositionRangeQuery;
+import org.apache.lucene.search.spans.SpanQuery;
+import org.apache.lucene.search.spans.SpanTermQuery;
+import org.elasticsearch.common.lucene.all.AllTermQuery;
+import org.elasticsearch.common.lucene.search.XConstantScoreQuery;
+import org.elasticsearch.test.ElasticsearchTestCase;
+import org.junit.Test;
+
+public class SafeifierNoopQueriesTest extends ElasticsearchTestCase {
+ @Test
+ public void noopQueries() {
+ Term t = new Term("test", "foo");
+ Query[] queries = new Query[] {
+ new TermQuery(t),
+ new FuzzyQuery(t, 1, 1, 1, false),
+ new RegexpQuery(t),
+ new WildcardQuery(t),
+ new PrefixQuery(t),
+ new CommonTermsQuery(Occur.SHOULD, Occur.MUST, .5f),
+ new ExtendedCommonTermsQuery(Occur.SHOULD, Occur.MUST, .5f,
false, null),
+ new XConstantScoreQuery(new TermFilter(t)),
+
+ pq("1", "2", "3"),
+ mpq(new String[] {"1", "2"}, new String[] {"3"}, new String[]
{"a", "Adsfa"}),
+ multiPhrasePrefixQuery(new String[] {"1", "2"}, new String[]
{"3"}, new String[] {"a", "Adsfa"}),
+
+ new FieldMaskingSpanQuery(new SpanTermQuery(t), "test"),
+ new SpanMultiTermQueryWrapper<>(new PrefixQuery(t)),
+ new SpanNearQuery(new SpanQuery[] {new SpanTermQuery(t)}, 1,
true),
+ new PayloadNearQuery(new SpanQuery[] {new SpanTermQuery(t)},
1, true),
+ new SpanNotQuery(new SpanTermQuery(t), new SpanTermQuery(t)),
+ new SpanOrQuery(new SpanTermQuery(t), new SpanTermQuery(t)),
+ new SpanNearPayloadCheckQuery(new SpanNearQuery(new
SpanQuery[] {new SpanTermQuery(t)}, 1, true), Collections.<byte[]>emptyList()),
+ new SpanPayloadCheckQuery(new SpanTermQuery(t),
Collections.<byte[]>emptyList()),
+ new SpanPositionRangeQuery(new SpanTermQuery(t), 1, 20),
+ new SpanFirstQuery(new SpanTermQuery(t), 10),
+ new SpanTermQuery(t),
+ new AllTermQuery(t),
+ new PayloadTermQuery(t, new MaxPayloadFunction()),
+ };
+ for (Query query: queries) {
+ query.setBoost(getRandom().nextFloat());
+ assertEquals(query, new Safeifier(true).safeify(query));
+ }
+ }
+
+}
diff --git
a/src/test/java/org/wikimedia/search/extra/safer/SafeifierQueryExplodingTest.java
b/src/test/java/org/wikimedia/search/extra/safer/SafeifierQueryExplodingTest.java
new file mode 100644
index 0000000..3571334
--- /dev/null
+++
b/src/test/java/org/wikimedia/search/extra/safer/SafeifierQueryExplodingTest.java
@@ -0,0 +1,76 @@
+package org.wikimedia.search.extra.safer;
+import static org.wikimedia.search.extra.safer.SafeifierTest.flatten;
+import static org.wikimedia.search.extra.safer.SafeifierTest.pq;
+import static org.wikimedia.search.extra.safer.SafeifierTest.tq;
+
+import org.apache.lucene.search.BooleanClause.Occur;
+import org.apache.lucene.search.BooleanQuery;
+import org.apache.lucene.search.ConstantScoreQuery;
+import org.apache.lucene.search.DisjunctionMaxQuery;
+import org.apache.lucene.search.FilteredQuery;
+import org.elasticsearch.common.lucene.search.MatchAllDocsFilter;
+import org.elasticsearch.common.lucene.search.XFilteredQuery;
+import org.elasticsearch.test.ElasticsearchTestCase;
+import org.junit.Test;
+
+public class SafeifierQueryExplodingTest extends ElasticsearchTestCase {
+ /**
+ * Validate that phrase queries are flattened inside of boolean queries.
+ */
+ @Test
+ public void booleanQuery() {
+ BooleanQuery in = new BooleanQuery();
+ in.setBoost(getRandom().nextFloat());
+ in.add(pq("1", "2", "3"), Occur.MUST);
+ in.add(pq("1", "2", "4"), Occur.SHOULD);
+ in.add(pq("1", "2"), Occur.MUST_NOT);
+ BooleanQuery expected = new BooleanQuery();
+ expected.setBoost(in.getBoost());
+ expected.add(tq("1", "2", "3"), Occur.MUST);
+ expected.add(tq("1", "2", "4"), Occur.SHOULD);
+ expected.add(tq("1", "2"), Occur.MUST_NOT);
+ assertEquals(expected, flatten(in));
+ }
+
+ /**
+ * Validate that phrase queries are flattened inside of disjunction max
queries.
+ */
+ @Test
+ public void disjunctionMaxQuery() {
+ DisjunctionMaxQuery in = new
DisjunctionMaxQuery(getRandom().nextFloat());
+ in.setBoost(getRandom().nextFloat());
+ in.add(pq("1", "2", "3"));
+ in.add(pq("1", "2", "4"));
+ in.add(pq("1", "2"));
+ DisjunctionMaxQuery expected = new
DisjunctionMaxQuery(in.getTieBreakerMultiplier());
+ expected.setBoost(in.getBoost());
+ expected.add(tq("1", "2", "3"));
+ expected.add(tq("1", "2", "4"));
+ expected.add(tq("1", "2"));
+ assertEquals(expected, flatten(in));
+ }
+
+ /**
+ * Validate that phrase queries are flattened inside of filtered queries.
+ */
+ @Test
+ public void filteredQuery() {
+ FilteredQuery in = new FilteredQuery(pq("1", "2"), new
MatchAllDocsFilter());
+ in.setBoost(getRandom().nextFloat());
+ XFilteredQuery expected = new XFilteredQuery(tq("1", "2"), new
MatchAllDocsFilter());
+ expected.setBoost(in.getBoost());
+ assertEquals(expected, flatten(in));
+ }
+
+ /**
+ * Validate that phrase queries are flattened inside of constant score
queries.
+ */
+ @Test
+ public void xFilteredQuery() {
+ ConstantScoreQuery in = new ConstantScoreQuery(pq("1", "2"));
+ in.setBoost(getRandom().nextFloat());
+ ConstantScoreQuery expected = new ConstantScoreQuery(tq("1", "2"));
+ expected.setBoost(in.getBoost());
+ assertEquals(expected, flatten(in));
+ }
+}
diff --git a/src/test/java/org/wikimedia/search/extra/safer/SafeifierTest.java
b/src/test/java/org/wikimedia/search/extra/safer/SafeifierTest.java
new file mode 100644
index 0000000..fa4579b
--- /dev/null
+++ b/src/test/java/org/wikimedia/search/extra/safer/SafeifierTest.java
@@ -0,0 +1,137 @@
+package org.wikimedia.search.extra.safer;
+
+import org.apache.lucene.index.Term;
+import org.apache.lucene.search.BooleanClause.Occur;
+import org.apache.lucene.search.BooleanQuery;
+import org.apache.lucene.search.MultiPhraseQuery;
+import org.apache.lucene.search.PhraseQuery;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.TermQuery;
+import org.apache.lucene.util.TestUtil;
+import org.elasticsearch.common.lucene.search.MultiPhrasePrefixQuery;
+import org.elasticsearch.test.ElasticsearchTestCase;
+import org.junit.Test;
+import org.wikimedia.search.extra.safer.phrase.PhraseTooLargeAction;
+import org.wikimedia.search.extra.safer.phrase.PhraseTooLargeActionModule;
+
+public class SafeifierTest extends ElasticsearchTestCase {
+ @Test(expected = UnknownQueryException.class)
+ public void unknownQueryError() {
+ new Safeifier().safeify(new Query() {
+ @Override
+ public String toString(String field) {
+ return "I'm just here to cause trouble.";
+ }
+ });
+ }
+
+ @Test(expected = UnknownQueryException.class)
+ public void unknownSubclassQueryError() {
+ new Safeifier().safeify(new TermQuery(new Term("test", "foo")) {
+ // Intentionally creating my own subclass to cause this not to be
+ // recognized.
+ });
+ }
+
+ @Test
+ public void unknownQueryNoError() {
+ Query q = new Query() {
+ @Override
+ public String toString(String field) {
+ return "I'm just here to cause trouble.";
+ }
+ };
+ assertEquals(q, new Safeifier(false).safeify(q));
+ }
+
+ /**
+ * Quick and dirty performance test just used to get a sense of how
+ * expensive this whole operation is. Disabled because this is not
something
+ * you can assert against unfortunately.
+ */
+ // @Test
+ public void quickAndDirtyPerfTest() {
+ BooleanQuery q = new BooleanQuery();
+ for (int i = 0; i < 10; i++) {
+ q.add(pq(10), Occur.MUST);
+ }
+ long start = System.currentTimeMillis();
+ for (int i = 0; i < 1000000; i++) {
+ PhraseTooLargeActionModule pa = new PhraseTooLargeActionModule();
+
pa.phraseTooLargeAction(PhraseTooLargeAction.CONVERT_TO_TERM_QUERIES);
+ pa.maxTermsInAllQueries(getRandom().nextInt(128));
+ new Safeifier(pa).safeify(q);
+ }
+ logger.info("Total: {}", System.currentTimeMillis() - start);
+ }
+
+ /**
+ * Generate a phrase query.
+ */
+ public static PhraseQuery pq(String... terms) {
+ PhraseQuery p = new PhraseQuery();
+ for (String term : terms) {
+ p.add(new Term("test", term));
+ }
+ return p;
+ }
+
+ /**
+ * Generate a bool of term queries as though you flattened a phrase query
+ * containing those terms.
+ */
+ public static BooleanQuery tq(String... terms) {
+ BooleanQuery bq = new BooleanQuery();
+ for (String term : terms) {
+ bq.add(new TermQuery(new Term("test", term)), Occur.MUST);
+ }
+ return bq;
+ }
+
+ /**
+ * Generate a multi phrase query.
+ */
+ public static MultiPhraseQuery mpq(String[]... positions) {
+ MultiPhraseQuery p = new MultiPhraseQuery();
+ for (String[] position : positions) {
+ Term[] terms = new Term[position.length];
+ for (int t = 0; t < position.length; t++) {
+ terms[t] = new Term("test", position[t]);
+ }
+ p.add(terms);
+ }
+ return p;
+ }
+
+ /**
+ * Generate a multi phrase query.
+ */
+ public static MultiPhrasePrefixQuery multiPhrasePrefixQuery(String[]...
positions) {
+ MultiPhrasePrefixQuery p = new MultiPhrasePrefixQuery();
+ for (String[] position : positions) {
+ Term[] terms = new Term[position.length];
+ for (int t = 0; t < position.length; t++) {
+ terms[t] = new Term("test", position[t]);
+ }
+ p.add(terms);
+ }
+ return p;
+ }
+
+ /**
+ * Throw a query through the safeifier to flatten all phrase queries into
+ * {@linkplain BooleanQuery}s of term queries.
+ */
+ public static Query flatten(Query q) {
+ return new Safeifier(true, new
PhraseTooLargeActionModule().maxTermsInAllQueries(0).phraseTooLargeAction(
+ PhraseTooLargeAction.CONVERT_TO_TERM_QUERIES)).safeify(q);
+ }
+
+ private PhraseQuery pq(int count) {
+ String[] terms = new String[count];
+ for (int i = 0; i < count; i++) {
+ terms[i] = TestUtil.randomSimpleString(getRandom());
+ }
+ return pq(terms);
+ }
+}
diff --git
a/src/test/java/org/wikimedia/search/extra/safer/SaferQueryStringTest.java
b/src/test/java/org/wikimedia/search/extra/safer/SaferQueryStringTest.java
new file mode 100644
index 0000000..34d7130
--- /dev/null
+++ b/src/test/java/org/wikimedia/search/extra/safer/SaferQueryStringTest.java
@@ -0,0 +1,234 @@
+package org.wikimedia.search.extra.safer;
+
+import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder;
+import static org.elasticsearch.index.query.QueryBuilders.queryString;
+import static
org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked;
+import static
org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertFailures;
+import static
org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertHitCount;
+import static
org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertSearchHits;
+import static org.hamcrest.Matchers.both;
+import static org.hamcrest.Matchers.containsString;
+
+import java.io.IOException;
+import java.util.concurrent.ExecutionException;
+
+import org.apache.lucene.queryparser.classic.ParseException;
+import org.elasticsearch.action.search.SearchRequestBuilder;
+import org.elasticsearch.common.xcontent.XContentBuilder;
+import org.elasticsearch.index.query.QueryStringQueryBuilder;
+import org.elasticsearch.index.query.QueryStringQueryBuilder.Operator;
+import org.elasticsearch.rest.RestStatus;
+import org.junit.Before;
+import org.junit.Test;
+import org.wikimedia.search.extra.AbstractPluginIntegrationTest;
+import org.wikimedia.search.extra.safer.phrase.PhraseTooLargeAction;
+
+/**
+ * Tests the safer query wrapping query_string queries similar to how <a
>CirrusSearch</a> works.
+ */
+public class SaferQueryStringTest extends AbstractPluginIntegrationTest {
+ @Before
+ public void setup() throws InterruptedException, ExecutionException,
IOException {
+ XContentBuilder mapping = jsonBuilder().startObject();
+ mapping.startObject("test").startObject("properties");
+ mapping.startObject("findme");
+ {
+ mapping.field("type", "string");
+ mapping.field("analyzer", "custom");
+ }
+ mapping.endObject();
+
+ XContentBuilder settings =
jsonBuilder().startObject().startObject("index");
+ settings.startObject("analysis");
+ settings.startObject("analyzer");
+ settings.startObject("custom");
+ {
+ settings.field("type", "custom");
+ settings.field("tokenizer", "standard");
+ settings.field("filter", "standard", "capture_A", "lowercase");
+ settings.field("char_filter", new String[] {"dots"});
+ }
+ settings.endObject();
+ settings.endObject();
+ settings.startObject("filter");
+ settings.startObject("capture_A");
+ {
+ settings.field("type", "pattern_capture");
+ settings.startArray("patterns").value("(A)").endArray();
+ }
+ settings.endObject();
+ settings.endObject();
+ settings.startObject("char_filter");
+ settings.startObject("dots");
+ {
+ settings.field("type", "mapping");
+ settings.field("mappings", new String[] {".=>\\u0020" });
+ }
+ settings.endObject();
+ settings.endObject();
+ settings.endObject();
+ settings.endObject();
+
assertAcked(prepareCreate("test").setSettings(settings).addMapping("test",
mapping));
+ ensureYellow();
+ indexRandom(true,
+ client().prepareIndex("test", "test", "1").setSource("findme",
"0 0 0 0 0 0", "otherfindme", "0"),
+ client().prepareIndex("test", "test", "2").setSource("findme",
"0 0 0 0"),
+ client().prepareIndex("test", "test",
"delimited1").setSource("findme", "CaptureAAAA Test"),
+ client().prepareIndex("test", "test",
"delimited2").setSource("findme", "CaptureAAA Test"));
+ }
+
+ @Test
+ public void error() throws ParseException, InterruptedException,
ExecutionException {
+ // Everything works when you expect it to
+ QueryStringQueryBuilder qs = queryString("\"0 0 0 0 0 0\"");
+ SaferQueryBuilder b = new SaferQueryBuilder(qs);
+ if (getRandom().nextBoolean()) {
+ b.phraseTooLargeAction(PhraseTooLargeAction.ERROR);
+ }
+ SearchRequestBuilder search =
client().prepareSearch("test").setQuery(b);
+ assertSearchHits(search.get(), "1");
+ b.maxTermsPerPhraseQuery(6);
+ assertSearchHits(search.get(), "1");
+
+ // Even with phrases made by lack of spaces
+ qs = queryString("findme:0.0.0.0.0.0");
+ b = new SaferQueryBuilder(qs);
+ if (getRandom().nextBoolean()) {
+ b.phraseTooLargeAction(PhraseTooLargeAction.ERROR);
+ }
+ qs.autoGeneratePhraseQueries(true);
+ search = client().prepareSearch("test").setQuery(b);
+ assertSearchHits(search.get(), "1");
+ b.maxTermsPerPhraseQuery(6);
+ assertSearchHits(search.get(), "1");
+
+ // And also with MultiPhraseQueries
+ qs = queryString("findme:\"CaptureAAAA Test\"");
+ b = new SaferQueryBuilder(qs);
+ if (getRandom().nextBoolean()) {
+ b.phraseTooLargeAction(PhraseTooLargeAction.ERROR);
+ }
+ search = client().prepareSearch("test").setQuery(b);
+ assertSearchHits(search.get(), "delimited1", "delimited2");
+ b.maxTermsPerPhraseQuery(6);
+ assertSearchHits(search.get(), "delimited1", "delimited2");
+
+ // And everything fails when you expect it to
+ // Single field big enough to trip the max terms per phrase query
+ assertSearchByFieldsFails(false, "\"0 0 0 0 0 0\"", "_all");
+ // One field big enough to trip the max terms per phrase query and one
that doesn't exist
+ assertSearchByFieldsFails(false, "\"0 0 0 0 0 0\"", "findme",
"doesnotexist");
+ // Two fields that add up to tripping the max terms in all phrase
queries issue
+ assertSearchByFieldsFails(true, "\"0 0 0 0 0\"", "findme",
"otherfindme");
+ // Multiple small phrase queries on a single field that add up to trip
the large phrase query max
+ assertSearchByFieldsFails(true, "\"0 0 0\" \"0 0 0\" \"0 0 0\" \"0 0
0\"", "_all");
+ assertSearchByFieldsFails(false, "0.0.0.0.0.0", "findme",
"otherfindme");
+ assertSearchByString("_all:\"0 0 0 0 0 0\"");
+ assertSearchByString("findme:\"0 0 0 0 0 0\"");
+ assertSearchByString("findme:\"0 0 0 0 0\" _all:\"0 0 0 0 0 0\"");
+ assertSearchByString("findme:\"0 0 0 0 0\" otherfindme:\"0 0 0 0 0
0\"");
+ assertSearchByString("findme:\"0 0 0 0 0\" | otherfindme:\"0 0 0 0 0
0\"");
+ assertSearchByString("(findme:\"0 0 0 0 0\" | otherfindme:\"0 0 0 0 0
0\")");
+ assertSearchByString("(findme:\"0 0 0 0 0\" | otherfindme:\"0 0 0 0 0
0\") 0");
+ assertSearchByString("findme:\"CaptureAAAA Test\"");
+ }
+
+ private void assertSearchByFieldsFails(boolean total, String query,
String... fields) {
+ QueryStringQueryBuilder qs = queryString(query);
+ SaferQueryBuilder b = new
SaferQueryBuilder(qs).maxTermsPerPhraseQuery(5).maxTermsInAllPhraseQueries(8);
+ if (getRandom().nextBoolean()) {
+ b.phraseTooLargeAction(PhraseTooLargeAction.ERROR);
+ }
+ if (getRandom().nextBoolean()) {
+ qs.useDisMax(false);
+ }
+ qs.autoGeneratePhraseQueries(true);
+ for (String field : fields) {
+ qs.field(field);
+ }
+ SearchRequestBuilder search =
client().prepareSearch("test").setQuery(b);
+ if (total) {
+ assertFailures(search, RestStatus.BAD_REQUEST,
both(containsString("Query has ")).and(containsString(" total terms but only 8
total terms are allowed")));
+ } else {
+ assertFailures(search, RestStatus.BAD_REQUEST,
containsString("Query has 6 terms but only 5 are allowed"));
+ }
+ }
+
+ private void assertSearchByString(String string) {
+ QueryStringQueryBuilder qs = queryString(string);
+ SaferQueryBuilder b = new
SaferQueryBuilder(qs).maxTermsPerPhraseQuery(5);
+ if (getRandom().nextBoolean()) {
+ b.phraseTooLargeAction(PhraseTooLargeAction.ERROR);
+ }
+ SearchRequestBuilder search =
client().prepareSearch("test").setQuery(b);
+ assertFailures(search, RestStatus.BAD_REQUEST, containsString("Query
has 6 terms but only 5 are allowed"));
+ }
+
+ @Test
+ public void convert() {
+ QueryStringQueryBuilder qs = queryString("\"0 0 0 0 0 0\"");
+ SaferQueryBuilder b = new
SaferQueryBuilder(qs).phraseTooLargeAction(PhraseTooLargeAction.CONVERT_TO_TERM_QUERIES);
+ SearchRequestBuilder search =
client().prepareSearch("test").setQuery(b);
+ assertSearchHits(search.get(), "1");
+ b.maxTermsPerPhraseQuery(6);
+ assertSearchHits(search.get(), "1");
+
+ b.maxTermsPerPhraseQuery(5);
+ assertSearchHits(search.get(), "1", "2"); //Unorderd
+
+ qs = queryString("\"0 0 0\" \"0 0 0 0 0
0\"").defaultOperator(Operator.AND);
+ b = new
SaferQueryBuilder(qs).phraseTooLargeAction(PhraseTooLargeAction.CONVERT_TO_TERM_QUERIES);
+ search = client().prepareSearch("test").setQuery(b);
+ assertSearchHits(search.get(), "1");
+ b.maxTermsInAllPhraseQueries(9);
+ assertSearchHits(search.get(), "1");
+
+ // If you go over the limit then the last phrase query is converted to
term queries
+ b.maxTermsInAllPhraseQueries(8);
+ assertSearchHits(search.get(), "1", "2"); //Unorderd
+
+ qs = queryString("findme:\"CaptureAAAA Test\"");
+ b = new
SaferQueryBuilder(qs).phraseTooLargeAction(PhraseTooLargeAction.CONVERT_TO_TERM_QUERIES);
+ search = client().prepareSearch("test").setQuery(b);
+ assertSearchHits(search.get(), "delimited1", "delimited2");
+ b.maxTermsPerPhraseQuery(6);
+ assertSearchHits(search.get(), "delimited1", "delimited2");
+
+ b.maxTermsPerPhraseQuery(5);
+ assertSearchHits(search.get(), "delimited1", "delimited2");
+
+ qs = queryString("findme:0.0.0.0.0.0");
+ b = new
SaferQueryBuilder(qs).phraseTooLargeAction(PhraseTooLargeAction.CONVERT_TO_TERM_QUERIES);
+ search = client().prepareSearch("test").setQuery(b);
+ assertSearchHits(search.get(), "1", "2");
+ b.maxTermsPerPhraseQuery(6);
+ assertSearchHits(search.get(), "1", "2");
+
+ b.maxTermsPerPhraseQuery(5);
+ assertSearchHits(search.get(), "1", "2"); //Unorderd
+ }
+
+ @Test
+ public void matchNone() {
+ QueryStringQueryBuilder qs = queryString("\"0 0 0 0 0 0\"");
+ SaferQueryBuilder b = new
SaferQueryBuilder(qs).phraseTooLargeAction(PhraseTooLargeAction.CONVERT_TO_MATCH_NONE_QUERY);
+ SearchRequestBuilder search =
client().prepareSearch("test").setQuery(b);
+ assertSearchHits(search.get(), "1");
+ b.maxTermsPerPhraseQuery(6);
+ assertSearchHits(search.get(), "1");
+ b.maxTermsPerPhraseQuery(5);
+ assertHitCount(search.get(), 0);
+ }
+
+ @Test
+ public void matchAll() {
+ QueryStringQueryBuilder qs = queryString("\"0 0 0 0 0 0\"");
+ SaferQueryBuilder b = new
SaferQueryBuilder(qs).phraseTooLargeAction(PhraseTooLargeAction.CONVERT_TO_MATCH_ALL_QUERY);
+ SearchRequestBuilder search =
client().prepareSearch("test").setQuery(b);
+ assertSearchHits(search.get(), "1");
+ b.maxTermsPerPhraseQuery(6);
+ assertSearchHits(search.get(), "1");
+ b.maxTermsPerPhraseQuery(5);
+ assertHitCount(search.get(), 4);
+ }
+}
--
To view, visit https://gerrit.wikimedia.org/r/175853
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I9560105c31d429df24d13232222740acba9b8e9a
Gerrit-PatchSet: 12
Gerrit-Project: search/extra
Gerrit-Branch: master
Gerrit-Owner: Manybubbles <[email protected]>
Gerrit-Reviewer: BearND <[email protected]>
Gerrit-Reviewer: BryanDavis <[email protected]>
Gerrit-Reviewer: Chad <[email protected]>
Gerrit-Reviewer: Manybubbles <[email protected]>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
