[jira] [Updated] (LUCENE-7747) QueryBuilder should build side-paths query (graph query) lazily
[ https://issues.apache.org/jira/browse/LUCENE-7747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Ferenczi updated LUCENE-7747: - Priority: Blocker (was: Major) Fix Version/s: 6.5 > QueryBuilder should build side-paths query (graph query) lazily > --- > > Key: LUCENE-7747 > URL: https://issues.apache.org/jira/browse/LUCENE-7747 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi >Priority: Blocker > Fix For: 6.5 > > Attachments: LUCENE-7747.patch > > > In LUCENE-7638 we generate a query for each multi-token path in the graph and > combine them at the end in a boolean query. > This can lead to OOM when the number of path is big, instead we should build > the disjunction of these paths lazily in order to throw "too many clauses" > early if the number of paths is bigger than max boolean clauses. > For instance a shingle filter with shingles of different size produces a > graph with multiple side paths at each position. If the input query has a lot > of tokens, the number of paths (query) created is exponential. For this use > case it is maybe preferable to disallow graph query analysis completely but > when allowed we should also be protected against combinatorial explosion. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-7747) QueryBuilder should build side-paths query (graph query) lazily
[ https://issues.apache.org/jira/browse/LUCENE-7747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Ferenczi updated LUCENE-7747: - Attachment: LUCENE-7747.patch Here is a patch that builds an lazy iterator over the different paths. The graph boolean query is built by consuming this iterator and throws a "too many clauses" exception when the number of paths is greater than the max number of clauses allowed. > QueryBuilder should build side-paths query (graph query) lazily > --- > > Key: LUCENE-7747 > URL: https://issues.apache.org/jira/browse/LUCENE-7747 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jim Ferenczi > Attachments: LUCENE-7747.patch > > > In LUCENE-7638 we generate a query for each multi-token path in the graph and > combine them at the end in a boolean query. > This can lead to OOM when the number of path is big, instead we should build > the disjunction of these paths lazily in order to throw "too many clauses" > early if the number of paths is bigger than max boolean clauses. > For instance a shingle filter with shingles of different size produces a > graph with multiple side paths at each position. If the input query has a lot > of tokens, the number of paths (query) created is exponential. For this use > case it is maybe preferable to disallow graph query analysis completely but > when allowed we should also be protected against combinatorial explosion. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
