[
https://issues.apache.org/jira/browse/LUCENE-8202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16411066#comment-16411066
]
Adrien Grand commented on LUCENE-8202:
--------------------------------------
bq. The decompounding filter was producing up to 50 tokens in the same
position, which lead to 50^11 shingles being generated, resulting in OOM.
I agree that 50 tokens per position and a shingle length of 11 would create
50^11 shingles, but the filter would only keep 50*11 tokens in memory and
generate shingles on the fly. So I would expect it to be slow, but not to use
lots of memory? So maybe we have a bug?
bq. I'm not sure of the best way of dealing with this one though - we could
just limit shingle length to a maximum of 3 or 4, but that seems like too harsh
a restriction for this. The other possibility would be to have a
(configurable) maximum number of shingles emitted at a single position, and
throw IllegalStateException if this is hit.
I would add both. I don't think restricting the shingle size would be too
harsh, I don't remember of someone using a max shingle size that was greater
than 3? Also limits are always easier to increase than decrease.
> Add a FixedShingleFilter
> ------------------------
>
> Key: LUCENE-8202
> URL: https://issues.apache.org/jira/browse/LUCENE-8202
> Project: Lucene - Core
> Issue Type: New Feature
> Reporter: Alan Woodward
> Assignee: Alan Woodward
> Priority: Major
> Fix For: 7.4
>
> Attachments: LUCENE-8202.patch, LUCENE-8202.patch, LUCENE-8202.patch
>
>
> In LUCENE-3475 I tried to make a ShingleGraphFilter that could accept and
> emit arbitrary graphs, while duplicating all the functionality of the
> existing ShingleFilter. This ends up being extremely hairy, and doesn't play
> well with query parsers.
> I'd like to step back and try and create a simpler shingle filter that can be
> used for index-time phrase tokenization only. It will have a single fixed
> shingle size, can deal with single-token synonyms, and won't emit unigrams.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
