[jira] [Commented] (LUCENE-3475) ShingleFilter should handle positionIncrement of zero, e.g. synonyms
[ https://issues.apache.org/jira/browse/LUCENE-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396416#comment-16396416 ] David Smiley commented on LUCENE-3475: -- bq. Some of this behaviour is different to the standard ShingleFilter, so I think we should probably keep them separate (while deprecating the old version and removing in master) Wouldn't this be a Version switch situation instead? Why bother the users with a new name instead? > ShingleFilter should handle positionIncrement of zero, e.g. synonyms > > > Key: LUCENE-3475 > URL: https://issues.apache.org/jira/browse/LUCENE-3475 > Project: Lucene - Core > Issue Type: New Feature > Components: modules/analysis >Reporter: Cameron >Assignee: Alan Woodward >Priority: Minor > Labels: newdev > Attachments: LUCENE-3475.patch, LUCENE-3475.patch > > > ShingleFilter is creating shingles for a single term that has been expanded > by synonyms when it shouldn't. The position increment is 0. > As an example, I have an Analyzer with a SynonymFilter followed by a > ShingleFilter. Assuming car and auto are synonyms, the SynonymFilter produces > two tokens and position 1: car, auto. The ShingleFilter is then producing 3 > tokens, when there should only be two: car, car auto, auto. This behavior > seems incorrect. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3475) ShingleFilter should handle positionIncrement of zero, e.g. synonyms
[ https://issues.apache.org/jira/browse/LUCENE-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395140#comment-16395140 ] Alan Woodward commented on LUCENE-3475: --- Updated patch: * trailing terms that would produce shingles < minShingleLength are emitted as single tokens * all shingles have positionLength=1, which fixes issues with QueryBuilder * added tests using CannedTokenStream from FlattenGraphFilter Some of this behaviour is different to the standard ShingleFilter, so I think we should probably keep them separate (while deprecating the old version and removing in master) > ShingleFilter should handle positionIncrement of zero, e.g. synonyms > > > Key: LUCENE-3475 > URL: https://issues.apache.org/jira/browse/LUCENE-3475 > Project: Lucene - Core > Issue Type: New Feature > Components: modules/analysis >Reporter: Cameron >Assignee: Alan Woodward >Priority: Minor > Labels: newdev > Attachments: LUCENE-3475.patch, LUCENE-3475.patch > > > ShingleFilter is creating shingles for a single term that has been expanded > by synonyms when it shouldn't. The position increment is 0. > As an example, I have an Analyzer with a SynonymFilter followed by a > ShingleFilter. Assuming car and auto are synonyms, the SynonymFilter produces > two tokens and position 1: car, auto. The ShingleFilter is then producing 3 > tokens, when there should only be two: car, car auto, auto. This behavior > seems incorrect. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3475) ShingleFilter should handle positionIncrement of zero, e.g. synonyms
[
https://issues.apache.org/jira/browse/LUCENE-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358207#comment-16358207
]
Alan Woodward commented on LUCENE-3475:
---
{quote}but won't generate a graph unless the input is a graph too
{quote}
Correct
{quote}Should we just replace it?
{quote}
I'll need to rejig it slightly so that it produces identical output, but I
think that's worth doing. And a drop-in replacement would be easier for
upgrades as well.
I'll add some more tests, mixing up CannedTokenStream and real analyzer outputs.
> ShingleFilter should handle positionIncrement of zero, e.g. synonyms
>
>
> Key: LUCENE-3475
> URL: https://issues.apache.org/jira/browse/LUCENE-3475
> Project: Lucene - Core
> Issue Type: New Feature
> Components: modules/analysis
>Reporter: Cameron
>Assignee: Alan Woodward
>Priority: Minor
> Labels: newdev
> Attachments: LUCENE-3475.patch
>
>
> ShingleFilter is creating shingles for a single term that has been expanded
> by synonyms when it shouldn't. The position increment is 0.
> As an example, I have an Analyzer with a SynonymFilter followed by a
> ShingleFilter. Assuming car and auto are synonyms, the SynonymFilter produces
> two tokens and position 1: car, auto. The ShingleFilter is then producing 3
> tokens, when there should only be two: car, car auto, auto. This behavior
> seems incorrect.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3475) ShingleFilter should handle positionIncrement of zero, e.g. synonyms
[ https://issues.apache.org/jira/browse/LUCENE-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357375#comment-16357375 ] Adrien Grand commented on LUCENE-3475: -- Exciting! I'll need some time to digest this new implementation but I already have some questions/remarks :) - Is my understanding correct that this filter can consume graphs but won't generate a graph unless the input is a graph too (ie. it won't require FlattenGraphTokenFilter at index time unless the input is a graph)? - At first sight, this impl looks compatible with the existing ShingleFilter, or do I miss something? Should we just replace it? - Can the tests create token stream graphs explicitly with CannedTokenStream instead of implicitly with SynonymGraphTokenStream? - I'd be curious to see tests on more complex token stream graphs eg. like those used in TestFlattenGraphFilter.testWTF1/testWTF2, does your impl support such complex graphs? > ShingleFilter should handle positionIncrement of zero, e.g. synonyms > > > Key: LUCENE-3475 > URL: https://issues.apache.org/jira/browse/LUCENE-3475 > Project: Lucene - Core > Issue Type: New Feature > Components: modules/analysis >Reporter: Cameron >Assignee: Alan Woodward >Priority: Minor > Labels: newdev > Attachments: LUCENE-3475.patch > > > ShingleFilter is creating shingles for a single term that has been expanded > by synonyms when it shouldn't. The position increment is 0. > As an example, I have an Analyzer with a SynonymFilter followed by a > ShingleFilter. Assuming car and auto are synonyms, the SynonymFilter produces > two tokens and position 1: car, auto. The ShingleFilter is then producing 3 > tokens, when there should only be two: car, car auto, auto. This behavior > seems incorrect. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3475) ShingleFilter should handle positionIncrement of zero, e.g. synonyms
[ https://issues.apache.org/jira/browse/LUCENE-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347069#comment-16347069 ] Mayya Sharipova commented on LUCENE-3475: - [~jpountz] thank so much for suggestions. [~romseygeek] is going to work on this issue. I will study his solution. > ShingleFilter should handle positionIncrement of zero, e.g. synonyms > > > Key: LUCENE-3475 > URL: https://issues.apache.org/jira/browse/LUCENE-3475 > Project: Lucene - Core > Issue Type: New Feature > Components: modules/analysis >Affects Versions: 3.4 >Reporter: Cameron >Priority: Minor > Labels: newdev > > ShingleFilter is creating shingles for a single term that has been expanded > by synonyms when it shouldn't. The position increment is 0. > As an example, I have an Analyzer with a SynonymFilter followed by a > ShingleFilter. Assuming car and auto are synonyms, the SynonymFilter produces > two tokens and position 1: car, auto. The ShingleFilter is then producing 3 > tokens, when there should only be two: car, car auto, auto. This behavior > seems incorrect. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3475) ShingleFilter should handle positionIncrement of zero, e.g. synonyms
[
https://issues.apache.org/jira/browse/LUCENE-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346807#comment-16346807
]
Adrien Grand commented on LUCENE-3475:
--
[~mayyas] I think we need to do something like this indeed. We might also need
\{{getNextToken}} I don't think you should try to recreate a token stream from
the synonyms, however it is true that you need to record their state somewhere.
There are multiple ways to do this, for instance ShingleFilter uses
\{{cloneAttributes}}, but there is also {{captureState/\{{restoreState}}.
You can look at \{{EdgeNGramTokenFilter}} for an example of how to use them.
I haven't looked deeply at how the token filter currently works, but I wouldn't
be surprised that significant refactorings are required to make things work, so
feel free to reorganize the code if it makes this problem any easier to fix!
> ShingleFilter should handle positionIncrement of zero, e.g. synonyms
>
>
> Key: LUCENE-3475
> URL: https://issues.apache.org/jira/browse/LUCENE-3475
> Project: Lucene - Core
> Issue Type: New Feature
> Components: modules/analysis
>Affects Versions: 3.4
>Reporter: Cameron
>Priority: Minor
> Labels: newdev
>
> ShingleFilter is creating shingles for a single term that has been expanded
> by synonyms when it shouldn't. The position increment is 0.
> As an example, I have an Analyzer with a SynonymFilter followed by a
> ShingleFilter. Assuming car and auto are synonyms, the SynonymFilter produces
> two tokens and position 1: car, auto. The ShingleFilter is then producing 3
> tokens, when there should only be two: car, car auto, auto. This behavior
> seems incorrect.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3475) ShingleFilter should handle positionIncrement of zero, e.g. synonyms
[
https://issues.apache.org/jira/browse/LUCENE-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344192#comment-16344192
]
Mayya Sharipova commented on LUCENE-3475:
-
[~jpountz] Hi Adrien! I wonder what would be the best approach to handle
positionIncrement=0?
I was thinking that in *ShingleFilter:getNextToken* we can do something like
this:
{code:java}
if (input.incrementToken()) {
while (posIncrAtt.getPositionIncrement() == 0) { // we may have multiple
synonyms
if (input.incrementToken()) { // go to next token
// store synonyms tokens and following tokens somewhere and create a
new input TokenStream from them?
{code}
I guess I am wondering if we have any other sample code that already doing
that, and which I can reference?
> ShingleFilter should handle positionIncrement of zero, e.g. synonyms
>
>
> Key: LUCENE-3475
> URL: https://issues.apache.org/jira/browse/LUCENE-3475
> Project: Lucene - Core
> Issue Type: New Feature
> Components: modules/analysis
>Affects Versions: 3.4
>Reporter: Cameron
>Priority: Minor
> Labels: newdev
>
> ShingleFilter is creating shingles for a single term that has been expanded
> by synonyms when it shouldn't. The position increment is 0.
> As an example, I have an Analyzer with a SynonymFilter followed by a
> ShingleFilter. Assuming car and auto are synonyms, the SynonymFilter produces
> two tokens and position 1: car, auto. The ShingleFilter is then producing 3
> tokens, when there should only be two: car, car auto, auto. This behavior
> seems incorrect.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3475) ShingleFilter should handle positionIncrement of zero, e.g. synonyms
[
https://issues.apache.org/jira/browse/LUCENE-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343073#comment-16343073
]
Adrien Grand commented on LUCENE-3475:
--
I'm not going to work on this in the short term but I just wrote a test case in
case someone else has interest in fixing this issue:
{code}
public void testSynonyms1() throws IOException {
Token t1 = new Token("small", 0, 5);
Token t2 = new Token("little", 0, 5);
t2.setPositionIncrement(0);
Token t3 = new Token("thing", 6, 11);
TokenStream tk = new CannedTokenStream(t1, t2, t3);
ShingleFilter f = new ShingleFilter(tk, 2, 2);
f.setOutputUnigrams(false);
assertTokenStreamContents(f,
new String[] { "small thing", "little thing" },
new int[] {0, 0},
new int[] {11, 11},
new int[] {1, 0});
}
public void testSynonyms2() throws IOException {
Token t1 = new Token("small", 0, 5);
Token t2 = new Token("thing", 6, 11);
Token t3 = new Token("object", 6, 11);
t3.setPositionIncrement(0);
TokenStream tk = new CannedTokenStream(t1, t2, t3);
ShingleFilter f = new ShingleFilter(tk, 2, 2);
f.setOutputUnigrams(false);
assertTokenStreamContents(f,
new String[] { "small thing", "small object" },
new int[] {0, 0},
new int[] {11, 11},
new int[] {1, 0});
}
{code}
> ShingleFilter should handle positionIncrement of zero, e.g. synonyms
>
>
> Key: LUCENE-3475
> URL: https://issues.apache.org/jira/browse/LUCENE-3475
> Project: Lucene - Core
> Issue Type: New Feature
> Components: modules/analysis
>Affects Versions: 3.4
>Reporter: Cameron
>Priority: Minor
> Labels: newdev
>
> ShingleFilter is creating shingles for a single term that has been expanded
> by synonyms when it shouldn't. The position increment is 0.
> As an example, I have an Analyzer with a SynonymFilter followed by a
> ShingleFilter. Assuming car and auto are synonyms, the SynonymFilter produces
> two tokens and position 1: car, auto. The ShingleFilter is then producing 3
> tokens, when there should only be two: car, car auto, auto. This behavior
> seems incorrect.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3475) ShingleFilter should handle positionIncrement of zero, e.g. synonyms
[ https://issues.apache.org/jira/browse/LUCENE-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13116829#comment-13116829 ] Cameron commented on LUCENE-3475: - It does not handle a position increment of 0. ...I had interpreted that to mean it would just 'ignore' or not try to do anything with tokens where posIncr=0. I guess that is an incorrect assumption? ShingleFilter should handle positionIncrement of zero, e.g. synonyms Key: LUCENE-3475 URL: https://issues.apache.org/jira/browse/LUCENE-3475 Project: Lucene - Java Issue Type: New Feature Components: modules/analysis Affects Versions: 3.4 Reporter: Cameron Priority: Minor ShingleFilter is creating shingles for a single term that has been expanded by synonyms when it shouldn't. The position increment is 0. As an example, I have an Analyzer with a SynonymFilter followed by a ShingleFilter. Assuming car and auto are synonyms, the SynonymFilter produces two tokens and position 1: car, auto. The ShingleFilter is then producing 3 tokens, when there should only be two: car, car auto, auto. This behavior seems incorrect. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3475) ShingleFilter should handle positionIncrement of zero, e.g. synonyms
[ https://issues.apache.org/jira/browse/LUCENE-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13116833#comment-13116833 ] Steven Rowe commented on LUCENE-3475: - That sentence was intended to warn users that ShingleFilter behavior was undefined over streams with position increment of zero. ShingleFilter should handle positionIncrement of zero, e.g. synonyms Key: LUCENE-3475 URL: https://issues.apache.org/jira/browse/LUCENE-3475 Project: Lucene - Java Issue Type: New Feature Components: modules/analysis Affects Versions: 3.4 Reporter: Cameron Priority: Minor ShingleFilter is creating shingles for a single term that has been expanded by synonyms when it shouldn't. The position increment is 0. As an example, I have an Analyzer with a SynonymFilter followed by a ShingleFilter. Assuming car and auto are synonyms, the SynonymFilter produces two tokens and position 1: car, auto. The ShingleFilter is then producing 3 tokens, when there should only be two: car, car auto, auto. This behavior seems incorrect. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
