[
https://issues.apache.org/jira/browse/LUCENE-5269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13790940#comment-13790940
]
Robert Muir commented on LUCENE-5269:
-------------------------------------
Now i see stuff like this:
{noformat}
EdgeNGramTokenizer.reset()
ShingleFilter.reset()
NGramTokenFilter.reset()
EdgeNGramTokenizer->term=⥞ ,bytes=[e2 a5 9e
20],positionIncrement=1,positionLength=1,startOffset=0,endOffset=2,type=word,clearCalled=true
EdgeNGramTokenizer->term=⥞ 𐤋,bytes=[e2 a5 9e 20 f0 90 a4
8b],positionIncrement=1,positionLength=1,startOffset=0,endOffset=4,type=word,clearCalled=true
EdgeNGramTokenizer->term=⥞ 𐤋𐤟,bytes=[e2 a5 9e 20 f0 90 a4 8b f0 90 a4
9f],positionIncrement=1,positionLength=1,startOffset=0,endOffset=6,type=word,clearCalled=true
EdgeNGramTokenizer->term=⥞ 𐤋𐤟 ,bytes=[e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f
20],positionIncrement=1,positionLength=1,startOffset=0,endOffset=7,type=word,clearCalled=true
EdgeNGramTokenizer->term=⥞ 𐤋𐤟 x,bytes=[e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f 20
78],positionIncrement=1,positionLength=1,startOffset=0,endOffset=8,type=word,clearCalled=true
EdgeNGramTokenizer->term=⥞ 𐤋𐤟 xq,bytes=[e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f 20
78
71],positionIncrement=1,positionLength=1,startOffset=0,endOffset=9,type=word,clearCalled=true
EdgeNGramTokenizer->term=⥞ 𐤋𐤟 xqx,bytes=[e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f 20
78 71
78],positionIncrement=1,positionLength=1,startOffset=0,endOffset=10,type=word,clearCalled=true
EdgeNGramTokenizer->term=⥞ 𐤋𐤟 xqxp,bytes=[e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f
20 78 71 78
70],positionIncrement=1,positionLength=1,startOffset=0,endOffset=11,type=word,clearCalled=true
EdgeNGramTokenizer->term=⥞ 𐤋𐤟 xqxp ,bytes=[e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f
20 78 71 78 70
20],positionIncrement=1,positionLength=1,startOffset=0,endOffset=12,type=word,clearCalled=true
EdgeNGramTokenizer->term=⥞ 𐤋𐤟 xqxp ,bytes=[e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f
20 78 71 78 70 20
16],positionIncrement=1,positionLength=1,startOffset=0,endOffset=13,type=word,clearCalled=true
EdgeNGramTokenizer.end()
ShingleFilter->term=⥞ ,bytes=[e2 a5 9e
20],positionIncrement=1,positionLength=1,startOffset=0,endOffset=2,type=word,clearCalled=true
ShingleFilter->term=⥞ ⥞ 𐤋,bytes=[e2 a5 9e 20 20 e2 a5 9e 20 f0 90 a4
8b],positionIncrement=0,positionLength=2,startOffset=0,endOffset=4,type=shingle,clearCalled=true
ShingleFilter->term=⥞ ⥞ 𐤋 ⥞ 𐤋𐤟,bytes=[e2 a5 9e 20 20 e2 a5 9e 20 f0 90 a4 8b
20 e2 a5 9e 20 f0 90 a4 8b f0 90 a4
9f],positionIncrement=0,positionLength=3,startOffset=0,endOffset=6,type=shingle,clearCalled=true
ShingleFilter->term=⥞ ⥞ 𐤋 ⥞ 𐤋𐤟 ⥞ 𐤋𐤟 ,bytes=[e2 a5 9e 20 20 e2 a5 9e 20 f0 90
a4 8b 20 e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f 20 e2 a5 9e 20 f0 90 a4 8b f0 90
a4 9f
20],positionIncrement=0,positionLength=4,startOffset=0,endOffset=7,type=shingle,clearCalled=true
ShingleFilter->term=⥞ ⥞ 𐤋 ⥞ 𐤋𐤟 ⥞ 𐤋𐤟 ⥞ 𐤋𐤟 x,bytes=[e2 a5 9e 20 20 e2 a5 9e 20
f0 90 a4 8b 20 e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f 20 e2 a5 9e 20 f0 90 a4 8b
f0 90 a4 9f 20 20 e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f 20
78],positionIncrement=0,positionLength=5,startOffset=0,endOffset=8,type=shingle,clearCalled=true
ShingleFilter->term=⥞ ⥞ 𐤋 ⥞ 𐤋𐤟 ⥞ 𐤋𐤟 ⥞ 𐤋𐤟 x ⥞ 𐤋𐤟 xq,bytes=[e2 a5 9e 20 20 e2
a5 9e 20 f0 90 a4 8b 20 e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f 20 e2 a5 9e 20 f0
90 a4 8b f0 90 a4 9f 20 20 e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f 20 78 20 e2 a5
9e 20 f0 90 a4 8b f0 90 a4 9f 20 78
71],positionIncrement=0,positionLength=6,startOffset=0,endOffset=9,type=shingle,clearCalled=true
ShingleFilter->term=⥞ ⥞ 𐤋 ⥞ 𐤋𐤟 ⥞ 𐤋𐤟 ⥞ 𐤋𐤟 x ⥞ 𐤋𐤟 xq ⥞ 𐤋𐤟 xqx,bytes=[e2 a5 9e
20 20 e2 a5 9e 20 f0 90 a4 8b 20 e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f 20 e2 a5
9e 20 f0 90 a4 8b f0 90 a4 9f 20 20 e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f 20 78
20 e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f 20 78 71 20 e2 a5 9e 20 f0 90 a4 8b f0
90 a4 9f 20 78 71
78],positionIncrement=0,positionLength=7,startOffset=0,endOffset=10,type=shingle,clearCalled=true
ShingleFilter->term=⥞ ⥞ 𐤋 ⥞ 𐤋𐤟 ⥞ 𐤋𐤟 ⥞ 𐤋𐤟 x ⥞ 𐤋𐤟 xq ⥞ 𐤋𐤟 xqx ⥞ 𐤋𐤟
xqxp,bytes=[e2 a5 9e 20 20 e2 a5 9e 20 f0 90 a4 8b 20 e2 a5 9e 20 f0 90 a4 8b
f0 90 a4 9f 20 e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f 20 20 e2 a5 9e 20 f0 90 a4
8b f0 90 a4 9f 20 78 20 e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f 20 78 71 20 e2 a5
9e 20 f0 90 a4 8b f0 90 a4 9f 20 78 71 78 20 e2 a5 9e 20 f0 90 a4 8b f0 90 a4
9f 20 78 71 78
70],positionIncrement=0,positionLength=8,startOffset=0,endOffset=11,type=shingle,clearCalled=true
ShingleFilter->term=⥞ ⥞ 𐤋 ⥞ 𐤋𐤟 ⥞ 𐤋𐤟 ⥞ 𐤋𐤟 x ⥞ 𐤋𐤟 xq ⥞ 𐤋𐤟 xqx ⥞ 𐤋𐤟 xqxp ⥞ 𐤋𐤟
xqxp ,bytes=[e2 a5 9e 20 20 e2 a5 9e 20 f0 90 a4 8b 20 e2 a5 9e 20 f0 90 a4 8b
f0 90 a4 9f 20 e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f 20 20 e2 a5 9e 20 f0 90 a4
8b f0 90 a4 9f 20 78 20 e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f 20 78 71 20 e2 a5
9e 20 f0 90 a4 8b f0 90 a4 9f 20 78 71 78 20 e2 a5 9e 20 f0 90 a4 8b f0 90 a4
9f 20 78 71 78 70 20 e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f 20 78 71 78 70
20],positionIncrement=0,positionLength=9,startOffset=0,endOffset=12,type=shingle,clearCalled=true
NGramTokenFilter->term=⥞ ⥞ 𐤋 ⥞ 𐤋𐤟 ⥞ 𐤋𐤟 ⥞ 𐤋𐤟 x ⥞ 𐤋𐤟 xq ⥞ 𐤋𐤟 xqx ⥞ 𐤋𐤟 xqxp ⥞
𐤋,bytes=[e2 a5 9e 20 20 e2 a5 9e 20 f0 90 a4 8b 20 e2 a5 9e 20 f0 90 a4 8b f0
90 a4 9f 20 e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f 20 20 e2 a5 9e 20 f0 90 a4 8b
f0 90 a4 9f 20 78 20 e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f 20 78 71 20 e2 a5 9e
20 f0 90 a4 8b f0 90 a4 9f 20 78 71 78 20 e2 a5 9e 20 f0 90 a4 8b f0 90 a4 9f
20 78 71 78 70 20 e2 a5 9e 20 f0 90 a4
8b],positionIncrement=0,positionLength=9,startOffset=0,endOffset=12,type=word,clearCalled=true
TEST FAIL: useCharFilter=false text='\u295e \u1090b\u1091f xqxp \u0016'
java.lang.AssertionError: first posIncrement must be >= 1
at
__randomizedtesting.SeedInfo.seed([6CC8BD35A010E1FF:714032FA1B8FBB60]:0)
{noformat}
> TestRandomChains failure
> ------------------------
>
> Key: LUCENE-5269
> URL: https://issues.apache.org/jira/browse/LUCENE-5269
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Robert Muir
> Attachments: LUCENE-5269_test.patch, LUCENE-5269_test.patch,
> LUCENE-5269_test.patch
>
>
> One of EdgeNGramTokenizer, ShingleFilter, NGramTokenFilter is buggy, or
> possibly only the combination of them conspiring together.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
