Re: When zero offsets are not bad - a.k.a. multi-token synonyms yet again
Hi Mike,
Sorry for the delay, I was away last week. Now that I get back to it
again my plan is to write a test for the WordDelimiterFilter and
pinpoint the problem.
Cheers,
Roman
On Thu, Aug 20, 2020 at 11:21 AM Michael McCandless
wrote:
>
> Hi Roman,
>
> No need for anyone to be falling on swords here! This is really complicated
> stuff, no worries. And I think we have a compelling plan to move forwards so
> that we can index multi-token synonyms AND have 100% correct positional
> queries at search time, thanks to Michael Gibney's cool approach on
> https://issues.apache.org/jira/browse/LUCENE-4312.
>
> So it looks like WordDelimiterGraphFilter is producing buggy (out of order
> offsets) tokens here?
>
> Or are you running SynonymGraphFilter after WordDelimiterFilter?
>
> Looking at that failing example, it should have output'd that spacetime token
> immediately after the space token, not after the time token.
>
> Maybe use TokenStreamToDot to visualize what the heck token graph you are
> getting ...
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Tue, Aug 18, 2020 at 9:41 PM Roman Chyla wrote:
>>
>> Hi Mike,
>>
>> I'm sorry, the problem all the time is inside related to a
>> word-delimiter filter factory. This is embarrassing but I have to
>> admit publicly and self-flagellate.
>>
>> A word-delimiter filter is used to split tokens, these then are used
>> to find multi-token synonyms (hence the connection). In my desire to
>> simplify, I have omitted that detail while writing my first email.
>>
>> I went to generate the stack trace:
>>
>> ```
>> assertU(adoc("id", "603", "bibcode", "xx603",
>> "title", "THE HUBBLE constant: a summary of the HUBBLE SPACE
>> TELESCOPE program"));```
>>
>> stage:indexer term=xx603 pos=1 type=word offsetStart=0 offsetEnd=13
>> stage:indexer term=acr::the pos=1 type=ACRONYM offsetStart=0 offsetEnd=3
>> stage:indexer term=hubble pos=1 type=word offsetStart=4 offsetEnd=10
>> stage:indexer term=acr::hubble pos=0 type=ACRONYM offsetStart=4 offsetEnd=10
>> stage:indexer term=constant pos=1 type=word offsetStart=11 offsetEnd=20
>> stage:indexer term=summary pos=1 type=word offsetStart=23 offsetEnd=30
>> stage:indexer term=hubble pos=1 type=word offsetStart=38 offsetEnd=44
>> stage:indexer term=syn::hubble space telescope pos=0 type=SYNONYM
>> offsetStart=38 offsetEnd=60
>> stage:indexer term=syn::hst pos=0 type=SYNONYM offsetStart=38 offsetEnd=60
>> stage:indexer term=space pos=1 type=word offsetStart=45 offsetEnd=50
>> stage:indexer term=telescope pos=1 type=word offsetStart=51 offsetEnd=60
>> stage:indexer term=program pos=1 type=word offsetStart=61 offsetEnd=68
>>
>> that worked, only the next one failed:
>>
>> ```assertU(adoc("id", "605", "bibcode", "xx604",
>> "title", "MIT and anti de sitter space-time"));```
>>
>>
>> stage:indexer term=xx604 pos=1 type=word offsetStart=0 offsetEnd=13
>> stage:indexer term=mit pos=1 type=word offsetStart=0 offsetEnd=3
>> stage:indexer term=acr::mit pos=0 type=ACRONYM offsetStart=0 offsetEnd=3
>> stage:indexer term=syn::massachusetts institute of technology pos=0
>> type=SYNONYM offsetStart=0 offsetEnd=3
>> stage:indexer term=syn::mit pos=0 type=SYNONYM offsetStart=0 offsetEnd=3
>> stage:indexer term=anti pos=1 type=word offsetStart=8 offsetEnd=12
>> stage:indexer term=syn::ads pos=0 type=SYNONYM offsetStart=8 offsetEnd=28
>> stage:indexer term=syn::anti de sitter space pos=0 type=SYNONYM
>> offsetStart=8 offsetEnd=28
>> stage:indexer term=syn::antidesitter spacetime pos=0 type=SYNONYM
>> offsetStart=8 offsetEnd=28
>> stage:indexer term=de pos=1 type=word offsetStart=13 offsetEnd=15
>> stage:indexer term=sitter pos=1 type=word offsetStart=16 offsetEnd=22
>> stage:indexer term=space pos=1 type=word offsetStart=23 offsetEnd=28
>> stage:indexer term=time pos=1 type=word offsetStart=29 offsetEnd=33
>> stage:indexer term=spacetime pos=0 type=word offsetStart=23 offsetEnd=33
>>
>> ```325677 ERROR
>> (TEST-TestAdsabsTypeFulltextParsing.testNoSynChain-seed#[ADFAB495DA8F6F40])
>> [] o.a.s.h.RequestHandlerBase
>> org.apache.solr.common.SolrException: Exception writing document id
>> 605 to the index; possible analysis error: startOffset must be
>> non-negative, and endOffset must be >= startOffset, and offsets must
>> not go backwards startOffset=23,endOffset=33,lastStartOffset=29 for
>> field 'title'
>> at
>> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:242)
>> at
>> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
>> at
>> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
>> at
>> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:1002)
>> at
>> org.apache.solr.update.processor.DistributedUpdateProcessor.doVersionAdd(DistributedUpdateProcessor.java:1233)
>> at
>>
Re: When zero offsets are not bad - a.k.a. multi-token synonyms yet again
Hi Roman,
No need for anyone to be falling on swords here! This is really
complicated stuff, no worries. And I think we have a compelling plan to
move forwards so that we can index multi-token synonyms AND have 100%
correct positional queries at search time, thanks to Michael Gibney's cool
approach on https://issues.apache.org/jira/browse/LUCENE-4312.
So it looks like WordDelimiterGraphFilter is producing buggy (out of order
offsets) tokens here?
Or are you running SynonymGraphFilter after WordDelimiterFilter?
Looking at that failing example, it should have output'd that spacetime
token immediately after the space token, not after the time token.
Maybe use TokenStreamToDot to visualize what the heck token graph you are
getting ...
Mike McCandless
http://blog.mikemccandless.com
On Tue, Aug 18, 2020 at 9:41 PM Roman Chyla wrote:
> Hi Mike,
>
> I'm sorry, the problem all the time is inside related to a
> word-delimiter filter factory. This is embarrassing but I have to
> admit publicly and self-flagellate.
>
> A word-delimiter filter is used to split tokens, these then are used
> to find multi-token synonyms (hence the connection). In my desire to
> simplify, I have omitted that detail while writing my first email.
>
> I went to generate the stack trace:
>
> ```
> assertU(adoc("id", "603", "bibcode", "xx603",
> "title", "THE HUBBLE constant: a summary of the HUBBLE SPACE
> TELESCOPE program"));```
>
> stage:indexer term=xx603 pos=1 type=word offsetStart=0 offsetEnd=13
> stage:indexer term=acr::the pos=1 type=ACRONYM offsetStart=0 offsetEnd=3
> stage:indexer term=hubble pos=1 type=word offsetStart=4 offsetEnd=10
> stage:indexer term=acr::hubble pos=0 type=ACRONYM offsetStart=4
> offsetEnd=10
> stage:indexer term=constant pos=1 type=word offsetStart=11 offsetEnd=20
> stage:indexer term=summary pos=1 type=word offsetStart=23 offsetEnd=30
> stage:indexer term=hubble pos=1 type=word offsetStart=38 offsetEnd=44
> stage:indexer term=syn::hubble space telescope pos=0 type=SYNONYM
> offsetStart=38 offsetEnd=60
> stage:indexer term=syn::hst pos=0 type=SYNONYM offsetStart=38 offsetEnd=60
> stage:indexer term=space pos=1 type=word offsetStart=45 offsetEnd=50
> stage:indexer term=telescope pos=1 type=word offsetStart=51 offsetEnd=60
> stage:indexer term=program pos=1 type=word offsetStart=61 offsetEnd=68
>
> that worked, only the next one failed:
>
> ```assertU(adoc("id", "605", "bibcode", "xx604",
> "title", "MIT and anti de sitter space-time"));```
>
>
> stage:indexer term=xx604 pos=1 type=word offsetStart=0 offsetEnd=13
> stage:indexer term=mit pos=1 type=word offsetStart=0 offsetEnd=3
> stage:indexer term=acr::mit pos=0 type=ACRONYM offsetStart=0 offsetEnd=3
> stage:indexer term=syn::massachusetts institute of technology pos=0
> type=SYNONYM offsetStart=0 offsetEnd=3
> stage:indexer term=syn::mit pos=0 type=SYNONYM offsetStart=0 offsetEnd=3
> stage:indexer term=anti pos=1 type=word offsetStart=8 offsetEnd=12
> stage:indexer term=syn::ads pos=0 type=SYNONYM offsetStart=8 offsetEnd=28
> stage:indexer term=syn::anti de sitter space pos=0 type=SYNONYM
> offsetStart=8 offsetEnd=28
> stage:indexer term=syn::antidesitter spacetime pos=0 type=SYNONYM
> offsetStart=8 offsetEnd=28
> stage:indexer term=de pos=1 type=word offsetStart=13 offsetEnd=15
> stage:indexer term=sitter pos=1 type=word offsetStart=16 offsetEnd=22
> stage:indexer term=space pos=1 type=word offsetStart=23 offsetEnd=28
> stage:indexer term=time pos=1 type=word offsetStart=29 offsetEnd=33
> stage:indexer term=spacetime pos=0 type=word offsetStart=23 offsetEnd=33
>
> ```325677 ERROR
> (TEST-TestAdsabsTypeFulltextParsing.testNoSynChain-seed#[ADFAB495DA8F6F40])
> [] o.a.s.h.RequestHandlerBase
> org.apache.solr.common.SolrException: Exception writing document id
> 605 to the index; possible analysis error: startOffset must be
> non-negative, and endOffset must be >= startOffset, and offsets must
> not go backwards startOffset=23,endOffset=33,lastStartOffset=29 for
> field 'title'
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:242)
> at
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:1002)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.doVersionAdd(DistributedUpdateProcessor.java:1233)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.lambda$2(DistributedUpdateProcessor.java:1082)
> at org.apache.solr.update.VersionBucket.runWithLock(VersionBucket.java:50)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1082)
> at
>
Re: When zero offsets are not bad - a.k.a. multi-token synonyms yet again
Hi Mike,
I'm sorry, the problem all the time is inside related to a
word-delimiter filter factory. This is embarrassing but I have to
admit publicly and self-flagellate.
A word-delimiter filter is used to split tokens, these then are used
to find multi-token synonyms (hence the connection). In my desire to
simplify, I have omitted that detail while writing my first email.
I went to generate the stack trace:
```
assertU(adoc("id", "603", "bibcode", "xx603",
"title", "THE HUBBLE constant: a summary of the HUBBLE SPACE
TELESCOPE program"));```
stage:indexer term=xx603 pos=1 type=word offsetStart=0 offsetEnd=13
stage:indexer term=acr::the pos=1 type=ACRONYM offsetStart=0 offsetEnd=3
stage:indexer term=hubble pos=1 type=word offsetStart=4 offsetEnd=10
stage:indexer term=acr::hubble pos=0 type=ACRONYM offsetStart=4 offsetEnd=10
stage:indexer term=constant pos=1 type=word offsetStart=11 offsetEnd=20
stage:indexer term=summary pos=1 type=word offsetStart=23 offsetEnd=30
stage:indexer term=hubble pos=1 type=word offsetStart=38 offsetEnd=44
stage:indexer term=syn::hubble space telescope pos=0 type=SYNONYM
offsetStart=38 offsetEnd=60
stage:indexer term=syn::hst pos=0 type=SYNONYM offsetStart=38 offsetEnd=60
stage:indexer term=space pos=1 type=word offsetStart=45 offsetEnd=50
stage:indexer term=telescope pos=1 type=word offsetStart=51 offsetEnd=60
stage:indexer term=program pos=1 type=word offsetStart=61 offsetEnd=68
that worked, only the next one failed:
```assertU(adoc("id", "605", "bibcode", "xx604",
"title", "MIT and anti de sitter space-time"));```
stage:indexer term=xx604 pos=1 type=word offsetStart=0 offsetEnd=13
stage:indexer term=mit pos=1 type=word offsetStart=0 offsetEnd=3
stage:indexer term=acr::mit pos=0 type=ACRONYM offsetStart=0 offsetEnd=3
stage:indexer term=syn::massachusetts institute of technology pos=0
type=SYNONYM offsetStart=0 offsetEnd=3
stage:indexer term=syn::mit pos=0 type=SYNONYM offsetStart=0 offsetEnd=3
stage:indexer term=anti pos=1 type=word offsetStart=8 offsetEnd=12
stage:indexer term=syn::ads pos=0 type=SYNONYM offsetStart=8 offsetEnd=28
stage:indexer term=syn::anti de sitter space pos=0 type=SYNONYM
offsetStart=8 offsetEnd=28
stage:indexer term=syn::antidesitter spacetime pos=0 type=SYNONYM
offsetStart=8 offsetEnd=28
stage:indexer term=de pos=1 type=word offsetStart=13 offsetEnd=15
stage:indexer term=sitter pos=1 type=word offsetStart=16 offsetEnd=22
stage:indexer term=space pos=1 type=word offsetStart=23 offsetEnd=28
stage:indexer term=time pos=1 type=word offsetStart=29 offsetEnd=33
stage:indexer term=spacetime pos=0 type=word offsetStart=23 offsetEnd=33
```325677 ERROR
(TEST-TestAdsabsTypeFulltextParsing.testNoSynChain-seed#[ADFAB495DA8F6F40])
[] o.a.s.h.RequestHandlerBase
org.apache.solr.common.SolrException: Exception writing document id
605 to the index; possible analysis error: startOffset must be
non-negative, and endOffset must be >= startOffset, and offsets must
not go backwards startOffset=23,endOffset=33,lastStartOffset=29 for
field 'title'
at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:242)
at
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:1002)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doVersionAdd(DistributedUpdateProcessor.java:1233)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.lambda$2(DistributedUpdateProcessor.java:1082)
at org.apache.solr.update.VersionBucket.runWithLock(VersionBucket.java:50)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1082)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:694)
at
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:261)
at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:188)
at
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2551)
at
org.apache.solr.servlet.DirectSolrConnection.request(DirectSolrConnection.java:125)
at org.apache.solr.util.TestHarness.update(TestHarness.java:285)
at
org.apache.solr.util.BaseTestHarness.checkUpdateStatus(BaseTestHarness.java:274)
at org.apache.solr.util.BaseTestHarness.validateUpdate(BaseTestHarness.java:244)
at
Re: When zero offsets are not bad - a.k.a. multi-token synonyms yet again
Hi Roman,
Can you share the full exception / stack trace that IndexWriter throws on
that one *'d token in your first example? I thought IndexWriter checks 1)
startOffset >= last token's startOffset, and 2) endOffset >= startOffset
for the current token.
But you seem to be hitting an exception due to endOffset check across
tokens, which I didn't remember/realize IW was enforcing.
Could you share a small standalone test case showing the first example?
Maybe attach it to the issue (
http://issues.apache.org/jira/browse/LUCENE-8776)?
Thanks,
Mike McCandless
http://blog.mikemccandless.com
On Fri, Aug 14, 2020 at 12:09 PM Roman Chyla wrote:
> Hi Mike,
>
> Thanks for the question! And sorry for the delay, I haven't managed to
> get to it yesterday. I have generated better output, marked with (*)
> where it currently fails the first time and also included one extra
> case to illustrate the PositionLength attribute.
>
> assertU(adoc("id", "603", "bibcode", "xx603",
> "title", "THE HUBBLE constant: a summary of the hubble space
> telescope program"));
>
>
> term=hubble posInc=2 posLen=1 type=word offsetStart=4 offsetEnd=10
> term=acr::hubble posInc=0 posLen=1 type=ACRONYM offsetStart=4 offsetEnd=10
> term=constant posInc=1 posLen=1 type=word offsetStart=11 offsetEnd=20
> term=summary posInc=1 posLen=1 type=word offsetStart=23 offsetEnd=30
> term=hubble posInc=1 posLen=1 type=word offsetStart=38 offsetEnd=44
> term=syn::hubble space telescope posInc=0 posLen=3 type=SYNONYM
> offsetStart=38 offsetEnd=60
> term=syn::hst posInc=0 posLen=3 type=SYNONYM offsetStart=38 offsetEnd=60
> term=acr::hst posInc=0 posLen=3 type=ACRONYM offsetStart=38 offsetEnd=60
> * term=space posInc=1 posLen=1 type=word offsetStart=45 offsetEnd=50
> term=telescope posInc=1 posLen=1 type=word offsetStart=51 offsetEnd=60
> term=program posInc=1 posLen=1 type=word offsetStart=61 offsetEnd=68
>
> * - fails because of offsetEnd < lastToken.offsetEnd; If reordered
> (the multi-token synonym emitted as a last token) it would fail as
> well, because of the check for lastToken.beginOffset <
> currentToken.beginOffset. Basically, any reordering would result in a
> failure (unless offsets are trimmed).
>
>
>
> The following example has additional twist because of `space-time`;
> the tokenizer first splits the word and generate two new tokens --
> those alternative tokens are then used to find synonyms (space ==
> universe)
>
> assertU(adoc("id", "605", "bibcode", "xx604",
> "title", "MIT and anti de sitter space-time"));
>
>
> term=xx604 posInc=1 posLen=1 type=word offsetStart=0 offsetEnd=13
> term=mit posInc=1 posLen=1 type=word offsetStart=0 offsetEnd=3
> term=acr::mit posInc=0 posLen=1 type=ACRONYM offsetStart=0 offsetEnd=3
> term=syn::massachusetts institute of technology posInc=0 posLen=1
> type=SYNONYM offsetStart=0 offsetEnd=3
> term=syn::mit posInc=0 posLen=1 type=SYNONYM offsetStart=0 offsetEnd=3
> term=acr::mit posInc=0 posLen=1 type=ACRONYM offsetStart=0 offsetEnd=3
> term=anti posInc=1 posLen=1 type=word offsetStart=8 offsetEnd=12
> term=syn::ads posInc=0 posLen=4 type=SYNONYM offsetStart=8 offsetEnd=28
> term=acr::ads posInc=0 posLen=4 type=ACRONYM offsetStart=8 offsetEnd=28
> term=syn::anti de sitter space posInc=0 posLen=4 type=SYNONYM
> offsetStart=8 offsetEnd=28
> term=syn::antidesitter spacetime posInc=0 posLen=4 type=SYNONYM
> offsetStart=8 offsetEnd=28
> term=syn::antidesitter space posInc=0 posLen=4 type=SYNONYM
> offsetStart=8 offsetEnd=28
> * term=de posInc=1 posLen=1 type=word offsetStart=13 offsetEnd=15
> term=sitter posInc=1 posLen=1 type=word offsetStart=16 offsetEnd=22
> term=space posInc=1 posLen=1 type=word offsetStart=23 offsetEnd=28
> term=syn::universe posInc=0 posLen=1 type=SYNONYM offsetStart=23
> offsetEnd=28
> term=time posInc=1 posLen=1 type=word offsetStart=29 offsetEnd=33
> term=spacetime posInc=0 posLen=1 type=word offsetStart=23 offsetEnd=33
>
> So far, all of these cases could be handled with the new position
> length attribute. But let us look at a case where that would fail too.
>
> assertU(adoc("id", "606", "bibcode", "xx604",
> "title", "Massachusetts Institute of Technology and
> antidesitter space-time"));
>
>
> term=massachusetts posInc=1 posLen=1 type=word offsetStart=0 offsetEnd=12
> term=syn::massachusetts institute of technology posInc=0 posLen=4
> type=SYNONYM offsetStart=0 offsetEnd=36
> term=syn::mit posInc=0 posLen=4 type=SYNONYM offsetStart=0 offsetEnd=36
> term=acr::mit posInc=0 posLen=4 type=ACRONYM offsetStart=0 offsetEnd=36
> term=institute posInc=1 posLen=1 type=word offsetStart=13 offsetEnd=22
> term=technology posInc=1 posLen=1 type=word offsetStart=26 offsetEnd=36
> term=antidesitter posInc=1 posLen=1 type=word offsetStart=41 offsetEnd=53
> term=syn::ads posInc=0 posLen=2 type=SYNONYM offsetStart=41 offsetEnd=59
> term=acr::ads posInc=0 posLen=2 type=ACRONYM offsetStart=41 offsetEnd=59
> term=syn::anti de sitter space
Re: When zero offsets are not bad - a.k.a. multi-token synonyms yet again
Hi Mike,
Thanks for the question! And sorry for the delay, I haven't managed to
get to it yesterday. I have generated better output, marked with (*)
where it currently fails the first time and also included one extra
case to illustrate the PositionLength attribute.
assertU(adoc("id", "603", "bibcode", "xx603",
"title", "THE HUBBLE constant: a summary of the hubble space
telescope program"));
term=hubble posInc=2 posLen=1 type=word offsetStart=4 offsetEnd=10
term=acr::hubble posInc=0 posLen=1 type=ACRONYM offsetStart=4 offsetEnd=10
term=constant posInc=1 posLen=1 type=word offsetStart=11 offsetEnd=20
term=summary posInc=1 posLen=1 type=word offsetStart=23 offsetEnd=30
term=hubble posInc=1 posLen=1 type=word offsetStart=38 offsetEnd=44
term=syn::hubble space telescope posInc=0 posLen=3 type=SYNONYM
offsetStart=38 offsetEnd=60
term=syn::hst posInc=0 posLen=3 type=SYNONYM offsetStart=38 offsetEnd=60
term=acr::hst posInc=0 posLen=3 type=ACRONYM offsetStart=38 offsetEnd=60
* term=space posInc=1 posLen=1 type=word offsetStart=45 offsetEnd=50
term=telescope posInc=1 posLen=1 type=word offsetStart=51 offsetEnd=60
term=program posInc=1 posLen=1 type=word offsetStart=61 offsetEnd=68
* - fails because of offsetEnd < lastToken.offsetEnd; If reordered
(the multi-token synonym emitted as a last token) it would fail as
well, because of the check for lastToken.beginOffset <
currentToken.beginOffset. Basically, any reordering would result in a
failure (unless offsets are trimmed).
The following example has additional twist because of `space-time`;
the tokenizer first splits the word and generate two new tokens --
those alternative tokens are then used to find synonyms (space ==
universe)
assertU(adoc("id", "605", "bibcode", "xx604",
"title", "MIT and anti de sitter space-time"));
term=xx604 posInc=1 posLen=1 type=word offsetStart=0 offsetEnd=13
term=mit posInc=1 posLen=1 type=word offsetStart=0 offsetEnd=3
term=acr::mit posInc=0 posLen=1 type=ACRONYM offsetStart=0 offsetEnd=3
term=syn::massachusetts institute of technology posInc=0 posLen=1
type=SYNONYM offsetStart=0 offsetEnd=3
term=syn::mit posInc=0 posLen=1 type=SYNONYM offsetStart=0 offsetEnd=3
term=acr::mit posInc=0 posLen=1 type=ACRONYM offsetStart=0 offsetEnd=3
term=anti posInc=1 posLen=1 type=word offsetStart=8 offsetEnd=12
term=syn::ads posInc=0 posLen=4 type=SYNONYM offsetStart=8 offsetEnd=28
term=acr::ads posInc=0 posLen=4 type=ACRONYM offsetStart=8 offsetEnd=28
term=syn::anti de sitter space posInc=0 posLen=4 type=SYNONYM
offsetStart=8 offsetEnd=28
term=syn::antidesitter spacetime posInc=0 posLen=4 type=SYNONYM
offsetStart=8 offsetEnd=28
term=syn::antidesitter space posInc=0 posLen=4 type=SYNONYM
offsetStart=8 offsetEnd=28
* term=de posInc=1 posLen=1 type=word offsetStart=13 offsetEnd=15
term=sitter posInc=1 posLen=1 type=word offsetStart=16 offsetEnd=22
term=space posInc=1 posLen=1 type=word offsetStart=23 offsetEnd=28
term=syn::universe posInc=0 posLen=1 type=SYNONYM offsetStart=23 offsetEnd=28
term=time posInc=1 posLen=1 type=word offsetStart=29 offsetEnd=33
term=spacetime posInc=0 posLen=1 type=word offsetStart=23 offsetEnd=33
So far, all of these cases could be handled with the new position
length attribute. But let us look at a case where that would fail too.
assertU(adoc("id", "606", "bibcode", "xx604",
"title", "Massachusetts Institute of Technology and
antidesitter space-time"));
term=massachusetts posInc=1 posLen=1 type=word offsetStart=0 offsetEnd=12
term=syn::massachusetts institute of technology posInc=0 posLen=4
type=SYNONYM offsetStart=0 offsetEnd=36
term=syn::mit posInc=0 posLen=4 type=SYNONYM offsetStart=0 offsetEnd=36
term=acr::mit posInc=0 posLen=4 type=ACRONYM offsetStart=0 offsetEnd=36
term=institute posInc=1 posLen=1 type=word offsetStart=13 offsetEnd=22
term=technology posInc=1 posLen=1 type=word offsetStart=26 offsetEnd=36
term=antidesitter posInc=1 posLen=1 type=word offsetStart=41 offsetEnd=53
term=syn::ads posInc=0 posLen=2 type=SYNONYM offsetStart=41 offsetEnd=59
term=acr::ads posInc=0 posLen=2 type=ACRONYM offsetStart=41 offsetEnd=59
term=syn::anti de sitter space posInc=0 posLen=2 type=SYNONYM
offsetStart=41 offsetEnd=59
term=syn::antidesitter spacetime posInc=0 posLen=2 type=SYNONYM
offsetStart=41 offsetEnd=59
term=syn::antidesitter space posInc=0 posLen=2 type=SYNONYM
offsetStart=41 offsetEnd=59
term=space posInc=1 posLen=1 type=word offsetStart=54 offsetEnd=59
term=syn::universe posInc=0 posLen=1 type=SYNONYM offsetStart=54 offsetEnd=59
term=time posInc=1 posLen=1 type=word offsetStart=60 offsetEnd=64
term=spacetime posInc=0 posLen=1 type=word offsetStart=54 offsetEnd=64
Notice the posLen=4 of MIT; it would cover tokens `massachusetts
institute technology antidesitter` while offsets are still correct.
This would, I think, affect not only highlighting, but also search
(which is, at least for us, more important). But I can imagine that in
more NLP-related
Re: When zero offsets are not bad - a.k.a. multi-token synonyms yet again
Hi Roman, Sorry for the late reply! I think there remains substantial confusion about multi-token synonyms and IW's enforcement of offsets. It really is worth thoroughly iterating/understanding your examples so we can get to the bottom of this. It looks to me it is possible to emit tokens whose offsets do not go backwards and that properly model your example synonyms, so I do not yet see what the problem is. Maybe I am being blind/tired ... What do you mean by pos=2, pos=0, etc.? I think that is really the position increment? Can you re-do the examples with posInc instead? (Alternatively, you could keep "pos" but make it the absolute position, not the increment?). Could you also add posLength to each token? This helps (me?) visualize the resulting graph, even though IW does not enforce it today. Looking at your first example, "THE HUBBLE constant: a summary of the hubble space telescope program", it looks to me like those tokens would all be accepted by IW's checks as they are? startOffset never goes backwards, and for every token, endOffset >= startOffset. Where in that first example does IW throw an exception? Maybe insert a "** IW fails here" under the problematic token? Or, maybe write a simple test case using e.g. CannedTokenStream? Your second example should also be fine, and not at all weird, but could you enumerate it into the specific tokens with posInc, posLength, start/end offset, "** IW fails here", etc., so we have a concrete example to discuss? Lucene's TokenStreams are really serializing a directed acyclic graph (DAG), in a specific order, one transition at a time. Ironically/strangely, it is similar to the graph that git history maintains, and how "git log" then serializes that graph into an ordered series of transitions. The simple int position in Lucene's TokenStream corresponds to git's githashes, to uniquely identify each "node", though, I do not think there is an analog in git to Lucene's offsets. Hmm, maybe a timestamp? Mike McCandless http://blog.mikemccandless.com On Thu, Aug 6, 2020 at 10:49 AM Roman Chyla wrote: > Hi Mike, > > Yes, they are not zero offsets - I was instinctively avoiding > "negative offsets"; but they are indeed backward offsets. > > Here is the token stream as produced by the analyzer chain indexing > "THE HUBBLE constant: a summary of the hubble space telescope program" > > term=hubble pos=2 type=word offsetStart=4 offsetEnd=10 > term=acr::hubble pos=0 type=ACRONYM offsetStart=4 offsetEnd=10 > term=constant pos=1 type=word offsetStart=11 offsetEnd=20 > term=summary pos=1 type=word offsetStart=23 offsetEnd=30 > term=hubble pos=1 type=word offsetStart=38 offsetEnd=44 > term=syn::hubble space telescope pos=0 type=SYNONYM offsetStart=38 > offsetEnd=60 > term=syn::hst pos=0 type=SYNONYM offsetStart=38 offsetEnd=60 > term=acr::hst pos=0 type=ACRONYM offsetStart=38 offsetEnd=60 > term=space pos=1 type=word offsetStart=45 offsetEnd=50 > term=telescope pos=1 type=word offsetStart=51 offsetEnd=60 > term=program pos=1 type=word offsetStart=61 offsetEnd=68 > > Sometimes, we'll even have a situation when synonyms overlap: for > example "anti de sitter space time" > > "anti de sitter space time" -> "antidesitter space" (one token > spanning offsets 0-26; it gets emitted with the first token "anti" > right now) > "space time" -> "spacetime" (synonym 16-26) > "space" -> "universe" (25-26) > > Yes, weird, but useful if people want to search for `universe NEAR > anti` -- but another usecase which would be prohibited by the "new" > rule. > > DefaultIndexingChain checks new token offset against the last emitted > token, so I don't see a way to emit the multi-token synonym with > offsetts spanning multiple tokens if even one of these tokens was > already emitted. And the complement is equally true: if multi-token is > emitted as last of the group - it trips over `startOffset < > invertState.lastStartOffset` > > > https://github.com/apache/lucene-solr/blame/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L915 > > > -roman > > On Thu, Aug 6, 2020 at 6:17 AM Michael McCandless > wrote: > > > > Hi Roman, > > > > Hmm, this is all very tricky! > > > > First off, why do you call this "zero offsets"? Isn't it "backwards > offsets" that your analysis chain is trying to produce? > > > > Second, in your first example, if you output the tokens in the right > order, they would not violate the "offsets do not go backwards" check in > IndexWriter? I thought IndexWriter is just checking that the startOffset > for a token is not lower than the previous token's startOffset? (And that > the token's endOffset is not lower than its startOffset). > > > > So I am confused why your first example is tripping up on IW's offset > checks. Could you maybe redo the example, listing single token per line > with the start/end offsets they are producing? > > > > Mike McCandless > > > > http://blog.mikemccandless.com > > > > > > On Wed, Aug 5, 2020 at
Re: When zero offsets are not bad - a.k.a. multi-token synonyms yet again
oh,thanks! that saves everybody some time. I have commented in there, pleading to be allowed to do something - if that proposal sounds even little bit reasonable, please consider amplifying the signal On Mon, Aug 10, 2020 at 4:22 PM David Smiley wrote: > > There already is one: https://issues.apache.org/jira/browse/LUCENE-8776 > > ~ David Smiley > Apache Lucene/Solr Search Developer > http://www.linkedin.com/in/davidwsmiley > > > On Mon, Aug 10, 2020 at 1:30 PM Roman Chyla wrote: >> >> I'll have to somehow find a solution for this situation, giving up >> offsets seems like too big a price to pay, I see that overriding >> DefaultIndexingChain is not exactly easy -- the only thing I can think >> of is to just trick the classloader into giving it a different version >> of the chain (praying this can be done without compromising security, >> I have not followed JDK evolutions for some time...) - aside from >> forking lucene and editing that; which I decidedly don't want to do >> (monkey-patching it, ok, i can live with that... :-)) >> >> It *seems* to me that the original reason for negative offset checks >> stemmed from the fact that vint could have been written (and possibly >> vlong too) - https://issues.apache.org/jira/browse/LUCENE-3738 >> >> but the underlying issue and some of the patches seem to have been >> addressing those problems; but a much shorter version of the patch was >> committed -- despite the perf results not being indicative (i.e. it >> could have been good with the longer patch) -- but to really >> understand it, one would have to spend more than 10mins reading the >> comments >> >> Further to the point, I think negative offsets can be produced only on >> the very first token, unless there is a bug in a filter (there was/is >> a separate check for that in 6x and perhaps it is still there in 7x). >> That would be much less restrictive than the current condition which >> disallows all backward offsets. We never ran into an index corruption >> in lucene 4-6x, so I really wonder if the "forbid all backwards >> offsets" approach might be too restrictive. >> >> Looks like I should create an issue... >> >> On Thu, Aug 6, 2020 at 11:28 AM Gus Heck wrote: >> > >> > I've had a nearly identical experience to what Dave describes, I also >> > chafe under this restriction. >> > >> > On Thu, Aug 6, 2020 at 11:07 AM David Smiley wrote: >> >> >> >> I sympathize with your pain, Roman. >> >> >> >> It appears we can't really do index-time multi-word synonyms because of >> >> the offset ordering rule. But it's not just synonyms, it's other forms >> >> of multi-token expansion. Where I work, I've seen an interesting >> >> approach to mixed language text analysis in which a sophisticated >> >> Tokenizer effectively re-tokenizes an input multiple ways by producing a >> >> token stream that is a concatenation of different interpretations of the >> >> input. On a Lucene upgrade, we had to "coarsen" the offsets to the point >> >> of having highlights that point to a whole sentence instead of the words >> >> in that sentence :-(. I need to do something to fix this; I'm trying >> >> hard to resist modifying our Lucene fork for this constraint. Maybe >> >> instead of concatenating, it might be interleaved / overlapped but the >> >> interpretations aren't necessarily aligned to make this possible without >> >> risking breaking position-sensitive queries. >> >> >> >> So... I'm not a fan of this constraint on offsets. >> >> >> >> ~ David Smiley >> >> Apache Lucene/Solr Search Developer >> >> http://www.linkedin.com/in/davidwsmiley >> >> >> >> >> >> On Thu, Aug 6, 2020 at 10:49 AM Roman Chyla wrote: >> >>> >> >>> Hi Mike, >> >>> >> >>> Yes, they are not zero offsets - I was instinctively avoiding >> >>> "negative offsets"; but they are indeed backward offsets. >> >>> >> >>> Here is the token stream as produced by the analyzer chain indexing >> >>> "THE HUBBLE constant: a summary of the hubble space telescope program" >> >>> >> >>> term=hubble pos=2 type=word offsetStart=4 offsetEnd=10 >> >>> term=acr::hubble pos=0 type=ACRONYM offsetStart=4 offsetEnd=10 >> >>> term=constant pos=1 type=word offsetStart=11 offsetEnd=20 >> >>> term=summary pos=1 type=word offsetStart=23 offsetEnd=30 >> >>> term=hubble pos=1 type=word offsetStart=38 offsetEnd=44 >> >>> term=syn::hubble space telescope pos=0 type=SYNONYM offsetStart=38 >> >>> offsetEnd=60 >> >>> term=syn::hst pos=0 type=SYNONYM offsetStart=38 offsetEnd=60 >> >>> term=acr::hst pos=0 type=ACRONYM offsetStart=38 offsetEnd=60 >> >>> term=space pos=1 type=word offsetStart=45 offsetEnd=50 >> >>> term=telescope pos=1 type=word offsetStart=51 offsetEnd=60 >> >>> term=program pos=1 type=word offsetStart=61 offsetEnd=68 >> >>> >> >>> Sometimes, we'll even have a situation when synonyms overlap: for >> >>> example "anti de sitter space time" >> >>> >> >>> "anti de sitter space time" -> "antidesitter space" (one token >> >>> spanning offsets 0-26; it gets
Re: When zero offsets are not bad - a.k.a. multi-token synonyms yet again
There already is one: https://issues.apache.org/jira/browse/LUCENE-8776 ~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Mon, Aug 10, 2020 at 1:30 PM Roman Chyla wrote: > I'll have to somehow find a solution for this situation, giving up > offsets seems like too big a price to pay, I see that overriding > DefaultIndexingChain is not exactly easy -- the only thing I can think > of is to just trick the classloader into giving it a different version > of the chain (praying this can be done without compromising security, > I have not followed JDK evolutions for some time...) - aside from > forking lucene and editing that; which I decidedly don't want to do > (monkey-patching it, ok, i can live with that... :-)) > > It *seems* to me that the original reason for negative offset checks > stemmed from the fact that vint could have been written (and possibly > vlong too) - https://issues.apache.org/jira/browse/LUCENE-3738 > > but the underlying issue and some of the patches seem to have been > addressing those problems; but a much shorter version of the patch was > committed -- despite the perf results not being indicative (i.e. it > could have been good with the longer patch) -- but to really > understand it, one would have to spend more than 10mins reading the > comments > > Further to the point, I think negative offsets can be produced only on > the very first token, unless there is a bug in a filter (there was/is > a separate check for that in 6x and perhaps it is still there in 7x). > That would be much less restrictive than the current condition which > disallows all backward offsets. We never ran into an index corruption > in lucene 4-6x, so I really wonder if the "forbid all backwards > offsets" approach might be too restrictive. > > Looks like I should create an issue... > > On Thu, Aug 6, 2020 at 11:28 AM Gus Heck wrote: > > > > I've had a nearly identical experience to what Dave describes, I also > chafe under this restriction. > > > > On Thu, Aug 6, 2020 at 11:07 AM David Smiley wrote: > >> > >> I sympathize with your pain, Roman. > >> > >> It appears we can't really do index-time multi-word synonyms because of > the offset ordering rule. But it's not just synonyms, it's other forms of > multi-token expansion. Where I work, I've seen an interesting approach to > mixed language text analysis in which a sophisticated Tokenizer effectively > re-tokenizes an input multiple ways by producing a token stream that is a > concatenation of different interpretations of the input. On a Lucene > upgrade, we had to "coarsen" the offsets to the point of having highlights > that point to a whole sentence instead of the words in that sentence :-(. > I need to do something to fix this; I'm trying hard to resist modifying our > Lucene fork for this constraint. Maybe instead of concatenating, it might > be interleaved / overlapped but the interpretations aren't necessarily > aligned to make this possible without risking breaking position-sensitive > queries. > >> > >> So... I'm not a fan of this constraint on offsets. > >> > >> ~ David Smiley > >> Apache Lucene/Solr Search Developer > >> http://www.linkedin.com/in/davidwsmiley > >> > >> > >> On Thu, Aug 6, 2020 at 10:49 AM Roman Chyla > wrote: > >>> > >>> Hi Mike, > >>> > >>> Yes, they are not zero offsets - I was instinctively avoiding > >>> "negative offsets"; but they are indeed backward offsets. > >>> > >>> Here is the token stream as produced by the analyzer chain indexing > >>> "THE HUBBLE constant: a summary of the hubble space telescope program" > >>> > >>> term=hubble pos=2 type=word offsetStart=4 offsetEnd=10 > >>> term=acr::hubble pos=0 type=ACRONYM offsetStart=4 offsetEnd=10 > >>> term=constant pos=1 type=word offsetStart=11 offsetEnd=20 > >>> term=summary pos=1 type=word offsetStart=23 offsetEnd=30 > >>> term=hubble pos=1 type=word offsetStart=38 offsetEnd=44 > >>> term=syn::hubble space telescope pos=0 type=SYNONYM offsetStart=38 > offsetEnd=60 > >>> term=syn::hst pos=0 type=SYNONYM offsetStart=38 offsetEnd=60 > >>> term=acr::hst pos=0 type=ACRONYM offsetStart=38 offsetEnd=60 > >>> term=space pos=1 type=word offsetStart=45 offsetEnd=50 > >>> term=telescope pos=1 type=word offsetStart=51 offsetEnd=60 > >>> term=program pos=1 type=word offsetStart=61 offsetEnd=68 > >>> > >>> Sometimes, we'll even have a situation when synonyms overlap: for > >>> example "anti de sitter space time" > >>> > >>> "anti de sitter space time" -> "antidesitter space" (one token > >>> spanning offsets 0-26; it gets emitted with the first token "anti" > >>> right now) > >>> "space time" -> "spacetime" (synonym 16-26) > >>> "space" -> "universe" (25-26) > >>> > >>> Yes, weird, but useful if people want to search for `universe NEAR > >>> anti` -- but another usecase which would be prohibited by the "new" > >>> rule. > >>> > >>> DefaultIndexingChain checks new token offset against the last emitted > >>> token, so I don't see a way
Re: When zero offsets are not bad - a.k.a. multi-token synonyms yet again
I'll have to somehow find a solution for this situation, giving up offsets seems like too big a price to pay, I see that overriding DefaultIndexingChain is not exactly easy -- the only thing I can think of is to just trick the classloader into giving it a different version of the chain (praying this can be done without compromising security, I have not followed JDK evolutions for some time...) - aside from forking lucene and editing that; which I decidedly don't want to do (monkey-patching it, ok, i can live with that... :-)) It *seems* to me that the original reason for negative offset checks stemmed from the fact that vint could have been written (and possibly vlong too) - https://issues.apache.org/jira/browse/LUCENE-3738 but the underlying issue and some of the patches seem to have been addressing those problems; but a much shorter version of the patch was committed -- despite the perf results not being indicative (i.e. it could have been good with the longer patch) -- but to really understand it, one would have to spend more than 10mins reading the comments Further to the point, I think negative offsets can be produced only on the very first token, unless there is a bug in a filter (there was/is a separate check for that in 6x and perhaps it is still there in 7x). That would be much less restrictive than the current condition which disallows all backward offsets. We never ran into an index corruption in lucene 4-6x, so I really wonder if the "forbid all backwards offsets" approach might be too restrictive. Looks like I should create an issue... On Thu, Aug 6, 2020 at 11:28 AM Gus Heck wrote: > > I've had a nearly identical experience to what Dave describes, I also chafe > under this restriction. > > On Thu, Aug 6, 2020 at 11:07 AM David Smiley wrote: >> >> I sympathize with your pain, Roman. >> >> It appears we can't really do index-time multi-word synonyms because of the >> offset ordering rule. But it's not just synonyms, it's other forms of >> multi-token expansion. Where I work, I've seen an interesting approach to >> mixed language text analysis in which a sophisticated Tokenizer effectively >> re-tokenizes an input multiple ways by producing a token stream that is a >> concatenation of different interpretations of the input. On a Lucene >> upgrade, we had to "coarsen" the offsets to the point of having highlights >> that point to a whole sentence instead of the words in that sentence :-(. I >> need to do something to fix this; I'm trying hard to resist modifying our >> Lucene fork for this constraint. Maybe instead of concatenating, it might >> be interleaved / overlapped but the interpretations aren't necessarily >> aligned to make this possible without risking breaking position-sensitive >> queries. >> >> So... I'm not a fan of this constraint on offsets. >> >> ~ David Smiley >> Apache Lucene/Solr Search Developer >> http://www.linkedin.com/in/davidwsmiley >> >> >> On Thu, Aug 6, 2020 at 10:49 AM Roman Chyla wrote: >>> >>> Hi Mike, >>> >>> Yes, they are not zero offsets - I was instinctively avoiding >>> "negative offsets"; but they are indeed backward offsets. >>> >>> Here is the token stream as produced by the analyzer chain indexing >>> "THE HUBBLE constant: a summary of the hubble space telescope program" >>> >>> term=hubble pos=2 type=word offsetStart=4 offsetEnd=10 >>> term=acr::hubble pos=0 type=ACRONYM offsetStart=4 offsetEnd=10 >>> term=constant pos=1 type=word offsetStart=11 offsetEnd=20 >>> term=summary pos=1 type=word offsetStart=23 offsetEnd=30 >>> term=hubble pos=1 type=word offsetStart=38 offsetEnd=44 >>> term=syn::hubble space telescope pos=0 type=SYNONYM offsetStart=38 >>> offsetEnd=60 >>> term=syn::hst pos=0 type=SYNONYM offsetStart=38 offsetEnd=60 >>> term=acr::hst pos=0 type=ACRONYM offsetStart=38 offsetEnd=60 >>> term=space pos=1 type=word offsetStart=45 offsetEnd=50 >>> term=telescope pos=1 type=word offsetStart=51 offsetEnd=60 >>> term=program pos=1 type=word offsetStart=61 offsetEnd=68 >>> >>> Sometimes, we'll even have a situation when synonyms overlap: for >>> example "anti de sitter space time" >>> >>> "anti de sitter space time" -> "antidesitter space" (one token >>> spanning offsets 0-26; it gets emitted with the first token "anti" >>> right now) >>> "space time" -> "spacetime" (synonym 16-26) >>> "space" -> "universe" (25-26) >>> >>> Yes, weird, but useful if people want to search for `universe NEAR >>> anti` -- but another usecase which would be prohibited by the "new" >>> rule. >>> >>> DefaultIndexingChain checks new token offset against the last emitted >>> token, so I don't see a way to emit the multi-token synonym with >>> offsetts spanning multiple tokens if even one of these tokens was >>> already emitted. And the complement is equally true: if multi-token is >>> emitted as last of the group - it trips over `startOffset < >>> invertState.lastStartOffset` >>> >>>
Re: When zero offsets are not bad - a.k.a. multi-token synonyms yet again
I've had a nearly identical experience to what Dave describes, I also chafe under this restriction. On Thu, Aug 6, 2020 at 11:07 AM David Smiley wrote: > I sympathize with your pain, Roman. > > It appears we can't really do index-time multi-word synonyms because of > the offset ordering rule. But it's not just synonyms, it's other forms of > multi-token expansion. Where I work, I've seen an interesting approach to > mixed language text analysis in which a sophisticated Tokenizer effectively > re-tokenizes an input multiple ways by producing a token stream that is a > concatenation of different interpretations of the input. On a Lucene > upgrade, we had to "coarsen" the offsets to the point of having highlights > that point to a whole sentence instead of the words in that sentence :-(. > I need to do something to fix this; I'm trying hard to resist modifying our > Lucene fork for this constraint. Maybe instead of concatenating, it might > be interleaved / overlapped but the interpretations aren't necessarily > aligned to make this possible without risking breaking position-sensitive > queries. > > So... I'm not a fan of this constraint on offsets. > > ~ David Smiley > Apache Lucene/Solr Search Developer > http://www.linkedin.com/in/davidwsmiley > > > On Thu, Aug 6, 2020 at 10:49 AM Roman Chyla wrote: > >> Hi Mike, >> >> Yes, they are not zero offsets - I was instinctively avoiding >> "negative offsets"; but they are indeed backward offsets. >> >> Here is the token stream as produced by the analyzer chain indexing >> "THE HUBBLE constant: a summary of the hubble space telescope program" >> >> term=hubble pos=2 type=word offsetStart=4 offsetEnd=10 >> term=acr::hubble pos=0 type=ACRONYM offsetStart=4 offsetEnd=10 >> term=constant pos=1 type=word offsetStart=11 offsetEnd=20 >> term=summary pos=1 type=word offsetStart=23 offsetEnd=30 >> term=hubble pos=1 type=word offsetStart=38 offsetEnd=44 >> term=syn::hubble space telescope pos=0 type=SYNONYM offsetStart=38 >> offsetEnd=60 >> term=syn::hst pos=0 type=SYNONYM offsetStart=38 offsetEnd=60 >> term=acr::hst pos=0 type=ACRONYM offsetStart=38 offsetEnd=60 >> term=space pos=1 type=word offsetStart=45 offsetEnd=50 >> term=telescope pos=1 type=word offsetStart=51 offsetEnd=60 >> term=program pos=1 type=word offsetStart=61 offsetEnd=68 >> >> Sometimes, we'll even have a situation when synonyms overlap: for >> example "anti de sitter space time" >> >> "anti de sitter space time" -> "antidesitter space" (one token >> spanning offsets 0-26; it gets emitted with the first token "anti" >> right now) >> "space time" -> "spacetime" (synonym 16-26) >> "space" -> "universe" (25-26) >> >> Yes, weird, but useful if people want to search for `universe NEAR >> anti` -- but another usecase which would be prohibited by the "new" >> rule. >> >> DefaultIndexingChain checks new token offset against the last emitted >> token, so I don't see a way to emit the multi-token synonym with >> offsetts spanning multiple tokens if even one of these tokens was >> already emitted. And the complement is equally true: if multi-token is >> emitted as last of the group - it trips over `startOffset < >> invertState.lastStartOffset` >> >> >> https://github.com/apache/lucene-solr/blame/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L915 >> >> >> -roman >> >> On Thu, Aug 6, 2020 at 6:17 AM Michael McCandless >> wrote: >> > >> > Hi Roman, >> > >> > Hmm, this is all very tricky! >> > >> > First off, why do you call this "zero offsets"? Isn't it "backwards >> offsets" that your analysis chain is trying to produce? >> > >> > Second, in your first example, if you output the tokens in the right >> order, they would not violate the "offsets do not go backwards" check in >> IndexWriter? I thought IndexWriter is just checking that the startOffset >> for a token is not lower than the previous token's startOffset? (And that >> the token's endOffset is not lower than its startOffset). >> > >> > So I am confused why your first example is tripping up on IW's offset >> checks. Could you maybe redo the example, listing single token per line >> with the start/end offsets they are producing? >> > >> > Mike McCandless >> > >> > http://blog.mikemccandless.com >> > >> > >> > On Wed, Aug 5, 2020 at 6:41 PM Roman Chyla >> wrote: >> >> >> >> Hello devs, >> >> >> >> I wanted to create an issue but the helpful message in red letters >> >> reminded me to ask first. >> >> >> >> While porting from lucene 6.x to 7x I'm struggling with a change that >> >> was introduced in LUCENE-7626 >> >> (https://issues.apache.org/jira/browse/LUCENE-7626) >> >> >> >> It is believed that zero offset tokens are bad bad - Mike McCandles >> >> made the change which made me automatically doubt myself. I must be >> >> wrong, hell, I was living in sin the past 5 years! >> >> >> >> Sadly, we have been indexing and searching large volumes of data >> >> without any corruption in index whatsover, but also without
Re: When zero offsets are not bad - a.k.a. multi-token synonyms yet again
I sympathize with your pain, Roman. It appears we can't really do index-time multi-word synonyms because of the offset ordering rule. But it's not just synonyms, it's other forms of multi-token expansion. Where I work, I've seen an interesting approach to mixed language text analysis in which a sophisticated Tokenizer effectively re-tokenizes an input multiple ways by producing a token stream that is a concatenation of different interpretations of the input. On a Lucene upgrade, we had to "coarsen" the offsets to the point of having highlights that point to a whole sentence instead of the words in that sentence :-(. I need to do something to fix this; I'm trying hard to resist modifying our Lucene fork for this constraint. Maybe instead of concatenating, it might be interleaved / overlapped but the interpretations aren't necessarily aligned to make this possible without risking breaking position-sensitive queries. So... I'm not a fan of this constraint on offsets. ~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Thu, Aug 6, 2020 at 10:49 AM Roman Chyla wrote: > Hi Mike, > > Yes, they are not zero offsets - I was instinctively avoiding > "negative offsets"; but they are indeed backward offsets. > > Here is the token stream as produced by the analyzer chain indexing > "THE HUBBLE constant: a summary of the hubble space telescope program" > > term=hubble pos=2 type=word offsetStart=4 offsetEnd=10 > term=acr::hubble pos=0 type=ACRONYM offsetStart=4 offsetEnd=10 > term=constant pos=1 type=word offsetStart=11 offsetEnd=20 > term=summary pos=1 type=word offsetStart=23 offsetEnd=30 > term=hubble pos=1 type=word offsetStart=38 offsetEnd=44 > term=syn::hubble space telescope pos=0 type=SYNONYM offsetStart=38 > offsetEnd=60 > term=syn::hst pos=0 type=SYNONYM offsetStart=38 offsetEnd=60 > term=acr::hst pos=0 type=ACRONYM offsetStart=38 offsetEnd=60 > term=space pos=1 type=word offsetStart=45 offsetEnd=50 > term=telescope pos=1 type=word offsetStart=51 offsetEnd=60 > term=program pos=1 type=word offsetStart=61 offsetEnd=68 > > Sometimes, we'll even have a situation when synonyms overlap: for > example "anti de sitter space time" > > "anti de sitter space time" -> "antidesitter space" (one token > spanning offsets 0-26; it gets emitted with the first token "anti" > right now) > "space time" -> "spacetime" (synonym 16-26) > "space" -> "universe" (25-26) > > Yes, weird, but useful if people want to search for `universe NEAR > anti` -- but another usecase which would be prohibited by the "new" > rule. > > DefaultIndexingChain checks new token offset against the last emitted > token, so I don't see a way to emit the multi-token synonym with > offsetts spanning multiple tokens if even one of these tokens was > already emitted. And the complement is equally true: if multi-token is > emitted as last of the group - it trips over `startOffset < > invertState.lastStartOffset` > > > https://github.com/apache/lucene-solr/blame/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L915 > > > -roman > > On Thu, Aug 6, 2020 at 6:17 AM Michael McCandless > wrote: > > > > Hi Roman, > > > > Hmm, this is all very tricky! > > > > First off, why do you call this "zero offsets"? Isn't it "backwards > offsets" that your analysis chain is trying to produce? > > > > Second, in your first example, if you output the tokens in the right > order, they would not violate the "offsets do not go backwards" check in > IndexWriter? I thought IndexWriter is just checking that the startOffset > for a token is not lower than the previous token's startOffset? (And that > the token's endOffset is not lower than its startOffset). > > > > So I am confused why your first example is tripping up on IW's offset > checks. Could you maybe redo the example, listing single token per line > with the start/end offsets they are producing? > > > > Mike McCandless > > > > http://blog.mikemccandless.com > > > > > > On Wed, Aug 5, 2020 at 6:41 PM Roman Chyla > wrote: > >> > >> Hello devs, > >> > >> I wanted to create an issue but the helpful message in red letters > >> reminded me to ask first. > >> > >> While porting from lucene 6.x to 7x I'm struggling with a change that > >> was introduced in LUCENE-7626 > >> (https://issues.apache.org/jira/browse/LUCENE-7626) > >> > >> It is believed that zero offset tokens are bad bad - Mike McCandles > >> made the change which made me automatically doubt myself. I must be > >> wrong, hell, I was living in sin the past 5 years! > >> > >> Sadly, we have been indexing and searching large volumes of data > >> without any corruption in index whatsover, but also without this new > >> change: > >> > >> > https://github.com/apache/lucene-solr/commit/64b86331c29d074fa7b257d65d3fda3b662bf96a#diff-cbdbb154cb6f3553edff2fcdb914a0c2L774 > >> > >> With that change, our multi-token synonyms house of cards is falling. > >> > >> Mike has this wonderful blogpost
Re: When zero offsets are not bad - a.k.a. multi-token synonyms yet again
Hi Mike,
Yes, they are not zero offsets - I was instinctively avoiding
"negative offsets"; but they are indeed backward offsets.
Here is the token stream as produced by the analyzer chain indexing
"THE HUBBLE constant: a summary of the hubble space telescope program"
term=hubble pos=2 type=word offsetStart=4 offsetEnd=10
term=acr::hubble pos=0 type=ACRONYM offsetStart=4 offsetEnd=10
term=constant pos=1 type=word offsetStart=11 offsetEnd=20
term=summary pos=1 type=word offsetStart=23 offsetEnd=30
term=hubble pos=1 type=word offsetStart=38 offsetEnd=44
term=syn::hubble space telescope pos=0 type=SYNONYM offsetStart=38 offsetEnd=60
term=syn::hst pos=0 type=SYNONYM offsetStart=38 offsetEnd=60
term=acr::hst pos=0 type=ACRONYM offsetStart=38 offsetEnd=60
term=space pos=1 type=word offsetStart=45 offsetEnd=50
term=telescope pos=1 type=word offsetStart=51 offsetEnd=60
term=program pos=1 type=word offsetStart=61 offsetEnd=68
Sometimes, we'll even have a situation when synonyms overlap: for
example "anti de sitter space time"
"anti de sitter space time" -> "antidesitter space" (one token
spanning offsets 0-26; it gets emitted with the first token "anti"
right now)
"space time" -> "spacetime" (synonym 16-26)
"space" -> "universe" (25-26)
Yes, weird, but useful if people want to search for `universe NEAR
anti` -- but another usecase which would be prohibited by the "new"
rule.
DefaultIndexingChain checks new token offset against the last emitted
token, so I don't see a way to emit the multi-token synonym with
offsetts spanning multiple tokens if even one of these tokens was
already emitted. And the complement is equally true: if multi-token is
emitted as last of the group - it trips over `startOffset <
invertState.lastStartOffset`
https://github.com/apache/lucene-solr/blame/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L915
-roman
On Thu, Aug 6, 2020 at 6:17 AM Michael McCandless
wrote:
>
> Hi Roman,
>
> Hmm, this is all very tricky!
>
> First off, why do you call this "zero offsets"? Isn't it "backwards offsets"
> that your analysis chain is trying to produce?
>
> Second, in your first example, if you output the tokens in the right order,
> they would not violate the "offsets do not go backwards" check in
> IndexWriter? I thought IndexWriter is just checking that the startOffset for
> a token is not lower than the previous token's startOffset? (And that the
> token's endOffset is not lower than its startOffset).
>
> So I am confused why your first example is tripping up on IW's offset checks.
> Could you maybe redo the example, listing single token per line with the
> start/end offsets they are producing?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Wed, Aug 5, 2020 at 6:41 PM Roman Chyla wrote:
>>
>> Hello devs,
>>
>> I wanted to create an issue but the helpful message in red letters
>> reminded me to ask first.
>>
>> While porting from lucene 6.x to 7x I'm struggling with a change that
>> was introduced in LUCENE-7626
>> (https://issues.apache.org/jira/browse/LUCENE-7626)
>>
>> It is believed that zero offset tokens are bad bad - Mike McCandles
>> made the change which made me automatically doubt myself. I must be
>> wrong, hell, I was living in sin the past 5 years!
>>
>> Sadly, we have been indexing and searching large volumes of data
>> without any corruption in index whatsover, but also without this new
>> change:
>>
>> https://github.com/apache/lucene-solr/commit/64b86331c29d074fa7b257d65d3fda3b662bf96a#diff-cbdbb154cb6f3553edff2fcdb914a0c2L774
>>
>> With that change, our multi-token synonyms house of cards is falling.
>>
>> Mike has this wonderful blogpost explaining troubles with multi-token
>> synonyms:
>> http://blog.mikemccandless.com/2012/04/lucenes-tokenstreams-are-actually.html
>>
>> Recommended way to index multi-token synonyms appears to be this:
>> https://stackoverflow.com/questions/19927537/multi-word-synonyms-in-solr
>>
>> BUT, but! We don't want to place multi-token synonym into the same
>> position as the other words. We want to preserve their positions! We
>> want to preserve informaiton about offsets!
>>
>> Here is an example:
>>
>> * THE HUBBLE constant: a summary of the HUBBLE SPACE TELESCOPE program
>>
>> This is how it gets indexed
>>
>> [(0, []),
>> (1, ['acr::hubble']),
>> (2, ['constant']),
>> (3, ['summary']),
>> (4, []),
>> (5, ['acr::hubble', 'syn::hst', 'syn::hubble space telescope', 'hubble'']),
>> (6, ['acr::space', 'space']),
>> (7, ['acr::telescope', 'telescope']),
>> (8, ['program']),
>>
>> Notice the position 5 - multi-token synonym `syn::hubble space
>> telescope` token is on the first token which started the group
>> (emitted by Lucene's synonym filter). hst is another synonym; we also
>> index the 'hubble' word there.
>>
>> if you were to search for a phrase "HST program" it will be found
>> because our search parser will search for ("HST ? ? program" | "Hubble
>> Space Telescope program")
>>
Re: When zero offsets are not bad - a.k.a. multi-token synonyms yet again
Hi Roman,
Hmm, this is all very tricky!
First off, why do you call this "zero offsets"? Isn't it "backwards
offsets" that your analysis chain is trying to produce?
Second, in your first example, if you output the tokens in the right order,
they would not violate the "offsets do not go backwards" check in
IndexWriter? I thought IndexWriter is just checking that the startOffset
for a token is not lower than the previous token's startOffset? (And that
the token's endOffset is not lower than its startOffset).
So I am confused why your first example is tripping up on IW's offset
checks. Could you maybe redo the example, listing single token per line
with the start/end offsets they are producing?
Mike McCandless
http://blog.mikemccandless.com
On Wed, Aug 5, 2020 at 6:41 PM Roman Chyla wrote:
> Hello devs,
>
> I wanted to create an issue but the helpful message in red letters
> reminded me to ask first.
>
> While porting from lucene 6.x to 7x I'm struggling with a change that
> was introduced in LUCENE-7626
> (https://issues.apache.org/jira/browse/LUCENE-7626)
>
> It is believed that zero offset tokens are bad bad - Mike McCandles
> made the change which made me automatically doubt myself. I must be
> wrong, hell, I was living in sin the past 5 years!
>
> Sadly, we have been indexing and searching large volumes of data
> without any corruption in index whatsover, but also without this new
> change:
>
>
> https://github.com/apache/lucene-solr/commit/64b86331c29d074fa7b257d65d3fda3b662bf96a#diff-cbdbb154cb6f3553edff2fcdb914a0c2L774
>
> With that change, our multi-token synonyms house of cards is falling.
>
> Mike has this wonderful blogpost explaining troubles with multi-token
> synonyms:
>
> http://blog.mikemccandless.com/2012/04/lucenes-tokenstreams-are-actually.html
>
> Recommended way to index multi-token synonyms appears to be this:
> https://stackoverflow.com/questions/19927537/multi-word-synonyms-in-solr
>
> BUT, but! We don't want to place multi-token synonym into the same
> position as the other words. We want to preserve their positions! We
> want to preserve informaiton about offsets!
>
> Here is an example:
>
> * THE HUBBLE constant: a summary of the HUBBLE SPACE TELESCOPE program
>
> This is how it gets indexed
>
> [(0, []),
> (1, ['acr::hubble']),
> (2, ['constant']),
> (3, ['summary']),
> (4, []),
> (5, ['acr::hubble', 'syn::hst', 'syn::hubble space telescope', 'hubble'']),
> (6, ['acr::space', 'space']),
> (7, ['acr::telescope', 'telescope']),
> (8, ['program']),
>
> Notice the position 5 - multi-token synonym `syn::hubble space
> telescope` token is on the first token which started the group
> (emitted by Lucene's synonym filter). hst is another synonym; we also
> index the 'hubble' word there.
>
> if you were to search for a phrase "HST program" it will be found
> because our search parser will search for ("HST ? ? program" | "Hubble
> Space Telescope program")
>
> It simply found that by looking at synonyms: HST -> Hubble Space Telescope
>
> And because of those funny 'syn::' prefixes, we don't suffer from the
> other problem that Mike described -- "hst space" phrase search will
> NOT find this paper (and that is a correct behaviour)
>
> But all of this is possible only because lucene was indexing tokens
> with offsets that can be lower than the last emitted token; for
> example 'hubble space telescope' wil have offset 21-45; and the next
> emitted token "space" will have offset 28-33
>
> And it just works (lucene 6.x)
>
> Here is another proof with the appropriate verbiage ("crazy"):
>
>
> https://github.com/romanchyla/montysolr/blob/master/contrib/adsabs/src/test/org/apache/solr/analysis/TestAdsabsTypeFulltextParsing.java#L618
>
> Zero offsets have been working wonderfully for us so far. And I
> actually cannot imagine how it can work without them - i.e. without
> the ability to emit a token stream with offsets that are lower than
> the last seen token.
>
> I haven't tried SynonymFlatten filter, but because of this line in the
> DefaultIndexingChain - I'm convinced the flatten symbol is not going
> to do what we need (as seen in the example above)
>
>
> https://github.com/apache/lucene-solr/blame/master/lucene/core/src/java/org/apache/lucene/index/DefaultIndexingChain.java#L915
>
> What would you say? Is it a bug, is it not a bug but just some special
> usecase? If it is a special usecase, what do we need to do? Plug in
> our own indexing chain?
>
> Thanks!
>
> -roman
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>
