mikemccand commented on PR #12875: URL: https://github.com/apache/lucene/pull/12875#issuecomment-1845294522
Thanks for tackling this @lukas-vlcek and @msfroh! I left a couple small comments, but otherwise it looks great. Given that this alters the indexed tokens (makes them non-overlapping), I think this should be a 10.0 only change? It's highly unlikely any users are relying on how `PhraseQuery` or synonyms, etc. (any positional tokenfilters / queries) behave on fieleds analyzed with these tokenizers, but, still, could you add an entry in `MIGRATE.txt` explaining this change? Note that this should make highlighting based on postings offsets (e.g. `UnifiedHighlighter`, in certain modes) work on such fields when it does not today. Also please add a `CHANGES.txt` entry under 10.0 as well. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
