[ https://issues.apache.org/jira/browse/SOLR-5426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13885364#comment-13885364 ]
Arun Kumar commented on SOLR-5426: ---------------------------------- I did investigate this issue and found that it is a desired behavior of the highlighter. The default chars to analyze for highlighter is hard coded and set to 51200 chars max. DEFAULT_MAX_CHARS_TO_ANALYZE = 50*1024; But the document you indexed has more characters to analyze in one of the field value that causes this issue. But there is a way to increase this max chars to analyze by sending an additional query param like this hl.maxAnalyzedChars=52300 So if I fire a query as below then the error is not seen. http://localhost:8983/solr/global/select?q=aa&indent=true&hl=true&hl.fl=*_stx&hl.simple.pre=%3Cem%3E&hl.simple.post=%3C/em%3E&hl.maxAnalyzedChars=52300 Hope this helps. > org.apache.solr.common.SolrException; org.apache.solr.common.SolrException: > org.apache.lucene.search.highlight.InvalidTokenOffsetsException: Token 0 > exceeds length of provided text sized 840 > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: SOLR-5426 > URL: https://issues.apache.org/jira/browse/SOLR-5426 > Project: Solr > Issue Type: Bug > Components: highlighter > Affects Versions: 4.4, 4.5.1 > Reporter: Nikolay > Priority: Minor > Attachments: highlighter.zip > > > Highlighter does not work correctly on test-data. > I added index- and config- files (see attached highlighter.zip) for > reproducing this issue. > Everything works fine if I search without highlighting: > http://localhost:8983/solr/global/select?q=aa&wt=json&indent=true > But if search with highlighting: > http://localhost:8983/solr/global/select?q=aa&wt=json&indent=true&hl=true&hl.fl=*_stx&hl.simple.pre=<em>&hl.simple.post=<%2Fem> > I'm get the error: > ERROR - 2013-11-07 10:17:15.797; org.apache.solr.common.SolrException; > null:org.apache.solr.common.SolrException: > org.apache.lucene.search.highlight.InvalidTokenOffsetsException: Token 0 > exceeds length of provided text sized 840 > at > org.apache.solr.highlight.DefaultSolrHighlighter.doHighlightingByHighlighter(DefaultSolrHighlighter.java:542) > at > org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(DefaultSolrHighlighter.java:414) > at > org.apache.solr.handler.component.HighlightComponent.process(HighlightComponent.java:139) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:703) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:406) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > at org.eclipse.jetty.server.Server.handle(Server.java:368) > at > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) > at > org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53) > at > org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942) > at > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004) > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640) > at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) > at > org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72) > at > org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) > at java.lang.Thread.run(Unknown Source) > Caused by: org.apache.lucene.search.highlight.InvalidTokenOffsetsException: > Token 0 exceeds length of provided text sized 840 > at > org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:225) > at > org.apache.solr.highlight.DefaultSolrHighlighter.doHighlightingByHighlighter(DefaultSolrHighlighter.java:527) > ... 33 more -- This message was sent by Atlassian JIRA (v6.1.5#6160) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org